Claude Fable 5.1 & GPT-6 Astra packages are live

Test Strategy

Free

Deciding what to test and at which level — the shape of a suite, what to do with legacy code, and the signals that a strategy is failing.

208 lines8.0 KB Kimi Testing
targetModels
Kimi K3Kimi K2.6Kimi K2 FamilyFuture Kimi Models
name
test-strategy
category
Testing
description
Deciding what to test and at which level — the shape of a suite, what to do with legacy code, and the signals that a strategy is failing.
license
MIT
author
Agent.md maintainers
last-verified
reviewed-by
unreviewed
<!-- Generated from models/_canonical by scripts/build-model-variants.js. Edit the canonical source, not this file. Behavioural profile for Kimi: scripts/model-profiles.json -->

#Scope contract

FILE_ISOLATION: Modify only files inside the scope the task names; report any out-of-scope change instead of making it.


#Purpose

Rules for choosing what to test, at which level, and when to stop.

The purpose of a test suite is confidence to change the code. Any test that does not increase that confidence — or that must be edited every time the code is refactored without a behaviour change — is a liability on the balance sheet, not an asset.


#The shape of the suite

LevelShareRuntimeCatches
Unit~70%millisecondsLogic, edge cases, error paths
Integration~20%secondsQueries, migrations, HTTP contract, wiring
E2E~10%minutesCritical journeys, real browser behaviour

Two failure modes:

  • Ice-cream cone — mostly E2E. Slow, flaky, expensive to maintain; failures point at a page rather than a line. Usually appears when unit tests were hard to write, which is itself a design signal.
  • Hourglass — many unit and E2E, no integration. Wiring bugs reach production because nothing tests the seam.

Match the shape to the risk, not to a rule. A payments service justifies more integration tests than a marketing site.

bash
        /\        E2E — few, critical journeys
       /  \
      /----\      Integration — queries, contracts, wiring
     /      \
    /--------\    Unit — logic, edges, errors

#Choosing a level

Ask what could break, then choose the cheapest test that would catch it.

Concretely: a discount calculation is unit; "does findFirst filter by organisationId" is integration; "can a user complete checkout" is e2e. If you can express the risk as a pure function of inputs, it is a unit test — reach for describe/it and expect, not a browser.

RiskLevel
A calculation is wrongUnit
A query returns another tenant's rowsIntegration
A migration fails on populated dataIntegration
Checkout breaks after a deployE2E
A dependency upgrade changes behaviourIntegration + contract
A layout regressesTesting/visual
The system falls over at loadTesting/load

Never write an E2E test for something a unit test can catch. It costs 1,000× the runtime for the same information.


#What not to test

Every test has a maintenance cost. Skip:

  • Framework and library behaviour. Trust that Array.map, JSON.parse and express.Router work. Testing zod validates, or that prisma.findMany returns rows, tests someone else's suite.
  • Trivial getters and setters with no logic.
  • Generated code, unless you wrote the generator.
  • Exact log strings, private methods, internal call counts.
  • Third-party internals — test your adapter, not their SDK.

The question to ask: if this test fails, will it be because of a real bug, or because someone renamed something?


#Legacy code

Do not attempt to retrofit coverage everywhere at once. It will not finish.

  1. Characterise before changing. Write a test that asserts current behaviour, even where that behaviour is wrong. It is a safety net for the refactor.
  2. Cover the seam you are about to touch, not the whole module.
  3. Add a test with every bug fix. The regression test is the deliverable that outlives the fix.
  4. Ratchet coverage on changed lines, not on the whole repository. A global threshold on a legacy codebase blocks every pull request until someone lowers it to zero and stops caring.

#Enforcing the shape

yaml
# Run the cheap tiers on every push; gate merges on the critical journey only.
jobs:
  unit:         { run: "npm run test:unit -- --coverage" }
  integration:  { run: "npm run test:integration" }
  e2e-critical: { run: "npx playwright test --grep @critical" }
  e2e-full:     { if: "github.event_name == 'schedule'" }
json
// Ratchet on changed lines, not the whole repository — a global threshold on a
// legacy codebase gets lowered to zero and then ignored.
{
  "coverageThreshold": {
    "global": { "branches": 0, "lines": 0 },
    "./src/billing/": { "branches": 80, "lines": 90 }
  }
}

Tools worth naming: vitest / jest for unit, supertest and testcontainers for integration, playwright for E2E, stryker for mutation testing, and c8 / istanbul for coverage reporting.

#Signals the strategy is failing

Treat these as evidence, not as a reason to write more tests:

SignalLikely cause
Tests break on every refactorTesting implementation, not behaviour
Nobody runs the suite locallyToo slow — usually the wrong shape
Bugs reach production despite green CIWrong level or missing integration layer
Tests are retried until they passNon-determinism → Testing/unit
Coverage is high, escapes are frequentAssertions are weak; try mutation testing
Writing a test requires extensive mockingThe unit has too many collaborators

That last one is the most valuable: hard-to-test code is usually badly designed code. The instinct to reach for a heavier mocking framework is the wrong response; extracting the dependency is the right one.


#Practical rules

  • Every bug fix ships with a failing-then-passing test. No exceptions — this is the single highest-value rule in the document.
  • Keep the unit suite under a minute so it runs on save — vitest --watch or jest --watch should be usable while writing code, not a CI-only step.
  • Run unit and integration on every pull request; E2E critical-path on every merge; the full E2E suite less often.
  • Make failures legible: a good name and a clear assertion diff mean a reviewer does not need to read the test to know what broke.
  • Delete tests that no longer earn their keep. A deleted redundant test is a net gain.

#Anti-patterns

Anti-patternWhy it failsFix
Coverage percentage as a goalAssertion-free tests reach 100%Mutation testing; cover changed lines
E2E for logic a unit test could cover1,000× the cost, more flakePush down the pyramid
No integration layerWiring bugs reach productionTest the seams
Retrofitting coverage everywhereNever finishes; blocks deliveryCharacterise the seam you touch
Global coverage gate on legacy codeThreshold gets lowered to zeroRatchet on changed lines
Heavy mocking to make a test possibleHides a design problemExtract the dependency
Bug fixed without a regression testThe same bug returnsTest with every fix
Slow suite nobody runsFeedback arrives after mergeKeep unit under a minute
Keeping every test foreverMaintenance cost compoundsDelete redundant tests

#Checklist

  • Verify: The suite shape matches the risk, not a fixed ratio
  • Verify: Each test sits at the cheapest level that would catch its failure
  • Verify: No E2E test covers logic a unit test could
  • Verify: An integration layer exists and covers queries, migrations and HTTP contract
  • Verify: Framework behaviour, trivial accessors and third-party internals are untested
  • Verify: Every bug fix includes a test that failed before the fix
  • Verify: Coverage is measured on changed lines, never as a global target
  • Verify: Unit suite runs in under a minute
  • Verify: Unit and integration run on every pull request
  • Verify: Failure output identifies the problem without reading the test
  • Verify: Hard-to-test code is treated as a design signal, not a mocking problem