Claude Fable 5.1 & GPT-6 Astra packages are live

E2e

Free

End-to-end tests that are worth their cost — user-visible selectors, deterministic waiting, and keeping the suite small enough to trust.

210 lines8.0 KB Qwen Testing
targetModels
Qwen3.8-MaxQwen3.8-Flash-NextQwen3.8-27BQwen3.8 FamilyFuture Qwen Models
name
e2e
category
Testing
description
End-to-end tests that are worth their cost — user-visible selectors, deterministic waiting, and keeping the suite small enough to trust.
license
MIT
author
Agent.md maintainers
last-verified
reviewed-by
unreviewed
<!-- Generated from models/_canonical by scripts/build-model-variants.js. Edit the canonical source, not this file. Behavioural profile for Qwen: scripts/model-profiles.json -->

#Task boundary

  1. Implement only what the task names; no extra abstractions or files.
  2. English-only comments and identifiers.
  3. Stop when the checklist passes.

#Purpose

Rules for tests that drive a real browser against a running application.

E2E tests are the slowest, flakiest and most expensive tier. Keep few of them and make each one earn its place. A suite of 400 end-to-end tests is a suite nobody trusts, run on a schedule, ignored when red.

Test the journeys that generate revenue or lose data: signup, login, checkout, the primary create-and-save path. Everything else belongs lower in the pyramid.


#Selectors

Selector choice is the single largest cause of E2E maintenance cost.

js
// Best — how a user finds it; survives restyling and DOM changes
await page.getByRole("button", { name: "Place order" }).click();
await page.getByLabel("Email address").fill("a@example.com");

// Acceptable — explicit contract with the test
await page.getByTestId("checkout-submit").click();

// Fragile — breaks on any restyle or refactor
await page.click(".btn.btn-primary > span:nth-child(2)");
await page.click("//div[3]/form/button[1]");

Priority: role and accessible name → label → text → test id → CSS → XPath. Role-based queries double as an accessibility check: if getByRole cannot find your button, a screen reader cannot either.

Never select by CSS class or DOM position. Both encode presentation, which is the thing most likely to change without any behaviour changing.

Where a test id is needed, use a dedicated attribute (data-testid) so it is obviously a contract and nobody deletes it during a cleanup.


#Waiting

Flakiness in E2E is almost always a waiting bug.

js
// WRONG — a fixed sleep is a race that passes locally and fails on CI
await page.waitForTimeout(2000);

// RIGHT — wait for the condition that actually matters
await expect(page.getByText("Order confirmed")).toBeVisible();
await page.waitForResponse((r) => r.url().includes("/api/orders") && r.ok());

Modern frameworks (Playwright, Cypress) auto-wait for actionability — attached, visible, stable, enabled — before acting. Fighting that with manual sleeps reintroduces the race they removed.

Never use a fixed timeout to wait for anything. If you cannot express the condition, the application is missing an observable signal — add one rather than guessing at a duration.

Watch for these specific races: animations still running, a toast that auto-dismisses before the assertion, a list that re-renders after a background refetch, and focus moving during a form fill.


#Test data and isolation

  1. Create the data the test needs, in the test, through an API or a factory endpoint — not through the UI. Signing up via the interface to test checkout makes checkout failures indistinguishable from signup failures, and triples the runtime.
  2. Never depend on a shared staging database. Another test, or a colleague, will change the row you assert on.
  3. Use a unique identifier per run (user-${runId}@example.com) so parallel runs cannot collide.
  4. Seed authentication via storage state rather than logging in through the form in every test:
js
// Log in once, reuse the session for every test in the project
await page.context().storageState({ path: "auth.json" });
// playwright.config: use: { storageState: "auth.json" }

Keep exactly one test that exercises the real login form. The rest start authenticated.


#Scope and structure

Test at E2E levelTest lower
Signup, login, checkout, paymentField validation rules → Testing/unit
Critical multi-page journeysAPI status codes → Testing/integration
Third-party redirect flows (OAuth, payment)Business calculations → Testing/unit
Anything that has broken in production beforeEvery permutation of a form
  1. One journey per test. A test asserting six unrelated things fails opaquely and hides the later failures.
  2. Do not chain tests. Each starts from a known state and can run alone.
  3. Tag by criticality (@smoke, @critical) so a fast subset gates deployment and the full suite runs less often.

#Running in CI

  1. Run headless in CI, headed locally for debugging.
  2. Capture trace, video and screenshot on failure. A failed E2E test with no artifact costs an hour of local reproduction.
  3. Never paper over flakiness with blanket retries. One retry to absorb genuine infrastructure noise is defensible; three retries hide a real race, and the bug reaches production.
  4. Quarantine, do not skip. A flaky test moved to a quarantined suite still runs and still reports; a skipped test is deleted coverage nobody notices.
  5. Track flake rate per test. A test failing 5% of the time is not passing — it is costing every engineer who sees it red.
  6. Pin the browser version so an upstream update does not turn into a mystery failure.

#Configuration that matters

js
// playwright.config.ts — the settings that decide flake rate
export default defineConfig({
  timeout: 30_000,                  // per test, not per action
  expect: { timeout: 5_000 },       // auto-retrying assertions
  retries: process.env.CI ? 1 : 0,  // one retry absorbs infra noise, not races
  fullyParallel: true,
  workers: process.env.CI ? 4 : undefined,
  use: {
    trace: "on-first-retry",        // artifact only when it matters
    video: "retain-on-failure",
    screenshot: "only-on-failure",
    baseURL: process.env.BASE_URL,
    actionTimeout: 10_000,
  },
});

Key APIs worth knowing: getByRole, getByLabel, getByTestId, toBeVisible, toHaveURL, waitForResponse, route for request interception, storageState for session reuse, and test.step to make traces readable.

In Cypress the equivalents are cy.findByRole, cy.intercept, cy.session and cy.wait("@alias") — never cy.wait(2000).

#Anti-patterns

Anti-patternWhy it failsFix
waitForTimeout(2000)Race that fails on loaded CIWait for the condition
CSS class or XPath selectorsBreak on any restylegetByRole / getByLabel
Creating data through the UISlow; couples unrelated featuresAPI or factory setup
Logging in via the form in every testMultiplies runtime and flakeReuse storageState
Shared staging dataAnother run changes your rowUnique data per run
Chained, order-dependent testsOne failure cascadesIndependent tests
Retrying until greenHides a real raceFix it or quarantine it
Testing every form permutation E2ESlow suite nobody trustsPush down to unit tests
No trace or video on failureUnreproducible failuresCapture artifacts
Skipping a flaky testSilent loss of coverageQuarantine and track

#Checklist

  • The suite covers critical journeys only, not exhaustive permutations
  • Selectors use role, label or data-testid — never CSS position or XPath
  • No fixed timeout appears anywhere; waits express a real condition
  • Test data is created via API or factory, uniquely per run
  • Authentication is seeded from stored state, with one real login test
  • Each test is independent and can run in isolation
  • Tests are tagged so a fast critical subset can gate deployment
  • Trace, video and screenshots are captured on failure
  • Retries are at most one, and flake rate is tracked per test
  • Flaky tests are quarantined rather than skipped
  • Browser versions are pinned in CI