Claude Fable 5.1 & GPT-6 Astra packages are live

Visual

Free

Visual regression testing that is not permanently red — deterministic rendering, masking dynamic content, and reviewing diffs instead of regenerating…

196 lines7.9 KB Glm Testing
targetModels
GLM-5.3GLM-5.2GLM-5 FamilyGLM-4.6Future GLM Models
name
visual
category
Testing
description
Visual regression testing that is not permanently red — deterministic rendering, masking dynamic content, and reviewing diffs instead of regenerating them.
license
MIT
author
Agent.md maintainers
last-verified
reviewed-by
unreviewed
<!-- Generated from models/_canonical by scripts/build-model-variants.js. Edit the canonical source, not this file. Behavioural profile for GLM: scripts/model-profiles.json -->

#Task boundary

  1. Implement only what the task names; no extra abstractions or files.
  2. English-only comments and identifiers.
  3. Stop when the checklist passes.

#Purpose

Rules for catching unintended visual change by comparing rendered output against approved baselines.

Visual tests catch what assertions cannot: a CSS change that breaks an unrelated page, a font fallback, a component overlapping at one breakpoint. They fail badly when non-determinism makes them noisy — the common outcome is a suite that is always slightly red and therefore approved without looking, which is worse than having none.


#Determinism is the whole problem

Every source of variation must be removed or masked before the first baseline.

SourceFix
Dates and timesFreeze the clock; inject a fixed timestamp
Random dataSeed the generator; use fixed fixtures
Animations and transitionsDisable globally in the test stylesheet
Font loadingWait for document.fonts.ready; self-host fonts
Images from a networkStub them or use local fixtures
ScrollbarsConsistent OS or a container-based screenshot
GPU and font renderingRender in a container — the biggest single fix
Carousels, skeletons, spinnersMask the region
Blinking text caretcaret-color: transparent
Lazy images below the foldloading="eager" in the test build
Math.random() in componentsInject a seeded generator
Locale-dependent formattingPin TZ=UTC and LANG=en_US.UTF-8
css
/* Loaded only in visual tests: nothing may be mid-animation at capture time */
*, *::before, *::after {
  animation: none !important;
  transition: none !important;
  caret-color: transparent !important;
  scroll-behavior: auto !important;
}

Never generate baselines on a developer laptop and compare them in CI. Font hinting and GPU rasterisation differ, so every screenshot differs. Generate and compare in the same container image, pinned by digest.


Tooling: @playwright/test toHaveScreenshot, percy, chromatic, argos-ci, backstopjs, loki. Playwright's built-in comparison needs no service and stores baselines in the repository; hosted tools add review workflows and cross-browser rendering at a cost.

Pin the runner explicitly — mcr.microsoft.com/playwright:v1.49.0-jammy by digest, not :latest. A browser or font-package update inside the image invalidates every baseline at once.

#Capture

js
test("invoice card renders", async ({ page }) => {
  await page.goto("/components/invoice-card");
  await page.evaluate(() => document.fonts.ready);      // fonts settled

  await expect(page.getByTestId("invoice-card")).toHaveScreenshot("invoice-card.png", {
    maxDiffPixelRatio: 0.01,        // absorb sub-pixel noise, catch real change
    animations: "disabled",
    mask: [page.getByTestId("relative-time")],   // dynamic region excluded
  });
});
  1. Screenshot the component, not the page, where possible. A page-level baseline fails for every unrelated change and tells you nothing about which.
  2. Set a small but non-zero diff tolerance. Zero is brittle; anything above a percent or two hides real regressions.
  3. Mask genuinely dynamic regions rather than trying to freeze them.
  4. Cover the states that break in production: empty, loading, error, overflow and long-text. A component tested only with a two-word label passes forever and breaks on the first real customer name.
  5. Test the breakpoints that matter — typically 375px, 768px and 1280px — plus dark mode if you support it. Every extra viewport is another baseline to review.

yaml
# Pin the image by digest. A font or browser update inside :latest silently
# invalidates every baseline in the repository.
jobs:
  visual:
    container:
      image: mcr.microsoft.com/playwright@sha256:<digest>
    steps:
      - run: npx playwright test --grep @visual
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: visual-diffs
          path: test-results/**/*-diff.png    # reviewers need the diff image

#Reviewing diffs

This is where the practice succeeds or fails.

  1. Every baseline update is a code review. The diff image goes in the pull request and someone looks at it.

  2. Never run the update command to make CI green. Doing so commits the regression as the new expected appearance — the exact failure mode this suite exists to prevent.

  3. Approvals belong to the person who owns the visual change, not whoever is unblocking the build. Commands worth knowing: --update-snapshots regenerates (dangerous — see above), --grep @visual runs only this tier, and --reporter=html produces the side-by-side expected / actual / diff view that makes review possible at all.

  4. Store baselines in the repository (Git LFS if they grow) or in the tool's managed store — but they must be versioned with the code, so checking out an old commit gives its correct baselines.


#Scope

Worth a visual testNot worth it
Design-system components in each stateEvery page of a content site
Critical pages: pricing, checkout, landingText-only changes
Complex layouts: tables, grids, dashboardsAnything a unit test asserts better
Dark mode and RTL variantsRapidly iterating prototypes
Empty, loading, error and overflow states

Component-level baselines in a Storybook-style harness give the best ratio: they are stable, fast, and the failure points directly at the component.

Never treat a visual test as an accessibility check. Identical pixels can hide a missing label or an unreachable control → Testing/accessibility.


#Anti-patterns

Anti-patternWhy it failsFix
Baselines from a laptop, compared in CIFont and GPU differencesSame pinned container
Zero diff toleranceSub-pixel noise fails every runSmall maxDiffPixelRatio
Updating baselines to go greenCommits the regression as expectedReview every diff image
Full-page shots for everythingOne change fails every testComponent-level captures
No animation disablingCaptures mid-transitionGlobal disable stylesheet
Live dates and random dataDifferent every runFreeze and seed
Screenshotting every pageEnormous review burdenTarget components and key pages
Baselines outside version controlOld commits fail against new baselinesVersion alongside the code
Treating pixels as accessibilityIdentical pixels hide missing labelsRun axe separately
Ignoring a persistently red testReal regressions become invisibleFix determinism or delete it

#Checklist

  • Baselines are generated and compared in the same pinned container image
  • Animations, transitions and carets are disabled during capture
  • Clocks are frozen and random data seeded
  • Fonts are self-hosted and awaited before capture
  • Dynamic regions are masked rather than left to vary
  • Captures target components; page-level shots are reserved for key pages
  • Diff tolerance is small but non-zero
  • Key breakpoints and dark mode are covered
  • Every baseline update is reviewed as a diff image by the change owner
  • Baselines are versioned with the code
  • Accessibility is tested separately, never inferred from pixels