Claude Fable 5.1 & GPT-6 Astra packages are live

Performance

Free · MIT

Testing performance as a regression gate — budgets in CI, lab versus field data, and measuring the metrics users actually feel.

185 lines6.7 KB Qwen Testing
Target models
Qwen3.8-MaxQwen3.8-Flash-NextQwen3.8-27BQwen3.8 FamilyFuture Qwen Models
Name
performance
Category
Testing
Description
Testing performance as a regression gate — budgets in CI, lab versus field data, and measuring the metrics users actually feel.
License
MIT
Author
Agent.md maintainers
Last verified
2026-08-23
Reviewed by
unreviewed

#Task boundary

  1. Implement only what the task names; no extra abstractions or files.
  2. English-only comments and identifiers.
  3. Stop when the checklist passes.

#Purpose

Rules for catching performance regressions before users do. Capacity under concurrency is Testing/load; this package is about the speed of a single experience and keeping it from decaying.

Performance degrades one pull request at a time. A budget enforced in CI is the only mechanism that stops it, because nobody notices 40ms.


#Measure what users feel

For web interfaces, the Core Web Vitals plus one:

MetricGoodMeasures
LCP — Largest Contentful Paint< 2.5sWhen the main content appears
INP — Interaction to Next Paint< 200msResponsiveness to input
CLS — Cumulative Layout Shift< 0.1Visual stability
TTFB — Time to First Byte< 800msServer and network

INP replaced FID because it measures every interaction, not just the first — a page that responds instantly once and stutters afterwards now scores honestly.

For APIs, latency percentiles at the boundary: p50, p95, p99. Never the mean — see Testing/load.


#Lab and field are both required

Lab (synthetic)Field (RUM)
SourceLighthouse, WebPageTest, CIReal users, web-vitals
StrengthReproducible; can gate a PRTruthful; real devices and networks
WeaknessIdealised device and networkCannot block a regression pre-merge

Use lab data as the gate and field data as the truth. A lab score of 98 on a simulated fast connection tells you nothing about a mid-range Android on 4G, which is what most of the world is using.

js
// Field collection — send real user measurements, don't guess at them
import { onLCP, onINP, onCLS, onTTFB } from "web-vitals";

const send = (metric) =>
  navigator.sendBeacon("/rum", JSON.stringify({
    name: metric.name, value: metric.value, rating: metric.rating,
    id: metric.id, path: location.pathname,
  }));

onLCP(send); onINP(send); onCLS(send); onTTFB(send);

navigator.sendBeacon survives page unload; a fetch in visibilitychange frequently does not.


#Budgets in CI

A budget only works if crossing it fails the build.

js
// lighthouserc.js — assert, don't just report
module.exports = {
  ci: {
    collect: { url: ["http://localhost:3000/", "http://localhost:3000/checkout"], numberOfRuns: 3 },
    assert: {
      assertions: {
        "categories:performance": ["error", { minScore: 0.9 }],
        "largest-contentful-paint": ["error", { maxNumericValue: 2500 }],
        "cumulative-layout-shift": ["error", { maxNumericValue: 0.1 }],
        "total-byte-weight": ["error", { maxNumericValue: 500_000 }],
      },
    },
  },
};
  1. Run at least 3 iterations and take the median. Single runs on shared CI runners are too noisy to gate on.
  2. Budget the bundle as well as the timings — size-limit or bundlesize catches a 300 KB dependency at the pull request that added it, which is the only time it is cheap to remove.
  3. Set budgets from current measured values, slightly tightened. An aspirational budget that fails on day one gets disabled on day two.

#Comparing fairly

Performance numbers are noisy; most reported "regressions" are measurement error.

  1. Compare against the base commit, not against an absolute from last quarter.
  2. Pin CPU throttling and network conditions so runs are comparable.
  3. Prefer a relative threshold ("no more than 10% slower than base") over an absolute one for CI gating.
  4. Re-run before believing a single failure. Then look at a trend, not a point.
  5. For microbenchmarks use a real harness — benchmark.js, mitata, hyperfine — which handles warmup and statistical significance. A Date.now() difference around a loop measures the JIT warming up.

#Profile before optimising

A regression test tells you that it slowed down. Finding where needs a profile.

SymptomTool
Slow page loadChrome DevTools Performance, WebPageTest filmstrip
Slow interactionPerformance panel, React Profiler, INP attribution
Large bundlewebpack-bundle-analyzer, source-map-explorer
Slow endpointAPM traces, flame graph, --cpu-prof
Slow queryEXPLAIN ANALYZE, pg_stat_statements
bash
# Node: capture a CPU profile of the real workload, then read the flame graph
node --cpu-prof --cpu-prof-dir=./profiles server.js
npx speedscope ./profiles/*.cpuprofile

Never optimise from a guess. The bottleneck is routinely somewhere nobody predicted, and the time spent on the wrong thing is unrecoverable.


#Anti-patterns

Anti-patternWhy it failsFix
Lab data onlyIdealised device and networkCollect field RUM
Reporting the meanHides the tailp95, p99
Budgets that only warnNobody reads a warningFail the build
A single CI runRunner noise reads as regressionMedian of 3+
Absolute budget on noisy CIFlaky gate gets disabledRelative to base commit
Aspirational budgetFails immediately, gets removedSet from measured values
Date.now() microbenchmarksMeasures JIT warmupUse a real harness
Optimising without a profileEffort spent on the wrong codeProfile first
No bundle budgetDependencies accrete unnoticedsize-limit in CI
Measuring only the homepageRegressions hide on other routesBudget key routes

#Checklist

  • LCP, INP, CLS and TTFB are measured, not just a Lighthouse score
  • Field data is collected from real users via web-vitals and sendBeacon
  • Lab budgets run in CI and fail the build when exceeded
  • At least three runs are taken and the median used
  • Budgets were derived from current measurements, then tightened
  • Bundle size is budgeted separately and gated per pull request
  • Comparisons are relative to the base commit
  • Key routes are budgeted, not only the homepage
  • API latency is reported as percentiles
  • Microbenchmarks use a harness that handles warmup
  • Optimisation follows a profile, never a guess