Claude Fable 5.1 & GPT-6 Astra packages are live

Performance

Free

Testing performance as a regression gate — budgets in CI, lab versus field data, and measuring the metrics users actually feel.

182 lines6.8 KB Sarvam Ai Testing
targetModels
Sarvam-105BSarvam-30BSarvam FamilyFuture Sarvam Models
name
performance
category
Testing
description
Testing performance as a regression gate — budgets in CI, lab versus field data, and measuring the metrics users actually feel.
license
MIT
author
Agent.md maintainers
last-verified
reviewed-by
unreviewed
<!-- Generated from models/_canonical by scripts/build-model-variants.js. Edit the canonical source, not this file. Behavioural profile for Sarvam: scripts/model-profiles.json -->

#Locale

Examples use Indian conventions: ₹ amounts, IST, dd/mm/yyyy, Aadhaar and DPDP Act where a standard mentions identity or privacy law. Keep them when you copy an example.


#Purpose

Rules for catching performance regressions before users do. Capacity under concurrency is Testing/load; this package is about the speed of a single experience and keeping it from decaying.

Performance degrades one pull request at a time. A budget enforced in CI is the only mechanism that stops it, because nobody notices 40ms.


#Measure what users feel

For web interfaces, the Core Web Vitals plus one:

MetricGoodMeasures
LCP — Largest Contentful Paint< 2.5sWhen the main content appears
INP — Interaction to Next Paint< 200msResponsiveness to input
CLS — Cumulative Layout Shift< 0.1Visual stability
TTFB — Time to First Byte< 800msServer and network

INP replaced FID because it measures every interaction, not just the first — a page that responds instantly once and stutters afterwards now scores honestly.

For APIs, latency percentiles at the boundary: p50, p95, p99. Never the mean — see Testing/load.


#Lab and field are both required

Lab (synthetic)Field (RUM)
SourceLighthouse, WebPageTest, CIReal users, web-vitals
StrengthReproducible; can gate a PRTruthful; real devices and networks
WeaknessIdealised device and networkCannot block a regression pre-merge

Use lab data as the gate and field data as the truth. A lab score of 98 on a simulated fast connection tells you nothing about a mid-range Android on 4G, which is what most of the world is using.

js
// Field collection — send real user measurements, don't guess at them
import { onLCP, onINP, onCLS, onTTFB } from "web-vitals";

const send = (metric) =>
  navigator.sendBeacon("/rum", JSON.stringify({
    name: metric.name, value: metric.value, rating: metric.rating,
    id: metric.id, path: location.pathname,
  }));

onLCP(send); onINP(send); onCLS(send); onTTFB(send);

navigator.sendBeacon survives page unload; a fetch in visibilitychange frequently does not.


#Budgets in CI

A budget only works if crossing it fails the build.

js
// lighthouserc.js — assert, don't just report
module.exports = {
  ci: {
    collect: { url: ["http://localhost:3000/", "http://localhost:3000/checkout"], numberOfRuns: 3 },
    assert: {
      assertions: {
        "categories:performance": ["error", { minScore: 0.9 }],
        "largest-contentful-paint": ["error", { maxNumericValue: 2500 }],
        "cumulative-layout-shift": ["error", { maxNumericValue: 0.1 }],
        "total-byte-weight": ["error", { maxNumericValue: 500_000 }],
      },
    },
  },
};
  • Run at least 3 iterations and take the median. Single runs on shared CI runners are too noisy to gate on.
  • Budget the bundle as well as the timings — size-limit or bundlesize catches a 300 KB dependency at the pull request that added it, which is the only time it is cheap to remove.
  • Set budgets from current measured values, slightly tightened. An aspirational budget that fails on day one gets disabled on day two.

#Comparing fairly

Performance numbers are noisy; most reported "regressions" are measurement error.

  • Compare against the base commit, not against an absolute from last quarter.
  • Pin CPU throttling and network conditions so runs are comparable.
  • Prefer a relative threshold ("no more than 10% slower than base") over an absolute one for CI gating.
  • Re-run before believing a single failure. Then look at a trend, not a point.
  • For microbenchmarks use a real harness — benchmark.js, mitata, hyperfine — which handles warmup and statistical significance. A Date.now() difference around a loop measures the JIT warming up.

#Profile before optimising

A regression test tells you that it slowed down. Finding where needs a profile.

SymptomTool
Slow page loadChrome DevTools Performance, WebPageTest filmstrip
Slow interactionPerformance panel, React Profiler, INP attribution
Large bundlewebpack-bundle-analyzer, source-map-explorer
Slow endpointAPM traces, flame graph, --cpu-prof
Slow queryEXPLAIN ANALYZE, pg_stat_statements
bash
# Node: capture a CPU profile of the real workload, then read the flame graph
node --cpu-prof --cpu-prof-dir=./profiles server.js
npx speedscope ./profiles/*.cpuprofile

Never optimise from a guess. The bottleneck is routinely somewhere nobody predicted, and the time spent on the wrong thing is unrecoverable.


#Anti-patterns

Anti-patternWhy it failsFix
Lab data onlyIdealised device and networkCollect field RUM
Reporting the meanHides the tailp95, p99
Budgets that only warnNobody reads a warningFail the build
A single CI runRunner noise reads as regressionMedian of 3+
Absolute budget on noisy CIFlaky gate gets disabledRelative to base commit
Aspirational budgetFails immediately, gets removedSet from measured values
Date.now() microbenchmarksMeasures JIT warmupUse a real harness
Optimising without a profileEffort spent on the wrong codeProfile first
No bundle budgetDependencies accrete unnoticedsize-limit in CI
Measuring only the homepageRegressions hide on other routesBudget key routes

#Checklist

  • Verify: LCP, INP, CLS and TTFB are measured, not just a Lighthouse score
  • Verify: Field data is collected from real users via web-vitals and sendBeacon
  • Verify: Lab budgets run in CI and fail the build when exceeded
  • Verify: At least three runs are taken and the median used
  • Verify: Budgets were derived from current measurements, then tightened
  • Verify: Bundle size is budgeted separately and gated per pull request
  • Verify: Comparisons are relative to the base commit
  • Verify: Key routes are budgeted, not only the homepage
  • Verify: API latency is reported as percentiles
  • Verify: Microbenchmarks use a harness that handles warmup
  • Verify: Optimisation follows a profile, never a guess