<!-- Generated from models/_canonical by scripts/build-model-variants.js. Edit the canonical source, not this file. Behavioural profile for ChatGPT: scripts/model-profiles.json -->

#Scope contract

FILE_ISOLATION: Modify only files inside the scope the task names. Reading elsewhere is allowed; writing outside it is not, and a needed out-of-scope change is reported, not made. SIGNATURE_PINNING: Before implementing, write the exact signatures you will add or change (name, parameters, return type). Implement to those signatures; if one must change, say so before changing it. TYPE_CONTRACTS: Every public function carries explicit parameter and return types. No any, untyped dict, or interface{} at a module boundary.

---

#Purpose

Rules for instrumenting a service so that you can tell whether it is healthy, and find out why when it is not.

Two different questions, two different tools:

• Is it broken? Metrics and alerts. Low cardinality, cheap, always on.
• Why is it broken? Traces and logs. High cardinality, expensive, sampled.

Instrumenting for the first and hoping it answers the second is the most common mistake. → Backend/logging

---

#Measure the four golden signals

ts
const httpDuration = new Histogram({
  name: "http_request_duration_seconds",
  help: "Request duration",
  labelNames: ["method", "route", "status"],   // route TEMPLATE, bounded set
  buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10],
});

Measure a histogram, not an average. An average hides the tail entirely: a p50 of 40 ms and a p99 of 9 s average out to something that looks fine while 1% of users cannot use the product.

Buckets must be chosen for your SLO — you cannot compute a p99 accurately if all your data lands in one bucket.

Saturation is the leading indicator. A database connection pool at 95% utilisation is an outage in a few minutes; latency has not moved yet.

---

#Control cardinality

A time series exists for every unique label combination. Cardinality is multiplicative, and it is what makes monitoring bills explode and queries time out.

ts
// 1 series per route — correct
{ route: "/orders/:id" }

// 1 series per order — millions of series, and the collector falls over
{ route: `/orders/${id}` }

Never use as a metric label: user id, order id, email, session id, full URL path, raw error message, or a timestamp. Those are log fields and trace attributes, where high cardinality is the point.

Keep total series per service in the thousands, not millions. Audit label sets in review — a new label multiplies every existing series.

---

#Instrument what the business cares about

Infrastructure metrics tell you a host is unhealthy. Business metrics tell you the product is broken, which is the thing users notice.

• payments_captured_total, signups_completed_total, orders_created_total
• Queue depth and oldest-message age per queue → Backend/queues
• Job success/failure counts and durations by job type
• Dependency call rate, error rate and latency, per dependency
• Cache hit ratio

A deploy that breaks checkout while every host stays green is a routine outage. The business metric is what catches it.

---

#Traces

Distributed tracing is the only practical way to answer "where did the 4 seconds go" across services.

• Use OpenTelemetry. Vendor-neutral, and every backend consumes it.
• Auto-instrument HTTP, database and queue clients; add manual spans for significant internal work.
• Propagate W3C traceparent across every service and queue boundary — a trace that stops at the first hop is nearly useless.
• Put high-cardinality identifiers on span attributes (user.id, order.id), which is exactly where they belong.
• Sample tail-based where available: keep 100% of errors and slow requests, a small fraction of fast successes.
• Record the trace_id in logs so the two link.

ts
// OpenTelemetry: auto-instrument the edges, add spans for real work
const sdk = new NodeSDK({
  resource: resourceFromAttributes({
    [ATTR_SERVICE_NAME]: "orders-api",
    [ATTR_SERVICE_VERSION]: process.env.GIT_SHA,
    [ATTR_DEPLOYMENT_ENVIRONMENT_NAME]: process.env.ENV,
  }),
  traceExporter: new OTLPTraceExporter({ url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT }),
  instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();

await tracer.startActiveSpan("order.price", async (span) => {
  span.setAttribute("order.id", orderId);        // high cardinality belongs here
  span.setAttribute("tenant.id", tenantId);
  try { return await price(order); }
  catch (err) { span.recordException(err); span.setStatus({ code: SpanStatusCode.ERROR }); throw err; }
  finally { span.end(); }
});

---

#SLOs and alerts

Define the objective before the alert. An alert with no SLO behind it is a threshold someone guessed.

yaml
# 99.9% of checkout requests succeed within 500ms over 28 days.
# Error budget: 0.1% ≈ 40 minutes per month.

Alert on burn rate, not on instantaneous thresholds:

This is what removes the 3am page for a two-minute blip while still catching a slow burn that a static threshold never trips.

Every paging alert must satisfy all four:

1. A user is affected — not a proxy metric.
2. It is actionable — there is something the responder can do now.
3. It is urgent — it cannot wait until morning.
4. It links to a runbook with the first three diagnostic steps.

Anything failing one of these is a ticket or a dashboard, not a page. Review alerts monthly and delete the ones nobody acted on — an alert nobody trusts makes every other alert weaker. → DevOps/monitoring

---

#Health checks

Separate the two, because they mean different things to the orchestrator:

• Liveness — is the process wedged? Must not check dependencies. A liveness check that fails when the database is down makes Kubernetes restart every pod during a database incident, turning a degradation into a full outage.
• Readiness — can it serve traffic now? May check the database and required dependencies, and should fail during startup and graceful shutdown.

---

#Anti-patterns

---

#Checklist

• Latency, traffic, errors and saturation are instrumented for every service
• Latency is a histogram with buckets chosen for the SLO
• Metric labels are bounded; no ids or raw paths as labels
• Business-level metrics exist alongside infrastructure metrics
• Dependency call rate, errors and latency are tracked per dependency
• Queue depth and oldest-message age are monitored
• OpenTelemetry traces cover every service, with context propagated
• High-cardinality identifiers are span attributes, not metric labels
• Sampling keeps all errors and slow requests
• Logs carry the trace_id
• An SLO with an explicit error budget is defined for each critical journey
• Alerts fire on multi-window burn rate, not static thresholds
• An alert fires on unexpected absence of traffic
• Every paging alert is user-affecting, actionable, urgent and has a runbook
• Alerts are reviewed monthly and unused ones deleted
• Liveness and readiness probes are separate; liveness checks no dependencies