<!-- Generated from models/_canonical by scripts/build-model-variants.js. Edit the canonical source, not this file. Behavioural profile for DeepSeek: scripts/model-profiles.json -->

#Task boundary

1. Implement exactly the task as stated. Do not add abstractions, options, config, or files the task did not name.
2. Comments, identifiers, commit messages and log strings are English only.
3. Stop when the checklist at the end passes. Do not refactor or "improve" surrounding code.
4. Every checklist item below is backed by an assertion in a test or by pasted command output, never by a sentence.

---

#Purpose

Rules for monitoring infrastructure: nodes, clusters, databases, queues, networking and third-party dependencies. Application instrumentation is Backend/monitoring.

The distinction that matters: infrastructure metrics tell you a component is unhealthy; only user-facing signals tell you the product is broken. Alert on the second, use the first to diagnose.

---

#Collect the layers that fail

Two that are quietly fatal and routinely uncollected:

1. Disk and inode exhaustion. A full disk stops writes, breaks logging, and can corrupt state. Inodes exhaust separately — a directory full of tiny files fills them while df still shows free space.
2. Certificate expiry. An expired certificate is a total outage with a known date. Alert 30 and 7 days out.

Also monitor from outside: a synthetic check on the real user path, from another network, catches DNS, CDN and certificate failures that internal metrics cannot see.

---

#Alert on symptoms, page on user impact

An alert should mean a user is affected, and you can do something now. Define the SLO first; the alert follows from it.

yaml
# Multi-window burn rate: fast burn pages, slow burn opens a ticket
- alert: CheckoutFastBurn
  expr: |
    (1 - sum(rate(sli_good_total{journey="checkout"}[1h]))
       / sum(rate(sli_total{journey="checkout"}[1h]))) > 14.4 * 0.001
  for: 2m
  labels: { severity: page }
  annotations: { runbook: "https://runbooks.example.com/checkout" }

Every paging alert needs: a runbook link with the first three diagnostic steps, a clear owner, and a reason it cannot wait until morning. Anything failing those is a ticket.

promql
# Disk: predict, do not threshold. Four hours of warning is actionable.
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4*3600) < 0

# Inodes exhaust independently of bytes — `df -h` still shows free space
node_filesystem_files_free / node_filesystem_files < 0.1

# Certificates: a scheduled outage with a known date
probe_ssl_earliest_cert_expiry - time() < 7 * 86400

# Replication lag, in seconds behind the primary → `Database/replication`
pg_stat_replication_replay_lag_seconds > 30

Predict rather than threshold where you can: predict_linear on disk usage warns four hours ahead, which is actionable; "disk 90% full" on a slowly-growing volume is noise, and on a fast-filling one it is already too late.

---

#Dashboards people actually open

Build from the questions asked during an incident, not from every available metric.

1. Service overview — the four golden signals per service, one screen.
2. Dependency health — every downstream, with error rate and latency.
3. Capacity — saturation of each finite resource: pool, disk, memory, quota.
4. Deploy correlation — deploy markers annotated on the graphs, so "what changed?" is answerable in one glance.

json
// Grafana: annotate every panel with deploy markers, so "what changed?"
// is answered without leaving the dashboard.
{ "annotations": { "list": [{
  "name": "Deploys",
  "datasource": "prometheus",
  "expr": "changes(kube_deployment_status_observed_generation{deployment=\"api\"}[1m]) > 0",
  "iconColor": "rgba(255, 96, 96, 1)"
}]}}

Rules: default to the last hour, keep the top row to what matters, and delete dashboards nobody opens. A wall of unread graphs trains people to ignore all of them.

---

#Cost and retention

Observability spend grows superlinearly with traffic, and metric cardinality is the usual cause.

1. Cardinality kills. A label with a pod name, request id or user id multiplies series by that value's range. Audit label sets; keep high-cardinality data in logs and traces where it belongs. → Backend/logging
2. Retention by tier: high resolution for days, downsampled for months, aggregates for years. Nobody queries second-resolution data from March.
3. Sample high-volume success paths; never sample errors.
4. Alert on the observability bill itself — a cardinality explosion shipped on a Friday is discovered on the invoice otherwise.

---

#Operational hygiene

1. Monitoring must not share a failure domain with what it monitors. An alerting system hosted in the cluster it watches goes down with it.
2. Have a dead-man's switch: a heartbeat alert that fires when monitoring stops reporting. Silence is indistinguishable from health otherwise.
3. Test alerts when you write them — trigger the condition and confirm the page arrives at the right person.
4. Review alerts monthly: delete those nobody acted on, and add one for anything an incident revealed you were blind to.
5. Define escalation: who is paged, after how long unacknowledged, and to whom it escalates. → DevOps/disaster-recovery

---

#Anti-patterns

---

#Checklist

• Node, cluster, database, queue, cache and load-balancer signals are collected
• Disk usage and inode usage are monitored with predictive alerts
• Certificate and domain expiry alert at 30 and 7 days
• Third-party dependency error rate and latency are measured from your side
• Synthetic checks run against the real user path from outside the network
• An SLO exists for each critical user journey
• Paging alerts fire on multi-window burn rate, not static thresholds
• Every paging alert is user-affecting, actionable and has a runbook
• Non-urgent conditions create tickets, not pages
• Alerts are tested when written and reviewed monthly
• A dead-man's switch detects monitoring failure
• Monitoring runs outside the failure domain it observes
• Metric label cardinality is bounded and audited
• Retention is tiered by resolution and age
• Observability cost is monitored and alerted on
• Dashboards answer specific incident questions and show deploy markers
• An escalation policy defines who is paged and when it escalates