<!-- Generated from models/_canonical by scripts/build-model-variants.js. Edit the canonical source, not this file. Behavioural profile for Kimi: scripts/model-profiles.json -->

#Scope contract

FILE_ISOLATION: Modify only files inside the scope the task names; report any out-of-scope change instead of making it.

---

#Purpose

Rules for surviving a major failure: a region outage, a destructive mistake, a ransomware event, or the loss of a critical third party.

The plan is not the deliverable — the rehearsed capability is. A recovery document that has never been executed is a hypothesis, and it is usually wrong in ways only a drill reveals.

---

#Start with two numbers, per service

Agree them with the business, per service tier, and write them down. They are the only basis on which any of the following decisions can be made.

If measured recovery time does not meet the stated RTO, either change the architecture or change the number. An aspirational RTO nobody has measured is worse than an honest one.

---

#Enumerate the scenarios

Plan for causes, because the response differs:

Two are commonly missed and are the most likely to actually happen:

• Human error is more frequent than infrastructure failure. Point-in-time recovery matters more than multi-region, for most organisations. A delayed replica (an hour behind on purpose) is cheap insurance.
• Account compromise defeats every backup that shares the account. → DevOps/backups

A delayed replica is configured with recovery_min_apply_delay = '1h' in Postgres, or --delay on a MySQL replica. It is an hour behind on purpose, so a DROP TABLE or a destructive UPDATE can be caught before it applies — and the recovery is a promotion rather than a multi-hour restore. → Database/replication

---

#Write runbooks that work at 3am

markdown
# Runbook: Primary database region failure

## Detect
- `pg_up == 0` for 2 minutes in `eu-west-1`, or the RDS event stream shows failover
- Confirm: `pg_isready -h $PRIMARY_HOST` from a bastion outside the region

## Decide
- Promote if the primary is unreachable for > 5 minutes. Decision owner: on-call.
- No approval required.

## Act
1. Verify the standby's replay position:  SELECT pg_last_wal_replay_lsn();
2. Promote:  aws rds failover-db-cluster --db-cluster-identifier prod
3. Update the connection secret and restart consumers
4. Verify writes:  psql -c "INSERT INTO healthcheck …"
5. Re-point surviving replicas

## Verify
- Error rate returns to baseline within 5 minutes
- Write path confirmed by the synthetic check

## Communicate
- Status page within 10 minutes; update every 30

Rules: exact commands, no ambiguity, decision criteria before the incident, and a named owner. Anything requiring judgement should be pre-decided.

Runbooks live in version control and somewhere reachable when your systems are down — a runbook hosted only in the wiki that runs in the failed cluster is useless exactly when needed.

---

#Define incident roles

Under pressure, unassigned work does not happen and everyone debugs at once.

The commander not debugging is the rule people resist and the one that matters most: someone must hold the overall picture and decide when to roll back.

Have a communications channel that does not depend on your infrastructure, and know how to reach people out of hours.

Break-glass credentials are the one people skip: if authentication itself is down, every recovery action requiring SSO is blocked. Store them offline, require two people to open them, and alert on use.

---

#Drill, or it does not work

bash
# Time every drill. The measured number replaces the aspirational RTO.
START=$(date -u +%s)
./runbooks/restore-db.sh --target-time "2026-08-23T14:32:59Z" --into scratch
psql "$SCRATCH_URL" -c "SELECT count(*) FROM orders;"        # sanity, not proof
./scripts/smoke-test.sh --base-url "$SCRATCH_APP_URL"        # proof
echo "measured RTO: $(( $(date -u +%s) - START ))s"

Run drills against the runbook as written, with someone who did not write it, and time them. Every drill's output is a corrected runbook and a corrected RTO.

yaml
# A chaos experiment states its hypothesis and its abort condition up front.
# Without both, it is not an experiment — it is an outage you caused.
hypothesis: "Losing one of three API pods does not raise the 5xx rate above 0.1%."
method:     { action: pod-delete, namespace: prod, label: app=api, count: 1 }
abort_if:   "sum(rate(http_requests_total{status=~'5..'}[1m])) / sum(rate(http_requests_total[1m])) > 0.01"
blast_radius: "one pod, staging first, then 1% of production traffic"

Chaos engineering — deliberately killing instances, injecting latency, blocking a dependency — finds the assumptions nobody wrote down. Start in staging, with a hypothesis and an abort condition.

After every real incident, write a blameless postmortem: timeline, impact, contributing causes, and action items with owners and dates. Track them to completion; a postmortem whose actions are never done is theatre. → Review/postmortem

---

#Anti-patterns

---

#Checklist

• Verify: RPO and RTO are agreed and documented per service tier
• Verify: The architecture actually meets the stated numbers, verified by drill
• Verify: Failure scenarios are enumerated, including human error and compromise
• Verify: Point-in-time recovery covers destructive mistakes
• Verify: Backups are isolated and immutable against account compromise
• Verify: Runbooks exist for every scenario with exact commands and decision criteria
• Verify: Runbooks are reachable when the primary platform is down
• Verify: Incident roles are defined, with a commander who does not debug
• Verify: A communications channel independent of the infrastructure exists
• Verify: Database restore is drilled monthly; failover quarterly
• Verify: Region failover is drilled or tabletopped annually
• Verify: Secret and configuration recovery is drilled
• Verify: Drills are executed by someone who did not write the runbook, and timed
• Verify: Every drill produces runbook corrections
• Verify: Chaos experiments run with a hypothesis and an abort condition
• Verify: Every incident produces a blameless postmortem with tracked actions