<!-- Generated from models/_canonical by scripts/build-model-variants.js. Edit the canonical source, not this file. Behavioural profile for Claude: scripts/model-profiles.json -->

<critical_constraints> FORBIDDEN: Truncating code or writing placeholders such as "// ... existing code ..." or "# rest unchanged". Every edit is complete and applies as written. FORBIDDEN: Reporting a check as passed without showing the command and its output. REQUIRED: Reason through the rules below before the first edit; when two rules conflict, the one stated first wins. </critical_constraints>

---

#Purpose

Rules for the logging platform: how logs get from a process to somewhere searchable, how long they are kept, and what they cost. What an application should write is Backend/logging.

Logs are usually the largest observability line item and the one that grows fastest. Most of these rules are about keeping them useful and affordable at the same time.

---

#Applications write to stdout; the platform does the rest

arduino
process → stdout (JSON) → collector (Vector / Fluent Bit / OTel) → store → query

An application that opens log files, rotates them, or ships them directly is doing the platform's job badly:

• File logging in a container writes to the ephemeral layer and disappears on restart — exactly when you need it.
• Unrotated files fill the disk, which takes the service down.
• A direct-to-backend shipper couples the application to a vendor and blocks on the network during an outage.

Write JSON to stdout, one object per line, and let the collector attach pod, namespace, node, service and version metadata.

Handle multi-line output (stack traces) at the collector — a Java trace arriving as forty separate lines is unsearchable. Most runtimes can emit the trace inside one JSON field instead, which is better.

---

#Structure at the source, not with regex later

A collector-side regex parsing unstructured text is fragile, expensive, and breaks the moment a message changes.

If a legacy system emits plain text you cannot change, parse it once at the collector, keep the parser in version control, and test it. Otherwise: JSON at the source.

toml
# Vector: parse once, redact, drop noise, then route by class.
[transforms.parse]
type = "remap"
inputs = ["kubernetes_logs"]
source = '''
  . = parse_json!(.message)
  .service = .service || %kubernetes.pod_labels."app.kubernetes.io/name"
  del(.headers.authorization)
  del(.password)
'''

[transforms.drop_noise]
type = "filter"
inputs = ["parse"]
condition = '!match(string!(.http.path), r'''^/(healthz|readyz|metrics)$''')'

Enforce a shared field schema across services:

A field named three ways cannot be queried across services, which defeats the purpose of centralising logs at all.

---

#Retention by value, not one policy for everything

• Hot storage is indexed and expensive; archive is object storage and cheap.
• Audit logs are a separate stream with a separate lifecycle, write-once where possible. Never mix them with application logs whose retention is days.
• Deletion obligations apply: a GDPR erasure request covers logs containing personal data, which is the strongest argument for logging identifiers only.

---

#Control the cost

Logging bills grow with traffic and with verbosity, and both grow silently.

• Sample high-volume success paths — keep 1–10% of healthy 2xx request lines, 100% of errors. Record the sampling rate so counts can be reconstructed.
• Drop known-worthless lines at the collector: health-check requests, static asset hits, framework startup noise.
• Never derive a metric by counting log lines. It is expensive and breaks the moment sampling changes. Emit a counter.
• Make log level runtime-configurable per service, so debug can be raised during an incident and lowered afterwards — a permanently-debug service is usually the single largest cost line.
• Alert on log volume per service. A logging loop shipped on a Friday is otherwise discovered on the invoice.

---

#Access, PII and integrity

• Log storage is a sensitive data store. Access-control it, and audit reads — it frequently contains more personal data than the database does.
• Redact at the collector as a second line of defence; the application must redact first. Collector-side redaction alone means the secret already crossed the network. → Security/secret-management
• A credential in a log is a disclosed credential: log storage is replicated, backed up and widely readable. Rotate it.
• Audit logs need integrity: append-only, hash-chained or WORM storage, so a compromised account cannot erase its own trail.
• Ship logs off the host promptly, and cap the local buffer so a backend outage cannot fill the node disk:

yaml
# Fluent Bit: bounded on-disk buffering, so a downstream outage degrades
# logging rather than taking the node down with a full filesystem.
[SERVICE]
    storage.path              /var/log/flb-storage/
    storage.max_chunks_up     128
[OUTPUT]
    Name                      opensearch
    storage.total_limit_size  2G
    Retry_Limit               5

Logs that only exist on a node are lost with the node — which is common in exactly the incidents you most want to investigate.

---

#Anti-patterns

---

#Checklist

• Applications write structured JSON to stdout and nothing else
• A collector attaches environment, service and version metadata
• Multi-line stack traces arrive as a single record
• All services share one field schema
• Every record carries service, version, environment and correlation ids
• Retention is tiered by log class, hot and archive separated
• Audit logs are a separate stream with their own long retention
• High-volume success paths are sampled; errors never are
• Sampling rates are recorded in the records
• Worthless lines are dropped at the collector
• Metrics are emitted as counters, not derived from log lines
• Log level is runtime-configurable per service
• Log volume per service is monitored and alerted on
• Log storage is access-controlled and reads are audited
• Redaction happens at the source and again at the collector
• Audit logs are append-only or stored in WORM storage
• Logs are shipped off the host promptly