<!-- Generated from models/_canonical by scripts/build-model-variants.js. Edit the canonical source, not this file. Behavioural profile for Qwen: scripts/model-profiles.json -->

#Task boundary

1. Implement only what the task names; no extra abstractions or files.
2. English-only comments and identifiers.
3. Stop when the checklist passes.

---

#Purpose

Rules for producing and consuming messages. A queue decouples a slow or unreliable operation from a request, and in exchange hands you a distributed-systems problem: at-least-once delivery, no ordering guarantees across partitions, and failures that happen after the work but before the acknowledgement.

Design for the two facts that are always true:

1. Every message may be delivered more than once.
2. Every message may arrive out of order.

---

#Acknowledge after the work, not before

ts
// Wrong — a crash after ack loses the job silently
await ack(msg);
await process(msg);

// Right — a crash before ack redelivers it
await process(msg);
await ack(msg);

This is why duplicates exist, and it is the correct trade: at-least-once with idempotent consumers is achievable; exactly-once is not, across a network.

Set the acknowledgement deadline (visibilityTimeout, ackDeadline, lockDuration) above the p99 processing time, and extend it for long jobs rather than setting a very large default — a large default means a crashed consumer's message is invisible for that long.

---

#Consumers must be idempotent

Deduplicate on a business key, not a delivery id.

sql
-- The unique constraint is the guarantee; the check is the nice error.
INSERT INTO processed_messages (idempotency_key, processed_at)
VALUES ($1, now())
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING id;                     -- no row returned means already processed

Better still, make the effect itself idempotent — UPDATE orders SET status = 'shipped' WHERE id = $1 AND status = 'paid' is safe to run twice by construction.

Where the handler both writes to the database and enqueues another message, use the transactional outbox: write the message to a table inside the same transaction as the state change, and publish from that table asynchronously. Otherwise the two can diverge — the row commits and the publish fails, or the reverse. → Database/transactions

---

#Retries and dead letters

A dead-letter queue with no alert and no replay tool is a data-loss bucket. Alert on DLQ depth greater than zero, and provide a way to inspect, fix and replay messages. Every system eventually needs to reprocess a window.

Distinguish the failure classes: retrying a message that will never parse burns your retry budget and delays healthy work behind it.

---

#Ordering

Most queues guarantee ordering only within a partition or group key, and only when a single consumer processes that key at a time.

1. If order matters, use an ordering key (MessageGroupId, Kafka partition key, RabbitMQ single active consumer) — and accept that it caps parallelism for that key.
2. Better: make handlers order-independent. Include a version or updated_at in the payload and discard messages older than the state you already hold.
3. Best for entity updates: treat the message as a notification and re-read current state from the source. Ordering stops mattering entirely.

Never assume that publishing A then B means A is processed first. With multiple consumers, it usually is not.

---

#Payloads

1. Small. Send an id and a version, not a 2 MB document. Large payloads hit broker limits and become stale between publish and consume.
2. Where the body is genuinely large, use the claim-check pattern: store the blob in object storage and send its key.
3. Versioned and additive. Consumers deploy at different times to producers, so a new required field breaks in-flight messages. Add optional fields; never repurpose an existing one.
4. Include messageId, idempotencyKey, occurredAt, traceparent and a schema version in every message.
5. Propagate trace context so a job links back to the request that created it. → Backend/monitoring

---

#Operations

Monitor these four; the first two are the ones that matter:

ts
// BullMQ: bounded concurrency, backoff with jitter, and a real ack window
new Worker("orders", handler, {
  connection,
  concurrency: 8,                       // ≤ free slots in the database pool
  lockDuration: 60_000,                 // > p99 handler duration
  limiter: { max: 100, duration: 1000 },
});
await queue.add("send-receipt", { orderId }, {
  jobId: `receipt:${orderId}`,          // deduplication key
  attempts: 6,
  backoff: { type: "exponential", delay: 2000 },
  removeOnComplete: { age: 86_400, count: 10_000 },
});

1. Bound consumer concurrency, especially where the handler touches the database — a queue is very good at exhausting a connection pool. → Database/postgres
2. Handle shutdown gracefully: stop fetching, finish in-flight messages, then exit. A SIGKILL mid-handler relies on redelivery to avoid losing work.
3. Separate queues by priority and by workload shape. One slow job type must not block a fast one behind it.
4. Rate-limit calls to external services from consumers; a backlog draining at full speed will exceed a partner's rate limit instantly.

---

#Anti-patterns

---

#Checklist

• Messages are acknowledged only after successful processing
• Acknowledgement deadlines exceed p99 processing time, extended for long jobs
• Every consumer is idempotent, keyed on a business identifier
• State changes and message publication share a transaction via an outbox
• Retries use exponential backoff with jitter and a bounded attempt count
• Retryable and non-retryable failures are distinguished
• Every queue has a dead-letter queue
• DLQ depth alerts, and a replay path exists
• Ordering requirements are explicit; handlers are order-independent by default
• Payloads are small, versioned and additively evolved
• Every message carries id, idempotency key, timestamp, schema version and trace context
• Consumer concurrency is bounded against downstream capacity
• Oldest-message age is monitored and alerted on
• Consumers shut down gracefully, draining in-flight work
• Queues are separated by priority and workload shape