#Non-negotiable

The constraints hoisted below override anything later in this document. Read them first; the rest is rationale.

---

#Non-negotiable constraints

These override anything later in this document.

• Never assume that publishing A then B means A is processed first. With multiple consumers, it usually is not.

---

#Purpose

Rules for producing and consuming messages. A queue decouples a slow or unreliable operation from a request, and in exchange hands you a distributed-systems problem: at-least-once delivery, no ordering guarantees across partitions, and failures that happen after the work but before the acknowledgement.

Design for the two facts that are always true:

• Every message may be delivered more than once.
• Every message may arrive out of order.

---

#Acknowledge after the work, not before

ts
// Wrong — a crash after ack loses the job silently
await ack(msg);
await process(msg);

// Right — a crash before ack redelivers it
await process(msg);
await ack(msg);

This is why duplicates exist, and it is the correct trade: at-least-once with idempotent consumers is achievable; exactly-once is not, across a network.

Set the acknowledgement deadline (visibilityTimeout, ackDeadline, lockDuration) above the p99 processing time, and extend it for long jobs rather than setting a very large default — a large default means a crashed consumer's message is invisible for that long.

---

#Consumers must be idempotent

Deduplicate on a business key, not a delivery id.

sql
-- The unique constraint is the guarantee; the check is the nice error.
INSERT INTO processed_messages (idempotency_key, processed_at)
VALUES ($1, now())
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING id;                     -- no row returned means already processed

Better still, make the effect itself idempotent — UPDATE orders SET status = 'shipped' WHERE id = $1 AND status = 'paid' is safe to run twice by construction.

Where the handler both writes to the database and enqueues another message, use the transactional outbox: write the message to a table inside the same transaction as the state change, and publish from that table asynchronously. Otherwise the two can diverge — the row commits and the publish fails, or the reverse. → Database/transactions

---

#Retries and dead letters

A dead-letter queue with no alert and no replay tool is a data-loss bucket. Alert on DLQ depth greater than zero, and provide a way to inspect, fix and replay messages. Every system eventually needs to reprocess a window.

Distinguish the failure classes: retrying a message that will never parse burns your retry budget and delays healthy work behind it.

---

#Ordering

Most queues guarantee ordering only within a partition or group key, and only when a single consumer processes that key at a time.

• If order matters, use an ordering key (MessageGroupId, Kafka partition key, RabbitMQ single active consumer) — and accept that it caps parallelism for that key.
• Better: make handlers order-independent. Include a version or updated_at in the payload and discard messages older than the state you already hold.
• Best for entity updates: treat the message as a notification and re-read current state from the source. Ordering stops mattering entirely.

Never assume that publishing A then B means A is processed first. With multiple consumers, it usually is not.

---

#Payloads

• Small. Send an id and a version, not a 2 MB document. Large payloads hit broker limits and become stale between publish and consume.
• Where the body is genuinely large, use the claim-check pattern: store the blob in object storage and send its key.
• Versioned and additive. Consumers deploy at different times to producers, so a new required field breaks in-flight messages. Add optional fields; never repurpose an existing one.
• Include messageId, idempotencyKey, occurredAt, traceparent and a schema version in every message.
• Propagate trace context so a job links back to the request that created it. → Backend/monitoring

---

#Operations

Monitor these four; the first two are the ones that matter:

ts
// BullMQ: bounded concurrency, backoff with jitter, and a real ack window
new Worker("orders", handler, {
  connection,
  concurrency: 8,                       // ≤ free slots in the database pool
  lockDuration: 60_000,                 // > p99 handler duration
  limiter: { max: 100, duration: 1000 },
});
await queue.add("send-receipt", { orderId }, {
  jobId: `receipt:${orderId}`,          // deduplication key
  attempts: 6,
  backoff: { type: "exponential", delay: 2000 },
  removeOnComplete: { age: 86_400, count: 10_000 },
});

• Bound consumer concurrency, especially where the handler touches the database — a queue is very good at exhausting a connection pool. → Database/postgres
• Handle shutdown gracefully: stop fetching, finish in-flight messages, then exit. A SIGKILL mid-handler relies on redelivery to avoid losing work.
• Separate queues by priority and by workload shape. One slow job type must not block a fast one behind it.
• Rate-limit calls to external services from consumers; a backlog draining at full speed will exceed a partner's rate limit instantly.

---

#Anti-patterns

---

#Checklist

• Verify: Messages are acknowledged only after successful processing
• Verify: Acknowledgement deadlines exceed p99 processing time, extended for long jobs
• Verify: Every consumer is idempotent, keyed on a business identifier
• Verify: State changes and message publication share a transaction via an outbox
• Verify: Retries use exponential backoff with jitter and a bounded attempt count
• Verify: Retryable and non-retryable failures are distinguished
• Verify: Every queue has a dead-letter queue
• Verify: DLQ depth alerts, and a replay path exists
• Verify: Ordering requirements are explicit; handlers are order-independent by default
• Verify: Payloads are small, versioned and additively evolved
• Verify: Every message carries id, idempotency key, timestamp, schema version and trace context
• Verify: Consumer concurrency is bounded against downstream capacity
• Verify: Oldest-message age is monitored and alerted on
• Verify: Consumers shut down gracefully, draining in-flight work
• Verify: Queues are separated by priority and workload shape