#Non-negotiable

The constraints hoisted below override anything later in this document. Read them first; the rest is rationale.

---

#Non-negotiable constraints

These override anything later in this document.

• Never write a job that processes an unbounded set in one execution. It will eventually exceed every timeout you have, and a failure at 90% loses all of it.

---

#Purpose

Rules for work that happens outside an HTTP request: emails, exports, imports, webhooks, nightly reconciliation, cleanup. Transport mechanics are Backend/queues; this package is about designing the jobs themselves.

Move work into a job when it is slow, retryable, or must survive the caller disconnecting. Keep it in the request when the user needs the result now.

---

#Enqueue after commit

ts
// Broken — the job may run before (or without) the transaction committing.
// The worker reads a row that does not exist yet.
await db.$transaction(async (tx) => {
  const order = await tx.order.create({ data });
  await queue.add("send-receipt", { orderId: order.id });   // ← wrong
});

// Correct — write the intent inside the transaction, publish after it commits
await db.$transaction(async (tx) => {
  const order = await tx.order.create({ data });
  await tx.outbox.create({ data: { type: "send-receipt", payload: { orderId: order.id } } });
});

The race is real and intermittent: with a fast worker, the job starts before the commit lands. Use a transactional outbox, or enqueue strictly after the transaction returns and accept that a crash in between loses the job. → Database/transactions

Pass identifiers, not objects. A serialised entity in a payload is stale by the time the worker runs, and it grows without bound as the model grows.

---

#Design of a job

ts
// Fan-out: the parent job enqueues children and finishes fast.
// One failure retries one item, not the whole batch.
for (const id of await findDueSubscriptionIds({ limit: 10_000 })) {
  await queue.add("renew-subscription", { id }, { jobId: `renew:${id}:${period}` });
}

The deterministic jobId is the deduplication: enqueuing the same renewal twice for the same period is a no-op.

Never write a job that processes an unbounded set in one execution. It will eventually exceed every timeout you have, and a failure at 90% loses all of it.

---

#Scheduling

Cron in a distributed system has three failure modes that a single server does not:

1. Multiple instances run it. N replicas means N executions. Use the scheduler's own leader election, a distributed lock with a TTL, or a platform scheduler that guarantees a single invocation.
2. A missed window is silent. If the scheduler was down at 02:00, nothing reports that the job never ran. Alert on absence of a successful run, not only on failure.
3. Overlap. A run that takes longer than the interval collides with the next. Configure non-overlapping execution explicitly.

Other rules:

• Schedule in UTC. A cron in local time runs twice or zero times on daylight-saving transitions.
• Jitter schedules across tenants and instances. Every job at 0 0 * * * creates a thundering herd at midnight.
• Prefer event-driven work to polling. Where you must poll, poll a marker (WHERE processed_at IS NULL), not the whole table.

An in-process cron in a horizontally scaled deployment is the most common version of this bug: it works in staging on one replica and triple-charges customers in production on three.

---

#Timeouts, resources and failure

• Set a timeout per job type, and make it shorter than the acknowledgement deadline so a hung job is reclaimed rather than run twice concurrently.
• Bound worker concurrency against the narrowest downstream resource — usually the database connection pool, sometimes a partner's rate limit.
• Cap memory: a job that loads a whole table into memory works in staging and OOMs in production. Stream and paginate.
• Isolate workloads. Long exports and quick emails on the same worker pool means the export starves the email.
• On repeated failure, dead-letter with the full context needed to diagnose it, and alert. → Backend/queues

---

#Observability

A job that fails silently is worse than no job — the system appears to work.

Emit for every job: type, id, outcome, duration, attempt number, and the trace context of the request that created it.

Log at start and completion with the same job id so a run can be reconstructed, and propagate traceparent from the enqueuing request.

ts
const jobDuration = new Histogram({
  name: "job_duration_seconds",
  labelNames: ["job_type", "outcome"],          // bounded label set
  buckets: [0.1, 0.5, 1, 5, 15, 60, 300],
});
const jobAttempts = new Counter({ name: "job_attempts_total", labelNames: ["job_type", "outcome"] });
const oldestPending = new Gauge({ name: "job_oldest_pending_seconds", labelNames: ["job_type"] });

yaml
# Alert on a scheduled job that never ran — failure alerts cannot catch absence.
- alert: NightlyReconciliationMissing
  expr: |
    time() - max(job_last_success_timestamp_seconds{job_type="reconcile"}) > 93600
  for: 15m
  annotations:
    runbook: https://runbooks.example.com/reconcile-missing

→ Backend/monitoring

---

#Operating

• Deploys must drain: stop accepting new jobs, finish in-flight work with a bounded timeout, then exit. Workers being SIGKILLed mid-job relies on redelivery every single deploy.
• Version compatibility: workers and producers deploy at different times, so a worker must tolerate both the old and new payload shape during the rollout.
• Keep a manual replay and cancel path. Every system eventually needs to re-run yesterday's failed batch or stop a runaway job.
• Retain job history long enough to answer "did this customer's export run?" — usually 7–30 days. In BullMQ that is removeOnComplete: { age }; in Sidekiq it is the dead_max_jobs and dead_timeout_in_seconds settings. Unbounded history is a slow memory leak in the broker.

---

#Anti-patterns

---

#Checklist

• Verify: Jobs are enqueued after commit, or via a transactional outbox
• Verify: Payloads carry identifiers and a schema version, not serialised entities
• Verify: Every job is idempotent under repeated execution
• Verify: Large workloads fan out into small, independently retryable jobs
• Verify: Long jobs checkpoint progress so retries resume
• Verify: Every job type has an explicit timeout below the ack deadline
• Verify: Scheduled jobs run once across all replicas
• Verify: Schedules are in UTC, jittered, and non-overlapping
• Verify: Absence of a successful scheduled run raises an alert
• Verify: Worker concurrency is bounded by the narrowest downstream resource
• Verify: Memory use is bounded; large datasets are streamed
• Verify: Fast and slow workloads run on separate pools
• Verify: Job type, outcome, duration and attempt are emitted as metrics
• Verify: Trace context propagates from the enqueuing request
• Verify: Workers drain gracefully on deploy
• Verify: Workers tolerate both payload versions during a rollout
• Verify: A manual replay and cancel path exists