Background jobs: a practical path to a reliable queue
Writing a background job is easy. Generate a report, send an email, compress an image — all of it happens outside the main request and fits in a few lines of code.
The hard part starts afterwards: what happens if the job dies halfway? If it runs twice? If the external service does not answer? If the server reboots?
This article is about those questions. More precisely, about making a queue reliable.
Why setTimeout is not enough
The simplest "background job" looks like this:
app.post('/orders', async (c) => {
const order = await createOrder(...)
setTimeout(() => sendReceipt(order), 0) // background
return c.json(order)
})This works — right up until the server restarts. If the process goes down, everything queued disappears and nobody finds out.
The core principle is this: work that must happen has to live on disk, not in memory. The job must be written somewhere, then picked up, then marked done.
That is, in fact, the definition of the word "queue".
Idempotency is the first rule
In any system with retries, a job can run more than once. That is not a fallback case, it is the normal state of affairs.
The reason is simple: the worker finished the job but crashed before marking it complete. The queue still considers it open and hands it out again.
So every job must produce the same result when it runs twice as when it runs once. That property is idempotency.
In practice one column usually settles it:
create table receipts (
id uuid primary key,
order_id uuid not null,
idempotency_key text not null unique,
created_at timestamptz not null default now()
);And inside the job:
await db.insert(receipts)
.values({ orderId, idempotencyKey: `receipt:${orderId}` })
.onConflictDoNothing()The second attempt quietly does nothing. This is the most reliable shape, because the protection lives in a database constraint rather than in application logic.
There is an important rule when choosing the key: it must derive from the work itself, not be random. receipt:${orderId} is correct. crypto.randomUUID() is wrong — it differs on every attempt and protects against nothing.
Retries: the right and wrong errors
The second big mistake is treating all failures alike.
Errors come in two kinds.
Transient errors may pass on their own: a network blip, a 503 from an external API, a database connection limit. These deserve a retry.
Permanent errors will never pass: bad data, a nonexistent user, a validation failure. Retrying those wastes resources and floods the logs.
The distinction should be explicit in code:
try {
await sendReceipt(order)
} catch (err) {
if (err.status >= 400 && err.status < 500) {
throw new PermanentError(err.message) // do not retry
}
throw err // transient — retry
}The retry interval should be exponential: 1 second, 2, 4, 8, 16. The reason is practical — if an external service is down, hammering it every second stops it from recovering.
A random jitter helps too. If a hundred jobs fail at once and all retry after exactly 2 seconds, you have arranged your own attack.
The dead letter queue
What do you do with a job once the attempts are exhausted?
The worst option is to delete it. The job vanishes and nobody knows.
The second worst is retrying forever. One broken job clogs the queue and buries your logs.
The right answer is a dead letter queue: failed jobs move to a separate table and stay there.
create table jobs_dead (
id uuid primary key,
type text not null,
payload jsonb not null,
attempts int not null,
last_error text,
failed_at timestamptz not null default now()
);The value of this table is that it tells you a problem exists. I attach a simple check to it: every morning, verify the row count is zero. If it is not, somebody looks.
Remember: a dead letter queue nobody watches is just a slower delete.
Postgres or Redis
Now the main choice. In many projects "we need a queue" turns instantly into "let's install Redis". I would delay that step.
A Postgres queue needs no extra infrastructure if you already run Postgres — and you almost always do. All the magic is in one query:
update jobs
set status = 'running', started_at = now()
where id = (
select id from jobs
where status = 'pending' and run_after <= now()
order by run_after
for update skip locked
limit 1
)
returning *;for update skip locked is the key to that whole statement. It guarantees that even when several workers query simultaneously, each gets a different job. No blocking, no duplicates.
Postgres also gives you transactions: you can create the job and write the business data in one transaction. That is a large advantage — the state "order created but its job got lost" becomes impossible.
Redis cannot give you that guarantee: the database and the queue are two separate systems, and there is always a small window between them.
When do you need a Redis queue (BullMQ and similar)? Three signs:
- High volume — thousands per second. Postgres can handle plenty, but table growth and vacuum become a question.
- You need ready-made infrastructure: scheduling, priorities, rate limiting, progress tracking. Writing that yourself is several days of work.
- You want an off-the-shelf dashboard for inspecting jobs.
If none of the three apply, start with Postgres. Migrating later is not hard, because the job logic does not change — only the storage does.
One trap with scheduled jobs
If your application runs in several instances, a plain cron or @Cron decorator fires in every one of them. The daily report goes out three times.
I have hit this more than once and wrote about it in detail in a separate article. In short: scheduling should also go through the queue — one instance enqueues the job, one worker picks it up.
In Postgres you can do it with a simple lock:
select pg_try_advisory_lock(12345);Only one instance takes the lock; the rest skip the step.
Observability: three numbers are enough
An invisible queue will not tell you when it breaks. I watch three numbers:
- Queue depth — how many jobs are waiting. If it grows, you are short of workers.
- Age of the oldest job — how many minutes it has been waiting. More useful than depth, because it shows latency directly.
- Dead queue size — should be zero.
All three come from a single SQL query and hook into a simple alert. This works without any elaborate monitoring stack.
Conclusion
The minimum for a reliable queue:
- Jobs live on disk, not in memory.
- Every job is idempotent, with a key derived from the work itself.
- Transient and permanent errors are separated.
- Retries are exponential, with jitter.
- Failed jobs land in a dead letter queue, and somebody looks at it.
- Three numbers are monitored.
There is no complicated technology in that list. Postgres, one table and for update skip locked are enough for a great many projects. You move to Redis when a clear signal appears — not because it is fashionable.