Background jobs that do not vanish
a queue built for retries, not hope

What it is

A background job queue takes work out of the request-response cycle, sending an email, generating a report, resizing an image, calling a slow third-party API, and runs it separately, with retries if it fails and visibility into what happened. Without one, that work either blocks the user’s request until it finishes, which is slow, or runs as a fire-and-forget call that fails silently if anything goes wrong, which is worse.

When you need it (and when you do not)

You need this once your application has any task that can fail for reasons outside your control, an email provider having a bad minute, a PDF generation library choking on an unusual input, a third-party API timing out, and that failure currently either blocks a user or disappears without a trace. It is also the right build once you have scheduled tasks, nightly reports, hourly syncs, cleanup jobs, scattered across cron entries with no shared monitoring or overlap protection.

You do not need this for a handful of jobs that genuinely never fail and run instantly, a cron job calling a script is fine until it is not. The tell that you have outgrown it is the first time someone asks “did that export actually run last night?” and nobody can answer without SSHing into a server and reading raw logs.

How we build it

For Python backends we use Celery with Redis or RabbitMQ as the broker, the most battle-tested combination for this, with tasks organized by priority so a user-triggered export does not sit behind a bulk nightly job. For Node.js backends, BullMQ on Redis does the same job with less overhead. Every job type gets a defined retry policy: exponential backoff for things like a flaky third-party API, no retry at all for things where retrying would cause harm, like sending the same email twice. Jobs that exhaust their retries move to a dead-letter queue with full context attached, not just an error message, so debugging does not require reproducing the failure from scratch. We built exactly this kind of job infrastructure underneath a digital goods marketplace’s fulfillment flow, where a delivery job failing silently means a paid customer does not receive their product.

What to watch

The most common failure mode in a queue system is not the queue itself, it is a worker silently falling behind under load while everything still looks fine from the outside, jobs are “running,” just slower every day. We build monitoring on queue depth and processing time into the initial setup specifically to catch this before it becomes a backlog measured in days. The other thing to watch is job idempotency: if a job can retry, it has to be safe to run twice, which sometimes means redesigning a job’s logic, not just adding a retry decorator around it. We flag any job where that is not naturally true before building it. Lock-in is minimal, Celery, RQ and BullMQ are all open source, and the job logic itself lives in your application code, not in a vendor’s platform.

Worker scaling is worth planning before the first real load spike, not during one: a queue that backs up because only one worker process is running is a capacity problem, not a software bug, and the fix is adding workers, not debugging code that was never broken in the first place. We size initial worker counts against realistic peak load rather than average load specifically to avoid discovering this gap live.

Price and timeline

Related

Built as part of custom development. Often paired with webhook and event bus infrastructure when jobs are triggered by external events, and monitoring and observability to watch worker health in production. See it running underneath ProBay’s AI agent team and digital goods marketplace automation. Tell us what work needs to move off the request path: get in touch.