← Back to home

Flagship case study

Go Distributed Queue

A personal systems project built to make background work survive the uncomfortable cases: worker death, duplicate ownership, delayed retries, and poison jobs.

Go · Redis · Lua · Prometheus 2025 GitHub repo ↗

SUMMARY

Redis-backed task queue with leases, atomic ownership transitions, retries, delayed jobs, priority queues, DLQ behavior, chaos testing, and Prometheus metrics.

Problem

A queue demo is easy until workers die mid-task, duplicate ownership appears, retries overwhelm the system, or expired leases never return work to pending.

What I built

I built a Redis-backed queue with at-least-once delivery, atomic claim/extend/reclaim scripts, priority queues, delayed scheduling, exponential backoff, DLQ behavior, and Prometheus instrumentation.

Architecture / system design

The core follows the Reliable Queue Pattern: pending queues, processing leases, Lua-backed atomic transitions, reaper sweeps, delayed retries, and worker APIs around those state transitions.

Failure modes / what broke

The key bug was lease extension racing with reclaim: without a single atomic check-and-extend operation, two consumers could believe they owned the same task.

Proof / metrics / tests

With a 30-second lease and 5-second reaper sweep, the documented reclaim bound is about 35 seconds. The page also includes a live browser simulation of worker death, requeue, retry, and DLQ flow.

Lessons learned

The important part of queue design is not pushing jobs into Redis; it is naming every state transition and making ownership changes atomic.

field notes

Expanded field notes

WHY THIS PROJECT EXISTS

This is a personal systems project - built to study the full lifecycle of reliable background work: enqueue, lease, process, retry, reclaim, fail, inspect, recover. Not presented as a company-scale production service. It's a production-style prototype that intentionally tries to hurt itself so the failure modes have somewhere to show up.

A fault-tolerant task queue built on the Reliable Queue Pattern: at-least-once delivery through atomic Redis LMOVE operations, multi-tier priority queues, delayed scheduling via sorted sets, batch ingestion, and dead-letter queues for poison messages. CI is green; the README has the architecture diagram in Mermaid.

The interesting part is failure. A lease-based reaper watches for workers that died mid-task and atomically reclaims their work via Lua, and the whole system is instrumented with native Prometheus metrics. With a 30-second lease and 5-second reaper sweep, the reclaim bound is provable:

reclaim_latencymax ≈ LEASE_TTL + REAPER_INTERVAL ≈ 35s

For non-idempotent handlers, producers can attach an idempotency_key - workers use a claim-then-confirm pattern to guarantee at-most-once execution within the configured TTL window. I validated all of this the only way that counts: automated chaos tests that kill workers at random while the queue is under load.

WHAT BROKE

Chaos testing exposed a race between a slow worker renewing its lease and the reaper reclaiming the same task — two consumers, one job. The fix was making check-and-extend a single atomic Lua script on Redis, closing the window entirely.

35s
max reclaim after worker death
≥1
delivery guarantee
CI ✓
github actions green
Distributed queue architecture producers redis pending → processing delayed zset priority tiers worker 1 worker 2 worker n reaper (lua) dlq reclaims dead leases chaos tests kill workers at random under load
fig. 1 — reliable queue pattern with lease reaper and dead-letter path for poison messages.

hover or tab through any node to trace the flow

processed 0 · reclaimed 0 · dlq 0

fig. 1b — live in-browser simulation, modeled on the repo's two separate failure paths. Reclaim path: press kill worker - the job freezes, its lease counts down (red bar), then the reaper sweep atomically moves it back to pending with its retry count unchanged (matching the Lua reclaim script). Retry path: workers also fail ~30% of completions (mirroring the repo's WORKER_FAILURE_PCT=25). A failed job is retried with an incremented count; once it has failed four times, it lands in the dead-letter queue. High-priority jobs (darker, tagged H) jump the queue, mirroring the high → default → low poll order. Lease TTL and reaper interval are compressed for the demo — the real reclaim bound is ~35s, and the delayed-queue + exponential-backoff scheduler is not modeled here.

contact

Open to backend systems, AI infrastructure, and product engineering roles.

yehiasaidgewily@gmail.com GitHub LinkedIn