Cloud · Messaging · DWG NO. 030

Competing Consumers

Run several workers pulling from one queue so processing throughput scales by adding more workers.

TypeMessaging
ScopeConsumer side
ComplexityLow–Medium
Common inBackground job processing, video transcoding, batch pipelines
01 · The problem

One consumer can't keep up, and a queue alone doesn't fix that

Queue-Based Load Leveling smooths bursts, but if the average arrival rate simply exceeds what one consumer can process, the queue just grows without bound. What's needed is a way to add processing capacity, not just buffering.

02 · Structure

Many workers, one shared queue

Multiple consumer instances pull from the same queue. Each message is delivered to exactly one worker — the queue itself (or broker) guarantees that, so workers don't need to coordinate with each other. Adding throughput is just adding another worker process.

QueueWorker AWorker BWorker C
03 · Reference implementation

Multiple workers competing for the same queue

Each `dequeue()` call removes the item for whichever worker calls it first — no message is processed twice, no coordination needed between workers.

competing-consumers.js 
class JobQueue {
  #items = [];
  enqueue(job) { this.#items.push(job); }
  dequeue() { return this.#items.shift(); } // atomic hand-off — one job, one worker
}

const queue = new JobQueue();
for (let i = 1; i <= 6; i++) queue.enqueue(`job-${i}`);

function startWorker(name) {
  return setInterval(() => {
    const job = queue.dequeue();
    if (job) console.log(`${name} processing ${job}`);
  }, 100);
}

const workers = [startWorker("worker-A"), startWorker("worker-B"), startWorker("worker-C")];
// All three pull from the same queue — throughput scales roughly with worker count.

setTimeout(() => workers.forEach(clearInterval), 500);
04 · When to use it

Trade-offs