Rate Limiting / Throttling

One noisy client can starve every other client

Without a cap, one misbehaving script — or one tenant with an unusually bursty workload — can consume enough of a shared API's capacity that other, well-behaved clients start seeing slow responses or errors, even though they did nothing wrong.

A token bucket per client

Each client gets a bucket that refills with tokens at a fixed rate, up to a cap. Every request consumes one token; a request with no tokens available is rejected (or delayed) until more refill. This allows small bursts while still capping the sustained rate.

A token bucket rate limiter

Tokens refill continuously based on elapsed time; a request is only allowed if at least one token is currently available.

class TokenBucket {
  #capacity; #tokens; #refillPerMs; #lastRefill;

  constructor(capacity, refillPerSecond) {
    this.#capacity = capacity;
    this.#tokens = capacity;
    this.#refillPerMs = refillPerSecond / 1000;
    this.#lastRefill = Date.now();
  }

  #refill() {
    const now = Date.now();
    const elapsed = now - this.#lastRefill;
    this.#tokens = Math.min(this.#capacity, this.#tokens + elapsed * this.#refillPerMs);
    this.#lastRefill = now;
  }

  tryConsume() {
    this.#refill();
    if (this.#tokens >= 1) {
      this.#tokens -= 1;
      return true;
    }
    return false;
  }
}

const bucket = new TokenBucket(5, 1); // 5 burst capacity, refills 1/sec

for (let i = 1; i <= 7; i++) {
  console.log(`request ${i}: ${bucket.tryConsume() ? "200 OK" : "429 rate limited"}`);
}

Trade-offs

• Allows bursts, caps sustained rate — A client can spend a small saved-up burst quickly, but can't sustain a high rate indefinitely.
• Protects capacity fairly — Per-client buckets prevent one tenant's traffic from degrading service for others.
• Needs to be distributed too — A single process's in-memory bucket doesn't limit traffic correctly across multiple service instances — that needs a shared store like Redis.