Cloud · Scale & Traffic · DWG NO. 039

Sharding

Partition data across multiple stores by a key, so the system scales past what a single database node can hold or serve.

TypeScale/Traffic
ScopeData storage layer
ComplexityMedium–High
Common inMulti-tenant databases, large user bases, high-write systems
01 · The problem

One database node has a ceiling

A single database instance has finite storage, memory, and throughput. Once a table's data volume or write rate outgrows what one node can handle well, adding more application servers in front of it doesn't help — the database itself is the bottleneck.

02 · Structure

Split data across nodes by a shard key

Data is partitioned across multiple database instances (shards) based on a chosen key — often a user ID or tenant ID. A routing layer computes which shard owns a given key and directs reads and writes there. Each shard only holds a fraction of the total data.

Routerhash(key) % NShard 0users 0-999Shard 1users 1000-1999Shard 2users 2000-2999
03 · Reference implementation

A simple hash-based shard router

`shardFor(key)` deterministically picks the same shard for the same key every time, so reads and writes for that key always land in the same place.

shard-router.js 
const shards = [
  { name: "shard-0", data: new Map() },
  { name: "shard-1", data: new Map() },
  { name: "shard-2", data: new Map() },
];

function hash(key) {
  let h = 0;
  for (const char of key) h = (h * 31 + char.charCodeAt(0)) >>> 0;
  return h;
}

function shardFor(key) {
  return shards[hash(key) % shards.length];
}

function writeUser(id, data) {
  const shard = shardFor(id);
  shard.data.set(id, data);
  console.log(`wrote ${id} to ${shard.name}`);
}

function readUser(id) {
  return shardFor(id).data.get(id); // routes to the same shard deterministically
}

writeUser("user-42", { name: "Alex" });
writeUser("user-99", { name: "Sam" });
console.log(readUser("user-42"));
04 · When to use it

Trade-offs