What is Database Sharding?
Instagram stores over 2 billion photos. Facebook handles 4 petabytes of new data daily. Discord manages 4 billion messages per day. How do you store this much data when a single database server maxes out at a few terabytes?
The answer is sharding - splitting your database horizontally across multiple servers.
SINGLE DATABASE
- All data on one server
- Limited by hardware
- Single point of failure
- Write bottleneck
SHARDED DATABASE
- Data split across servers
- Horizontally scalable
- No single point of failure
- Distributed writes
The Problem: Database Limits
Every database has limits. Even the most powerful server eventually hits a wall:
~64TB max per server
~100K writes/sec
~10K concurrent
~1-2TB RAM max
Vertical Partitioning vs Horizontal Partitioning
Before sharding, let's clarify terminology:
Vertical Partitioning
Split by columns. Put frequently accessed columns in one table, rarely accessed in another.
users_profile: id, bio, avatar, settings
Horizontal Partitioning (Sharding)
Split by rows. Distribute rows across multiple database servers.
Shard 2: users 1M-2M
Shard 3: users 2M-3M
How Sharding Works
In a sharded architecture, data is distributed across multiple database servers called shards. Each shard holds a subset of the total data.
┌─────────────────┐
│ Application │
└────────┬────────┘
│
┌────────▼────────┐
│ Shard Router │
│ (Which shard?) │
└────────┬────────┘
│
┌────────────────────┼────────────────────┐
│ │ │
┌───────▼───────┐ ┌───────▼───────┐ ┌───────▼───────┐
│ Shard 1 │ │ Shard 2 │ │ Shard 3 │
│ Users A-H │ │ Users I-P │ │ Users Q-Z │
└───────────────┘ └───────────────┘ └───────────────┘The Shard Key
The shard key determines which shard a piece of data belongs to. For example:
# Using user_id as shard key
def get_shard(user_id, num_shards):
return user_id % num_shards
# User 12345 with 3 shards
shard = 12345 % 3 # = 0, goes to Shard 0Choosing the right shard key is the most important sharding decision. A bad shard key leads to hotspots, complex queries, and operational nightmares. We'll cover this in detail later.
When to Shard
Sharding adds significant complexity. Don't shard prematurely.
- Optimize queries and indexes
- Add read replicas
- Implement caching
- Vertical scaling (bigger server)
- Archive old data
- Data exceeds single server capacity
- Write throughput is bottleneck
- Vertical scaling costs are prohibitive
- Geographic distribution needed
Real-World Example: Discord
Discord stores trillions of messages. In 2017, they moved from MongoDB to Cassandra, and later migrated to ScyllaDB with a clever sharding strategy.
Discord shards messages by channel_id and buckets them by time. This keeps messages from the same channel together while distributing load across nodes. A channel's recent messages are always on the same shard, making reads fast.
Sharding Challenges
Sharding isn't free. Here's what you sign up for:
| Challenge | Description |
|---|---|
| Cross-shard queries | JOINs across shards are expensive or impossible |
| Transactions | ACID transactions don't work across shards |
| Rebalancing | Adding/removing shards requires data migration |
| Hotspots | Uneven data distribution causes bottlenecks |
| Operational complexity | Backups, monitoring, failover are harder |
What You'll Learn in This Course
Hash, Range, Directory-based
Choosing the right key
Consistent hashing, migrations
JOINs and aggregations
Key Takeaways
- Sharding = horizontal partitioning - splitting rows across multiple database servers
- The shard key determines which shard holds each piece of data
- Don't shard prematurely - try optimization, caching, and replicas first
- Sharding adds complexity - cross-shard queries, transactions, and operations become harder
Next up: Sharding Strategies - Learn the different approaches to distributing your data.