Load Balancing
You've scaled horizontally - you now have multiple servers. But how do users' requests get distributed across them? That's where load balancers come in.
The Traffic Cop Analogy
Think of a load balancer as a traffic cop at a busy intersection. Without one, all cars (requests) would try to go down the same street (server), causing gridlock. The traffic cop directs cars down different streets to keep traffic flowing.
Load Balancing Algorithms
1. Round Robin
The simplest approach: distribute requests in a circular order.
Request 1 → Server A
Request 2 → Server B
Request 3 → Server C
Request 4 → Server A (back to start)
...2. Weighted Round Robin
Like round robin, but servers with more capacity get more requests.
Server A (weight: 3) → gets 3 requests
Server B (weight: 1) → gets 1 request
Server C (weight: 2) → gets 2 requests3. Least Connections
Send requests to the server with the fewest active connections.
4. IP Hash / Consistent Hashing
Route based on client IP address hash. Same client always hits same server.
server_index = hash(client_ip) % num_serversL4 vs L7 Load Balancing
L4 (Transport Layer)
- Works at TCP/UDP level
- Can't see HTTP content
- Very fast, low latency
- Routes based on IP + port
L7 (Application Layer)
- Works at HTTP level
- Can inspect URLs, headers, cookies
- Slightly higher latency
- Route based on content
L7 Routing Example
# Route /api/* to API servers
location /api/ {
proxy_pass http://api_servers;
}
# Route /images/* to CDN
location /images/ {
proxy_pass http://cdn_servers;
}
# Route everything else to web servers
location / {
proxy_pass http://web_servers;
}Health Checks
Load balancers continuously monitor server health. If a server stops responding, it's removed from the pool.
| Check Type | How It Works | Use Case |
|---|---|---|
| TCP Check | Can we open a TCP connection? | Basic availability |
| HTTP Check | Does /health return 200? | App is running |
| Deep Health | Check DB, cache connections | Full system health |
Real-World: Meta's Katran
Meta (Facebook) built Katran, an open-source L4 load balancer that uses eBPF for ultra-high performance:
- Uses XDP (eXpress Data Path) for packet processing in kernel
- Maglev hash for consistent backend selection
- Configurable LRU cache for connection tracking
- Handles millions of packets per second
Real-World: Instagram's Journey
"Every request to Instagram servers goes through load balancing machines; we used to run 2 nginx machines and DNS Round-Robin between them. The downside was the time for DNS to update if a machine needed decommissioning. We moved to using Amazon's Elastic Load Balancer, with 3 NGINX instances behind it that can be swapped in and out."
Common Load Balancers
Key Takeaways
- Load balancers distribute traffic across multiple servers
- Algorithms matter: Round-robin for simplicity, least connections for variable workloads, IP hash for stickiness
- L4 vs L7: L4 is faster, L7 is smarter (content-based routing)
- Health checks automatically remove failing servers from the pool
Next up: Caching Strategies - How to dramatically speed up reads with caching.