Monitoring Backend Health
Your load balancer sends a request to Server A. Server A crashed 10 seconds ago. The request times out. User is unhappy.
Health checks solve this by continuously probing servers and removing unhealthy ones from the pool before real traffic is affected.How Health Checks Work
Load Balancer Backend Servers
┌─────────────┐ ┌─────────────┐
│ │ ── GET /health ────▶ │ Server A │ ✓ 200 OK
│ │ ◀─────────────────── │ HEALTHY │
│ Health │ └─────────────┘
│ Check │
│ Timer │ ┌─────────────┐
│ (every 5s) │ ── GET /health ────▶ │ Server B │ ✗ Timeout
│ │ ◀─── No response ── │ CRASHED │
│ │ └─────────────┘
│ │ → Remove from pool
└─────────────┘Types of Health Checks
TCP Health Check
Can we open a TCP connection?
Connect to 10.0.0.1:8080
If connection succeeds → Healthy
If connection refused → UnhealthyFast but shallow - only checks if port is open
HTTP Health Check
Does the application respond correctly?
GET /health HTTP/1.1
200 OK → Healthy
5xx or timeout → UnhealthyBetter - verifies application is running
Deep Health Check
Are all dependencies healthy?
Check database connection
Check Redis connection
Check external API
If all pass → 200 OKThorough - catches dependency failures
Designing Health Check Endpoints
Shallow (Liveness) Check
Just verifies the process is running:
// Express.js
app.get('/health/live', (req, res) => {
res.status(200).json({ status: 'alive' });
});Deep (Readiness) Check
Verifies the service can actually handle requests:
app.get('/health/ready', async (req, res) => {
const checks = {
database: await checkDatabase(),
cache: await checkRedis(),
diskSpace: checkDiskSpace()
};
const allHealthy = Object.values(checks).every(c => c.healthy);
res.status(allHealthy ? 200 : 503).json({
status: allHealthy ? 'ready' : 'not ready',
checks
});
});
async function checkDatabase() {
try {
await db.query('SELECT 1');
return { healthy: true };
} catch (error) {
return { healthy: false, error: error.message };
}
}Response Example
{
"status": "ready",
"checks": {
"database": {
"healthy": true,
"latency_ms": 5
},
"cache": {
"healthy": true,
"latency_ms": 2
},
"diskSpace": {
"healthy": true,
"available_gb": 45
}
},
"version": "2.1.0",
"uptime_seconds": 86400
}Health Check Configuration
| Parameter | Description | Typical Value |
|---|---|---|
| Interval | How often to check | 5-30 seconds |
| Timeout | How long to wait for response | 2-5 seconds |
| Unhealthy Threshold | Failures before marking unhealthy | 2-3 failures |
| Healthy Threshold | Successes before marking healthy | 2-3 successes |
NGINX Configuration
upstream backend {
server 10.0.0.1:8080 max_fails=3 fail_timeout=30s;
server 10.0.0.2:8080 max_fails=3 fail_timeout=30s;
server 10.0.0.3:8080 max_fails=3 fail_timeout=30s;
}
# Active health checks (NGINX Plus)
upstream backend_active {
zone backend 64k;
server 10.0.0.1:8080;
server 10.0.0.2:8080;
health_check interval=5s fails=3 passes=2 uri=/health;
}AWS ALB Configuration
# CloudFormation
TargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
HealthCheckEnabled: true
HealthCheckIntervalSeconds: 30
HealthCheckPath: /health
HealthCheckProtocol: HTTP
HealthCheckTimeoutSeconds: 5
HealthyThresholdCount: 2
UnhealthyThresholdCount: 3
Matcher:
HttpCode: 200Passive vs Active Health Checks
ACTIVE HEALTH CHECKS
Load balancer proactively sends health check requests at regular intervals.
- Detects failures quickly
- Works even with no traffic
- Adds some overhead
PASSIVE HEALTH CHECKS
Load balancer monitors actual request/response traffic for failures.
- No extra traffic overhead
- Detects real-world failures
- Requires actual traffic to work
The Thundering Herd Problem
Watch Out!
A server fails. It's removed from the pool. Remaining servers get more traffic. They fail under increased load. More servers removed. Cascade failure!
Solution: Use gradual re-introduction. When a server becomes healthy, slowly increase its traffic instead of immediately sending full load.
Graceful Degradation Pattern
app.get('/health/ready', async (req, res) => {
const checks = await runHealthChecks();
// Degraded but functional
if (checks.database.healthy && !checks.cache.healthy) {
return res.status(200).json({
status: 'degraded',
message: 'Cache unavailable, operating with database only',
checks
});
}
// Critical failure
if (!checks.database.healthy) {
return res.status(503).json({
status: 'unhealthy',
message: 'Database unavailable',
checks
});
}
// Fully healthy
res.status(200).json({
status: 'healthy',
checks
});
});Real-World: Health Check Best Practices
- Keep health checks fast - A slow health check can timeout and mark healthy servers as down
- Separate liveness from readiness - Liveness = "am I alive?", Readiness = "can I serve traffic?"
- Don't check external dependencies in liveness - External API down shouldn't kill your pods
- Include version info - Helps with debugging and deployment verification
Connection Draining
Before removing a server, drain existing connections gracefully:
1. Stop sending NEW requests to server
2. Wait for existing requests to complete (e.g., 30 seconds)
3. If timeout, force close remaining connections
4. Remove server from pool
Timeline:
─────────────────────────────────────────────
│ Active │ Draining │ Removed
│ (serving) │ (no new reqs) │
─────────────────────────────────────────────
▲ ▲
Health check Drain
fails timeoutKey Takeaways
- Health checks prevent traffic to dead servers - Automatic failover
- TCP vs HTTP vs Deep - Choose based on what failures you need to detect
- Configure thresholds wisely - Avoid flapping (healthy/unhealthy/healthy)
- Design endpoints carefully - Fast, accurate, and informative
Next up: L4 vs L7 Load Balancers - Understanding the layers of load balancing.