Why Circuit Breakers?
On November 24, 2014, AWS experienced a major outage. A small issue in one service cascaded through dependencies, taking down Netflix, Tinder, IMDb, and dozens of other sites. The problem? Cascading failures.
When services depend on each other, one failure can bring down everything.
WITHOUT CIRCUIT BREAKER
- Service A calls failing Service B
- Requests pile up waiting for timeouts
- Thread pool exhausted
- Service A starts failing too
- Cascade continues upstream
WITH CIRCUIT BREAKER
- Service A calls failing Service B
- Circuit breaker detects failures
- Circuit "opens" - stops calling B
- Service A returns fallback response
- System remains functional
The Anatomy of a Cascading Failure
Let's trace how a small problem becomes a major outage:
1. Payment Service has a bug, starts returning errors after 30 seconds
2. Checkout Service calls Payment Service
- Each request waits 30 seconds for timeout
- 100 requests/sec × 30 sec = 3,000 requests waiting
- Thread pool: 200 threads → all blocked!
3. Checkout Service can't handle ANY requests now
4. Product Page calls Checkout for "Buy Now" button
- Checkout times out → Product Page times out
- Product Page thread pool exhausted
5. Homepage calls Product Page...
(entire site is now down)One slow service doesn't just affect its callers - it affects their callers, and so on. A 30-second timeout deep in your service graph can bring down your entire application in minutes.
The Circuit Breaker Metaphor
The pattern is inspired by electrical circuit breakers in your home:
Electrical Circuit Breaker
- Detects electrical overload
- "Trips" to stop current flow
- Prevents fire/damage
- Can be reset after problem fixed
Software Circuit Breaker
- Detects service failures
- "Trips" to stop requests
- Prevents cascading failures
- Auto-resets after service recovers
How Circuit Breakers Work
A circuit breaker wraps calls to external services and monitors for failures:
class CircuitBreaker:
def call(self, service_function):
if self.is_open():
# Circuit is open - don't even try
return fallback_response()
try:
result = service_function()
self.record_success()
return result
except Exception:
self.record_failure()
if self.failure_threshold_reached():
self.open_circuit()
raiseThe Three States
Normal operation. Requests flow through.
Failure detected. Requests rejected immediately.
Testing recovery. Limited requests allowed.
Why Not Just Use Timeouts?
Timeouts are necessary but not sufficient:
| Scenario | Timeout Only | Circuit Breaker |
|---|---|---|
| Service down for 5 minutes | Every request waits for timeout | Fails fast after first few failures |
| Resource usage | Threads blocked waiting | Threads freed immediately |
| User experience | Slow failure (30s wait) | Fast fallback response |
| Recovery detection | Keep hammering broken service | Periodic health checks |
Real-World Example: Netflix Hystrix
Netflix pioneered circuit breakers in microservices with their Hystrix library:
"In a distributed environment, failure of any given service is inevitable. Hystrix isolates the points of access between services, stops cascading failures across them, and provides fallback options."
— Netflix Tech Blog
Netflix uses circuit breakers on every service call. When the recommendations service is down, you still see the Netflix homepage - just with cached or generic recommendations.
When to Use Circuit Breakers
- Calls to external APIs
- Calls to other microservices
- Database connections
- Any remote call that might fail
- In-memory operations
- Local file access
- CPU-bound calculations
- Operations that can't have fallbacks
What You'll Learn in This Course
Closed, Open, Half-Open states
Build your own circuit breaker
Thresholds, timeouts, fallbacks
Real-world patterns
Key Takeaways
- Cascading failures - One failing service can bring down your entire system
- Timeouts aren't enough - They still waste resources waiting for failures
- Fail fast - Circuit breakers stop calling broken services immediately
- Graceful degradation - Return fallback responses instead of errors
Next up: Circuit Breaker States - Deep dive into Closed, Open, and Half-Open states.