Circuit Breaker States
A circuit breaker is a state machine with three states. Understanding these states and their transitions is key to using circuit breakers effectively.
┌─────────────────────────────────────────────────────────────────┐
│ │
│ ┌──────────┐ Failures exceed ┌──────────┐ │
│ │ CLOSED │ ────── threshold ───────▶ │ OPEN │ │
│ │ (Green) │ │ (Red) │ │
│ └────┬─────┘ └────┬─────┘ │
│ │ │ │
│ │ │ Timeout │
│ Requests │ expires │
│ succeed │ │
│ │ ▼ │
│ │ ┌───────────┐ │
│ └────────── Success ◀────────── │ HALF-OPEN │ │
│ │ (Yellow) │ │
│ └─────┬─────┘ │
│ │ │
│ Failure ─────────┘ │
│ (reopens) │
└─────────────────────────────────────────────────────────────────┘State 1: Closed (Normal Operation)
In the Closed state, the circuit breaker is "off" - all requests flow through normally.
CLOSED STATE
Behavior: All requests are allowed through to the downstream service.
Monitoring: Tracks success/failure of each request in a sliding window.
Transition: Opens when failure rate exceeds threshold.
What Happens in Closed State
def call_in_closed_state(request):
try:
response = downstream_service.call(request)
record_success()
return response
except Exception as e:
record_failure()
# Check if we should trip the circuit
if failure_rate() > THRESHOLD:
transition_to_open()
raise eTracking Failures
Circuit breakers typically use a sliding window to track recent failures:
Count-Based Window
Track the last N requests.
[✓, ✓, ✗, ✓, ✗, ✗, ✓, ✗, ✗, ✗]
Failure rate: 60%
Time-Based Window
Track requests in the last N seconds.
1000 requests total
350 failures
Failure rate: 35%
State 2: Open (Failing Fast)
In the Open state, the circuit breaker rejects all requests immediately without calling the downstream service.
OPEN STATE
Behavior: All requests fail immediately (no call to downstream).
Purpose: Protect system resources, give downstream time to recover.
Transition: Moves to Half-Open after timeout period.
What Happens in Open State
def call_in_open_state(request):
# Don't even try to call the downstream service
# Fail immediately with a circuit breaker exception
if time_since_opened() > RESET_TIMEOUT:
transition_to_half_open()
return call_in_half_open_state(request)
raise CircuitOpenException("Service unavailable, circuit is open")Benefits of Open State
Milliseconds instead of timeout seconds
No threads blocked waiting
Downstream can recover
Stop hammering broken service
State 3: Half-Open (Testing Recovery)
After the timeout period, the circuit enters Half-Open state to test if the downstream service has recovered.
HALF-OPEN STATE
Behavior: Allow limited requests through to test recovery.
If success: Transition to Closed (service recovered).
If failure: Transition back to Open (still broken).
What Happens in Half-Open State
def call_in_half_open_state(request):
# Only allow one request through at a time
if not acquire_test_lock():
raise CircuitOpenException("Circuit half-open, test in progress")
try:
response = downstream_service.call(request)
# Success! Service has recovered
transition_to_closed()
return response
except Exception as e:
# Still failing - go back to open
transition_to_open()
raise e
finally:
release_test_lock()Half-Open Strategies
Single Request Test
Allow one request. If it succeeds, close the circuit.
Con: One fluke success might close prematurely
Multiple Request Test
Allow N requests. Close only if all (or most) succeed.
Con: Slower recovery, more test traffic
State Transition Diagram
Here's the complete flow with triggers:
| From | To | Trigger |
|---|---|---|
| Closed | Open | Failure rate > threshold (e.g., 50%) |
| Open | Half-Open | Reset timeout expires (e.g., 30 seconds) |
| Half-Open | Closed | Test request(s) succeed |
| Half-Open | Open | Test request fails |
Timeline Example
Let's trace a circuit breaker through a real scenario:
Common Configurations
Different thresholds for different scenarios:
Critical Service
Payment processing - fail fast, recover carefully
reset_timeout: 60s
half_open_requests: 3
Non-Critical Service
Recommendations - tolerate more failures
reset_timeout: 30s
half_open_requests: 1
Key Takeaways
- Closed - Normal operation, monitoring failures
- Open - Failing fast, protecting resources, waiting to test
- Half-Open - Testing if service recovered
- Sliding window tracks recent failures for threshold decisions
Next up: Building a Circuit Breaker - Implement your own circuit breaker from scratch.