Configuration & Fallbacks
A circuit breaker is only as good as its configuration. Too sensitive and it trips unnecessarily. Too lenient and it doesn't protect you. Let's learn how to tune it right.
Key Configuration Parameters
When to open the circuit
How long to stay open
Sample size for calculations
Test requests before closing
Failure Threshold
The failure threshold determines when the circuit opens.
Too Low (10%)
Opens on minor fluctuations. Normal network jitter can trigger it.
Just Right (50%)
Opens when genuinely broken. Tolerates occasional failures.
Too High (90%)
Only opens when almost everything fails. Defeats the purpose.
Guidelines by Service Type
| Service Type | Threshold | Rationale |
|---|---|---|
| Payment/Critical | 25-30% | Fail fast, protect revenue |
| Standard Service | 50% | Balanced approach |
| Non-critical/Enhancement | 60-70% | Tolerate more failures |
| External Third-party | 40-50% | Less control, protect yourself |
Reset Timeout
How long the circuit stays open before testing recovery.
Too short: You'll hammer a recovering service with test requests.
Too long: Users experience degraded service longer than necessary.
Common Reset Timeout Values
# Fast recovery expected (in-memory cache service)
reset_timeout = 10 # seconds
# Standard service
reset_timeout = 30 # seconds
# Slow recovery (database, external API)
reset_timeout = 60 # seconds
# Third-party with rate limits
reset_timeout = 120 # secondsExponential Backoff for Persistent Failures
If a service keeps failing, increase the timeout:
class AdaptiveCircuitBreaker:
def __init__(self):
self.base_timeout = 30
self.max_timeout = 300
self.current_timeout = self.base_timeout
self.consecutive_failures = 0
def on_open(self):
self.consecutive_failures += 1
self.current_timeout = min(
self.base_timeout * (2 ** self.consecutive_failures),
self.max_timeout
)
# 30s -> 60s -> 120s -> 240s -> 300s (max)
def on_close(self):
self.consecutive_failures = 0
self.current_timeout = self.base_timeoutDesigning Fallbacks
The fallback is what happens when the circuit is open. Good fallbacks maintain a usable experience.
BAD FALLBACKS
- Throw generic error to user
- Return empty/null
- Retry indefinitely
- Hide the problem completely
GOOD FALLBACKS
- Return cached data
- Return default values
- Queue for later processing
- Graceful degradation message
Fallback Strategies by Use Case
# 1. CACHED DATA - Great for read operations
@circuit_breaker(fallback=get_cached_recommendations)
def get_recommendations(user_id):
return recommendations_service.get(user_id)
def get_cached_recommendations(user_id):
cached = redis.get(f"recs:{user_id}")
if cached:
return cached
return get_default_recommendations()
# 2. DEFAULT VALUES - For non-critical features
@circuit_breaker(fallback=lambda user_id: {"badge": "member", "tier": 1})
def get_user_badges(user_id):
return badges_service.get(user_id)
# 3. QUEUE FOR LATER - For write operations
@circuit_breaker(fallback=queue_notification)
def send_notification(user_id, message):
return notification_service.send(user_id, message)
def queue_notification(user_id, message):
queue.put({"user_id": user_id, "message": message, "retry_count": 0})
return {"status": "queued", "message": "Notification will be sent shortly"}
# 4. GRACEFUL DEGRADATION - For critical features
@circuit_breaker(fallback=manual_payment_fallback)
def process_payment(order_id, amount):
return payment_service.charge(order_id, amount)
def manual_payment_fallback(order_id, amount):
# Don't process, but give user options
return {
"status": "pending",
"message": "Payment processing is temporarily unavailable.",
"options": [
"Try again in a few minutes",
"Contact support at support@example.com"
]
}Fallback Decision Matrix
| Service | Fallback Strategy | Example |
|---|---|---|
| Recommendations | Cached + Default | Show popular items instead |
| User Profile | Cached | Show last known profile |
| Inventory Check | Optimistic | Assume available, verify later |
| Notifications | Queue | Send later via retry queue |
| Payment | Graceful Error | Clear message + retry option |
| Search | Degraded | Database search instead of Elasticsearch |
Real-World Configuration: Netflix
Netflix's Hystrix (now in maintenance mode, but principles remain):
circuitBreaker.errorThresholdPercentage: 50
circuitBreaker.sleepWindowInMilliseconds: 5000
metrics.rollingStats.timeInMilliseconds: 10000
Translation: Track last 10 seconds of requests. If >20 requests and >50% failures, open circuit. Test again after 5 seconds.
Common Pitfalls
Fallback Calls Failing Service
Your fallback shouldn't depend on the same broken service.
fallback = lambda: service.get_backup()
Not Counting Timeouts
Timeout exceptions should count as failures too.
except ConnectionError: record_failure()
Same Config Everywhere
Different services have different needs. Don't use one-size-fits-all.
DEFAULT_CONFIG = {...}
No Monitoring
You need to know when circuits trip and why.
circuit.on_open(lambda: None)
Configuration Checklist
Before deploying, verify:
- Threshold tested? - Does it trip at the right failure rate?
- Timeout appropriate? - Long enough for service to recover?
- Fallback working? - Does it return useful data?
- Fallback independent? - Doesn't call the failing service?
- Metrics in place? - Can you see when it trips?
- Alerts configured? - Will you know about extended outages?
Key Takeaways
- Tune thresholds per service - Critical services need lower thresholds
- Use exponential backoff for persistent failures
- Design thoughtful fallbacks - Cached data, defaults, queuing
- Ensure fallback independence - Don't call the broken service
- Monitor and alert - Know when circuits trip
Next up: Circuit Breaker Assessment - Test your understanding with real-world scenarios.