Async Processing & Message Queues
Your user just uploaded a video. Do they really need to wait while you:
- Transcode it to 5 different resolutions?
- Generate thumbnails?
- Run content moderation?
- Update the search index?
That could take minutes. The user would abandon your site.
Solution: Accept the upload, return immediately, and process everything else asynchronously.Sync vs Async
Synchronous
User → API → Process → Response
User waits for everything to complete. Simple but slow for heavy operations.
Asynchronous
User → API → Queue → Response
↓
Worker
User gets immediate response. Heavy work happens in background.
The Message Queue Pattern
- Producer: Sends messages to the queue (your API server)
- Queue: Stores messages until they're processed
- Consumer/Worker: Pulls messages and processes them
When to Use Message Queues
| Use Case | Why Queue? |
|---|---|
| Email/SMS notifications | User doesn't need to wait for email to send |
| Video/image processing | CPU-intensive, can take minutes |
| Analytics/logging | Batch processing is more efficient |
| Webhooks to external services | External services may be slow or down |
| Search indexing | Eventual consistency is acceptable |
Delivery Guarantees
Not all queues are created equal. The delivery guarantee defines reliability:
At-Most-Once
Fire and forget. Message may be lost. Fast but unreliable.
At-Least-Once
Retry until acknowledged. May have duplicates. Most common.
Exactly-Once
Delivered exactly once. Hard to achieve. Expensive.
Handling Duplicates (Idempotency)
With at-least-once delivery, your consumers must be idempotent - processing the same message twice should have the same effect as processing it once.
# Bad: Not idempotent
def process_payment(order_id, amount):
charge_card(amount) # Double charge on retry!
# Good: Idempotent
def process_payment(order_id, amount):
if not payment_exists(order_id): # Check first
charge_card(amount)
record_payment(order_id)Dead Letter Queues
What happens when a message keeps failing? Without handling, it blocks the queue forever.
Dead Letter Queue (DLQ): A separate queue for messages that couldn't be processed after N retries.
DLQs let you:
- Investigate what went wrong
- Replay messages after fixing bugs
- Prevent one bad message from blocking everything
Popular Message Queues
Real-World: Twitter's Kappa Architecture
Twitter moved from a traditional Lambda architecture (batch + real-time) to a streaming-only Kappa architecture:
"We are now in streaming-only mode, removing batch components." Twitter processes billions of events in real-time using Apache Kafka and custom stream processors.
Real-World: Discord's Scale
Discord handles massive real-time communication with a small team:
Discord uses Elixir/BEAM for its WebSocket gateway because of its excellent concurrency model for handling millions of persistent connections.
Pub/Sub Pattern
Beyond simple queues, Publish-Subscribe allows one message to reach multiple subscribers:
Key Takeaways
- Use queues for deferrable work - emails, processing, indexing
- At-least-once is most common - make consumers idempotent
- Dead letter queues prevent poison messages from blocking
- Pub/Sub for fan-out to multiple services
Next up: API Design & Rate Limiting - Building robust APIs that scale.