Real-World Consensus

Consensus algorithms power some of the most critical infrastructure in modern systems. Let's explore how etcd, Consul, and ZooKeeper bring theory to production.

The Big Three

etcd: The Kubernetes Brain

Every Kubernetes cluster runs on etcd. It stores:

etcd Architecture

                    +------------------+
                    |   Kubernetes     |
                    |   API Server     |
                    +--------+---------+
                             |
                    +--------v---------+
                    |      etcd        |
                    |   (Leader)       |
                    +--+----+----+-----+
                       |    |    |
              +--------+    |    +--------+
              |             |             |
        +-----v-----+ +-----v-----+ +-----v-----+
        |   etcd    | |   etcd    | |   etcd    |
        | Follower  | | Follower  | | Follower  |
        +-----------+ +-----------+ +-----------+

Using etcd

# Store a value
etcdctl put /config/database/host "db.example.com"

# Retrieve a value
etcdctl get /config/database/host

# Watch for changes
etcdctl watch /config/database --prefix

# Distributed lock
etcdctl lock mylock
# ... critical section ...
# lock released when process exits

etcd Lease: TTL for Keys

# Create a 60-second lease
etcdctl lease grant 60
# lease 694d71e3b6a1bc0d granted with TTL(60s)

# Attach key to lease
etcdctl put --lease=694d71e3b6a1bc0d /services/api/node1 "alive"

# Key automatically deleted when lease expires
# Use for service discovery / health checking

Consul: Service Mesh Native

Consul adds service discovery, health checking, and service mesh on top of consensus.

Consul Service Registration

{
  "service": {
    "name": "web",
    "port": 8080,
    "check": {
      "http": "http://localhost:8080/health",
      "interval": "10s"
    }
  }
}

# Register service
curl -X PUT -d @service.json http://localhost:8500/v1/agent/service/register

# Discover services
curl http://localhost:8500/v1/catalog/service/web

# Or via DNS
dig @127.0.0.1 -p 8600 web.service.consul

ZooKeeper: The Elder Statesman

ZooKeeper pioneered distributed coordination. It uses Zab (ZooKeeper Atomic Broadcast), a protocol similar to Paxos.

ZooKeeper Data Model

ZooKeeper uses a hierarchical namespace (like a filesystem):

/
├── services/
│   ├── api/
│   │   ├── node1  (data: "host1:8080")
│   │   └── node2  (data: "host2:8080")
│   └── db/
│       └── primary  (data: "db1:5432")
├── locks/
│   └── job-processor  (ephemeral, sequence)
└── config/
    └── feature-flags  (data: {...})

ZooKeeper Node Types

ZooKeeper for Leader Election

// Simplified leader election
public class LeaderElection {
    private ZooKeeper zk;
    private String electionPath = "/election";

    public void participate() {
        // Create ephemeral sequential node
        String myNode = zk.create(
            electionPath + "/candidate_",
            new byte[0],
            OPEN_ACL_UNSAFE,
            EPHEMERAL_SEQUENTIAL
        );
        // myNode = /election/candidate_0000000001

        checkLeadership();
    }

    private void checkLeadership() {
        List<String> children = zk.getChildren(electionPath, false);
        Collections.sort(children);

        String smallest = children.get(0);
        if (myNode.endsWith(smallest)) {
            // I'm the leader!
            becomeLeader();
        } else {
            // Watch the node before me
            watchPredecessor();
        }
    }
}

Comparison

Deployment Best Practices

Cluster Sizing

Geographic Distribution

Datacenter A (Primary)     Datacenter B (Secondary)
+----------+               +----------+
| Node 1   |               | Node 4   |
| Node 2   | <-- Raft -->  | Node 5   |
| Node 3   |               +----------+
+----------+

Majority in DC A ensures local writes are fast.
DC B provides disaster recovery.

Monitoring Key Metrics

Common Patterns

Distributed Lock

# Using etcd
import etcd3

client = etcd3.client()

with client.lock('my-lock') as lock:
    # Only one process can be here at a time
    process_critical_section()

Service Discovery

# Using Consul
import consul

c = consul.Consul()

# Register service
c.agent.service.register(
    'my-service',
    port=8080,
    check=consul.Check.http('http://localhost:8080/health', interval='10s')
)

# Discover services
_, services = c.health.service('my-service', passing=True)
for service in services:
    print(f"{service['Service']['Address']}:{service['Service']['Port']}")

Key Takeaways

Next up: Consensus Assessment - Test your understanding of consensus algorithms.