Redis Cluster Failover Split-Brain: Detection and Recovery

Redis Cluster uses a gossip protocol to propagate node state. Each node maintains a configuration epoch (configEpoch) that is incremented during failovers. When a master fails, replicas attempt to get a majority of votes from other masters to become the new master. During a network partition, both sides may independently achieve a majority within their partition if the partition splits the cluster into two groups that each have a majority of masters (e.g., 2+2 in a 4-master cluster, but Redis Cluster requires odd number of masters? Actually, it requires at least 3 masters for quorum; a 3-master cluster split 2-1 gives majority on one side; but if split 1-1-1? Not possible; typical split-brain occurs with 3 masters and 3 replicas, where a partition isolates 1 master and its replica, leaving 2 masters and 2 replicas on the other side. The majority side (2 masters) can elect a new master for the isolated master's slots because they have quorum (2 > 1). However, the isolated side also has a master (the original) and its replica; they cannot elect a new master because they lack a majority of masters, but the original master remains active and can still serve slots. This creates a conflict: the majority side has a new master with a higher configEpoch, while the isolated side has the old master with a lower configEpoch. When the partition heals, the cluster should ideally resolve this via conflict resolution: nodes with higher epochs are preferred. However, if the partition causes asymmetric connectivity or if the epochs are equal (unlikely), the cluster may not resolve automatically.

In practice, split-brain is rare but occurs when the partition is long enough that both sides consider each other dead, and the minority side still has enough nodes to maintain quorum? Actually, in a 6-node cluster (3 masters, 3 replicas), if 1 master and its replica are isolated, the remaining 2 masters and 2 replicas have a majority of masters (2 out of 3). They can elect a new master for the missing master's slots. The isolated side has 1 master and 1 replica — the master is still alive, so it continues to serve its slots. No new election happens on that side because the original master is still up. But the problem is that the other side has elected a new master for the same slots. So we have two masters claiming the same slots. The cluster's gossip will eventually propagate the higher epoch, but if the partition healed quickly, there might be a tie-breaking issue. Redis uses a deterministic algorithm: when a node learns about two conflicting masters for the same slot, it keeps the one with the higher configEpoch. If epochs are equal, it keeps the one with the lower node ID (lexicographically). This usually resolves split-brain within a few gossip rounds. However, if the partition causes complete isolation where no gossip messages get through for an extended period, the cluster may remain in a split-brain state until manual intervention.

The first sign is often application errors: MOVED redirections to multiple IPs for the same key. Use `redis-cli -h <any-node> cluster slots` to dump the slot-to-node mapping. If you see a slot range appearing under two different master entries, that's definitive. Also, `cluster nodes` output includes flags and epochs. Look for duplicate 'master' entries with the same slot range but different node IDs and configEpochs. Another indicator: `cluster info` shows cluster_state:fail when the cluster detects inconsistency. But note, cluster_state can be fail for other reasons like minority partition.

To quantify, you can write a script that collects `cluster nodes` from all nodes and compares slot assignments. Use `redis-cli --cluster call <host:port> cluster nodes` to run on all nodes. Parse the output to find slots with multiple owners. Also check configEpoch values: if two masters for the same slot have similar epochs (difference < 2), it's likely split-brain. If one epoch is much higher, the lower one should step down automatically, but if that didn't happen, there's a propagation delay or a network issue.

Preventing split-brain starts with network reliability: use redundant networking, multiple AZs, and avoid single points of failure. But at the config level, the most critical parameter is `cluster-node-timeout`. A common mistake is setting it too high (e.g., 30 seconds) to avoid false failovers, but this increases the window for split-brain. A value between 5 and 10 seconds is a reasonable trade-off. Also, set `cluster-slave-validity-factor` to a low value (e.g., 5) to prevent replicas with stale data from becoming masters. Another important setting: `cluster-require-full-coverage no` — with this set to yes, the minority partition will reject all writes, which can cause data loss but also makes the split-brain more apparent? Actually, it prevents writes on the minority side, reducing the chance of inconsistent data. However, it also means the minority side is completely unavailable. I recommend setting it to no in production to allow partial service, but then you must have a strategy to reconcile data after partition.

Additionally, implement client-side slot caching with fallback: clients should periodically refresh the cluster slots map from a known healthy node, and if they get conflicting maps, they should contact a majority of nodes to determine the canonical mapping. Tools like redis-cli's `--cluster check` can also validate cluster consistency.