Redis READONLY error: prove which endpoint accepted the write

The symptoms

• •Client receives the literal error "READONLY You can't write against a read only replica." or "MOVED ..." followed by a retry that itself returns the same READONLY message.
• •Write commands (SET, INCR, DEL, HSET, LPUSH, XADD, EXPIRE) fail immediately while the same connection succeeds for GET, EXISTS, and SCAN.
• •Application logs show the failure only on a subset of hosts, regions, or after a failover, while other deployments of the same code continue writing successfully.
• •Latency or dashboard anomalies show a spike at the moment a Sentinel or Cluster role change occurred, followed by sustained write failures from clients that did not resubscribe.
• •Cluster client returns ASK on a hash slot redirect to a node that then replies READONLY, indicating the redirected target is a replica rather than the slot owner.

Likely causes

• •The client connected to a replica IP or DNS name instead of the primary, often because a load balancer was configured to round-robin across the entire replica set rather than separating read and write pools.
• •Redis Sentinel clients are reading topology from a stale cache: after a master failure and promotion, the old "master" address points to a demoted replica, which now refuses writes with READONLY.
• •Redis Cluster client is in read-only mode (cluster_slots_ok or a feature flag) and is intentionally routing writes to replicas, which is not supported by default in Redis Cluster.
• •A proxy or sidecar (for example, a connection-sharing layer) is silently downgrading the connection to a replica during connection storms, failover windows, or failover-induced reconnect storms.
• •The "master" was reconfigured to be a replica of another node (REPLICAOF) without a corresponding client refresh, so the now-read-only instance continues to accept client connections but rejects writes.
• •A misconfigured DNS, Consul, or service-discovery record is pointing at the replica's endpoint because the discovery source labels primaries and replicas with the same service name.

First ten minutes

1. 01Capture the exact error string, the command that triggered it, and the client identifier (host, pod, process, client IP) from the application log.
2. 02Identify the candidate Redis endpoint the client believes it is connected to at the moment of the failure: hostname, IP, port, and the connection establishment timestamp.
3. 03From a diagnostic host with safe network access, use the candidate endpoint to run a read-only identity check (PING, CLIENT GETNAME, ROLE) to see whether the target self-reports as master, replica, or loading.
4. 04Compare the candidate endpoint's identity against the currently authoritative primary recorded by your orchestration, cloud provider, or Sentinel, and note any discrepancy.
5. 05Pull the role-change history: Sentinel "+switch-master", Cluster "failover-end", or cloud provider failover events, and align timestamps with the first failed write.
6. 06Decide whether the evidence already attributes the write to a specific endpoint; if not, retain the client connection metadata so post-mortem attribution does not depend on guesswork.

Evidence to collect

• •Exact error string and stack trace from the client, including the command name, key or slot hint, and the connection ID or remote address the client logged.
• •ROLE output from the suspect endpoint, showing the connection's "role" field and, if it is a replica, the upstream master host and port it believes in.
• •CLUSTER NODES or CLUSTER SLOTS output from the suspect endpoint (or the cluster view) showing the actual role and slot ownership of the node at the time of the failure.
• •SENTRY MASTER, REPLICAS, or "+switch-master" events, or cloud provider failover notifications, with timestamps before, during, and after the first READONLY failure.
• •Client-side topology cache: the "known master" address the client had before the incident, the address it has now, and when it last refreshed the topology.
• •Service discovery or load balancer configuration: which backend targets are registered under the write pool, which under the read pool, and whether any weight, label, or selector mismatches exist.

Where to look

• •At the client boundary: the Redis client library's connection pool, the "known master" config, and the topology refresh callbacks.
• •At the network boundary: the load balancer, DNS resolver, or service-mesh sidecar that fronts the Redis write endpoint.
• •At the Redis node boundary: the suspect instance's ROLE, INFO replication, and CONFIG GET replica-read-only output, plus the cluster or Sentinel view of the same node.
• •At the orchestration boundary: Sentinel +sdown, +odown, and +switch-master events, or the cloud provider's control plane events for replica promotion.
• •At the discovery boundary: the registry, key, or service entry that maps the "write" name to a concrete endpoint, and the timestamp of its last update relative to the failure.

Diagnostic steps

1. 01Run ROLE on the suspect endpoint and record the "role" field: "master", "replica", or "loading". If the role is "replica", the endpoint cannot accept writes and the attribution is resolved.
2. 02If ROLE reports "master", run INFO replication and check master_link_down_since_seconds, master_link_status, and the slave_read_only config; a master_link_status of "down" combined with a "master" role describes a degraded state that often returns READONLY intermittently.
3. 03Run CLUSTER NODES (Cluster topology) for the suspect node and compare its "flags" field with the client-side topology; a mismatch where the client thinks the node is a master but the cluster marks it as a replica indicates a stale client cache.
4. 04Run CLUSTER SLOTS and confirm the suspect endpoint's slot ownership; a node that owns no slots in the cluster view cannot accept writes to those keys.
5. 05Dump the Sentinel view (SENTINEL MASTER <name> and SENTINEL REPLICAS <name>) and align the "ip" and "port" fields of the authoritative master with the endpoint the client used.
6. 06Inspect the client's topology refresh: the time the client last received a topology update, the source (Sentinel, Cluster, DNS), and whether the client observed a +switch-master event.
7. 07Cross-reference the captured client connection metadata (host, port, remote address) with the candidates above to attribute the failed write to one specific endpoint.

Common mistakes

• •Assuming the error means the current node is the problem and re-pointing the client to "the same address" without checking whether the address now resolves to a demoted replica.
• •Treating READONLY as a transient hiccup and adding retry logic that re-issues the same write to the same endpoint, which extends the outage window without changing the destination.
• •Enabling replica-read-write globally on the replica to "make the error go away", which silently converts writes into lost writes that vanish when the replica is resynchronized.
• •Resetting Sentinel without resetting the client topology cache, so the client continues to use the previous master address even after Sentinel has converged on a new primary.
• •Diagnosing only the application and Redis boundary while ignoring the load balancer, service registry, or DNS layer that is actually steering the connection to a replica.

Safe fixes

• •Conditional on confirmed attribution: if the client connected to a replica endpoint, force the client to drop its topology cache and re-resolve the master address from the authoritative source, then verify with a sentinel write that succeeds and is acknowledged by ROLE.
• •Conditional on a stale Sentinel or Cluster client cache: trigger the client's built-in topology refresh (for example, by forcing the client to re-subscribe to Sentinel or by closing the pooled connection), then assert the new connection points at the current master.
• •Conditional on a misconfigured load balancer or service registry: separate the read and write pool registries so the write pool only contains primaries, and add a label- or tag-based filter that rejects replica backends from the write pool.
• •Conditional on a half-open failover window: pause the affected clients during the window, or set their write pool to a single, health-checked primary until the topology stabilizes, then resume.
• •Remediation is only considered "applied" once a sentinel write from the client produces a non-READONLY reply and the audit log shows the write landed on the authoritative primary.

Prove the fix

1. 01Issue a known sentinel write from the previously failing client and confirm the response is "OK" rather than READONLY, over a connection that ROLE identifies as master.
2. 02Capture INFO replication on the target node and confirm master_link_status remains "up" while the sentinel write is acknowledged.
3. 03Re-run the client workload that originally triggered READONLY and observe zero READONLY responses over a representative interval, with the write pool still pointing at the proven endpoint.
4. 04Cross-check the audit trail: the timestamped write in the application's structured log, the corresponding client OBJECT or log record on the Redis node, and the matching entry in the cluster or Sentinel view.
5. 05Cause a controlled failover and verify that the client converges on the new master within one topology refresh, without producing any READONLY responses in the interim.

Prevention and next steps

• •Maintain separate naming or labeling for the write pool and read pool so that service discovery cannot resolve a replica into the write pool.
• •Configure the client to refresh Sentinel or Cluster topology on a short interval and on every connection error, not only on primary failures.
• •Add a write-pool health check that runs a write-only command and rejects any backend that returns READONLY, with the rejection removed only after ROLE confirms master.
• •Practice failover regularly and include a post-failover check that the client topology cache is consistent with Sentinel or Cluster before resuming writes.
• •Alert on the presence of READONLY responses in production logs, since they indicate a misroute rather than a true data store failure.

Safe commands and checks

redis-cli -h <primary-host> -p <port> ROLE
redis-cli -h <primary-host> -p <port> INFO replication
redis-cli -h <primary-host> -p <port> CONFIG GET replica-read-only
redis-cli -h <primary-host> -p <port> CLIENT GETNAME
redis-cli -h <primary-host> -p <port> CLUSTER NODES
redis-cli -h <primary-host> -p <port> CLUSTER SLOTS
redis-cli -h <sentinel-host> -p <sentinel-port> SENTINEL MASTER <master-name>
redis-cli -h <sentinel-host> -p <sentinel-port> SENTINEL REPLICAS <master-name>