Redis Cluster · intermediate
Redis CROSSSLOT: find the multi-key command's slot mismatch
Redis Cluster CROSSSLOT errors occur when a multi-key command targets keys whose CRC16 hash slots differ, which the cluster forbids by design. This guide explains how to identify the offending keys, confirm the slot mismatch, and choose between hash tags, client-side grouping, or pipeline splitting as the corrective path.
The symptoms
- •Application logs contain a MOVED or CROSSSLOT reply with the prefix "CROSSSLOT" followed by "Keys in request don't hash to the same slot" for commands such as MGET, MSET, SUNION, ZUNIONSTORE, or DEL issued with multiple keys.
- •A previously working batch endpoint starts returning errors after a key prefix, schema migration, or TTL change altered the key naming convention without corresponding changes to multi-key call sites.
- •The same logical operation succeeds against a single-node Redis but fails against a Redis Cluster deployment, because the cluster enforces hash-slot routing that single-node deployments do not.
- •Operations that worked under one client library version begin failing after an upgrade that introduced stricter cluster-aware key validation or switched the connection mode to cluster mode.
- •Latency dashboards show failed transactions coinciding with multi-key Lua scripts or pipelines, while single-key operations on the same keyspace remain healthy.
Likely causes
- •The application issues a multi-key command (MGET, MSET, DEL with several keys, SET with NX followed by EXPIRE on different keys, transactions spanning keys) where the keys do not share the same hash slot under CRC16 modulo 16384.
- •Hash tags were not used: Redis only co-locates keys whose names contain the same substring inside curly braces, such as {user:42}:profile and {user:42}:orders; without that pattern the hash slot is computed over the whole key.
- •A client library transparently expands a single high-level call into multiple sub-commands on different keys, for example a cache-aside that reads a primary key and a sidecar index key together.
- •A Lua script executed via EVAL or EVALSHA references more than one key via KEYS without curly-brace hash tags, so KEYS[1] and KEYS[2] land on different slots.
- •The cluster topology changed (reshard, scale-out, or failover reassignment of slots) and a cached slot map held by the client no longer reflects the current slot ownership, so the client routes a multi-key request through a node that does not own all referenced slots.
First ten minutes
- 01Capture the full error reply verbatim from the application log, including the server IP, port, slot number, and the original command string; the message format "CROSSSLOT Keys in request don't hash to the same slot" is the primary signal.
- 02Enumerate every key mentioned in the failing command and compute each key's hash slot using CRC16(key) mod 16384 to confirm they truly differ; treat any two keys with different slot indices as the immediate cause.
- 03Identify whether the multi-key call site uses a library helper, raw MGET/MSET, a Lua script with KEYS, or a MULTI/EXEC transaction; each pattern has a different corrective path.
- 04Check the cluster's current slot map by reading the output of CLUSTER SLOTS against one reachable node, so you know which nodes own which slot ranges before making routing decisions.
- 05Determine whether the keys share a curly-brace substring such as {tenant}:foo and {tenant}:bar; if not, that is the root cause and the key naming convention is the variable to fix.
Evidence to collect
- •The exact CROSSSLOT error message, timestamp, client identity, and the full command line (key list and command verb) as recorded by the application's structured logger.
- •Per-key hash slot values derived from CRC16 modulo 16384, so the mismatch can be proven numerically rather than asserted by inspection.
- •The CLUSTER SLOTS output from a reachable master showing current slot-to-node ownership, used to rule out stale client routing maps as the cause.
- •The application key naming convention, including any prefix, tenant separator, or hash-tag pattern, captured from configuration or a representative key sample.
- •The client library version, cluster-mode flag, and any documented behavior for multi-key commands, since cluster-aware clients may suppress or transform the error in ways that obscure the original failure.
Where to look
- •Application logs at the boundary where the Redis client throws or logs the CROSSSLOT reply; look for the literal prefix "CROSSSLOT" and the key list that produced it.
- •Redis cluster control plane at the boundary where slot ownership is tracked: the output of CLUSTER SLOTS, CLUSTER NODES, or the cluster bus logs if resharding is in progress.
- •Source code at the boundary where multi-key commands are constructed: data-access repositories, cache-aside helpers, Lua script registration, and any place where a transaction or pipeline is built dynamically.
- •Configuration at the boundary where keys are named: routing prefixes, tenant identifiers, shard selectors, and the presence or absence of curly-brace hash tags.
- •Client library at the boundary where the connection mode is selected: cluster-mode versus single-node mode, connection string topology, and whether the library auto-pipelines multi-key requests.
Diagnostic steps
- 01Confirm the error class: extract the first reply from the failing request and verify it begins with "CROSSSLOT" followed by "Keys in request don't hash to the same slot"; this distinguishes CROSSSLOT from a MOVED redirection or an ASK redirection.
- 02Reproduce the slot calculation offline: take each key in the failing command, compute CRC16(key) mod 16384 for each, and verify that at least two keys produce different slot indices; equal indices mean the error is not a hash-slot mismatch and another cause applies.
- 03Inspect the keys for a shared curly-brace substring: scan each key for the pattern { ... } and, when present, recompute the slot using only the substring inside the braces; co-located slots confirm the design intent even if the full key strings differ.
- 04Cross-check against current ownership: issue CLUSTER SLOTS to a reachable master and confirm each computed slot is currently served by exactly one master, ruling out a slot migration mid-flight as a confounding factor.
- 05Audit the call site for transparent expansion: trace from the failing log entry back to the source file or query method and confirm whether the client library fabricated extra keys (such as a tag key, lock key, or secondary index) that the developer did not author.
- 06Test a corrected key shape in isolation: construct a representative key set with an explicit hash tag, run the same multi-key command against the cluster, and verify the same command returns success on the new key names.
Common mistakes
- •Assuming that single-node Redis behavior carries over to a cluster: single-node Redis accepts arbitrary multi-key commands, so a working test environment can mask a CROSSSLOT condition that only appears under sharded topology.
- •Adding curly braces in only one of several related keys, which causes the hash tag to be applied inconsistently and the slot mismatch to persist even though the developer believes the keys now share a tag.
- •Catching the CROSSSLOT exception and retrying blindly without changing the key set, which produces a tight error loop because retrying the same keys reproduces the same slot mismatch.
- •Confusing CROSSSLOT with MOVED: MOVED indicates the client contacted the wrong node for a single key and should redirect; CROSSSLOT indicates the keys themselves cannot coexist in one command and require a key-shape change.
- •Splitting the request into many single-key round trips as a default fix, which increases latency and removes the atomicity that the original multi-key command was chosen to provide.
Safe fixes
- •If the call site is yours to change, introduce a shared hash tag by wrapping the partition dimension common to all keys in curly braces, for example rename keys so a tenant identifier becomes {tenant}:profile and {tenant}:orders, then verify that CRC16 of the inner substring is identical across the set before redeploying.
- •If you cannot change key names because they are shared with other systems, split the multi-key command into a sequence of single-key commands or, when atomicity is required, a server-side Lua script via EVAL that explicitly enumerates KEYS whose hash tags guarantee co-location.
- •If the call site uses a Lua script, declare the KEYS array so the cluster proxy can verify co-location before execution, and ensure every key passed to the script shares the same hash-tag substring; otherwise the cluster will refuse the script with CROSSSLOT.
- •If the client library transparently expands a single call into multiple keys, configure or patch the helper to operate within one hash slot, or replace the helper with explicit per-slot operations that the calling code can reason about.
- •If a stale client routing map is suspected, force a fresh topology refresh by issuing CLUSTER SLOTS from the client and confirm that slot ownership in the client matches the masters' current ownership before drawing further conclusions.
Prove the fix
- 01The exact command that previously returned CROSSSLOT now returns a successful reply of the expected type (array of values for MGET, integer for DEL, and so on) when executed against the same cluster with the same parameters.
- 02A targeted regression test issues the multi-key command with the new key shape and asserts that CRC16 of the hash-tag substring is identical for every key, then asserts that the cluster reply is non-error; the test runs as part of the deployment pipeline.
- 03Application metrics for CROSSSLOT-class errors return to zero within one monitoring interval after the change, and the previously affected endpoint shows successful response codes for at least one full traffic window.
- 04No new MOVED or ASK redirections appear for the same key set, confirming that the fix addressed slot co-location rather than merely rerouting the request to a different node.
Prevention and next steps
- •Adopt a key naming convention document that mandates a curly-brace hash tag on every key, with the tag chosen on the dimension that most multi-key operations share, so future keys are co-located by construction.
- •Add a pre-commit or CI check that scans the repository for multi-key Redis commands and asserts via an offline CRC16 computation that every key in the command maps to the same slot; treat mismatches as build failures.
- •Require cluster-mode integration tests in staging that exercise the same multi-key paths against a real Redis Cluster topology, rather than only against single-node Redis, so CROSSSLOT conditions surface before production.
- •Pin the client library's cluster-mode behavior and document it in the operations runbook, so upgrades that change multi-key handling are evaluated explicitly rather than absorbed silently.
Safe commands and checks
redis-cli -h <node-host> -p <port> CLUSTER SLOTS redis-cli -h <node-host> -p <port> CLUSTER NODES redis-cli -h <node-host> -p <port> CLUSTER KEYSLOT <key> redis-cli -h <node-host> -p <port> CLUSTER COUNTKEYSINSLOT <slot> redis-cli -h <node-host> -p <port> --no-auth-warning DEBUG SLEEP 0