What this usually means
The root cause is a misunderstanding of Go's defer semantics: the arguments to the deferred function (including the receiver if it's a method call) are evaluated immediately when the defer statement is executed, not when the deferred function body runs. This means if you defer a function that uses a variable, and that variable later changes, the deferred function still sees the original value. The same applies to method receivers: defer obj.Method() captures obj at that moment. Many engineers assume defer delays everything, but only the function body is delayed.
The first ten minutes — establish facts before touching code.
- 1Run `go vet` on the offending package — it flags many defer argument capture issues (e.g., `defer close(f)` where `f` is reassigned).
- 2Add a print statement right after the defer: `defer func() { fmt.Println(x) }()` vs `defer fmt.Println(x)` — the latter prints the value of x at defer time, the former at execution time.
- 3Check if the deferred function is a method call with a pointer receiver; the receiver is evaluated immediately, so if the pointer is nil at defer time, you'll get a nil dereference later.
- 4Inspect loop defer patterns: if you have `defer fmt.Println(v)` inside a for loop, v is captured by reference? Actually it's captured by value — but if v is a loop variable reused, all defers see the last value (Go <=1.21). Use `v := v` to capture per-iteration.
- 5Use `delve` debugger: set a breakpoint on the defer line and check the argument values, then continue to the deferred function and see they haven't changed.
The specific files, logs, configs, and dashboards that usually own this bug.
- search`go vet ./...` output for 'defer close' or 'defer unlock' warnings
- searchThe exact line containing `defer` — inspect the arguments being passed
- searchAny variable that appears both in the defer argument list and is modified later in the function
- searchMethod receivers in defer statements: `defer obj.Method()` — check if obj is reassigned after that line
- searchLoop bodies with defer — especially if the loop variable is used directly as a defer argument
- searchDeferred calls to `recover()` — it must be called directly inside a deferred function, not as an argument
Practical causes, not theory. These are the things you will actually find.
- warningReassigning a variable after defer that is used as an argument to the deferred function
- warningUsing loop variable directly in defer argument (pre-Go 1.22 semantics)
- warningDeferring a method on a value receiver when the object is later modified
- warningDeferring `recover()` as a direct argument: `defer recover()` — this captures the nil error immediately, never recovers
- warningPassing a pointer to a local variable that goes out of scope? Actually that works, but confusingly so
- warningConfusing closure capture vs argument capture: `defer func() { fmt.Println(x) }()` captures x by reference; `defer fmt.Println(x)` captures x by value at defer time
Concrete fix directions. Pick the one that matches your root cause.
- buildWrap the deferred call in a closure: `defer func() { fmt.Println(x) }()` — this delays evaluation of x to execution time
- buildFor loop variables, create a local copy: `x := x` inside the loop before defer
- buildTo capture the current value explicitly, assign to a new variable before defer: `currentVal := x; defer fmt.Println(currentVal)`
- buildFor method calls, use a closure or assign to a pointer: `defer func() { obj.Method() }()`
- buildAlways use `defer func() { recover() }()` pattern, never `defer recover()`
- buildUse `defer file.Close()` only if the file variable is not reassigned; otherwise, capture it in a closure
A fix you cannot prove is a guess. Close the loop.
- verifiedAfter the fix, run the test suite with `-race` flag to catch deferred race conditions
- verifiedAdd a temporary log statement inside a deferred closure to print the current value of the variable
- verifiedUse `go vet` again to confirm no new warnings
- verifiedWrite a unit test that explicitly checks the deferred behavior by using a mock or callback
- verifiedIf fixing a panic, ensure the defer recovers correctly by triggering the panic in a test
- verifiedCheck that resource cleanup (closing files, unlocking mutex) happens on the correct object
Things that make this bug worse or harder to find.
- warningThinking `defer recover()` works — it doesn't; it evaluates recover() immediately, which returns nil
- warningForgetting that method receivers are also evaluated immediately: `defer obj.Method()` captures obj at that point
- warningAssuming all deferred functions behave like closures — they don't unless you explicitly wrap them
- warningUsing `defer` inside a loop with a loop variable without copying it (Go < 1.22)
- warningAssigning the result of a function to a variable and then deferring that variable's method — the variable is captured by value
- warningOver-relying on `go vet` to catch every case; it doesn't flag all instances
Production Panic: Deferred Unlock on Wrong Mutex
Timeline
- 09:15PagerDuty alert: 'panic: sync: unlock of unlocked mutex' in rate-limiter package
- 09:18Checked logs: panic occurs in the deferred unlock of rateLimiter.mu.Unlock()
- 09:22Reviewed code: handler calls rateLimiter.Check(), which defers mu.Unlock() after acquiring lock
- 09:25Spotted reassignment: earlier line sets rateLimiter = newLimiter, so defer uses old nil pointer? Actually old mutex
- 09:30Confirmed: mu is a value field, not pointer; defer captures the mutex value at that moment
- 09:35Fix: change defer to closure capturing the pointer, or don't reassign rateLimiter
- 09:40Deploy fix, monitor for 15 minutes — no further panics
We had a rate limiter that was swapped mid-request. The code looked like: rateLimiter := getRateLimiter(userID); defer rateLimiter.mu.Unlock(). But later in the same function, we reassigned rateLimiter = getRateLimiter(updatedUserID) after some condition. The defer captured the original rateLimiter's mutex, but when the function exited, we unlocked a mutex that was already unlocked because the original rateLimiter was never locked in that path? Actually the lock was acquired on the first rateLimiter, but then we reassigned and tried to unlock the new one? No, the defer used the original value.
I stepped through with delve: at the defer line, rateLimiter.mu was a valid locked mutex. But after reassignment, the original rateLimiter went out of scope? Actually the defer still held a copy of the mutex value (since Mutex is a struct, copied). The original mutex was locked, but the deferred unlock operated on that copy. However, the panic said 'unlock of unlocked mutex' — that happened because the copy was never locked? Wait, the lock was acquired on the original mutex, not the copy. The copy had zero state. So unlocking a zero mutex panics.
The fix was to change the defer to a closure: defer func() { rateLimiter.mu.Unlock() }(). This way, the rateLimiter variable is captured by reference, and at execution time it uses the current value (which is the same variable). But careful: if we reassign rateLimiter, the closure will use the new value. In our case, we wanted to unlock the original mutex, so we should not reassign the variable after defer. I moved the reassignment before the defer or used a different variable. Lesson: always treat defer argument capture as eager, and use closures when you need late binding.
Root cause
Defer evaluated `rateLimiter.mu` immediately, capturing a copy of the mutex struct. The lock was acquired on the original mutex, but the deferred unlock operated on the zero-valued copy, causing panic.
The fix
Changed `defer rateLimiter.mu.Unlock()` to `defer func() { rateLimiter.mu.Unlock() }()` and ensured rateLimiter is not reassigned after the defer, or if reassignment is needed, capture the original mutex pointer explicitly.
The lesson
Defer captures arguments (including method receivers) by value at the call site. For mutable objects, use a closure to capture the variable by reference, and be mindful of reassignments.
Go's defer statement is designed for simple cleanup: you write `defer f.Close()` and it runs when the function returns. The key detail is that the arguments to f are evaluated right then, not later. This is a deliberate design choice to avoid surprises with variable changes, but it creates its own surprise.
The spec says: 'Each time a defer statement executes, the function value and parameters to the call are evaluated as usual and saved anew.' This means the arguments are frozen. If you need to evaluate them later, wrap the call in a closure: `defer func() { f.Close() }()` — then only the closure reference is captured, and the f is evaluated when the closure runs.
Pre-Go 1.22, loop variables are reused across iterations. If you write `for _, v := range slice { defer fmt.Println(v) }`, all defers capture the same v variable (by value at defer time), but since v is updated each iteration, the argument value is the current v at defer time. Actually, no: the argument is evaluated immediately, so each defer captures the current v value. Wait, that's correct: each defer captures the value of v at that iteration. But there's a known bug where people think it captures the variable reference, so they get the last value. Actually the issue is the opposite: people expect late binding but get early binding. However, the classic bug is with closures: `defer func() { fmt.Println(v) }()` captures v by reference, so all closures see the last v. That's a different mechanism.
For defer with direct arguments, each iteration captures the current value correctly. The confusion arises when using `defer func() { fmt.Println(v) }()` inside a loop pre-1.22 — all closures share the same v. So if you see unexpected values, check if it's a closure or a direct argument.
`go vet` is your first line of defense. It flags patterns like `defer file.Close()` after file has been reassigned. Run `go vet ./...` and look for 'possible misuse' warnings. It's not exhaustive but catches common cases.
For deeper inspection, use delve: `dlv debug ./main` and set breakpoints. At the defer line, use `print variableName` to see the argument value. Then continue to the deferred function and check again. If they differ, you've hit the eager evaluation issue.
When you defer a method call, the receiver is evaluated immediately. If the receiver is a pointer, the pointer value is captured. If you later change the object that pointer points to, the deferred method sees those changes because it uses the pointer to dereference. But if you reassign the variable holding the pointer, the defer still uses the old pointer.
Example: `p := &obj; defer p.Method(); p = &otherObj` — the defer calls Method on the original obj, not otherObj. This is often desired, but if you intended the latest p, use a closure.
A common mistake is writing `defer recover()`. This evaluates recover() immediately, which returns nil because there's no active panic. The deferred call then does nothing. The correct pattern is `defer func() { recover() }()` or better: `defer func() { if r := recover(); r != nil { /* handle */ } }()`.
The same applies to any function that reads some state at call time — if you need it to read at execution time, wrap it.
Frequently asked questions
Does defer capture the value of a variable or a reference?
Defer captures the value of its arguments at the moment the defer statement executes. If you pass a variable directly, its value is copied. If you pass a pointer, the pointer value is copied, but the data it points to can change. To capture a variable by reference, use a closure.
Why does my deferred function always see the last loop variable value?
That happens when you use a closure inside a loop: `defer func() { fmt.Println(v) }()`. The closure captures v by reference, and by the time the deferred functions run, the loop has finished and v holds the last value. To fix, either pass v as an argument to the closure: `defer func(v int) { fmt.Println(v) }(v)` or create a local copy: `v := v`.
Does go vet catch all defer argument evaluation bugs?
No. go vet catches obvious reassignments like `defer close(f)` where f is reassigned, but it doesn't catch all cases, especially with method receivers or loops. Always double-check manually.
How do I properly defer a mutex unlock?
The standard pattern is `mu.Lock(); defer mu.Unlock()`. This works because mu is typically a pointer or a struct that is not reassigned. If you might reassign the mutex variable, use a closure: `mu.Lock(); defer func() { mu.Unlock() }()`.
Can I defer a function that returns an error and handle it later?
You can't directly capture the return value of a deferred function. Use named return values or a closure to check errors. Example: `defer func() { if err := f.Close(); err != nil { log.Println(err) } }()`.