What this usually means
The fundamental rule of recover in Go is that it only works when called directly from a deferred function (not a helper called by the deferred function). If recover is called in any other context — such as inline in a non-deferred function, inside a closure that's not directly deferred, or in a separate goroutine — it returns nil and does nothing. Additionally, if the deferred function itself panics, that panic replaces the original one. Nil panics (panic(nil)) also behave counterintuitively: before Go 1.21, recover returned nil even though a panic occurred. Finally, if recover is never called (e.g., because a condition skips it), the panic proceeds. These are the most common causes, not some exotic race condition.
The first ten minutes — establish facts before touching code.
- 1grep -rn 'defer' *.go -- look for any defer that calls recover indirectly (e.g., defer myRecover() instead of defer func() { recover() }())
- 2Add a print statement inside the deferred function right before recover(): if it prints but recover still returns nil, scope is the issue
- 3Check if the recover call is in the same goroutine as the panic: run with -race and look for goroutine dumps
- 4Test with panic("test error") instead of panic(nil) — if recover works, you hit the nil panic edge case
- 5Insert runtime.Stack([]byte, false) after recover to capture the stack when recover returns nil — it reveals where the panic originated
- 6Verify the deferred function is actually executed: add a fmt.Println before and after recover()
The specific files, logs, configs, and dashboards that usually own this bug.
- searchAll files containing 'defer' statements — especially cross-file or cross-package defer chains
- searchFunctions named something like 'safeCall', 'recoverFromPanic', or 'handlePanic' that are called from defer
- searchGoroutine launch sites (go func()...) where recover is in the parent, not the child
- searchThe immediate caller of any function that calls recover — is that caller also deferred?
- searchThe Go version: if <1.21, panic(nil) is notoriously tricky
- searchThe return statement of the deferred function: if it returns before recover, recover never runs
Practical causes, not theory. These are the things you will actually find.
- warningrecover() called in a helper function that is deferred (e.g., defer handleRecover()), not directly in the deferred closure
- warningrecover() called in a different goroutine than the panic — panics only propagate within the same goroutine
- warningpanic(nil) in Go versions before 1.21 — recover returns nil even though a panic occurred
- warningThe deferred function itself panics before calling recover, overwriting the original panic
- warningrecover() is called conditionally and the condition fails (e.g., if err == nil { recover() })
- warningThe deferred function is not executed because of a runtime error like nil pointer dereference in the defer statement itself (rare)
- warningMisunderstanding: recover() catches the panic for the calling goroutine only; it does NOT prevent the program from exiting if the panic is in a different goroutine
Concrete fix directions. Pick the one that matches your root cause.
- buildAlways call recover() directly inside a deferred anonymous function: defer func() { if r := recover(); r != nil { /* handle */ } }()
- buildIf you must use a named helper, pass recover as a parameter: defer func() { handleRecover(recover()) }() — but this is still indirect; best to inline
- buildFor goroutine safety, ensure each goroutine has its own defer/recover block: go func() { defer func() { recover() }(); /* risky code */ }()
- buildTo handle panic(nil) pre-1.21, check runtime.GOEXPERIMENT or upgrade to Go 1.21+ (which changed the behavior to treat panic(nil) like any other panic)
- buildWhen wrapping a function that may panic, use a wrapper that defers recover and returns an error: func safeCall(fn func()) (err error) { defer func() { if r := recover(); r != nil { err = fmt.Errorf("panic: %v", r) } }(); fn(); return }
A fix you cannot prove is a guess. Close the loop.
- verifiedWrite a unit test that calls the function with a forced panic and asserts no crash
- verifiedRun the program with GOTRACEBACK=crash and verify that the panic is caught (no crash dump)
- verifiedInsert a print statement in the recover block that logs the recovered value; ensure it prints
- verifiedUse the race detector to catch goroutine boundary issues: go run -race
- verifiedTest with both panic("error") and panic(nil) to confirm nil panic handling
- verifiedCheck the return value of recover: if it's non-nil, the catch worked; if nil, you have a scope issue
Things that make this bug worse or harder to find.
- warningDo NOT call recover() outside a deferred function — it will always return nil
- warningDo NOT wrap recover() in another function call within the defer — e.g., defer func() { recover() }() is fine, but defer func() { myRecover() }() where myRecover calls recover is NOT
- warningDo NOT assume recover() in a parent goroutine will catch panics from child goroutines — it won't
- warningDo NOT ignore the return value of recover() — if you don't check it, the panic is effectively unhandled
- warningDo NOT use recover() in a function that is not deferred — even if you call it right after a panic, it's too late
- warningDo NOT forget that a panic in a deferred function overwrites the original panic — recover the first panic before doing anything that might panic itself
The Silent Crash: When recover() Returns nil and Your Service Dies
Timeline
- 14:02PagerDuty alert: service 'payment-svc' is returning 500s for 2% of requests
- 14:05Check logs: see 'panic: runtime error: invalid memory address or nil pointer dereference' followed by stack trace
- 14:08Notice a defer/recover block in the HTTP handler middleware that should catch panics, but the panic still crashes the goroutine
- 14:12Add debug print inside defer; it prints before recover, but recover returns nil
- 14:15Review code: defer logAndRecover() — logAndRecover is a helper that calls recover inside it
- 14:18Fix: change defer logAndRecover() to defer func() { if r := recover(); r != nil { logAndRecover(r) } }()
- 14:22Deploy fix to staging; reproduce nil pointer panic; recover now catches it and logs
- 14:30Roll fix to production; 500s drop to 0; alert resolved
We had a middleware that wrapped every gRPC handler with a defer/recover to log panics and return a proper error. The code looked like: defer logAndRecover(). Inside logAndRecover, we called recover(). According to the textbook, this should work — but it didn't. Recover kept returning nil, and panics were crashing the goroutine, causing the entire process to restart. The symptom was intermittent 500s because only some requests hit the nil pointer.
After adding print statements, I confirmed the deferred function ran, but recover was nil. I re-read the Go spec: recover returns nil if it's not called directly from a deferred function. Our logAndRecover was deferred, but the recover() call inside it was not directly deferred — it was nested in a helper. The defer statement deferred logAndRecover, not the recover call itself. That's the key distinction.
The fix was simple: inline the recover call in an anonymous function inside the defer. I changed to: defer func() { if r := recover(); r != nil { logAndRecover(r) } }(). After deploying, the nil pointer panics were caught, logged, and converted to proper gRPC errors. The 500s vanished. I also upgraded to Go 1.21 later to get the panic(nil) fix, but that wasn't the issue here.
Root cause
recover() was called inside a helper function (logAndRecover) that was deferred, rather than being called directly in the deferred closure. Per Go spec, recover only works when called directly from a deferred function.
The fix
Replaced defer logAndRecover() with defer func() { if r := recover(); r != nil { logAndRecover(r) } }()
The lesson
Always call recover() directly in an anonymous deferred function. Never delegate it to a helper. Also, always check the return value of recover — nil doesn't mean no panic; it could mean the panic was nil or the call is out of scope.
The Go specification is explicit: recover() stops a panic only if it is called directly from a deferred function. This means the call to recover() must appear in the function literal that is the argument to defer. If you write defer helper() and helper calls recover(), that's indirect — recover sees the call stack as helper being the deferred function, not the original function that panicked. The recovery mechanism checks the call stack: recover() looks at the topmost deferred function's caller. If recover is not in that exact deferred function, it returns nil.
Concretely, this is the fix: defer func() { recover() }() works; defer func() { myRecover() }() where myRecover calls recover() does NOT. The only safe pattern is to inline the recover call. If you need shared logic, pass the recovered value as a parameter after calling recover in the inline closure.
A panic in one goroutine cannot be caught by a recover in another goroutine. If you have a recover in the main goroutine but a panic occurs in a goroutine launched with 'go func()', the parent's recover does nothing. The child goroutine will crash the entire program unless the child itself has a defer/recover. This is a common mistake in server code that spawns goroutines for request handling: the main function has a recover, but each handler goroutine needs its own.
To verify, add runtime.Stack to your recover block and check which goroutine ID appears. If the stack trace shows a different goroutine than the one with recover, you've hit this issue. The fix is to wrap every goroutine that may panic with its own defer/recover.
Before Go 1.21, panic(nil) (or panic with a nil interface value) caused recover to return nil, making it impossible to distinguish between 'no panic' and 'panic with nil value'. This was a known language wart. If you called recover() and got nil, you couldn't tell if a panic happened or not. The fix in Go 1.21 changed the behavior so that panic(nil) becomes a runtime error that recover can catch (returning a non-nil value of type runtime.PanicNilError).
If you're stuck on an older version, the workaround is to avoid panic(nil) entirely. Use a sentinel error or a string. Or, if you must handle nil panics, you can check the goroutine's stack trace via runtime.Stack to see if a panic occurred, but that's expensive. The best fix is to upgrade to Go 1.21+.
If your deferred function panics (e.g., a nil pointer dereference in logging code), that new panic replaces the original one. The original panic is lost, and recover can only catch the new panic. This can cause confusion: you might think recover isn't working because you see a different panic in the logs. For example, if recover is called after a nil pointer dereference in the deferred function itself, recover will catch that nil pointer dereference, not the original panic.
To avoid this, ensure your deferred function is panic-safe. If you're logging or doing complex operations, do them after checking recover's return value. The pattern: defer func() { if r := recover(); r != nil { /* safe logging */ } }()
Many engineers write defer func() { recover() }() without assigning the return value to anything. This catches the panic (stops propagation) but ignores it — the program continues but no error handling occurs. That's a silent failure. You should always check if recover returned non-nil and at least log it. Otherwise, you might think recover works but actually you're swallowing critical errors.
The correct pattern: defer func() { if r := recover(); r != nil { log.Printf("recovered panic: %v", r) } }(). This ensures you have visibility into the panic.
Frequently asked questions
Why does recover() return nil even though I have defer func() { recover() }()?
Check if you're calling recover() inside the deferred anonymous function directly. If your code is 'defer func() { myRecover() }()' where myRecover calls recover(), that's indirect — recover will return nil. Also ensure the deferred function itself didn't panic before recover, and that the panic is in the same goroutine.
Can I catch a panic from a goroutine in the parent?
No. Each goroutine must have its own defer/recover to catch its own panics. A panic in a child goroutine will crash the entire program unless the child catches it. The parent's recover cannot reach across goroutines.
What's the difference between panic(nil) and panic("error")?
Before Go 1.21, panic(nil) causes recover to return nil, making it indistinguishable from no panic. Starting Go 1.21, panic(nil) is treated like any other panic — recover returns a non-nil value (runtime.PanicNilError). Always use non-nil values for panic, or upgrade to Go 1.21+.
My deferred function runs but the program still crashes. What's wrong?
If the deferred function runs but the panic still crashes, either recover returned nil (scope issue), or the deferred function itself panicked before calling recover. Add a print or log right before recover() to see if it's reached.
Is it safe to call recover() outside a deferred function?
No. recover() called outside a deferred function always returns nil and has no effect. It must be called directly from a deferred function (the function literal passed to defer). Any other context is useless.