A core dump is a file containing the memory image of a process at the moment it crashed. It's not a log — it's a snapshot of everything: the call stack, register values, heap data, and the exact instruction that caused the fault. If you've ever had a production service vanish without a trace, the core dump is your first and best witness.
Most teams I've worked with treat core dumps as a last resort. They grep logs, add more metrics, maybe turn on strace. But when the bug is a race condition that hits once every 10,000 requests, logs won't help. A core dump will show you exactly what each thread was doing. Here's how to read one, and what to look for beyond the obvious.
Setting the Stage: Enable Core Dumps Before You Need Them
You cannot analyze a core dump if your system doesn't produce them. On Linux, check `ulimit -c`. If it's zero, nothing will be written. Set it with `ulimit -c unlimited` in the process's environment, or set `LimitCORE=infinity` in your systemd service file.
The kernel writes dumps to the path in `/proc/sys/kernel/core_pattern`. Default is `core` in the current directory. In production, I've used `mkdir -p /var/crash && echo '/var/crash/core.%e.%p.%t' > /proc/sys/kernel/core_pattern` to get name, PID, and timestamp. Don't forget to ensure the directory is writable by the crashing process.
# Enable unlimited core dumps for the current shell
ulimit -c unlimited
# Or for a systemd service, add to the [Service] section:
# LimitCORE=infinity
# Check kernel pattern and set a custom one
echo '/var/crash/core.%e.%p.%t' | sudo tee /proc/sys/kernel/core_patternThe Obvious Step: Loading the Core in GDB
You have the core file (maybe named `core.12345`) and the original binary (unstripped, with debug symbols). Run:
`gdb /path/to/binary /path/to/core`
GDB will load the core and tell you the signal that killed the process. For a segfault, you'll see `Program terminated with signal SIGSEGV, Segmentation fault.` Then type `bt` (backtrace) to see the call stack. But that's just the start.
A Segfault That Wasn't a Null Pointer
- 14:23Production payment service goes down. No error logs. Just dead.
- 14:25Core dump retrieved from /var/crash. File size: 2.3 GB.
- 14:28gdb loaded. bt shows crash in a string manipulation function.
- 14:30Inspect local variables: `info locals` shows a std::string with garbage length.
- 14:32Disassemble the crashing instruction: `x/i $rip`. It's `movaps` — an SSE instruction requiring 16-byte alignment.
- 14:35Check the string address: `p &str`. It's at an odd address (0x7f8a...c). Not 16-byte aligned.
- 14:40Root cause: a misconfigured custom allocator returned non-aligned memory.
Lesson
The crash wasn't a null pointer. It was an alignment fault triggered by a vectorized string operation. Core dump analysis revealed the exact instruction and the misaligned address. Fixing the allocator resolved the issue permanently.
Beyond the Backtrace: What to Inspect
The backtrace tells you where you are, but not why you got there. After `bt`, switch to the frame that crashed (usually frame 0) with `frame 0` or `f 0`. Then:
- **Registers**: `info registers` shows the CPU state. The instruction pointer (RIP) and the faulting memory address (often in a segment register like CR2, though GDB doesn't expose that directly) are key.
- **Local variables**: `info locals` gives you the values of function arguments and local variables. If you see a pointer that looks suspicious, examine it with `print *ptr` or `x/10xb ptr`.
- **Disassembly**: `disassemble $rip, $rip+20` shows the instructions around the crash. The exact instruction tells you what kind of access it was — read, write, or execute.
- **Memory**: `x/16gx $rsp` dumps the stack. Compare with the backtrace to see if the stack is corrupted.
# Load core dump
gdb /usr/local/bin/myapp /var/crash/core.myapp.12345.1678900000
# Inside GDB:
bt # backtrace
f 0 # select crashing frame
info registers # see registers (especially rsp, rbp, rip)
info locals # see local variables
x/i $rip # disassemble the instruction at RIP
x/16gx $rsp # examine stack memory
p mystruct->member # print a struct member
list # show source code around the crash (if compiled with -g)Threads and Parallel Crashes
In a multithreaded process, the crash might be in one thread, but the root cause could be in another (e.g., a data race). Use `info threads` to see all threads, then `thread apply all bt` to get backtraces for every thread. This is especially useful when a thread is waiting on a mutex held by the crashed thread, or when one thread corrupts memory that another later uses.
I once debugged a crash where a background thread freed a buffer while the main thread was still writing to it. The crash happened in the main thread — the backtrace showed a valid-looking pointer, but the data was garbage. The thread that freed the buffer had exited cleanly and left no log. Only the core dump showed the freed memory region.
When analyzing multithreaded core dumps, always run `thread apply all bt` first. The crash site is often a symptom, not the cause. The smoking gun is usually in another thread.
Non-Crash Core Dumps: SIGABRT and SIGQUIT
A core dump isn't only for segfaults. If a process catches SIGABRT (e.g., from an assertion failure) or SIGQUIT (Ctrl+\), it produces a core dump too. These are valuable because they show the state of the process when an invariant was violated, not necessarily when memory was corrupted.
For example, a core dump from a failed assertion `assert(ptr != nullptr)` will show the exact line where the null pointer was discovered. But the real bug is why the pointer was null — often because of a missing initialization or a race condition. The stack trace at the assertion point is the starting point.
of critical production bugs in my experience are first understood through core dump analysis, not logs
Using the Core to Reconstruct the Past
One of the most powerful but underused features of core dumps is the ability to walk data structures. If the crashed process was processing a request, the request object is still on the heap. You can navigate pointers from the stack, find the request ID, and correlate with logs. I've used this to reconstruct the exact inputs that led to a crash, even when the logs were incomplete.
In one case, a web server crashed on a malformed HTTP header. The core dump let me examine the buffer that contained the raw request: `x/200bx $rdi` showed the exact bytes. It turned out the header contained a null byte in the middle of a value, which confused a string parsing function that expected null-terminated strings. The core dump made the bug obvious in minutes.
- 1Enable core dumps before you need them. Set ulimit and configure core_pattern.
- 2Keep debug symbols for every release. Strip only for deployment but archive the unstripped binary.
- 3Load the core with GDB and start with `bt`, then `f 0` and `info registers`.
- 4Inspect the exact instruction with `x/i $rip` and understand what kind of memory access it performs.
- 5Check all threads with `thread apply all bt` — the real bug may be elsewhere.
- 6Navigate memory to reconstruct the state: variables, data structures, even raw request buffers.
Never strip debug symbols from the binary that will run in production. Ship the binary with symbols but restrict access, or use a separate debuginfo package. Without symbols, a core dump is just a collection of addresses and you'll be guessing what function they belong to.
When You Don't Have the Exact Binary
This happens more often than you'd think: you have a core dump from a customer or a staging environment, but the binary that produced it is long gone. You can still extract some information. Use `file core` to see the build ID, then search for a binary with that ID. Tools like `debuginfod` can fetch symbols if your build system publishes them. But without the exact binary, you're limited to raw addresses and instruction disassembly.
If you're desperate, you can try `strings core | grep -i error` to find any error messages that were in memory. But that's a hail mary. The lesson: always archive your binaries and debug symbols.
A core dump is a photograph of the crime scene. Without the binary, you're looking at a blurry image with no context.
Core dump analysis is a skill that pays off disproportionately. The first time you find a bug in ten minutes that the team had been chasing for weeks, you'll never skip setting ulimit again. Next time a process goes silent, don't just restart it — capture its last words.
Frequently asked questions
Why is my core dump file empty or not generated?
Check `ulimit -c` — if it's 0, run `ulimit -c unlimited`. Also verify the kernel core pattern: `cat /proc/sys/kernel/core_pattern`. If it's set to e.g. `|/usr/lib/systemd/systemd-coredump`, your dump is managed by systemd and may be stored elsewhere or compressed.
Can I analyze a core dump from a different machine?
Yes, but you need the exact same binary (same build, same commit) with debug symbols. Without matching symbols, GDB can show raw addresses but no function names. Ship your debuginfo packages or unstripped binaries alongside your releases.
What if the crash happened in a shared library?
GDB will show the library name and offset. You need the matching .so with debug symbols installed (e.g., debuginfod can fetch them). Use `info sharedlibrary` to list loaded libraries and their base addresses.
How do I debug a core dump from a container or Docker?
Enable core dumps inside the container: `--ulimit core=-1` and mount a volume for the dump. Then copy the core file and the binary (from the same image) to a host and run GDB. Alternatively, use `docker cp` and debug outside the container.