Linux Kernel Crash Dump Setup: Configuring kdump and crash

When a Linux kernel encounters an unrecoverable hardware exception, page fault in kernel space, or fatal kernel bug, it triggers a kernel panic. In default server configurations, a panic locks up the machine and leaves behind only ephemeral console output that disappears upon reboot. For mission-critical infrastructure, configuring kdump is mandatory: it leverages kexec to boot a lightweight secondary capture kernel that dumps physical memory (vmcore) to disk before rebooting.

Analyzing a vmcore image with the GNU crash utility allows systems engineers to reconstruct the exact CPU register states, inspect kernel stack backtraces, and examine locked mutexes across every active thread at the moment of failure.

Expand

Configuring Crashkernel Memory Reservation

To allow the capture kernel to boot without overwriting the memory state of the panicked kernel, a dedicated memory region must be reserved at boot time via GRUB.

# Edit GRUB configuration to reserve 256MB of RAM for crash dumps
sudo sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="/GRUB_CMDLINE_LINUX_DEFAULT="crashkernel=256M /' /etc/default/grub
sudo update-grub

# Install kdump and crash analysis tools on Ubuntu / Debian
sudo apt-get update && sudo apt-get install -y kdump-tools crash linux-image-$(uname -r)-dbgsym

# Enable the kdump-tools service
sudo systemctl enable --now kdump-tools
sudo kdump-config status

The status output should confirm: current state: ready to kdump, indicating that the capture kernel is loaded into memory via kexec.

Triggering a Controlled Test Panic

Always verify your dump pipeline in staging prior to production deployment. You can trigger an intentional panic via the Linux SysRq interface:

# Enable SysRq functions and trigger an intentional kernel crash
echo 1 | sudo tee /proc/sys/kernel/sysrq
echo c | sudo tee /proc/sysrq-trigger

Upon execution, the server boots instantly into the capture kernel, compresses the physical memory image using makedumpfile, writes it to /var/crash/vmcore, and automatically reboots back into the primary OS.

Interactive Root Cause Analysis with crash

Once the server has rebooted, launch the crash tool with the unstripped debug kernel (vmlinux) and the saved vmcore file:

# Launch interactive post-mortem debugging
sudo crash /usr/lib/debug/boot/vmlinux-$(uname -r) /var/crash/*/vmcore

# Inside the crash shell, inspect system state and stack traces
crash> sys
# Shows uptime, memory, panic string, and offending kernel module

crash> bt
# Emits full call trace of the panicking thread with function offsets

crash> ps
# Lists process table state, showing UNINTERRUPTIBLE (D) sleep tasks

Crash Diagnostics Video

Explore kernel internals and memory execution analysis in this overview:

Retention and Compression Tuning

In /etc/default/kdump-tools, configure MAKEDUMP_ARGS="-c -d 31" to exclude zero pages, user-space cache, and free memory blocks. This reduces dump file sizes from 32GB to under 400MB, conserving disk space while preserving full kernel diagnostic fidelity.

Deploy production-grade Intel Xeon Platinum servers with predictable stability at netbayhosts.in.