Corefiles

From Segfault
Jump to: navigation, search

Linux

To enable corefiles on a Linux system, we have to enable it via sysctl(8), create the directory to hold the corefiles and also set the ulimit accordingly:

mkdir -m1733 /var/crash || chmod 1733 /var/crash

echo 'kernel.core_pattern=/var/crash/core.%e.%p.sig-%s' >> /etc/sysctl.d/local.conf
sysctl -p /etc/sysctl.d/local.conf

This will set the corefile format to core.%processname.%pid.sig-%signal

Be sure to raise the corefile size limit, preferably via /etc/security/limits.conf:

ulimit -c 1048576                                              # 1024 MB max corefile size

Note: /proc/sys/kernel/core_uses_pid has been obsoleted by /proc/sys/kernel/core_pattern.

Linux kernel

To enable kernel crashdumps[1]:

sudo apt-get install kexec-tools kdump-tools crash        # Debian
sudo apt-get install linux-crashdump                      # Ubuntu[2]

At a minimum, the following should be set:

$ grep ^[A-Z] /etc/default/kexec

LOAD_KEXEC=true
KERNEL_IMAGE="/vmlinuz"
INITRD="/initrd.img"
APPEND=""
USE_GRUB_CONFIG=false

$ grep ^[A-Z] /etc/default/kdump-tools 
USE_KDUMP=1
KDUMP_COREDIR="/var/crash"
MAKEDUMP_ARGS="-c -d 31"

The kernel needs to support crashdumps too:

CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_KERNEL=y
...?

If all goes well, we should[3] have something like this:

$ kdump-config status
current state   : ready to kdump
$ kdump-config show
USE_KDUMP:        1
KDUMP_SYSCTL:     kernel.panic_on_oops=1
KDUMP_COREDIR:    /var/crash
crashkernel addr: 0x2f000000
current state:    ready to kdump

kernel link: 
  /usr/lib/debug/boot/vmlinux-3.16.0-4-amd64

kexec command:
  /sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-3.16.0-4-amd64 root=UUID=3222f73c-deb2-49d1-a74f-3c9438aebc91 ro console=ttyS0,115200 console=tty0 irqpoll maxcpus=1 nousb systemd.unit=kdump-tools.service" --initrd=/boot/initrd.img-3.16.0-4-amd64 /boot/vmlinuz-3.16.0-4-amd64

The debug kernel will be assumed to be installed in /usr/lib/debug/vmlinux-$(uname -r)

$ sudo apt-get install linux-image-amd64-dbg
$ ls -gohL /usr/lib/debug/boot/vmlinux-$(uname -r)
-rw-r--r-- 1 117M Jul 17 17:17 /usr/lib/debug/boot/vmlinux-3.16.0-4-amd64

With all that in place, we should be ready to dump:

sysctl vm.drop_caches=3
echo s > /proc/sysrq-trigger
echo c > /proc/sysrq-trigger

After the system is up again, a crashdump should exist in KDUMP_COREDIR and we can use crash to analyze the dump:

$ sudo crash /usr/lib/debug/vmlinux-3.16.0-4-amd64 /var/crash/201508030242/dump.201508030242
       KERNEL: /usr/lib/debug/vmlinux-3.16.0-4-amd64
    DUMPFILE: /var/crash/201508030242/dump.201508030242  [PARTIAL DUMP]
        CPUS: 4
        DATE: Mon Aug  3 02:41:48 2015
      UPTIME: 00:06:45
LOAD AVERAGE: 0.00, 0.04, 0.04
       TASKS: 130
    NODENAME: foobar
     RELEASE: 3.16.0-4-amd64
     VERSION: #1 SMP Debian 3.16.7-ckt11-1+deb8u2 (2015-07-17)
     MACHINE: x86_64  (2195 Mhz)
      MEMORY: 1 GB
       PANIC: "Oops: 0002 [#1] SMP " (check log for details)
         PID: 1278
     COMMAND: "bash"
        TASK: ffff88003cd78050  [THREAD_INFO: ffff88003bb0c000]
         CPU: 0
       STATE: TASK_RUNNING (PANIC)
crash>

Solaris

To enable corefiles on a Solaris system, we have to enable it via coreadm(1M), create the directory to hold the corefiles and also set the ulimit accordingly:

mkdir -m1733 /var/core                                       # Should be already in place

coreadm -g '/var/core/core.%f.%p' -i '/var/core/core.%f.%p' \
        -e global -e process -e global-setid -e proc-setid -e log
ulimit -c 524288

This will modify /etc/coreadm.conf as well, so it'll persistent across reboots.

MacOS

In MacOS 10.7, corefiles are usually enabled from a kernel POV, but a few things have to be prepared:

$ sysctl -a | grep kern.core
kern.corefile = /cores/core.%P
kern.coredump = 1

$ chmod 1733 /cores              # /cores should be already in place, but only root could write to it
$ ulimit -c 524288

Note: corefiles on MacOS X seem to be very large. Dumping core from a freshly started bash-process produces a ~360MB corefile.

Links

References

  1. Documentation for Kdump - The kexec-based Crash Dumping Solution
  2. Ubuntu: CrashdumpRecipe
  3. Difficulty configuring kdump