Debugging

Electric Fence

Electric Fence is a memory debugger, consisting of a library and a wrapper binary to link/wrap C programs. An example would be:

$ cat bob.c

#include <stdio.h>
#include <stdlib.h>
int main()
{
       int *x = (int *) malloc(2*sizeof(int));
       x[3] = 5;               // Violation here!!!
       printf("Die\n");
}

Compile with:

gcc -g bob.c -o bob.exe

Execute:

$ ulimit -c unlimited
$ ./bob.exe; echo $?
Die
4

Let's wrap this through Electric Fence:

$ ef ./bob.exe; echo $?
/bin/ef: line 20: 25187 Segmentation fault  (core dumped) \
                 (export LD_PRELOAD=libefence.so.0.0; exec "$@" )
139

Let's examine the coredump with gdb:

$ gdb bob.exe -c core.25187
Core was generated by `./bob.exe'.
Program terminated with signal 11, Segmentation fault.
#0  0x08048480 in main () at bob.c:6
6               x[3] = 5;               // Violation here!!!
Missing separate debuginfos, use: debuginfo-install ElectricFence-2.2.2-36.fc18.i686
(gdb) bt
#0  0x08048480 in main () at bob.c:6

We could have also linked against the Electric Fence library and ommiting the call to ef:

gcc -g -lefence bob.c -o bob.exe

Now bob.exe would exit with a segfault and a coredump all by itself. See efence(3) for environment variables to set to help debugging.

If the ef binary is missing and we cannot recompile the binary to link against libefence, we can use LD_PRELOAD:

$ ls -lLgo /usr/lib/libefence.so
-rwxr-xr-x. 1 14748 Jul 18  2012 /usr/lib/libefence.so
$ LD_PRELOAD=libefence.so ./bob.exe
Segmentation fault (core dumped)

• Electric Fence (Embedded Linux Wiki)

GDB

Use gdb to gather more information why your application crashes:

$ gdb /usr/sbin/apache2
(gdb) set height 0                                                                           # Disable the pager
(gdb) set args -X
(gdb) run
[ scroll down some startup messages.... ]
[ maybe trigger the bug with some client application (here: a webbrowser might do) ]
(gdb) bt full

Note: The "-X" is an argument to apache. You can set and unset any arguments your program understands with set args.

• 7.2 Example Debugging Session: Segmentation Fault Example

no debugging symbols found

$ gdb -p `pgrep alpine`
[...]
Reading symbols from /opt/alpine/bin/alpine...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libssl.so.1.1...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...(no debugging symbols found)...done.

Let's see if we can find some debugging packages from our PackageManager:

$ for a in `awk '/no debugging sym/ {print $4}' gdb_out | sed 's/\.\..*//' | sort -u | xargs dpkg -S | awk -F: '{print $1}' | sort -u`; do
   apt-cache search "$a" | grep dbg; done
libaudit1-dbgsym - Debug symbols for libaudit1
libc6-dbg - GNU C Library: detached debugging symbols
libc6-i386-dbgsym - Debug symbols for libc6-i386
libc6-x32-dbgsym - Debug symbols for libc6-x32
libcap-ng0-dbgsym - Debug symbols for libcap-ng0
libgcc1-dbg - GCC support library (debug symbols)
libpam0g-dbgsym - Debug symbols for libpam0g
libssl1.1-dbgsym - Debug symbols for libssl1.1
libtinfo5-dbg - debugging/profiling library for the low-level terminfo library

Installing (some of) those should provide the debug symbols that were missing before.

LD_DEBUG

LD_DEBUG can be used to debug ld-linux, the dynamic linker. A couple of options are available

LD_DEBUG=help ls

When set, it will print out a lot of debugging information:

LD_DEBUG=all /bin/ls

Set LD_DEBUG_OUTPUT to a filename to direct all debugging output to this file instead of standard output.

• SDB:Debugging with Glibc
• Linker and Libraries Guide: Debugging Aids

Valgrind

Valgrind can automatically detect many memory management and threading bugs, and profile your programs in detail. An example on how to use valgrind would be:

valgrind --tool=memcheck --leak-check=full --trace-children=yes ./executable

• Using Valgrind to Find Memory Leaks and Invalid Memory Use
• Debugging Segmentation Faults and Pointer Problems
• Basic Valgrind Tutorial
• Valgrind Tutorial

Links

• Dtrace / truss (Solaris)
• strace / ltrace / ltrace.git (Fedora)
• ktruss (BSD)
• ktrace (Darwin)
• Collecting and analyzing Linux kernel crashes - crash