From Segfault
Jump to navigation Jump to search

Electric Fence

Electric Fence is a memory debugger, consisting of a library and a wrapper binary to link/wrap C programs. An example[1] would be:

$ cat bob.c

#include <stdio.h>
#include <stdlib.h>
int main()
       int *x = (int *) malloc(2*sizeof(int));
       x[3] = 5;               // Violation here!!!

Compile with:

gcc -g bob.c -o bob.exe


$ ulimit -c unlimited
$ ./bob.exe; echo $?

Let's wrap this through Electric Fence:

$ ef ./bob.exe; echo $?
/bin/ef: line 20: 25187 Segmentation fault  (core dumped) \
                 (export; exec "$@" )

Let's examine the coredump with gdb:

$ gdb bob.exe -c core.25187
Core was generated by `./bob.exe'.
Program terminated with signal 11, Segmentation fault.
#0  0x08048480 in main () at bob.c:6
6               x[3] = 5;               // Violation here!!!
Missing separate debuginfos, use: debuginfo-install ElectricFence-2.2.2-36.fc18.i686
(gdb) bt
#0  0x08048480 in main () at bob.c:6

We could have also linked against the Electric Fence library and ommiting the call to ef:

gcc -g -lefence bob.c -o bob.exe

Now bob.exe would exit with a segfault and a coredump all by itself. See efence(3) for environment variables to set to help debugging.

If the ef binary is missing and we cannot recompile the binary to link against libefence, we can use LD_PRELOAD:

$ ls -lLgo /usr/lib/
-rwxr-xr-x. 1 14748 Jul 18  2012 /usr/lib/
$ ./bob.exe
Segmentation fault (core dumped)


Use gdb to gather more information why your application crashes:

$ gdb /usr/sbin/apache2
(gdb) set height 0                                                                           # Disable the pager
(gdb) set args -X
(gdb) run
[ scroll down some startup messages.... ]
[ maybe trigger the bug with some client application (here: a webbrowser might do) ]
(gdb) bt full

Note: The "-X" is an argument to apache. You can set and unset any arguments your program understands with set args.

no debugging symbols found

$ gdb -p `pgrep alpine`
Reading symbols from /opt/alpine/bin/alpine...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/ debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/ debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/ debugging symbols found)...done.

Let's see if we can find some debugging packages[2] from our PackageManager:

$ for a in `awk '/no debugging sym/ {print $4}' gdb_out | sed 's/\.\..*//' | sort -u | xargs dpkg -S | awk -F: '{print $1}' | sort -u`; do
   apt-cache search "$a" | grep dbg; done
libaudit1-dbgsym - Debug symbols for libaudit1
libc6-dbg - GNU C Library: detached debugging symbols
libc6-i386-dbgsym - Debug symbols for libc6-i386
libc6-x32-dbgsym - Debug symbols for libc6-x32
libcap-ng0-dbgsym - Debug symbols for libcap-ng0
libgcc1-dbg - GCC support library (debug symbols)
libpam0g-dbgsym - Debug symbols for libpam0g
libssl1.1-dbgsym - Debug symbols for libssl1.1
libtinfo5-dbg - debugging/profiling library for the low-level terminfo library

Installing (some of) those should provide the debug symbols that were missing before.


LD_DEBUG can be used to debug ld-linux[3], the dynamic linker. A couple of options are available

LD_DEBUG=help ls

When set, it will print out a lot of debugging information:

LD_DEBUG=all /bin/ls

Set LD_DEBUG_OUTPUT to a filename to direct all debugging output to this file instead of standard output.


Valgrind can automatically detect many memory management and threading bugs, and profile your programs in detail. An example on how to use valgrind would be:

valgrind --tool=memcheck --leak-check=full --trace-children=yes ./executable