How to figure out what is causing an apache segmentation fault

Having an apache segmentation fault on your server can drive you crazy trying to solve it… So, I’m gonna explain you how I figured out what was causing the last apache segfault I had.

My server was running under Debian Squeeze, with Apache 2.2 and PHP 5.3, and I got randomly segmentation faults:

[Jun 08 17:38:07 2012] [notice] child pid 5646 exit signal Segmentation fault (11)
[Jun 08 18:21:34 2012] [notice] child pid 8278 exit signal Segmentation fault (11)
[Jun 08 18:44:59 2012] [notice] child pid 5665 exit signal Segmentation fault (11)

I supposed the problem was on some php extension, so I deactivated some of them…but the problem was still there :(

After spending a lot of time looking for what was causing it, I decided to debug apache with gdb. I installed it through backports because I needed an upper version than gdb 7.0 (7.0.1-2 is the current version on squeeze stable repository) otherwise I got this: “warning: The current binary is a PIE (Position Independent Executable), which GDB does NOT currently support. Most debugger features will fail if used in this session.” when I was trying to debug apache:

$ echo "deb http://backports.debian.org/debian-backports squeeze-backports main" >> /etc/apt/sources.list && apt-get update
$ apt-get -t squeeze-backports install gdb

Well, I had the debugger installed, but I also needed the php and apache debug symbols:

$ apt-get install apache2-dbg php5-dbg

Once I had installed all I needed, I configured apache to create a core dump file every time it had a segfault by adding in /etc/apache2/httpd.conf the following line:

CoreDumpDirectory /tmp/apache-coredumps

Previously, I had created the /tmp/apache-coredumps directory

By default, my system core file size was 0, so I had to change it through ulimit:

$ ulimit -c unlimited
$ ulimit -a

core file size          (blocks, -c) unlimited
[...]

I restarted apache, and I only had to wait for the next Segmentation fault… and I didn’t have to wait too much …

[Jun 08 20:16:52 2012] [notice] child pid 7976 exit signal Segmentation fault (11), 

possible coredump in /tmp/apache-coredumps

Then, as the message said, I had a core file in /tmp/apache-coredumps, and I was closer to finding out what was causing segfaults :)

Therefore, it was time to use gdb:

$ gdb apache2 -core /tmp/apache-coredumps/core

GNU gdb (GDB) 7.3-debian
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/apache2...Reading symbols from 
/usr/lib/debug/usr/sbin/apache2-mpm-prefork...done.
done.
[New LWP 7961]
 
warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f9aa0bffa5d in do_bind_function (opline=0x7f9a8e9ea1a8, 
function_table=0x7f9aa842c230, compile_time=0 '\000')
     at /tmp/buildd/php5-5.3.3/Zend/zend_compile.c:2956
2956    /tmp/buildd/php5-5.3.3/Zend/zend_compile.c: No such file or directory.
     in /tmp/buildd/php5-5.3.3/Zend/zend_compile.c

I had a problem with php as I thought, and I was able to find more information about it. And after Googling for a few minutes, I found there is a bug with the PHP and APC version that I was using! (more info here)

After that, I could solve the problem and until now I haven’t got more segfaults on this server :)

Using gdb will not solve your segmentation faults, but at least, you will be able to figure out what’s causing it, and you’ll get more info to solve it.

Reference:

http://sysadmin.carlusgg.com/?p=197