How to debug a segmentation fault?

  108294
January 28, 2020 16:28 benjamin.morel@gmail.com (Benjamin Morel)
Hi internals,

I'm encountering a SIGSEGV in PHP-FPM on PHP 7.4.2 in a Symfony app. The
bug seems to happen during the rendering of a Twig template; this makes it
hard for me to pinpoint the code that triggers the segfault.

Could you please tell me what the procedure to collect information is, so
that I can file a useful bug report?

Thank you,
— Benjamin
  108297
January 28, 2020 17:14 tysonandre775@hotmail.com (tyson andre)
> I'm encountering a SIGSEGV in PHP-FPM on PHP 7.4.2 in a Symfony app. The > bug seems to happen during the rendering of a Twig template; this makes it > hard for me to pinpoint the code that triggers the segfault. > > Could you please tell me what the procedure to collect information is, so > that I can file a useful bug report?
I'm not sure if there are official recommendations, but what I've done before for httpd: 1. Do you have a line in your syslogs (e.g. /var/log/messages or elsewhere for ip (instruction pointer) segfaulted in libphp7.so or elsewhere, etc. If you run objdump --disassemble-all on the shared library where the crash occurs, you might know slightly more, but not very much, and would want to generate a core dump instead. For something like "segfault at ffffffff00000008 ip 00007f2ca2ef7121 sp 00007ffeef578850 error 5 in libphp7.so[7f2ca2a83000+b95000]", it'd have crashed at the instruction on 0x00007f2ca2ef7121 - 0x7f2ca2a83000 in libphp7.so 2. I haven't tried these instructions before, but you probably want to enable core dumps. https://ma.ttias.be/generate-php-core-dumps-segfaults-php-fpm/ (first google result) may help. Then, you can run 'gdb /usr/local/php/modules/libphp7.so /tmp/coredumpxxx' to see the stack trace where the dump occurred to include it, and include it in a bug report. (the path or shared library you want may differ) If you can install debugging symbols (I'm not sure what the package is called - something like php-debuginfo or running debuginfo-install ....?), or build php with --enable-debug and reproduce the crash, that would also help. Make sure to turn off core dumps and restart php-fpm when you're done. Using a build of php configured with --enable-debug would also tell you if a ZEND_ASSERT failed before the segmentation fault (I think, I forget how it works with fpm)
  108317
January 29, 2020 10:59 nikita.ppv@gmail.com (Nikita Popov)
On Tue, Jan 28, 2020 at 5:29 PM Benjamin Morel morel@gmail.com>
wrote:

> Hi internals, > > I'm encountering a SIGSEGV in PHP-FPM on PHP 7.4.2 in a Symfony app. The > bug seems to happen during the rendering of a Twig template; this makes it > hard for me to pinpoint the code that triggers the segfault. > > Could you please tell me what the procedure to collect information is, so > that I can file a useful bug report? >
As the first step, I would always suggest to reproduce this issue with the CLI binary (this includes the builtin server under -S localhost:8000 -t public). That way you don't have to deal with multi-process FPM. Once that is done, the easiest way to get some useful debugging information is to run PHP under valgrind, this is done using "USE_ZEND_ALLOC=0 valgrind php ...". Nikita
  108321
January 29, 2020 17:19 bishop@php.net (Bishop Bettini)
On Tue, Jan 28, 2020 at 11:28 AM Benjamin Morel morel@gmail.com>
wrote:

> I'm encountering a SIGSEGV in PHP-FPM on PHP 7.4.2 in a Symfony app. The > bug seems to happen during the rendering of a Twig template; this makes it > hard for me to pinpoint the code that triggers the segfault. > > Could you please tell me what the procedure to collect information is, so > that I can file a useful bug report? >
Great advice so far. I would add, since you're using PHP-FPM, that the pool configuration needs to have: process.dumpable = 1 to get core dumps out of workers. Without this you might not get core dumps, even when you have other core dump settings configured (ulimit -c unlimited, chmod +r phpfpm, etc.).
  108325
January 29, 2020 22:58 benjamin.morel@gmail.com (Benjamin Morel)
> > I would add, since you're using PHP-FPM, that the pool configuration needs > to have: > process.dumpable = 1
Thank you! I could not generate a core dump before adding this configuration option! The generating a gdb backtrace <https://bugs.php.net/bugs-generating-backtrace.php> page looks a big outdated, it would be nice if someone could update it will all the helpful info received in this thread. I'm not sure if I could gather helpful enough info about the issue, but anyway I filed a bug here: https://bugs.php.net/bug.php?id=79194 Thank you, Benjamin On Wed, 29 Jan 2020 at 18:19, Bishop Bettini <bishop@php.net> wrote:
> On Tue, Jan 28, 2020 at 11:28 AM Benjamin Morel morel@gmail.com> > wrote: > >> I'm encountering a SIGSEGV in PHP-FPM on PHP 7.4.2 in a Symfony app. The >> bug seems to happen during the rendering of a Twig template; this makes it >> hard for me to pinpoint the code that triggers the segfault. >> >> Could you please tell me what the procedure to collect information is, so >> that I can file a useful bug report? >> > > Great advice so far. I would add, since you're using PHP-FPM, that the > pool configuration needs to have: > > process.dumpable = 1 > > to get core dumps out of workers. Without this you might not get core > dumps, even when you have other core dump settings configured (ulimit -c > unlimited, chmod +r phpfpm, etc.). >
  108326
January 30, 2020 02:43 tysonandre775@hotmail.com (tyson andre)
>> I'm encountering a SIGSEGV in PHP-FPM on PHP 7.4.2 in a Symfony app. The >> bug seems to happen during the rendering of a Twig template; this makes it >> hard for me to pinpoint the code that triggers the segfault. > > I'm not sure if I could gather helpful enough info about the issue, but > anyway I filed a bug here: > https://bugs.php.net/bug.php?id=79194
>> gdb output for the segfault: (only showing the first 10 entries, it goes on and on for more than 20000 entries—there may be many more, I gave up after that)
> #0 0x00007f7ea5fdc6d1 in bf_zend_execute () from /usr/lib64/php/modules/blackfire.so > #1 0x0000560bec0b20c0 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER () at /usr/src/debug/php-7.4.2-1.fc31.remi.x86_64/Zend/zend_vm_execute.h:1714 > #2 execute_ex (ex=0x7f7e99d6b2c0) at /usr/src/debug/php-7.4.2-1.fc31.remi.x86_64/Zend/zend_vm_execute.h:53821 > #3 0x00007f7ea5fdc6f8 in bf_zend_execute () from /usr/lib64/php/modules/blackfire.so > #4 0x0000560bec0b20c0 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER () at /usr/src/debug/php-7.4.2-1.fc31.remi.x86_64/Zend/zend_vm_execute.h:1714 > #5 execute_ex (ex=0x7f7e99d6b2c0) at /usr/src/debug/php-7.4.2-1.fc31.remi.x86_64/Zend/zend_vm_execute.h:53821
The symptoms sounds like you have a global function (each call handled in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER) that's recursively calling itself, infinitely. The stack trace doesn't say which. When blackfire is used, it seems to use C's call stack, which leads to a stack overflow. When blackfire is used, the C PHP engine manages a different stack and doesn't recurse, so it doesn't have a stack overflow, and the memory leak would be slower. Infinite recursion leading to a segfault is **a known bug with no plans to be fixed that I know of,** which I've encountered many times. You may also want to try running the same code with xdebug - Xdebug will throw/log an error if you recurse more than max_nesting_level(defaults to 100). https://xdebug.org/docs/basic That will give you the php function that calls itself. If you can reproduce the issue consistently, you may also want to look into https://github.com/adsr/phpspy (It has a top-like mode:)