Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[???] Segfault in phalcon.so #2104

Closed
temuri416 opened this issue Feb 26, 2014 · 24 comments
Closed

[???] Segfault in phalcon.so #2104

temuri416 opened this issue Feb 26, 2014 · 24 comments

Comments

@temuri416
Copy link
Contributor

With the latest Phalcon v1.3.0 I started seeing this error in kern.log:

kernel: [608773.583588] php-fpm[20398]: segfault at 7f46a37b1a68 ip 00007f46a8292223 sp 00007fff6d579348 error 4 in phalcon.so[7f46a8256000+275000]

There are no errors/warnings dumped to php_errors.log or anywhere else.

The strange part is only one out of four HTTP requests fails.

I am quite certain that my code does not generate errors or warnings.

How can this be debugged?

Thanks!


Want to back this issue? Place a bounty on it! We accept bounties via Bountysource.

@temuri416
Copy link
Contributor Author

I am seeing this:

root@projectQ:/tmp# gdb php /tmp/core-php-fpm.28363
GNU gdb (GDB) 7.5.91.20130417-cvs-ubuntu
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /var/web/php/bin/php...done.

warning: core file may not match specified executable file.
[New LWP 28363]

warning: Error reading shared library list entry at 0x31ff3100000002be

warning: Error reading shared library list entry at 0x5f72158d48008b48

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff8ff7000
Core was generated by `php-fpm: pool www                                                             '.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fd2a920d354 in ?? ()

(gdb) bt full
#0  0x00007fd2a920d354 in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

@ghost
Copy link

ghost commented Feb 26, 2014

Do you have the code on which this can be reproduced?

Or you may want to try to rebuild Phalcon in debug mode:

cd ext
phpize
./configure CFLAGS="-O0 -g3"
make -j16
sudo make install

@temuri416
Copy link
Contributor Author

  1. I did compile Phalcon in debug mode, as it was indicated in the article
  2. I have an HTTP location crashing inside my app, there's no way to reproduce it from CLI.
  3. Why can't I generate the proper dump? Am I missing a step?
  4. Never had this problem with one month-old revision of Phalcon...

@temuri416
Copy link
Contributor Author

I am going to check out an earlier commits to find the troublemaker.

How do I now disable core dumps?

@ghost
Copy link

ghost commented Feb 27, 2014

ulimit -S -c 0

I have an HTTP location crashing inside my app, there's no way to reproduce it from CLI.

You can try this: create a file router.php and place it where your index.php is:

<?php
<?php

if (isset($_SERVER['PATH_INFO'])) {
        $_GET['_url'] = $_SERVER['PATH_INFO'];
        require 'index.php';
}
else {
        return false;
}
?>

Then start php-cli in the server mode:

php -S 127.0.0.1:10001

Navigate to http://127.0.0.1:10001/ and try to reproduce the crash.

If you have valgrind installed, then

USE_ZEND_ALLOC=0 valgrind /path/to/php -S 127.0.0.1:10001

will be more helpful.

Why can't I generate the proper dump? Am I missing a step?

The dump is OK, it just means that the error happened somewhere in the PHP core.

Never had this problem with one month-old revision of Phalcon

I wish I knew how to write programs without any bugs at all :-)

@temuri416
Copy link
Contributor Author

:) thanks, I'll post more results as soon as I have them.

@temuri416
Copy link
Contributor Author

What I think is happening is that somewhere in my app a non-fatal error occurs. A warning or notice is issued. I cannot find it in the logs, but maybe it's because my logging level is different.

Do you think that may somehow crash Phalcon?

@ghost
Copy link

ghost commented Feb 27, 2014

You will need to run this as root:

ulimit -c unlimited
service php-fpm stop
service php-fpm start

(or whatever commands are used to start and stop php-fpm).

@ghost
Copy link

ghost commented Feb 27, 2014

Do you think that may somehow crash Phalcon?

Not sure. Do you have any idea as to what code causes crash?

@ghost
Copy link

ghost commented Feb 27, 2014

Tests under valgrind pass: https://travis-ci.org/sjinks/cphalcon/builds/19701254

@temuri416
Copy link
Contributor Author

I've been verifying my code and found that there was a PHP warning being generated during the execution of the HTTP request. It was not being dumped to error log, that's why I could not catch it earlier.

Having fixed it, Phalcon has become stable.

So, just FYI, in a situation like this, Phalcon becomes unstable and crashes randomly. You may keep that in mind, perhaps you find an explanation one day.

Thanks!

@ghost
Copy link

ghost commented Feb 28, 2014

perhaps you find an explanation one day

Without the code to reproduce this on, this will be very difficult 😉

@temuri416
Copy link
Contributor Author

How about doing screensharing? I can guide you through the app and will show you any code that might be suspicious.

I have installed latest Ubuntu, recompiled everything, tried various config files - it is still crashing. Spent the whole day, no result.

@iby
Copy link
Contributor

iby commented Feb 28, 2014

Might sound like a silly question, but did you try setting a breakpoint in index.php and following to the point where it crashes?

@temuri416
Copy link
Contributor Author

I am using PHPed and the script does not crash. It always finished successfully, I traced it up until the moment of session destructor - no warnings, no errors.

I started checking out earlier commits and I see that they are stable. So, I'm gonna find the one that broke it.

@iby
Copy link
Contributor

iby commented Feb 28, 2014

I've been having a lot of pain lately with 1.3.0, too. Try recompiling it from the source, I've put some suggestions. That might help if you're after the latest commit.

@temuri416
Copy link
Contributor Author

@sjinks

OK, I FOUND IT.

The last stable commit is Round 1, 7db1056 , from 02/12/2014 03:51.

The next commit, Round 2, fa43e8f, from 02/12/2014 07:01 is crashing it.

I was compiling from ext using the following command:

phpize .clean && phpize && ./configure CFLAGS="-O0 -g3" CC=gcc && make -j75 && sudo make install

Again, there are no warnings or errors generated by my script.

I can send you over my source files as a last resort, but hopefully, we'll find other way to catch error.

Thank you!

@iby
Copy link
Contributor

iby commented Mar 1, 2014

Did you figure out how to get a dump by the way? Can you post it?

@iby
Copy link
Contributor

iby commented Mar 1, 2014

This helps? http://serverfault.com/a/304948

@temuri416
Copy link
Contributor Author

I got the dump, but sjinks said this:

"The dump is OK, it just means that the error happened somewhere in the PHP core."

@temuri416
Copy link
Contributor Author

@ianbytchek

oh, sorry misread your question. That's the link where I got the working solution from.

@iby
Copy link
Contributor

iby commented Mar 1, 2014

😐 missed that. Sorry. Can you post piece of code where it crashes? You said you narrowed it down to session being destroyed?

@iby
Copy link
Contributor

iby commented Mar 1, 2014

I'm somehow confident it's has something to do with Phalcon, especially if other commits are working.

@temuri416
Copy link
Contributor Author

You're confusing two issues that I'm currently having.

  1. After the mentioned commit Phalcon starts crashing on 1 out of 5 HTTP requests. I cannot post the code, because I cannot pinpoint it to any line and the execution stack is very long (my app also requires presence of a valid Google OAuth2 user, which does not make it easier to debug). The script does not crash. My code is perfectly valid, as it works and had worked for about 7 months with all previous commits on branch 1.3.0.
  2. There's another issue that's resurfaced - [BUG ... possibly?] Exceptions cause loss of user session when remote debugging is on. #1624

@phalcon
Copy link
Collaborator

phalcon commented Mar 1, 2014

Could you please try again using the current 1.3.0?

@iby
Copy link
Contributor

iby commented Mar 1, 2014

Got you. I had similar "floater" recently #2085. There still must be someway to pinpoint it either through dump or through php -S 127.0.0.1:10001 suggester above.

@temuri416
Copy link
Contributor Author

@phalcon

It is stable now!

BTW - what do I have to do to convince you to get rid of that sage of Phalcon\DI to store non-objects is deprecated, please use Phalcon\Registry instead error? I am pulling my hair out..

Thanks.

@phalcon
Copy link
Collaborator

phalcon commented Mar 1, 2014

We could add a php.ini option to keep backwards compatibility with the old behavior, but you must know you're giving an incorrect use to Phalcon\DI that's why you're getting that warning.

@temuri416
Copy link
Contributor Author

I do not think I am using the DI incorrectly. Let me explain using the code from the application I'm working on.

This is the WAMP Topic user-specific prefix generator that lives in the shared DI:

'WampPrefix' => function($app) {
    if ($app->getDI()->has('auth')) {
        $auth = $app->getDI()->get('auth');
        if ($auth->isAuthenticated()) {
            return $app->getDI()->get('params')['WampBasePrefix'] . $auth->getAccount()->getId();
        }
    }
    throw new \Exception('WAMP Prefix cannot be generated without authenticated User account.');
},

It returns a string and should only be executed once within the lifetime of a request. That string does not change, as it is specific to the currently signed in user. However, to generate that string, I need access to current $app object and its DI and some other components from DI.

I do not see anything wrong with the above approach. I don't want to use Phalcon\Registry as it will compose that prefix every time $registry->WampPrefix() is called. In addition, Phalcon\Registry does not extend Phalcon\DI\Injectable, and actually, that is where it all ends for me.

Is there a rational explanation of the dangers/problems that my approach has to returning strings from the DI?

Thanks.

@ghost
Copy link

ghost commented Mar 1, 2014

It is stable now!

It is not. fcall cache corruption is the result of another bug where the same variable is destructed more than once or when a function incorrectly uses parent function's memory frame.

Turning off the cache just masks the issue but does not fix it.

@lantian lantian mentioned this issue Mar 1, 2014
@temuri416
Copy link
Contributor Author

Is this still an open issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants