header corruption in when using gevent loop engine after harakiri

[kriswehner]

We're running a high volume site on uwsgi 2.0.10 with the gevent loop engine. The application is very long polling based, COMET style, and so has very long timeouts configured.

Configuration is:

• 4 workers per node
• 50 gevent cores per worker
• harakiri set at 900 seconds

Relevant settings are:

[uwsgi]
gevent=50
master=true
loop=gevent
vacuum=true
virtualenv=env
lazy-apps=true
harakiri=900
worker-reload-mercy=900

OS is ubuntu trusty, running python 2.7.6 with gevent 1.0. The application is a Django 1.7 app.

We've been observing header corruption in production when harakiri occurs. Once a harakiri event happens on a worker, the newly spawned worker appears to initialize normally, but once it's up, the http headers that a given request sees include headers from other requests.

The only things we've firmly proven is that headers that should be not present in a request are incorrectly present, such as Referer: being set to an impossible value for an API request. In this sense, it sounds very similar to what was fixed here:
e393f36

At this point, we've observed a 1:1 correlation of harakiri events to this header corruption; neither occurs without the other. Internal efforts to reproduce have not shown the issue, which we assume is due to the reason for the harakiri, which we don't have root cause for. Our tests triggering harakiris with moderate amounts of concurrency and a forced infinite loop were not successful.

[unbit]

Hi, harakiri is only reliable for async and threads starting from 2.1. The problem you describe is a bit strange as harakiri generates a new whole process. Can you try with uWSGI 2.1 to check if the problem is still here ? Have you checked if the headers corruption happens only in the new generated worker ?

[georgebashi]

Hey @unbit, I'm working on the same codebase that @kriswehner was in 2015. This issue sounds a lot like what was described in this mailing list thread from a few months after this ticket, which was fixed in 8906d5d. Do they sound related to you? If so I think we could go ahead and close this ticket.

Is harakiri still dangerous for 2.0.15 with threads?