Issue loading images when threading is enabled

[evelyndooley]

I have a Flask app to display profiles based on LDAP data. After updating to 1.0 I'm having crashes when loading the data. Running the app with threaded=False works, but I'd like to have threads.

---

Expected Behavior

The app would pull an image from the LDAP server and return it to the client. It would be displayed in HTML.

@app.route("/image/<uid>", methods=["GET"])
def image(uid):
    return get_image(uid)

def get_image(uid):
    account = ldap_get_member(uid)
    return account.jpegPhoto

{% for m in members %}
	<a href="/user/{{ m['uid'] }}" class="card-block clearfix">
		<div class="card user-card">
			<div class="card-profile-pic card-img-top card-img-top-crop" style="background-image: url('/image/{{m['uid']}}');" >
			</div>
		</div>
	</a>
{% endfor %}

Actual Behavior

The application crashes some way through loading the images. It usually loads 1 or 2 before exiting like this.

This behavior is only observed when attempting to load a large amount of images, such as on a page with a group of users. It does not happen on a single profile which only loads a single image.

flask_pyoidc.flask_pyoidc
 * Serving Flask app "profiles" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://localhost:8080/ (Press CTRL+C to quit)
 * Restarting with stat
flask_pyoidc.flask_pyoidc
 * Debugger is active!
 * Debugger PIN: 252-481-398
/home/nate/.local/lib/python3.6/site-packages/flask/sessions.py:208: UserWarning: "localhost" is not a valid cookie domain, it must contain a ".". Add an entry to your hosts file, for example "localhost.localdomain", and use that instead.
  ' "{rv}.localdomain", and use that instead.'.format(rv=rv)
127.0.0.1 - - [29/Apr/2018 15:55:42] "GET /group/active HTTP/1.1" 302 -
127.0.0.1 - - [29/Apr/2018 15:55:42] "GET /redirect_uri?state=sb25ldwdXoNE6xlu&session_state=3f2de3ad-d4f8-431e-8d9a-09cfb811322b&code=eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..p7LHT5Z6Bzljz08mV4TzeQ.eG0Tnq9LJJ4OZtspH2qBOyYQkPsqp1gKYUATkQ8ANJbSsRmfI7wUgcEEgvNrVWq5ZcchfYeYqpxixCvVv0I53DdRkrS9IfNa4hv0UOFa4i7txNv1kxHBWfDnjULMTsP7exdpVc0fWxvO1S3fIHeuIm0kxaAsPBdt83wEmj147RnMhrHS6MTjAEICsBRehphzmLi1qWDn9aSoMISCTyc7kjqPikLCyMlpcn-7NPGgGVtQ4olCerE3lmNl-UqZFtYH.yZF3w47VEuMs4-hrboYW2w HTTP/1.1" 302 -
INFO:werkzeug:127.0.0.1 - - [29/Apr/2018 15:55:42] "GET /redirect_uri?state=sb25ldwdXoNE6xlu&session_state=3f2de3ad-d4f8-431e-8d9a-09cfb811322b&code=eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..p7LHT5Z6Bzljz08mV4TzeQ.eG0Tnq9LJJ4OZtspH2qBOyYQkPsqp1gKYUATkQ8ANJbSsRmfI7wUgcEEgvNrVWq5ZcchfYeYqpxixCvVv0I53DdRkrS9IfNa4hv0UOFa4i7txNv1kxHBWfDnjULMTsP7exdpVc0fWxvO1S3fIHeuIm0kxaAsPBdt83wEmj147RnMhrHS6MTjAEICsBRehphzmLi1qWDn9aSoMISCTyc7kjqPikLCyMlpcn-7NPGgGVtQ4olCerE3lmNl-UqZFtYH.yZF3w47VEuMs4-hrboYW2w HTTP/1.1" 302 -
127.0.0.1 - - [29/Apr/2018 15:55:43] "GET /group/active HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [29/Apr/2018 15:55:43] "GET /group/active HTTP/1.1" 200 -
127.0.0.1 - - [29/Apr/2018 15:55:43] "GET /image/dooley HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [29/Apr/2018 15:55:43] "GET /image/dooley HTTP/1.1" 200 -
Fatal Python error: saving thread twice?

Thread 0x00007f0263fff700 (most recent call first):
  File "/usr/lib/python3.6/hmac.py", line 93 in update
  File "/usr/lib/python3.6/hmac.py", line 84 in __init__
  File "/usr/lib/python3.6/hmac.py", line 144 in new
  File "/home/nate/.local/lib/python3.6/site-packages/itsdangerous.py", line 268 in get_signature
  File "/home/nate/.local/lib/python3.6/site-packages/itsdangerous.py", line 243 in verify_signature
  File "/home/nate/.local/lib/python3.6/site-packages/itsdangerous.py", line 362 in verify_signature
  File "/home/nate/.local/lib/python3.6/site-packages/itsdangerous.py", line 371 in unsign
  File "/home/nate/.local/lib/python3.6/site-packages/itsdangerous.py", line 422 in unsign
  File "/home/nate/.local/lib/python3.6/site-packages/itsdangerous.py", line 643 in loads
  File "/home/nate/.local/lib/python3.6/site-packages/flask/sessions.py", line 343 in open_session
  File "/home/nate/.local/lib/python3.6/site-packages/flask/ctx.py", line 377 in push
  File "/home/nate/.local/lib/python3.6/site-packages/flask/app.py", line 2283 in wsgi_app
  File "/home/nate/.local/lib/python3.6/site-packages/flask/app.py", line 2301 in __call__
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/debug/__init__.py", line 288 in debug_application
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 260 in execute
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 270 in run_wsgi
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 328 in handle_one_request
  File "/usr/lib/python3.6/http/server.py", line 418 in handle
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 293 in handle
  File "/usr/lib/python3.6/socketserver.py", line 696 in __init__
  File "/usr/lib/python3.6/socketserver.py", line 361 in finish_request
  File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007f0278997700 (most recent call first):
  File "/usr/lib/python3.6/socket.py", line 586 in readinto
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 324 in handle_one_request
  File "/usr/lib/python3.6/http/server.py", line 418 in handle
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 293 in handle
  File "/usr/lib/python3.6/socketserver.py", line 696 in __init__
  File "/usr/lib/python3.6/socketserver.py", line 361 in finish_request
  File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007f0279198700 (most recent call first):
  File "/home/nate/.local/lib/python3.6/site-packages/ldap/ldapobject.py", line 293 in _ldap_call
  File "/home/nate/.local/lib/python3.6/site-packages/ldap/ldapobject.py", line 792 in search_ext
  File "/home/nate/.local/lib/python3.6/site-packages/ldap/ldapobject.py", line 796 in search_ext_s
  File "/home/nate/.local/lib/python3.6/site-packages/ldap/ldapobject.py", line 1148 in _apply_method_s
  File "/home/nate/.local/lib/python3.6/site-packages/ldap/ldapobject.py", line 1210 in search_ext_s
  File "/home/nate/.local/lib/python3.6/site-packages/ldap/ldapobject.py", line 803 in search_s
  File "/home/nate/.local/lib/python3.6/site-packages/csh_ldap/member.py", line 26 in __init__
  File "/home/nate/.local/lib/python3.6/site-packages/csh_ldap/__init__.py", line 43 in get_member
  File "/home/nate/src/profiles/profiles/ldap.py", line 49 in ldap_get_member
  File "/home/nate/src/profiles/profiles/ldap.py", line 312 in get_image
  File "/home/nate/src/profiles/profiles/__init__.py", line 133 in image
  File "/home/nate/.local/lib/python3.6/site-packages/flask/app.py", line 1791 in dispatch_request
  File "/home/nate/.local/lib/python3.6/site-packages/flask/app.py", line 1805 in full_dispatch_request
  File "/home/nate/.local/lib/python3.6/site-packages/flask/app.py", line 2284 in wsgi_app
  File "/home/nate/.local/lib/python3.6/site-packages/flask/app.py", line 2301 in __call__
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/debug/__init__.py", line 288 in debug_application
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 260 in execute
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 270 in run_wsgi
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 328 in handle_one_request
  File "/usr/lib/python3.6/http/server.py", line 418 in handle
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 293 in handle
  File "/usr/lib/python3.6/socketserver.py", line 696 in __init__
  File "/usr/lib/python3.6/socketserver.py", line 361 in finish_request
  File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Current thread 0x00007f0279999700 (most recent call first):
  File "/home/nate/.local/lib/python3.6/site-packages/ldap/ldapobject.py", line 293 in _ldap_call
  File "/home/nate/.local/lib/python3.6/site-packages/ldap/ldapobject.py", line 722 in result4
  File "/home/nate/.local/lib/python3.6/site-packages/ldap/ldapobject.py", line 715 in result3
  File "/home/nate/.local/lib/python3.6/site-packages/ldap/ldapobject.py", line 708 in result2
  File "/home/nate/.local/lib/python3.6/site-packages/ldap/ldapobject.py", line 704 in result
  File "/home/nate/.local/lib/python3.6/site-packages/ldap/ldapobject.py", line 797 in search_ext_s
  File "/home/nate/.local/lib/python3.6/site-packages/ldap/ldapobject.py", line 1148 in _apply_method_s
  File "/home/nate/.local/lib/python3.6/site-packages/ldap/ldapobject.py", line 1210 in search_ext_s
  File "/home/nate/.local/lib/python3.6/site-packages/ldap/ldapobject.py", line 803 in search_s
  File "/home/nate/.local/lib/python3.6/site-packages/csh_ldap/member.py", line 26 in __init__
  File "/home/nate/.local/lib/python3.6/site-packages/csh_ldap/__init__.py", line 43 in get_member
  File "/home/nate/src/profiles/profiles/ldap.py", line 49 in ldap_get_member
  File "/home/nate/src/profiles/profiles/ldap.py", line 312 in get_image
  File "/home/nate/src/profiles/profiles/__init__.py", line 133 in image
  File "/home/nate/.local/lib/python3.6/site-packages/flask/app.py", line 1791 in dispatch_request
  File "/home/nate/.local/lib/python3.6/site-packages/flask/app.py", line 1805 in full_dispatch_request
  File "/home/nate/.local/lib/python3.6/site-packages/flask/app.py", line 2284 in wsgi_app
  File "/home/nate/.local/lib/python3.6/site-packages/flask/app.py", line 2301 in __call__
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/debug/__init__.py", line 288 in debug_application
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 260 in execute
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 270 in run_wsgi
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 328 in handle_one_request
  File "/usr/lib/python3.6/http/server.py", line 418 in handle
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 293 in handle
  File "/usr/lib/python3.6/socketserver.py", line 696 in __init__
  File "/usr/lib/python3.6/socketserver.py", line 361 in finish_request
  File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007f027a431700 (most recent call first):
  File "/usr/lib/python3.6/ssl.py", line 689 in do_handshake
  File "/usr/lib/python3.6/ssl.py", line 1068 in do_handshake
  File "/usr/lib/python3.6/ssl.py", line 814 in __init__
  File "/usr/lib/python3.6/ssl.py", line 407 in wrap_socket
  File "/usr/lib/python3.6/http/client.py", line 1400 in connect
  File "/usr/lib/python3.6/http/client.py", line 964 in send
  File "/usr/lib/python3.6/http/client.py", line 1026 in _send_output
  File "/usr/lib/python3.6/http/client.py", line 1234 in endheaders
  File "/usr/lib/python3.6/http/client.py", line 1285 in _send_request
  File "/usr/lib/python3.6/http/client.py", line 1239 in request
  File "/usr/lib/python3.6/urllib/request.py", line 1318 in do_open
  File "/usr/lib/python3.6/urllib/request.py", line 1361 in https_open
  File "/usr/lib/python3.6/urllib/request.py", line 504 in _call_chain
  File "/usr/lib/python3.6/urllib/request.py", line 544 in _open
  File "/usr/lib/python3.6/urllib/request.py", line 526 in open
  File "/usr/lib/python3.6/urllib/request.py", line 223 in urlopen
  File "/home/nate/src/profiles/profiles/ldap.py", line 324 in get_image
  File "/home/nate/src/profiles/profiles/__init__.py", line 133 in image
  File "/home/nate/.local/lib/python3.6/site-packages/flask/app.py", line 1791 in dispatch_request
  File "/home/nate/.local/lib/python3.6/site-packages/flask/app.py", line 1805 in full_dispatch_request
  File "/home/nate/.local/lib/python3.6/site-packages/flask/app.py", line 2284 in wsgi_app
  File "/home/nate/.local/lib/python3.6/site-packages/flask/app.py", line 2301 in __call__
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/debug/__init__.py", line 288 in debug_application
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 260 in execute
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 270 in run_wsgi
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 328 in handle_one_request
  File "/usr/lib/python3.6/http/server.py", line 418 in handle
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 293 in handle
  File "/usr/lib/python3.6/socketserver.py", line 696 in __init__
  File "/usr/lib/python3.6/socketserver.py", line 361 in finish_request
  File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007f027ac32700 (most recent call first):
  File "/usr/lib/python3.6/threading.py", line 295 in wait
  File "/usr/lib/python3.6/threading.py", line 551 in wait
  File "/usr/lib/python3.6/threading.py", line 851 in start
  File "/usr/lib/python3.6/socketserver.py", line 650 in process_request
  File "/usr/lib/python3.6/socketserver.py", line 317 in _handle_request_noblock
  File "/usr/lib/python3.6/socketserver.py", line 238 in serve_forever
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 612 in serve_forever
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 777 in inner
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007f028a0fd540 (most recent call first):
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/_reloader.py", line 164 in run
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/_reloader.py", line 273 in run_with_reloader
  File "/home/nate/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 812 in run_simple
  File "/home/nate/.local/lib/python3.6/site-packages/flask/app.py", line 941 in run
  File "app.py", line 4 in <module>

Environment

• Python version: 3.6.5
• Flask version: 1.0
• Werkzeug version: 0.14.1

[davidism]

You haven't provided enough information to identify the problem. Does this happen if you run with multiple workers with another WSGI server, such as Gunicorn? gunicorn -w 4 myapp:app Does this occur on 0.12 when enabling threads?

[davidism]

This appears to be an issue related to making multiple concurrent LDAP calls. Perhaps the LDAP library you're using is not thread safe in some way? Do you get the error if you remove Flask from the example and try the same thing using a Python thread pool?