Deadlock when adding a job

[jjo-sec]

Testing against a new deployment of v1.6.0 from tarball and git checkout tags/v1.6.0 (also see when testing against master)

When adding a job that contains an attachment using local mode, I am encountering a deadlock (this was also reported and closed in September #1044)

Deadlock waiting for <Future 7f3ac59fad10 created by _put_async(model.py:3474) for tasklet put(context.py:787) suspended generator put(context.py:824); pending> (user@localhost)
Traceback (most recent call last):
  File "/usr/lib/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.3/webapp2.py", line 545, in dispatch
    return method(*args, **kwargs)
  File "/home/jjones/clusterfuzz-1.6.0/src/appengine/libs/handler.py", line 287, in wrapper
    return func(self, *args, **kwargs)
  File "/home/jjones/clusterfuzz-1.6.0/src/appengine/libs/handler.py", line 422, in wrapper
    return func(self, *args, **kwargs)
  File "/home/jjones/clusterfuzz-1.6.0/src/appengine/handlers/jobs.py", line 140, in post
    job.put()
  File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/model.py", line 3458, in _put
    return self._put_async(**ctx_options).get_result()
  File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/tasklets.py", line 383, in get_result
    self.check_success()
  File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/ext/ndb/tasklets.py", line 380, in check_success
    raise self._exception.__class__, self._exception, self._traceback
RuntimeError: Deadlock waiting for <Future 7f3ac59fad10 created by _put_async(model.py:3474) for tasklet put(context.py:787) suspended generator put(context.py:824); pending>

[inferno-chromium]

Regressed in f5e837b
from importing of "from builtins import str"

Basically this has broken local testing completely, since db.puts result in deadlock if the entity has an attribute with str() assigned value (e.g. jobs page, testcase upload page that do str on blob key).
@mbarbella-chromium @oliverchang - any idea on fix, can you please investigate and release a new version.

[inferno-chromium]

can be easily reproduced on jobs and testcase uploads page locally by adding a file in upload. This does not impact production.

[inferno-chromium]

@jjo-sec - you can unblock yourself by removing "from builtins import str" from src/appengine/handlers/jobs.py. sorry we regressed this for local mode in our 2->3 migration, we should fix this soon.

[inferno-chromium]

v1.6.1 is released with the fix.
https://github.com/google/clusterfuzz/releases/tag/v1.6.1