Error in timer: OperationalError(2006, 'MySQL server has gone away')

[marco-silva0000]

system runs normally until something happens and worker stops running tasks until manually restarted.

I've applied the fix #370, for issue #359, but the error persists.

I have:
Django==1.6
celery==3.1.5
django-celery==3.1.1
MySQL-python==1.2.4
kombu==3.0.6

My traceback is:

[2015-12-10 09:50:43,300: ERROR/MainProcess] Error in timer: OperationalError(2006, 'MySQL server has gone away')
Traceback (most recent call last):
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/kombu/async/timer.py", line 171, in apply_entry
    entry()
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/kombu/async/timer.py", line 64, in __call__
    return self.fun(*self.args, **self.kwargs)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/kombu/async/timer.py", line 132, in _reschedules
    return fun(*args, **kwargs)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/celery/events/snapshot.py", line 73, in capture
    self.state.freeze_while(self.shutter, clear_after=self.clear_after)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/celery/events/state.py", line 428, in freeze_while
    return fun(*args, **kwargs)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/celery/events/snapshot.py", line 70, in shutter
    self.on_shutter(self.state)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/djcelery/snapshot.py", line 139, in on_shutter
    _handle_tasks()
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/djcelery/snapshot.py", line 135, in _handle_tasks
    self.handle_task(task)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/djcelery/snapshot.py", line 82, in handle_task
    (task.worker.hostname, task.worker),
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/djcelery/snapshot.py", line 72, in handle_worker
    defaults={'last_heartbeat': self.get_heartbeat(worker)},
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/djcelery/managers.py", line 87, in update_or_create
    return get_queryset(self).update_or_create(**kwargs)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/djcelery/managers.py", line 70, in update_or_create
    obj, created = self.get_or_create(**kwargs)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/django/db/models/query.py", line 373, in get_or_create
    return self.get(**lookup), False
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/django/db/models/query.py", line 301, in get
    num = len(clone)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/django/db/models/query.py", line 77, in __len__
    self._fetch_all()
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/django/db/models/query.py", line 854, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/django/db/models/query.py", line 220, in iterator
    for row in compiler.results_iter():
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 710, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 781, in execute_sql
    cursor.execute(sql, params)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 124, in execute
    return self.cursor.execute(query, args)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
    self.errorhandler(self, exc, value)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
OperationalError: (2006, 'MySQL server has gone away')

[marco-silva0000]

different traceback at beginning, may be relevant

[2015-12-10 11:54:32,071: ERROR/MainProcess] Pool callback raised exception: OperationalError(2006, 'MySQL server has gone away')
Traceback (most recent call last):
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/billiard/pool.py", line 1694, in safe_apply_callback
    fun(*args, **kwargs)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/celery/worker/job.py", line 437, in on_failure
    'terminated', True, string(exc), False)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/celery/worker/job.py", line 321, in _announce_revoked
    self.task.backend.mark_as_revoked(self.id, reason, request=self)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/celery/backends/base.py", line 157, in mark_as_revoked
    request=request)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/celery/backends/base.py", line 256, in store_result
    request=request, **kwargs)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/djcelery/backends/database.py", line 29, in _store_result
    traceback=traceback, children=self.current_task_children(request),
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/djcelery/managers.py", line 42, in _inner
    return fun(*args, **kwargs)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/djcelery/managers.py", line 181, in store_result
    'meta': {'children': children}})
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/djcelery/managers.py", line 87, in update_or_create
    return get_queryset(self).update_or_create(**kwargs)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/djcelery/managers.py", line 70, in update_or_create
    obj, created = self.get_or_create(**kwargs)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/django/db/models/query.py", line 373, in get_or_create
    return self.get(**lookup), False
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/django/db/models/query.py", line 301, in get
    num = len(clone)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/django/db/models/query.py", line 77, in __len__
    self._fetch_all()
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/django/db/models/query.py", line 854, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/django/db/models/query.py", line 220, in iterator
    for row in compiler.results_iter():
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 710, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 781, in execute_sql
    cursor.execute(sql, params)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 124, in execute
    return self.cursor.execute(query, args)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
    self.errorhandler(self, exc, value)
  File "/opt/virtualenvs/simocEnv/local/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
OperationalError: (2006, 'MySQL server has gone away')

[marco-silva0000]

on the error that generates the second callback, the worker is able to recover and complete the forwarding tasks. not sure if it can recover every time...

[marco-silva0000]

I have made a new test environment in django 1.9, with everything updated(celery, kombo, billiardm etc.), and the problem persists. I've recreated it running several tasks and revoking them. doing this raises the error after a few tries.

this is the traceback of this new env

[2015-12-14 10:03:25,382: CRITICAL/MainProcess] Internal error: OperationalError(2006, 'MySQL server has gone away')
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/__init__.py", line 227, in _process_task
    req.execute_using_pool(self.pool)
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/job.py", line 240, in execute_using_pool
    if self.revoked():
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/job.py", line 337, in revoked
    'expired' if expired else 'revoked', False, None, expired,
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/job.py", line 321, in _announce_revoked
    self.task.backend.mark_as_revoked(self.id, reason, request=self)
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/base.py", line 157, in mark_as_revoked
    request=request)
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/base.py", line 256, in store_result
    request=request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/djcelery/backends/database.py", line 29, in _store_result
    traceback=traceback, children=self.current_task_children(request),
  File "/usr/local/lib/python2.7/dist-packages/djcelery/managers.py", line 42, in _inner
    return fun(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/djcelery/managers.py", line 181, in store_result
    'meta': {'children': children}})
  File "/usr/local/lib/python2.7/dist-packages/djcelery/managers.py", line 87, in update_or_create
    return get_queryset(self).update_or_create(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/djcelery/managers.py", line 70, in update_or_create
    obj, created = self.get_or_create(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 373, in get_or_create
    return self.get(**lookup), False
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 301, in get
    num = len(clone)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 77, in __len__
    self._fetch_all()
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 854, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 220, in iterator
    for row in compiler.results_iter():
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 710, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 781, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/util.py", line 69, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 99, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 124, in execute
    return self.cursor.execute(query, args)
  File "/usr/local/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 205, in execute
    self.errorhandler(self, exc, value)
  File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
OperationalError: (2006, 'MySQL server has gone away')
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/__init__.py", line 227, in _process_task
    req.execute_using_pool(self.pool)
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/job.py", line 240, in execute_using_pool
    if self.revoked():
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/job.py", line 337, in revoked
    'expired' if expired else 'revoked', False, None, expired,
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/job.py", line 321, in _announce_revoked
    self.task.backend.mark_as_revoked(self.id, reason, request=self)
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/base.py", line 157, in mark_as_revoked
    request=request)
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/base.py", line 256, in store_result
    request=request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/djcelery/backends/database.py", line 29, in _store_result
    traceback=traceback, children=self.current_task_children(request),
  File "/usr/local/lib/python2.7/dist-packages/djcelery/managers.py", line 42, in _inner
    return fun(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/djcelery/managers.py", line 181, in store_result
    'meta': {'children': children}})
  File "/usr/local/lib/python2.7/dist-packages/djcelery/managers.py", line 87, in update_or_create
    return get_queryset(self).update_or_create(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/djcelery/managers.py", line 70, in update_or_create
    obj, created = self.get_or_create(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 373, in get_or_create
    return self.get(**lookup), False
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 301, in get
    num = len(clone)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 77, in __len__
    self._fetch_all()
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 854, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 220, in iterator
    for row in compiler.results_iter():
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 710, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 781, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/util.py", line 69, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 99, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 124, in execute
    return self.cursor.execute(query, args)
  File "/usr/local/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 205, in execute
    self.errorhandler(self, exc, value)
  File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
OperationalError: (2006, 'MySQL server has gone away')

[marco-silva0000]

I've adapted the test env to run with postgres backend. After some testing the error did not ocour. Changed the settings to a new mysqlDB, and the error happened in 30 seconds. So my guess is that there is something wrong with celery and mysql.

[abramovtv]

Have the same problem

[marco-silva0000]

What is your setup?

On Wed, Dec 30, 2015, 8:38 AM Timur notifications@github.com wrote:

Have the same problem

—
Reply to this email directly or view it on GitHub
#406 (comment)
.

[abramovtv]

Django==1.8.5
celery==3.1.18
django-celery==3.1.16
MySQL-python==1.2.5
kombu==3.0.29

[marco-silva0000]

I think the problem is with celery integrating mysql db. maybe it's a timeout, I don't know. but i changed to postgres and there isn't this problem. That's not a fix, and I would realy like to keep using mysql as I have several mysql db deployed... If you find anything update here, as I'll keep a eye on this issue.

has anyone solved this one?

[bcoover]

Same problem here

[auvipy]

did you tried 3.2.1 from github tag?? or if you are using new project then you should try the split up new packages

[temoto]

Relevant pull request #370 was merged. This issue should be also fixed by that.

[nmgeek]

It doesn't seem possible that the second and third tracebacks shown in this issue were fixed by the patch since djcelery is entered in backends/database.py in those tracebacks and the exception handler that fixed the first traceback catches exceptions thrown in the scheduler.

The problem demonstrated by the second and third tracebacks is related to revoking and terminating tasks.

Since this issue covered two fairly different bugs I reported this second bug separately as #517 .