OperationalError: (2006, 'MySQL server has gone away')

[abhijo89]

When ever i restart my MySql celeryd and celerycam throw OperationalError: (2006, 'MySQL server has gone away') .

    Traceback (most recent call last):
      File "/home/webapp/webapp/../vendor/djcelery/schedulers.py", line 204, in sync
        with commit_on_success():
      File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
        return self.gen.next()
      File "/home/webapp/webapp/../vendor/djcelery/db.py", line 56, in commit_on_success
        with transaction.atomic(using):
      File "/usr/local/lib/python2.7/dist-packages/django/db/transaction.py", line 280, in __enter__
        connection.set_autocommit(False)
      File "/usr/local/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 333, in set_autocommit
        self._set_autocommit(autocommit)
      File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 460, in _set_autocommit
        self.connection.autocommit(autocommit)
    OperationalError: (2006, 'MySQL server has gone away')

Is it a bug ?

[adriaant]

Lots of helpful stuff you can find via Google:
https://www.google.com/search?client=safari&rls=en&q=celeryd+MySQL+server+has+gone+away&ie=UTF-8&oe=UTF-8

It might be better to use something else as broker backend, e.g. RabbitMQ or Redis.

[trunneml]

Hi, we are using django-celery with redis as broker and result backend and getting the same error as @abhijo89 on a mysql restart or fail-over-switch. So it isn't a brocker or result backend problem. I think it is related to the scheduler.

[trunneml]

Hi, I found the problem that we had in the DatabaseScheduler. It does not create a new connection when the old one is broken. The patch above fixes the problem for me.

[chrono]

Had a similar issue, when we did an RDS failover. Broker backend is RabbitMQ, but scheduler is DatabaseScheduler

OperationalError
Root Cause
(2006, 'MySQL server has gone away')
Stacktrace (most recent call last):

  File "djcelery/schedulers.py", line 176, in schedule_changed
    transaction.commit()
  File "django/db/transaction.py", line 139, in commit
    get_connection(using).commit()
  File "django/db/backends/__init__.py", line 171, in commit
    self._commit()
  File "django/db/backends/__init__.py", line 139, in _commit
    return self.connection.commit()
  File "django/db/utils.py", line 99, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "django/db/backends/__init__.py", line 139, in _commit
    return self.connection.commit()
  File "newrelic/hooks/database_dbapi2.py", line 71, in commit
    return self.__wrapped__.commit()

versions:

pip freeze|grep celer
celery==3.1.18
django-celery==3.1.1

As a result any changes to the schedule won't be picked up until celerybeat is restarted. The connection should be reestablished or the application should crash so that a supervisor or upstart [etc..] can handle the issue by restarting the daemon.

[abhijo89]

👍

[arturataide]

I have exactly the same error. The worker retrieves the "2006, 'MySQL server has gone away'" and freezes. I've applied the fix #359 but this issue continues to happen.
Any ideas on this?
Thanks

[vakorol]

This seems to be a problem with Celery, not django-celery. I have the same problem, although am not using django-celery at all. There is a ticket on this:
celery/celery#2919

[arturataide]

Thanks @vakorol. I am upgrading my app and I think I will change my database from MySQL to PostgreSQL. I've made some tests and with PostgreSQL the error seems to disappear.

[brianmay]

I am not sure I entirely understand the circumstances of this bug ...

To me this sounds suspiciously like a Django/mysql bug. mysql will close the connection if it has been idle for more then X seconds (value is configurable), and the client is suppose to attempt another connection to the server if this happens. It sounds very much like Django isn't doing this however, and generating an error instead.

http://dev.mysql.com/doc/refman/5.7/en/gone-away.html

[chrono]

I think the issue is that the exception handling here: https://github.com/celery/django-celery/blob/v3.1.17/djcelery/schedulers.py#L183-L185 is too eager and catching an exception that it is not handling properly. Instead it logs it and pretends nothing happened.

A proper way would be to do one of these things:

• handle it (reconnect)
• don't catch it
• catch, log, raise.

[samskeller]

Just adding that I still have this problem constantly. Every time I restart my celery workers, things look fine for a little bit but then I get a barrage of these errors ("MySQL server has gone away"). The errors eventually stop, though, and the celery workers will then work fine until they're restarted again. Anybody have any ideas? Thanks!

[arturataide]

@samskeller I haven't manage to solve this issue.. Like I said on 22 Dec 2015 I've migrated to PostgreSQL, and obviously MySQL errors stopped. This kind of issue has never happened again with my new configuration.

[revilwang]

I'm using celerybeat to periodically run tasks every mintue. The broker and result backend both are RabbitMQ, with celerybeat using the default scheduler. There are django DB queries in the task, but still ("MySQL server has gone away") exception starts to pop up in DB query after several hours.

Celery verison 3.1.23, Django 1.8.12