For CeleryExecutor, why do we need to install mysql?

[r39132]

We don't use mysql, but there is a requirement to have the mysql client installed. We get the exception when starting the airflow worker.

[mistercrunch]

stack trace pls

[r39132]

Sorry. This was reported by a colleague of mine and he does not have the stack trace. It is possibly pilot error. Closing unless we can reproduce.

[dermorz]

I think it was the stack trace below. This happens if you run

$ airflow worker

on a freshly installed airflow with the defaults

broker_url = sqla+mysql://airflow:airflow@localhost:3306/airflow
celery_result_backend = db+mysql://airflow:airflow@localhost:3306/airflow

in the configuration file (with no mysql server running). The Exception does not raise if those values are properly set to a running rabbitmq instance.

 -------------- celery@airflow v3.1.18 (Cipater)
---- **** -----
--- * ***  * -- Linux-3.2.0-4-amd64-x86_64-with-debian-7.5
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         airflow.executors.celery_executor:0x345a550
- ** ---------- .> transport:   sqla+mysql://airflow:airflow@localhost:3306/airflow
- ** ---------- .> results:     db+mysql://airflow:airflow@localhost:3306/airflow
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ----
--- ***** ----- [queues]
 -------------- .> default          exchange=celery(direct) key=celery


[2015-07-24 13:37:36,009: ERROR/MainProcess] Unrecoverable error: ImportError('No module named MySQLdb',)
Traceback (most recent call last):
  File "/home/vagrant/env/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
    self.blueprint.start(self)
  File "/home/vagrant/env/local/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "/home/vagrant/env/local/lib/python2.7/site-packages/celery/bootsteps.py", line 374, in start
    return self.obj.start()
  File "/home/vagrant/env/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 278, in start
    blueprint.start(self)
  File "/home/vagrant/env/local/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "/home/vagrant/env/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 608, in start
    c.connection, on_decode_error=c.on_decode_error,
  File "/home/vagrant/env/local/lib/python2.7/site-packages/celery/app/amqp.py", line 375, in __init__
    **kw
  File "/home/vagrant/env/local/lib/python2.7/site-packages/kombu/messaging.py", line 363, in __init__
    self.revive(self.channel)
  File "/home/vagrant/env/local/lib/python2.7/site-packages/kombu/messaging.py", line 375, in revive
    self.declare()
  File "/home/vagrant/env/local/lib/python2.7/site-packages/kombu/messaging.py", line 385, in declare
    queue.declare()
  File "/home/vagrant/env/local/lib/python2.7/site-packages/kombu/entity.py", line 515, in declare
    self.queue_declare(nowait, passive=False)
  File "/home/vagrant/env/local/lib/python2.7/site-packages/kombu/entity.py", line 541, in queue_declare
    nowait=nowait)
  File "/home/vagrant/env/local/lib/python2.7/site-packages/kombu/transport/virtual/__init__.py", line 446, in queue_declare
    self._new_queue(queue, **kwargs)
  File "/home/vagrant/env/local/lib/python2.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 80, in _new_queue
    self._get_or_create(queue)
  File "/home/vagrant/env/local/lib/python2.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 68, in _get_or_create
    obj = self.session.query(self.queue_cls) \
  File "/home/vagrant/env/local/lib/python2.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 63, in session
    _, Session = self._open()
  File "/home/vagrant/env/local/lib/python2.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 54, in _open
    engine = self._engine_from_config()
  File "/home/vagrant/env/local/lib/python2.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 49, in _engine_from_config
    return create_engine(conninfo.hostname, **transport_options)
  File "/home/vagrant/env/local/lib/python2.7/site-packages/sqlalchemy/engine/__init__.py", line 386, in create_engine
    return strategy.create(*args, **kwargs)
  File "/home/vagrant/env/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 75, in create
    dbapi = dialect_cls.dbapi(**dbapi_args)
  File "/home/vagrant/env/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 92, in dbapi
    return __import__('MySQLdb')
ImportError: No module named MySQLdb
Starting flask
 * Running on http://0.0.0.0:8793/ (Press CTRL+C to quit)

[mistercrunch]

Seems like the expected behavior. Please reopen if you don't think it is.