not compatible with Google Cloud Containers

[artooro]

I'm using the google/python-runtime Docker image to run an app on Google Managed VMs, and need to access the datastore. The obvious solution seems to be gcloud-python as it's so much easier to use than the datastore python library.

So here's what happens. In my requirements.txt file I have:
httplib2
tornado
gcloud

PIP installs everything OK except it fails when installing cffi. Here is a log:

Processing cffi-0.8.6.tar.gz

Writing /tmp/easy_install-MCEL5F/cryptography-0.6.1/temp/easy_install-ZPcCkB/cffi-0.8.6/setup.cfg

Running cffi-0.8.6/setup.py -q bdist_egg --dist-dir /tmp/easy_install-MCEL5F/cryptography-0.6.1/temp/easy_install-ZPcCkB/cffi-0.8.6/egg-dist-tmp-fmCpCO

compiling '_configtest.c':

__thread int some_threadlocal_variable_42;



gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -c _configtest.c -o _configtest.o

success!

removing: _configtest.c _configtest.o

c/_cffi_backend.c:13:17: fatal error: ffi.h: No such file or directory

compilation terminated.

Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/env/build/gcloud/setup.py", line 49, in <module>

    'Topic :: Internet',

  File "/usr/lib/python2.7/distutils/core.py", line 112, in setup

    _setup_distribution = dist = klass(attrs)

  File "/env/local/lib/python2.7/site-packages/setuptools/dist.py", line 260, in __init__

    self.fetch_build_eggs(attrs.pop('setup_requires'))

  File "/env/local/lib/python2.7/site-packages/setuptools/dist.py", line 285, in fetch_build_eggs

    replace_conflicting=True

  File "/env/local/lib/python2.7/site-packages/pkg_resources.py", line 631, in resolve

    dist = best[req.key] = env.best_match(req, ws, installer)

  File "/env/local/lib/python2.7/site-packages/pkg_resources.py", line 871, in best_match

    return self.obtain(req, installer)

  File "/env/local/lib/python2.7/site-packages/pkg_resources.py", line 883, in obtain

    return installer(requirement)

  File "/env/local/lib/python2.7/site-packages/setuptools/dist.py", line 335, in fetch_build_egg

    return cmd.easy_install(req)

  File "/env/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 595, in easy_install

    return self.install_item(spec, dist.location, tmpdir, deps)

  File "/env/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 625, in install_item

    dists = self.install_eggs(spec, download, tmpdir)

  File "/env/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 822, in install_eggs

    return self.build_and_install(setup_script, setup_base)

  File "/env/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1028, in build_and_install

    self.run_setup(setup_script, setup_base, args)

  File "/env/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1013, in run_setup

    run_setup(setup_script, args)

  File "/env/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 50, in run_setup

    lambda: execfile(

  File "/env/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 100, in run

    return func()

  File "/env/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 52, in <lambda>

    {'__file__':setup_script, '__name__':'__main__'}

  File "setup.py", line 338, in <module>



  File "/usr/lib/python2.7/distutils/core.py", line 112, in setup

    _setup_distribution = dist = klass(attrs)

  File "/env/local/lib/python2.7/site-packages/setuptools/dist.py", line 260, in __init__

    self.fetch_build_eggs(attrs.pop('setup_requires'))

  File "/env/local/lib/python2.7/site-packages/setuptools/dist.py", line 285, in fetch_build_eggs

    replace_conflicting=True

  File "/env/local/lib/python2.7/site-packages/pkg_resources.py", line 631, in resolve

    dist = best[req.key] = env.best_match(req, ws, installer)

  File "/env/local/lib/python2.7/site-packages/pkg_resources.py", line 871, in best_match

    return self.obtain(req, installer)

  File "/env/local/lib/python2.7/site-packages/pkg_resources.py", line 883, in obtain

    return installer(requirement)

  File "/env/local/lib/python2.7/site-packages/setuptools/dist.py", line 335, in fetch_build_egg

    return cmd.easy_install(req)

  File "/env/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 595, in easy_install

    return self.install_item(spec, dist.location, tmpdir, deps)

  File "/env/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 625, in install_item

    dists = self.install_eggs(spec, download, tmpdir)

  File "/env/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 822, in install_eggs

    return self.build_and_install(setup_script, setup_base)

  File "/env/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1028, in build_and_install

    self.run_setup(setup_script, setup_base, args)

  File "/env/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1016, in run_setup

    raise DistutilsError("Setup script exited with %s" % (v.args[0],))

distutils.errors.DistutilsError: Setup script exited with error: command 'gcc' failed with exit status 1

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /env/build/gcloud
Storing debug log for failure in /root/.pip/pip.log

This is kind of a road block for me, is there anything that can be done to make it installable on google containers?

[tseaver]

FWIW, cffi is an indirect dependency (via pyOpenSSL and cryptography), which are required to make the OAuth flows work.

[dhermes]

@artooro You need to apt-get install libffi-dev (and maybe also libffi6).

---

As for this library:

We could (should) make the crypto stuff a conditional dependency just like it is in oauth2client:
https://github.com/google/oauth2client/blob/7d3b1ae4f7a87de20dcf2a7e9517479cda6311dc/oauth2client/client.py#L43

Compare:
https://github.com/google/oauth2client/blob/7d3b1ae4f7a87de20dcf2a7e9517479cda6311dc/setup.py#L34
https://github.com/GoogleCloudPlatform/gcloud-python/blob/01a1eeab3d6d66acf4a6b83fb476a48a4c8c0247/setup.py#L20

With that fix, you'd need to get a token from the environment, which might be possible?

[silvolu]

#335 will fix all of this, also enabling support for Compute Engine Service Accounts (no need for keys).

@artooro for the time being you should use a custom dockerfile and follow @dhermes instructions.

[artooro]

I'll use the suggestion from @dhermes to install the required packages. Thanks!

[dhermes]

@artooro That's great!

The missing ffi.h file may not be the only issue, though adding libffi-dev will likely fix the cffi install issue. If you encounter more, please post here so we can have a better idea of what we're up against.

[artooro]

I added this to my Dockerfile with success:

RUN apt-get install -y libffi-dev
RUN /env/bin/pip install gcloud

Note: Can't have gcloud in requirements.txt because it is processed by the docker image before it runs the apt-get install.