DefaultCredentialsError (only sometimes) in Google Compute Engine with docker

[gjanvier]

Hello,

I run a python script hourly, embedded in a docker container hosted on a Google Compute Engine instance. That script includes a step where I use python lib google.cloud.storage to upload files in google storage.

I want to use the default service account of the compute engine instance. As written in the doc, authentication should “just work”. And most of the time it does.

Yet I quite often get the error:
File "/usr/local/lib/python2.7/dist-packages/google/auth/_default.py", line 282, in default raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credential and re-run the application. For more information, please see https://developers.google.com/accounts/docs/application-default-credentials.

My code:

import google.auth
from google.cloud import storage
...
credentials, projectId = google.auth.default()
# Next line raises sometimes the exception
client = storage.Client(projectId, credentials=credentials)
...

When I run a python CLI from that same docker container, and call the credentials, projectId = google.auth.default(), I can see that the credentials are a google.auth.compute_engine.credentials.Credentials object, so I guess that the "magic" works even in a container (the doc does not explain how it works though).

Any explanation on how to make it work, using the service account from the host compute instance ?

My env:

• OS: ubuntu:16:04 (compute instance and docker instance)
• python 2.7.12
• google-cloud-storage==0.22.0, google-auth==0.5.0

[theacodes]

@gjanvier do you have logging enabled? Do you see any logs from google.auth? If so, those can be helpful.

My initial guess is that either there's a network timeout occurring or docker is dropping the traffic to the metadata server. The auth library pings the metadata server to determine if it's available and then tries to acquire credentials. You can try setting the GCE_METADATA_TIMEOUT environment variable to something like 30 (the default is 3) to see if that helps.

[gjanvier]

I do see a "Compute Engine Metadata server unavailable" log just before the exception, nice guess.
Let me try that env variable, I'll give some feedbacks soon.

[gjanvier]

I set the logs in debug and here is what I can see:

On a regular upload (as mentionned, my script does several uploads and most of them are ok):

2017-02-14 06:54:34,205 | INFO | Connect to bucket 'xxxx'
2017-02-14 06:54:34,207 | DEBUG | Making request: GET http://169.254.***.***
2017-02-14 06:54:34,209 | DEBUG | Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
2017-02-14 06:54:34,213 | DEBUG | Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
2017-02-14 06:54:34,216 | DEBUG | Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/my_user@my_project.iam.gserviceaccount.com/token
2017-02-14 06:54:34,291 | INFO | Upload 'my_file.json' to 'xxx/my_file.json'

And when it fails:

2017-02-14 06:54:35,768 | INFO | Connect to bucket 'xxx'
2017-02-14 06:54:35,770 | DEBUG | Making request: GET http://169.254.***.***
2017-02-14 06:54:35,771 | INFO | Compute Engine Metadata server unavailable.
Traceback (most recent call last):
File "/usr/local/bin/aqmap-utils", line 82, in <module>
main()
File "/usr/local/bin/aqmap-utils", line 56, in main
disableCache=args['--disable-cache'],
File "/usr/local/lib/python2.7/dist-packages/aqtools/commands/aqmaputils.py", line 104, in publishGoogleStorage
_uploadGoogleStorage(bucketName, sources, destination, disableCache)
File "/usr/local/lib/python2.7/dist-packages/aqtools/commands/aqmaputils.py", line 149, in _uploadGoogleStorage
credentials, projectId = google.auth.default()
File "/usr/local/lib/python2.7/dist-packages/google/auth/_default.py", line 282, in default
raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or
explicitly create credential and re-run the application. For more
information, please see
https://developers.google.com/accounts/docs/application-default-credentials.

As you can see, the "Compute Engine Metadata server unavailable." error is raised instantly, before the 30s I configured as timeout. I don't know what kind of error it is (5xx, 4xx?).

Maybe the mechanism does not support the fact that I do several uploads one after another? More precisely, in my batch, I call 4 times the same script, each time generating 1 auth: first I upload ~300 images, then 24 .json, then 1 .json, finally 1 last .json. It appears that it is always the 4th call that fails, not surprisingly where the auth is performed ~1s after the 3rd.

[theacodes]

Consdering it's failing immediately, I'm a bit stumped as to why it's failing. If you do a simply retry does it succeed? I'm not sure there's much we can do to fix this, it seems like a network failure.

[gjanvier]

I added an automated retry:

2017-02-15 06:55:07,850 | INFO | Get google storage client...
2017-02-15 06:55:07,853 | DEBUG | Making request: GET http://169.254.169.254
2017-02-15 06:55:07,854 | INFO | Compute Engine Metadata server unavailable.
2017-02-15 06:55:07,854 | WARNING | Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or
explicitly create credential and re-run the application. For more
information, please see
https://developers.google.com/accounts/docs/application-default-credentials., retrying in 5 seconds...
2017-02-15 06:55:12,860 | INFO | Get google storage client...
2017-02-15 06:55:12,860 | DEBUG | Making request: GET http://169.254.169.254
2017-02-15 06:55:12,862 | DEBUG | Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
2017-02-15 06:55:12,866 | DEBUG | Making request: GET http://169.254.169.254
2017-02-15 06:55:12,866 | INFO | Compute Engine Metadata server unavailable.
2017-02-15 06:55:12,866 | WARNING | Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or
explicitly create credential and re-run the application. For more
information, please see
https://developers.google.com/accounts/docs/application-default-credentials., retrying in 5 seconds...
2017-02-15 06:55:17,867 | INFO | Get google storage client...
2017-02-15 06:55:17,867 | DEBUG | Making request: GET http://169.254.169.254
2017-02-15 06:55:17,869 | DEBUG | Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
2017-02-15 06:55:17,870 | WARNING | No project ID could be determined from the Compute Engine metadata service. Consider setting the GOOGLE_CLOUD_PROJECT environment variable.
2017-02-15 06:55:17,870 | DEBUG | Making request: GET http://169.254.169.254
2017-02-15 06:55:17,871 | INFO | Compute Engine Metadata server unavailable.

It is really strange that I have a network issue every 8 hours (my script run every 8 hours and fails all the time at the 4th upload).

[theacodes]

@gjanvier do you have any way to test to see if this occurs outside of docker?

[gjanvier]

I could, but actually I found a better "workaround" which is - I think - the right way to do it.

I wrote some python to handle the upload of my files on google storage because it made things simpler, as all of my operations were python. Now I've just abandonned that, and execute a gsutil cp instead. It works great, and even faster with parallel uploads.

I'll keep the google-cloud-python libs only for specific operations, not the batch uploads of hundreds of files.

Thanks anyway

[charlescapps]

I'm seeing this exact same issue. We are running a Python service in GAE Flex environment. We get this in our logs every once in a while and it's very frustrating as we expect getting the GAE standard credentials to be a stable operation. This is triggering our PagerDuty on a regular basis:

Sometimes it fails continuously for a few minutes in a row; it usually resolves after a few minutes. To be blunt, this is unacceptable. This means we don't have a stable environment.

[charlescapps]

Regarding this issue: do you recommend creating a singleton PubSub client with pubsub.Client()? If this is a valid approach, it may reduce this error since we wouldn't need to get the Application Default Credentials so often.

[dhermes]

@charlescapps Definitely create a Client() as infrequently as you can, if for no other reason than the overhead of creating a new transport (which will invoke gRPC helpers at the C level from a low-level / C helper).

[naomifridman]

Same issue.
Is the python environment stable ? with the java it went smoothly.
The python I don't manage to solve for days.

[theacodes]

If you are experiencing issues with the metadata service the suggest route is to provide a private key directly. See https://cloud.google.com/docs/authentication/getting-started.