Client's auth token times out (StackOverflow)

[tseaver]

@dhermes Seems like you have the best grasp on our oauthclient story.

http://stackoverflow.com/questions/37982336/unauthenticated-error-using-gcloud-python-to-connect-to-google-bigtable

/cc @sduskis

[dhermes]

Ouch. Seems we need to turn it into a 401 so oauth2client can natively handle the NetworkError. I'll try to replicate.

[sduskis]

ping. What's that status?

[dhermes]

Sorry for delay (had a baby). The entire behavior of oauth2client is based on catching a 401 status, but gRPC doesn't do HTTP status codes. But we're using get_access_token() which does a refresh if the token is expired.

I'm not sure how to replicate. It seems that a gRPC unauthenticated error could only occur if now >= self.token_expiry is false (when checking if the token is expired) just moments before the request but true during the request.

But it's possible I misunderstand how the Python gRPC stubs are using the token (maybe they cache it?). I'm going to do some checking and report back.

[dhermes]

OK I made a few tiny tweaks

diff --git a/gcloud/bigtable/client.py b/gcloud/bigtable/client.py
index 35ca65c..aa79733 100644
--- a/gcloud/bigtable/client.py
+++ b/gcloud/bigtable/client.py
@@ -284,6 +284,7 @@ class Client(_ClientFactoryMixin, _ClientProjectMixin):
         :rtype: :class:`grpc.beta._stub._AutoIntermediary`
         :returns: A gRPC stub object.
         """
+        print('Making data stub')
         return _make_stub(self, DATA_STUB_FACTORY_V2,
                           DATA_API_HOST_V2, DATA_API_PORT_V2)

@@ -293,6 +294,7 @@ class Client(_ClientFactoryMixin, _ClientProjectMixin):
         :rtype: :class:`grpc.beta._stub._AutoIntermediary`
         :returns: A gRPC stub object.
         """
+        print('Making instance stub')
         return _make_stub(self, INSTANCE_STUB_FACTORY_V2,
                           INSTANCE_ADMIN_HOST_V2, INSTANCE_ADMIN_PORT_V2)

@@ -305,6 +307,7 @@ class Client(_ClientFactoryMixin, _ClientProjectMixin):
         :rtype: :class:`grpc.beta._stub._AutoIntermediary`
         :returns: A gRPC stub object.
         """
+        print('Making operations stub')
         return _make_stub(self, OPERATIONS_STUB_FACTORY_V2,
                           OPERATIONS_API_HOST_V2, OPERATIONS_API_PORT_V2)

@@ -314,6 +317,7 @@ class Client(_ClientFactoryMixin, _ClientProjectMixin):
         :rtype: :class:`grpc.beta._stub._AutoIntermediary`
         :returns: A gRPC stub object.
         """
+        print('Making table stub')
         return _make_stub(self, TABLE_STUB_FACTORY_V2,
                           TABLE_ADMIN_HOST_V2, TABLE_ADMIN_PORT_V2)

@@ -432,13 +436,21 @@ class _MetadataPlugin(object):
                    Provides authorization and user agent.
     """

+    COUNT = 0
+
     def __init__(self, client):
+        _MetadataPlugin.COUNT += 1
+        self.curr_id = _MetadataPlugin.COUNT
+        print('MetadataPlugin %d created' % (self.curr_id,))
         self._credentials = client.credentials
         self._user_agent = client.user_agent

     def __call__(self, unused_context, callback):
         """Adds authorization header to request metadata."""
-        access_token = self._credentials.get_access_token().access_token
+        access_token = self._credentials.get_access_token()
+        print('plugin %s, expires in %r' % (self.curr_id,
+                                            access_token.expires_in))
+        access_token = access_token.access_token
         headers = [
             ('Authorization', 'Bearer ' + access_token),
             ('User-agent', self._user_agent),

and effectively verified there is no caching.

[dhermes]

@sduskis I don't think there is enough in the SO question to replicate, I asked the author to weigh in here. @jonparrott Can you confirm that Python Flexible on GAE will use GCE tokens when doing auth?

[herbertrlee]

@dhermes, I am using the gae flexible environment. If there are any steps you'd like me to take to collect more information, let me know.

[theacodes]

The Python compat runtime can end up using App Engine identity credentials,
iirc. The non-compat runtime uses GCE every time.

On Mon, Jul 18, 2016, 3:23 PM herbertrlee notifications@github.com wrote:

@dhermes https://github.com/dhermes, I am using the gae flexible
environment. If there are any steps you'd like me to take to collect more
information, let me know.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
#1917 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPUcxVkAQKRdBDAwU76OTL8rTN8M6irks5qW_z-gaJpZM4I_SC3
.

[dhermes]

@herbertrlee Can you consistently reproduce this? Can I send you some modifications so you can log the same bits I did above so we can tell where the bad token issue is occurring?

[herbertrlee]

@dhermes Sorry, I have been traveling the last week or so. Certainly, send them to me and I'll attempt to collect some logs.

[dhermes]

First, start with a fresh virtualenv so we can edit the files without feeling bad about it

$ virtualenv repro-1917
$ source repro-1917/bin/activate
(repro-1917) $ pip install --upgrade gcloud

then edit repro-1917/lib/python2.7/site-packages/gcloud/bigtable/client.py with my diff above.

You'll also want to wrap your code that fails with a trace so you can step in and find out the state of things

try:
    main()
except:
    import pdb
    pdb.set_trace()

[herbertrlee]

@dhermes - Using gcloud 0.18.0 (previously 0.17.0), oauthclient 3.0.0 (previously 2.2.0), and grpcio 1.0.0rc1 (previously 0.15.0), I no longer seem to be able to reproduce this issue. I do still occasionally get auth errors, but I recover from them on subsequent attempts without needing to redeploy my app, which I can handle on my side. Thanks for your help and patience!

[dhermes]

w00t! Thanks for the reply. Feel free to ping this issue if the problem comes back.