RuntimeError: maximum recursion depth exceeded

[HaipengSu]

Hi,

After updating the Datalab to the newest version, I got this Runtime Error.

RuntimeError: maximum recursion depth exceeded

Apparently, this is related to the depth of recursive calls.

For the same notebook, I didn't get the Runtime Error before, but only for the new Datalab. Is there any change related to this issue?

Thank you.

Or should I just use
import sys sys.setrecursionlimit(10000)

to increase the recursion limit?

[nikhilk]

Could you provide more information when this happens?

[HaipengSu]

Hi @nikhilk

From Datalab,
I have
notebooks = [ "notebook1.ipynb",
"notebook2.ipynb" ,
.
.
.
"notebook20.ipynb" ]
for notebook in notebooks:
%run $notebook

Never had any issues running this, but since updating to the newest version, I got the Runtime Error. Not still not sure about whether it is related.

Thx

[HaipengSu]

Hi @nikhilk,

I am trying to get the error reproducible, so that it wouldn't confuse you and others.

Will let you know once I get more information.

thx @parthea for pointing out.

[parthea]

@HaipengSu shared a link to a notebook session which is currently in a failed state. The issue only occurs in this specific notebook session. If I open a new notebook and run the same code the issue disappears.

from datalab.bigquery import _api
import datalab
table = bq.Table('geotab-embedded:Test_Haipeng.sgs')
api = _api.Api(datalab.context.Context.default())
api.tables_get(table._name_parts)

I'm still looking into this but at first glance it appears that the recursion is happening in oauth2client.

RuntimeErrorTraceback (most recent call last)
<ipython-input-47-5908590773f2> in <module>()
      2 table = bq.Table('geotab-embedded:Test_Haipeng.sgs')
      3 api = _api.Api(datalab.context.Context.default())
----> 4 api.tables_get(table._name_parts)

/usr/local/lib/python2.7/dist-packages/datalab/bigquery/_api.pyc in tables_get(self, table_name)
    357     """
    358     url = Api._ENDPOINT + (Api._TABLES_PATH % table_name)
--> 359     return datalab.utils.Http.request(url, credentials=self._credentials)
    360 
    361   def tables_list(self, dataset_name, max_results=0, page_token=None):

/usr/local/lib/python2.7/dist-packages/datalab/utils/_http.pyc in request(url, args, data, headers, method, credentials, raw_response, stats)
    132                                        method=method,
    133                                        body=data,
--> 134                                        headers=headers)
    135       if 200 <= response.status < 300:
    136         if raw_response:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614                 logger.info('Attempting refresh to obtain '
    615                             'initial access_token')
--> 616                 self._refresh(request_orig)
    617 
    618             # Clone and modify the request headers to add the appropriate

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         """
    128         response, content = http_request(
--> 129             META, headers={'Metadata-Flavor': 'Google'})
    130         content = _from_bytes(content)
    131         if response.status == http_client.OK:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614                 logger.info('Attempting refresh to obtain '
    615                             'initial access_token')
--> 616                 self._refresh(request_orig)
    617 
    618             # Clone and modify the request headers to add the appropriate

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         """
    128         response, content = http_request(
--> 129             META, headers={'Metadata-Flavor': 'Google'})
    130         content = _from_bytes(content)
    131         if response.status == http_client.OK:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614                 logger.info('Attempting refresh to obtain '
    615                             'initial access_token')
--> 616                 self._refresh(request_orig)
    617 
    618             # Clone and modify the request headers to add the appropriate

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         """
    128         response, content = http_request(
--> 129             META, headers={'Metadata-Flavor': 'Google'})
    130         content = _from_bytes(content)
    131         if response.status == http_client.OK:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614                 logger.info('Attempting refresh to obtain '
    615                             'initial access_token')
--> 616                 self._refresh(request_orig)
    617 
    618             # Clone and modify the request headers to add the appropriate

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         """
    128         response, content = http_request(
--> 129             META, headers={'Metadata-Flavor': 'Google'})
    130         content = _from_bytes(content)
    131         if response.status == http_client.OK:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614                 logger.info('Attempting refresh to obtain '
    615                             'initial access_token')
--> 616                 self._refresh(request_orig)
    617 
    618             # Clone and modify the request headers to add the appropriate

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         """
    128         response, content = http_request(
--> 129             META, headers={'Metadata-Flavor': 'Google'})
    130         content = _from_bytes(content)
    131         if response.status == http_client.OK:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614                 logger.info('Attempting refresh to obtain '
    615                             'initial access_token')
--> 616                 self._refresh(request_orig)
    617 
    618             # Clone and modify the request headers to add the appropriate

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         """
    128         response, content = http_request(
--> 129             META, headers={'Metadata-Flavor': 'Google'})
    130         content = _from_bytes(content)
    131         if response.status == http_client.OK:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614                 logger.info('Attempting refresh to obtain '
    615                             'initial access_token')
--> 616                 self._refresh(request_orig)
    617 
    618             # Clone and modify the request headers to add the appropriate

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         """
    128         response, content = http_request(
--> 129             META, headers={'Metadata-Flavor': 'Google'})
    130         content = _from_bytes(content)
    131         if response.status == http_client.OK:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614                 logger.info('Attempting refresh to obtain '
    615                             'initial access_token')
--> 616                 self._refresh(request_orig)
    617 
    618             # Clone and modify the request headers to add the appropriate

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         """
    128         response, content = http_request(
--> 129             META, headers={'Metadata-Flavor': 'Google'})
    130         content = _from_bytes(content)
    131         if response.status == http_client.OK:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614                 logger.info('Attempting refresh to obtain '
    615                             'initial access_token')
--> 616                 self._refresh(request_orig)
    617 
    618             # Clone and modify the request headers to add the appropriate

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         """
    128         response, content = http_request(
--> 129             META, headers={'Metadata-Flavor': 'Google'})
    130         content = _from_bytes(content)
    131         if response.status == http_client.OK:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614                 logger.info('Attempting refresh to obtain '
    615                             'initial access_token')
--> 616                 self._refresh(request_orig)
    617 
    618             # Clone and modify the request headers to add the appropriate

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         """
    128         response, content = http_request(
--> 129             META, headers={'Metadata-Flavor': 'Google'})
    130         content = _from_bytes(content)
    131         if response.status == http_client.OK:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614                 logger.info('Attempting refresh to obtain '
    615                             'initial access_token')
--> 616                 self._refresh(request_orig)
    617 
    618             # Clone and modify the request headers to add the appropriate

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         """
    128         response, content = http_request(
--> 129             META, headers={'Metadata-Flavor': 'Google'})
    130         content = _from_bytes(content)
    131         if response.status == http_client.OK:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    629             resp, content = request_orig(uri, method, body,
    630                                          clean_headers(headers),
--> 631                                          redirections, connection_type)
    632 
    633             # A stored token may expire between the time it is retrieved and

... last 1 frames repeated, from the frame below ...

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    629             resp, content = request_orig(uri, method, body,
    630                                          clean_headers(headers),
--> 631                                          redirections, connection_type)
    632 
    633             # A stored token may expire between the time it is retrieved and

RuntimeError: maximum recursion depth exceeded while calling a Python object

Here are the results after upgrading to oauth2client==4.0.0

Collecting oauth2client==4.0.0
  Downloading oauth2client-4.0.0-py2.py3-none-any.whl (184kB)
    100% |################################| 194kB 3.2MB/s 
Requirement already satisfied: httplib2>=0.9.1 in /usr/local/lib/python2.7/dist-packages (from oauth2client==4.0.0)
Requirement already satisfied: rsa>=3.1.4 in /usr/local/lib/python2.7/dist-packages (from oauth2client==4.0.0)
Requirement already satisfied: pyasn1>=0.1.7 in /usr/local/lib/python2.7/dist-packages (from oauth2client==4.0.0)
Requirement already satisfied: pyasn1-modules>=0.0.5 in /usr/local/lib/python2.7/dist-packages (from oauth2client==4.0.0)
Requirement already satisfied: six>=1.6.1 in /usr/local/lib/python2.7/dist-packages (from oauth2client==4.0.0)
Installing collected packages: oauth2client
  Found existing installation: oauth2client 2.2.0
    Uninstalling oauth2client-2.2.0:
      Successfully uninstalled oauth2client-2.2.0
Successfully installed oauth2client-4.0.0

RuntimeErrorTraceback (most recent call last)
<ipython-input-50-5908590773f2> in <module>()
      2 table = bq.Table('geotab-embedded:Test_Haipeng.sgs')
      3 api = _api.Api(datalab.context.Context.default())
----> 4 api.tables_get(table._name_parts)

/usr/local/lib/python2.7/dist-packages/datalab/bigquery/_api.pyc in tables_get(self, table_name)
    357     """
    358     url = Api._ENDPOINT + (Api._TABLES_PATH % table_name)
--> 359     return datalab.utils.Http.request(url, credentials=self._credentials)
    360 
    361   def tables_list(self, dataset_name, max_results=0, page_token=None):

/usr/local/lib/python2.7/dist-packages/datalab/utils/_http.pyc in request(url, args, data, headers, method, credentials, raw_response, stats)
    132                                        method=method,
    133                                        body=data,
--> 134                                        headers=headers)
    135       if 200 <= response.status < 300:
    136         if raw_response:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    614         retval = cls(
    615             data['access_token'],
--> 616             data['client_id'],
    617             data['client_secret'],
    618             data['refresh_token'],

/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
    127         try:
    128             self._retrieve_info(http)
--> 129             self.access_token, self.token_expiry = _metadata.get_token(
    130                 http, service_account=self.service_account_email)
    131         except http_client.HTTPException as err:

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    629 
    630     @property
--> 631     def access_token_expired(self):
    632         """True if the credential is expired or invalid.
    633 

... last 1 frames repeated, from the frame below ...

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    629 
    630     @property
--> 631     def access_token_expired(self):
    632         """True if the credential is expired or invalid.
    633 

RuntimeError: maximum recursion depth exceeded while calling a Python object

[HaipengSu]

Thx @parthea for keeping track on this issue.

[parthea]

Here is some code that @HaipengSu and I used to create the issue locally:

import datalab.storage as gs
bucketName = "uspto-pair"
bucket = gs.Bucket(bucketName)
for i in range(0,1000):
  for item in bucket.items():
    pass

results in

RuntimeErrorTraceback (most recent call last)
<ipython-input-9-5daf0d8d76e0> in <module>()
      3 bucket = gs.Bucket(bucketName)
      4 for i in range(0,1000):
----> 5   for item in bucket.items():
      6     pass

/usr/local/lib/python2.7/dist-packages/datalab/utils/_iterator.pyc in __iter__(self)
     34     """Provides iterator functionality."""
     35     while self._first_page or (self._page_token is not None):
---> 36       items, next_page_token = self._retriever(self._page_token, self._count)
     37 
     38       self._page_token = next_page_token

/usr/local/lib/python2.7/dist-packages/datalab/storage/_item.pyc in _retrieve_items(self, page_token, _)
    276                                          page_token=page_token)
    277     except Exception as e:
--> 278       raise e
    279 
    280     items = list_info.get('items', [])

RuntimeError: maximum recursion depth exceeded while calling a Python object

The issue appears to be resolved if we revert the changes made in googledatalab/pydatalab#195 .

cc @nikhilk
cc @yebrahim

[ojarjur]

It looks like the source of the bug is that every time an HTTP request is issued, the call to credentials.authorize adds another wrapper around the http instance's request method.

Then, when http.request(...) gets invoked, all of those layers of wrappers have to be passed through before the final request actually gets performed.

In essence, we have a leak where the level of method wrapping grows unbounded, and eventually hits the recursive depth limit.

We can avoid this if the number of request wrappers was fixed (e.g. at 1). That, in turn, might be doable by making sure that the request method in the shared http instance cannot be overwritten.

I'll try making a copy of the shared instance before adding the credentials, and see if that prevents the bug.

[HaipengSu]

Thank you all for looking into this.