Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RecursionError when pickling Table object #2001

Open
adinamarca opened this issue Aug 23, 2024 · 0 comments
Open

RecursionError when pickling Table object #2001

adinamarca opened this issue Aug 23, 2024 · 0 comments
Assignees
Labels
api: bigquery Issues related to the googleapis/python-bigquery API.

Comments

@adinamarca
Copy link

adinamarca commented Aug 23, 2024

Hello.

Environment details

  • OS type and version: Latest Mac OS and also in Linux, Ubuntu 24.04 amd/64
  • Python version: 3.12.4
  • pip version: 24.0
  • google-cloud-bigquery version: 3.18.0

Steps to reproduce

  1. Create a valid instance of Table object.
  2. Using pickle, dumps the Table object.
  3. Using pickle, loads the Table object.

Code example

from warehouse.connectors import AccountAnalytics4Adminapi
from pickle import dumps, loads

def test_pickle():
    conn = AccountAnalytics4Adminapi("someaccount")
    conn.fetch()
    conn.transform()
    conn.load()
    
    try:
        pickled = dumps(conn)
        unpickled = loads(pickled)
    except Exception as e:
        assert False, f"Failed to pickle/unpickle object: {e}"
        
    assert conn == unpickled, "Pickled and unpickled objects are not equal."
    

Stack trace

Traceback (most recent call last):
  File "/Users/some_user/Desktop/proyectos/warehouse/repo/warehouse/tests/test_pickle.py", line 12, in test_pickle
    unpickled = loads(pickled)
                ^^^^^^^^^^^^^^
  File "/Users/some_user/.pyenv/versions/3.12.4/envs/warehouse/lib/python3.12/site-packages/google/cloud/bigquery/table.py", line 1478, in __getattr__
    value = self._xxx_field_to_index.get(name)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/some_user/.pyenv/versions/3.12.4/envs/warehouse/lib/python3.12/site-packages/google/cloud/bigquery/table.py", line 1478, in __getattr__
    value = self._xxx_field_to_index.get(name)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/some_user/.pyenv/versions/3.12.4/envs/warehouse/lib/python3.12/site-packages/google/cloud/bigquery/table.py", line 1478, in __getattr__
    value = self._xxx_field_to_index.get(name)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  [Previous line repeated 995 more times]
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/some_user/Desktop/proyectos/warehouse/repo/tests.py", line 3, in <module>
    test_pickle.test_pickle()
  File "/Users/some_user/Desktop/proyectos/warehouse/repo/warehouse/tests/test_pickle.py", line 14, in test_pickle
    assert False, f"Failed to pickle/unpickle object: {e}"
           ^^^^^
AssertionError: Failed to pickle/unpickle object: maximum recursion depth exceeded

I was using Airflow, and in a task a maximum recursion depth exceeded exception was raised. Also I checked this in my personal computer.

Im not expert, but I was checking the code and I see that in the "Row" object (which probably is used by Table), there is a recursion when calling __getattr__ because it uses the get method from _xxx_field_to_index, and the get method also uses the same get method from _xxx_field_to_index, which leads to recursion (my guess).

Thanks!

@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery API. label Aug 23, 2024
@shollyman shollyman assigned chalmerlowe and unassigned yirutang Aug 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery API.
Projects
None yet
Development

No branches or pull requests

3 participants