pip install feast[gcp] for 0.34.1 does not work when running python test_workflow.py

[ponekscb]

Expected Behavior

python test_workflow.py from feast init -t gcp should run successfully

Current Behavior

Seems feast cannot import ARROW_SCALAR_IDS_TO_BQ from bigquery library as in

feast.errors.FeastExtrasDependencyImportError: cannot import name 'ARROW_SCALAR_IDS_TO_BQ' from 'google.cloud.bigquery._pandas_helpers'

Here is the full stacktrace

% python test_workflow.py 
/Users/ponekscb/Documents/Code/test-feast/.venv/lib/python3.10/site-packages/google/rpc/__init__.py:18: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  import pkg_resources
/Users/ponekscb/Documents/Code/test-feast/.venv/lib/python3.10/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/Users/ponekscb/Documents/Code/test-feast/.venv/lib/python3.10/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.cloud')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/Users/ponekscb/Documents/Code/test-feast/.venv/lib/python3.10/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.logging')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/Users/ponekscb/Documents/Code/test-feast/.venv/lib/python3.10/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.iam')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/Users/ponekscb/Documents/Code/test-feast/.venv/lib/python3.10/site-packages/google/rpc/__init__.py:20: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.rpc')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  pkg_resources.declare_namespace(__name__)
Traceback (most recent call last):
  File "/Users/ponekscb/Documents/Code/test-feast/.venv/lib/python3.10/site-packages/feast/infra/offline_stores/bigquery.py", line 62, in <module>
    from google.cloud.bigquery._pandas_helpers import ARROW_SCALAR_IDS_TO_BQ
ImportError: cannot import name 'ARROW_SCALAR_IDS_TO_BQ' from 'google.cloud.bigquery._pandas_helpers' (/Users/ponekscb/Documents/Code/test-feast/.venv/lib/python3.10/site-packages/google/cloud/bigquery/_pandas_helpers.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ponekscb/Documents/Code/test-feast/.venv/lib/python3.10/site-packages/feast/importer.py", line 26, in import_class
    module = importlib.import_module(module_name)
  File "/opt/homebrew/Cellar/python@3.10/3.10.13/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/ponekscb/Documents/Code/test-feast/.venv/lib/python3.10/site-packages/feast/infra/offline_stores/bigquery.py", line 68, in <module>
    raise FeastExtrasDependencyImportError("gcp", str(e))
feast.errors.FeastExtrasDependencyImportError: cannot import name 'ARROW_SCALAR_IDS_TO_BQ' from 'google.cloud.bigquery._pandas_helpers' (/Users/ponekscb/Documents/Code/test-feast/.venv/lib/python3.10/site-packages/google/cloud/bigquery/_pandas_helpers.py)
You may need run pip install 'feast[gcp]'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/ponekscb/Documents/Code/test-feast/free_ibex/feature_repo/test_workflow.py", line 180, in <module>
    run_demo()
  File "/Users/ponekscb/Documents/Code/test-feast/free_ibex/feature_repo/test_workflow.py", line 12, in run_demo
    store = FeatureStore(repo_path=".")
  File "/Users/ponekscb/Documents/Code/test-feast/.venv/lib/python3.10/site-packages/feast/usage.py", line 362, in wrapper
    raise exc.with_traceback(traceback)
  File "/Users/ponekscb/Documents/Code/test-feast/.venv/lib/python3.10/site-packages/feast/usage.py", line 348, in wrapper
    return func(*args, **kwargs)
  File "/Users/ponekscb/Documents/Code/test-feast/.venv/lib/python3.10/site-packages/feast/feature_store.py", line 154, in __init__
    self.config = load_repo_config(
  File "/Users/ponekscb/Documents/Code/test-feast/.venv/lib/python3.10/site-packages/feast/repo_config.py", line 549, in load_repo_config
    c = RepoConfig(**raw_config)
  File "/Users/ponekscb/Documents/Code/test-feast/.venv/lib/python3.10/site-packages/feast/repo_config.py", line 185, in __init__
    super().__init__(**data)
  File "pydantic/main.py", line 339, in pydantic.main.BaseModel.__init__
  File "pydantic/main.py", line 1050, in pydantic.main.validate_model
  File "/Users/ponekscb/Documents/Code/test-feast/.venv/lib/python3.10/site-packages/feast/repo_config.py", line 385, in _validate_offline_store_config
    offline_config_class = get_offline_config_from_type(offline_store_type)
  File "/Users/ponekscb/Documents/Code/test-feast/.venv/lib/python3.10/site-packages/feast/repo_config.py", line 530, in get_offline_config_from_type
    return import_class(module_name, config_class_name, config_class_name)
  File "/Users/ponekscb/Documents/Code/test-feast/.venv/lib/python3.10/site-packages/feast/importer.py", line 31, in import_class
    raise FeastModuleImportError(module_name, class_name) from e
feast.errors.FeastModuleImportError: Could not import module 'feast.infra.offline_stores.bigquery' while attempting to load class 'BigQueryOfflineStoreConfig'

Steps to reproduce

pip install 'feast[gcp]'
feast init -t gcp
cd <project-name>/feature_repo
python test_workflow.py

Specifications

• Version: feast==0.34.1, google-cloud-bigquery==3.13.0
• Platform: MacOS 14.1.1 M1 Silicon
• Subsystem:

Possible Solution

I tried to dig through the google-cloud-bigquery library and found out that ARROW_SCALAR_IDS_TO_BQ was removed from _pandas_helpers.py in version 3.13.0 google-cloud-bigquery src, but for 3.12.0 it is still available.

So a possible workaround for any end-user is to lock the google-cloud-bigquery dependency to 3.12.0, but that should be a quickfix for now.

For more concrete solution, I think this line 62 from "feast/infra/offline_stores/bigquery.py" bigquery.py src

    from google.cloud.bigquery._pandas_helpers import ARROW_SCALAR_IDS_TO_BQ

should not depend on google.cloud.bigquery._pandas_helpers as _pandas_helpers might be a private api and can be changed whenever.

[sudohainguyen]

hey we resolved in #3823, you can fix on your side by downgrading google-cloud-bigquery to 3.12.0