Unable to import tfds with tensorflow 2.12

[rivershah]

Short description
Unable to import tfds

Environment information

• Operating System:

• Python version:

• tensorflow-datasets/tfds-nightly version: tensorflow-datasets==4.9.1

• tensorflow/tf-nightly version: tensorflow==2.12.0

• Does the issue still exists with the last tfds-nightly package (pip install --upgrade tfds-nightly) ?

Reproduction instructions

python
Python 3.11.3 (main, Apr  5 2023, 14:15:06) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow_datasets as tfds
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/tensorflow_datasets/proto/__init__.py", line 20, in <module>
    from waymo_open_dataset import dataset_pb2 as waymo_dataset_pb2
ModuleNotFoundError: No module named 'waymo_open_dataset'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.11/dist-packages/tensorflow_datasets/__init__.py", line 43, in <module>
    import tensorflow_datasets.core.logging as _tfds_logging
  File "/usr/local/lib/python3.11/dist-packages/tensorflow_datasets/core/__init__.py", line 22, in <module>
    from tensorflow_datasets.core import community
  File "/usr/local/lib/python3.11/dist-packages/tensorflow_datasets/core/community/__init__.py", line 18, in <module>
    from tensorflow_datasets.core.community.huggingface_wrapper import mock_builtin_to_use_gfile
  File "/usr/local/lib/python3.11/dist-packages/tensorflow_datasets/core/community/huggingface_wrapper.py", line 31, in <module>
    from tensorflow_datasets.core import dataset_builder
  File "/usr/local/lib/python3.11/dist-packages/tensorflow_datasets/core/dataset_builder.py", line 34, in <module>
    from tensorflow_datasets.core import dataset_info
  File "/usr/local/lib/python3.11/dist-packages/tensorflow_datasets/core/dataset_info.py", line 52, in <module>
    from tensorflow_datasets.core.features import feature as feature_lib
  File "/usr/local/lib/python3.11/dist-packages/tensorflow_datasets/core/features/__init__.py", line 26, in <module>
    from tensorflow_datasets.core.features.dataset_feature import Dataset
  File "/usr/local/lib/python3.11/dist-packages/tensorflow_datasets/core/features/dataset_feature.py", line 23, in <module>
    from tensorflow_datasets.core.features import sequence_feature
  File "/usr/local/lib/python3.11/dist-packages/tensorflow_datasets/core/features/sequence_feature.py", line 25, in <module>
    from tensorflow_datasets.core.features import features_dict
  File "/usr/local/lib/python3.11/dist-packages/tensorflow_datasets/core/features/features_dict.py", line 26, in <module>
    from tensorflow_datasets.core.features import top_level_feature
  File "/usr/local/lib/python3.11/dist-packages/tensorflow_datasets/core/features/top_level_feature.py", line 24, in <module>
    from tensorflow_datasets.core import example_parser
  File "/usr/local/lib/python3.11/dist-packages/tensorflow_datasets/core/example_parser.py", line 30, in <module>
    from tensorflow_datasets.proto import tf_example_pb2
  File "/usr/local/lib/python3.11/dist-packages/tensorflow_datasets/proto/__init__.py", line 23, in <module>
    from tensorflow_datasets.proto import waymo_dataset_generated_pb2 as waymo_dataset_pb2  # pylint: disable=line-too-long
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/tensorflow_datasets/proto/waymo_dataset_generated_pb2.py", line 130, in <module>
    _descriptor.EnumValueDescriptor(
  File "/usr/local/lib/python3.11/dist-packages/google/protobuf/descriptor.py", line 796, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

[adamdavidcole]

I'm having the same problem with a lot of students in my class. If anyone finds a workaround, please do share it 🙏 Thanks!

[fineguy]

Hi, sorry for the inconvenience, we're working on a fix.

Could you try pip install --force-reinstall -v protobuf==3.20.3?

[cdimofte]

we got some builds failing due to tensorflow-metadata ( dependency of tensorflow-datasets==4.4.0 ) having had a new version released on 11th April that installed other versions of other packages used by our build

[Wanglongzhi2001]

I solved it by installing the tfds-nightly and pip install --force-reinstall -v protobuf==3.20.3

[cdimofte]

I solved it by installing the tfds-nightly and pip install --force-reinstall -v protobuf==3.20.3

yes, this seems to be the workaround

[rivershah]

Unfortunately this is not a great workaround. protobuf needs to be on later versions for many other parts of the more recent tensorflow ecosystem. Requesting that we please fix this issue in tfds with tensorflow>=2.12.0

[andreped]

I have the same problem wth tensorflow==2.10.

I believe the correct fix would be to change the protobuf versioning set here to protobuf<=3.20

At least that resolved the issue for me in the GradientAccumulator project.

[fineguy]

We published a new release, could you please confirm that it works for you?
pip install tensorflow_datasets==4.9.2 --upgrade

[coreyfournier]

We published a new release, could you please confirm that it works for you? pip install tensorflow_datasets==4.9.2 --upgrade
@fineguy Works for me

[beew]

We published a new release, could you please confirm that it works for you? pip install tensorflow_datasets==4.9.2 --upgrade

No, it doesn't work.

pip install tensorflow_datasets==4.9.2 --upgrade --dry-run
...
Would install protobuf-4.22.3 tensorflow-datasets-4.9.2 tensorflow-metadata-1.13.1

tensorfow-2.11.0 requires protobuf<3.20,>=3.9.2

[Laubeee]

I'm on tf 2.10 (for cuda on windows) and solved it by installing tf-metadata 1.13.0 first, which does not need newer protobuf. Then, I used an old version of tf-datasets which also does not need newer protobuf.

so it was:

pip install tensorflow-metadata==1.13.0 
pip install tensorflow_datasets==4.4.*

[Laubeee]

so tf datasets could in theory solve this by changing its requirements to tf-metadata <= 1.13.0 for its older versions such that the protobuf-requirements align in the end

and then perhaps it can even be made so when you just pip install the package it will choose the right one for the installed tf version?