"import kafka" fails with "ModuleNotFoundError: No module named 'kafka.vendor.six.moves'" under Python 3.12

[gtzampanakis]

I have verified this on Linux and Windows, even though the output below is from Linux:

user1@CY1-WL-337:~/dev$ python3.12 -m venv venv
user1@CY1-WL-337:~/dev$ source venv/bin/activate
(venv) user1@CY1-WL-337:~/dev$ pip install kafka-python
Collecting kafka-python
  Using cached kafka_python-2.0.2-py2.py3-none-any.whl (246 kB)
Installing collected packages: kafka-python
Successfully installed kafka-python-2.0.2

(venv) user1@CY1-WL-337:~/dev$ python -c 'import kafka'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/__init__.py", line 23, in <module>
    from kafka.consumer import KafkaConsumer
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/consumer/__init__.py", line 3, in <module>
    from kafka.consumer.group import KafkaConsumer
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/consumer/group.py", line 13, in <module>
    from kafka.consumer.fetcher import Fetcher
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/consumer/fetcher.py", line 19, in <module>
    from kafka.record import MemoryRecords
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/record/__init__.py", line 1, in <module>
    from kafka.record.memory_records import MemoryRecords, MemoryRecordsBuilder
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/record/memory_records.py", line 27, in <module>
    from kafka.record.legacy_records import LegacyRecordBatch, LegacyRecordBatchBuilder
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/record/legacy_records.py", line 50, in <module>
    from kafka.codec import (
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/codec.py", line 9, in <module>
    from kafka.vendor.six.moves import range
ModuleNotFoundError: No module named 'kafka.vendor.six.moves'

[wbarnha]

This was patched in #2398 and will be resolved in the next release. I apologize for the delay. Hopefully running pip install git+https://github.com/dpkp/kafka-python.git could fulfill your needs in the meantime.

Please understand that I am doing everything within my means to get a release out to the public.

[simcax]

This was patched in #2398 and will be resolved in the next release. I apologize for the delay. Hopefully running pip install git+https://github.com/dpkp/kafka-python.git could fulfill your needs in the meantime.

Please understand that I am doing everything within my means to get a release out to the public.

Is there an idea of when the next release will be out?

[wbarnha]

No update, sorry. 😔

[Anitheanalyst]

Hi
i am still having this issue. Is it resolved in the latest release.

here is my contect:
Python 3.12.2
kafka-python 2.0.2

#import procedure:
from kafka import KafkaProducer
Error:
ModuleNotFoundError Traceback (most recent call last)
Cell In[7], line 1
----> 1 from kafka import KafkaProducer

File ~\AppData\Lib\site-packages\kafka_init_.py:23
21 from kafka.admin import KafkaAdminClient
22 from kafka.client_async import KafkaClient
---> 23 from kafka.consumer import KafkaConsumer
24 from kafka.consumer.subscription_state import ConsumerRebalanceListener
25 from kafka.producer import KafkaProducer

File ~\AppData\Lib\site-packages\kafka\consumer_init_.py:3
1 from future import absolute_import
----> 3 from kafka.consumer.group import KafkaConsumer
5 all = [
6 'KafkaConsumer'
7 ]

File ~\AppData\Lib\site-packages\kafka\consumer\group.py:13
10 from kafka.vendor import six
12 from kafka.client_async import KafkaClient, selectors
---> 13 from kafka.consumer.fetcher import Fetcher
14 from kafka.consumer.subscription_state import SubscriptionState
15 from kafka.coordinator.consumer import ConsumerCoordinator

File ~\AppData\Lib\site-packages\kafka\consumer\fetcher.py:19
15 from kafka.protocol.fetch import FetchRequest
16 from kafka.protocol.offset import (
17 OffsetRequest, OffsetResetStrategy, UNKNOWN_OFFSET
18 )
---> 19 from kafka.record import MemoryRecords
20 from kafka.serializer import Deserializer
21 from kafka.structs import TopicPartition, OffsetAndTimestamp

File ~\AppData\Lib\site-packages\kafka\record_init_.py:1
----> 1 from kafka.record.memory_records import MemoryRecords, MemoryRecordsBuilder
3 all = ["MemoryRecords", "MemoryRecordsBuilder"]

File ~\AppData\Lib\site-packages\kafka\record\memory_records.py:27
25 from kafka.errors import CorruptRecordException
26 from kafka.record.abc import ABCRecords
---> 27 from kafka.record.legacy_records import LegacyRecordBatch, LegacyRecordBatchBuilder
28 from kafka.record.default_records import DefaultRecordBatch, DefaultRecordBatchBuilder
31 class MemoryRecords(ABCRecords):

File ~\AppData\Lib\site-packages\kafka\record\legacy_records.py:50
47 from kafka.record.abc import ABCRecord, ABCRecordBatch, ABCRecordBatchBuilder
48 from kafka.record.util import calc_crc32
---> 50 from kafka.codec import (
51 gzip_encode, snappy_encode, lz4_encode, lz4_encode_old_kafka,
52 gzip_decode, snappy_decode, lz4_decode, lz4_decode_old_kafka,
53 )
54 import kafka.codec as codecs
55 from kafka.errors import CorruptRecordException, UnsupportedCodecError

File ~\AppData\Lib\site-packages\kafka\codec.py:9
6 import struct
8 from kafka.vendor import six
----> 9 from kafka.vendor.six.moves import range
11 _XERIAL_V1_HEADER = (-126, b'S', b'N', b'A', b'P', b'P', b'Y', 0, 1, 1)
12 _XERIAL_V1_FORMAT = 'bccccccBii'

ModuleNotFoundError: No module named 'kafka.vendor.six.moves'

[VishnuBhaarath]

Hi @wbarnha ,
The issue still persists and I am facing the same issue, can you pls provide an update when the fix will be rolloed out.
Thanks

[wbarnha]

I am aware of the issue. Please refer to #2412 (comment).

[VishnuBhaarath]

Thanks for it @wbarnha , I downgraded my python version and got it rectified.

[ankibunkers]

Did we ever merge this fix in for 3.12?

[ankibunkers]

This was patched in #2398 and will be resolved in the next release. I apologize for the delay. Hopefully running pip install git+https://github.com/dpkp/kafka-python.git could fulfill your needs in the meantime.

Please understand that I am doing everything within my means to get a release out to the public.

@wbarnha Thank you William for your support here.
Installing python -m pip install --break-system-packages git+https://github.com/dpkp/kafka-python.git still is not resolving the issue.

i can not downgrade python 3.11 due to a HIGH vulnerability with ecdsa pacakge. Any work around solution for this problem at the moment?

[wbarnha]

This was patched in #2398 and will be resolved in the next release. I apologize for the delay. Hopefully running pip install git+https://github.com/dpkp/kafka-python.git could fulfill your needs in the meantime.
Please understand that I am doing everything within my means to get a release out to the public.

@wbarnha Thank you William for your support here. Installing python -m pip install --break-system-packages git+https://github.com/dpkp/kafka-python.git still is not resolving the issue.

i can not downgrade python 3.11 due to a HIGH vulnerability with ecdsa pacakge. Any work around solution for this problem at the moment?

The current workaround solution would be to replace your kafka-python installation with my fork at kafka-python-ng for now.

[ssbarnea]

Can we please reopen this bug because a critical bug like this should not be closed until the fix is released. I would even pin it after reopening it, just to remind everyone about it.