-
Notifications
You must be signed in to change notification settings - Fork 894
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot install confluent-kafka with provided instructions on Apple's M1 #1190
Comments
@jliunyu Hi! Would you mind providing some guidance on how to fix this? Or point me to someone that could give some pointers? Thanks! |
Hi @armenzg , this looks like a path issue. Because librdkafka is installed with brew, but python doesn't look into the brew directories. So our suggestion is using the workaround you already used or find a way that python can find the brew installed packages. |
There's no `arm64` wheel for confluent-kafka, thus, we need to build our own and install it from there. Alternatively, we would have to automate a convoluted build work around (see [issue][upstream] for details). Fixes #28429. [upstream]: confluentinc/confluent-kafka-python#1190
There's no `arm64` wheel for confluent-kafka, thus, it needs to be built as part of the `pip install` step. Instead of waiting for official support, I've uploaded the wheel I produced into a GC bucket. Alternatively, we would automate a convoluted build process (see [issue][upstream] for details). Fixes #28429. [upstream]: confluentinc/confluent-kafka-python#1190
There's no `arm64` wheel for confluent-kafka, thus, it needs to be built as part of the `pip install` step. Instead of waiting for official support, I've uploaded the wheel I produced into a GC bucket. Alternatively, we would automate a convoluted build process (see [issue][upstream] for details). Fixes #28429. [upstream]: confluentinc/confluent-kafka-python#1190
There's no `arm64` wheel for confluent-kafka, thus, it needs to be built as part of the `pip install` step. Instead of waiting for official support, I've uploaded the wheel I produced into a GC bucket. Alternatively, we would automate a convoluted build process (see [issue][upstream] for details). Fixes #28429. [upstream]: confluentinc/confluent-kafka-python#1190
@jliunyu I figured this out. On Apple M1 hosts, all its needed is this:
|
Is this issue fixed? I'm seeing issues using confluent-kafka lib on M1
|
@edenhill is working on this, it will be more compatible with M1 |
CPATH was not quite working for me, but this did:
Remember to change the version to the librdkafka version you have installed through brew. |
This works. Unfortunately, I can't seem to get it to work in a Python Docker image, something like
fails, as I cannot pass those environment variables to the pip install command. I also do not want to change the Dockerfile since this is shared with the entire team, not just me. |
This did not work on my MAC. I still see the same error
|
Link the same issue with discussion to this one: #1025 |
Hi @yutotakano 👋 Hope this will help 🙂 Useful links |
@pthalasta thanks |
also not working for me. I have set CPATH, C_INCLUDE_PATH, and LIBRARY_PATH, but it still doesn't find the brew installed librdkafka headers on an M1 mac.
yet
|
I had to set this environment variable to get this to work Then I was able to successfully run |
I am getting the following error while i try to install × Running setup.py install for confluent_kafka did not run successfully. note: This error originates from a subprocess, and is likely not a problem with pip. × Encountered error while trying to install package. Please help |
I'm also getting the same error above when pip installing in a dockerfile! Works fine when emulating per: #1190 (comment) |
If you put the lines in the shell script file such as export C_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/include
export LIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/lib |
It's works for my M1 ! |
Even with the suggestions above, I get ❯ brew reinstall librdkafka
==> Downloading https://ghcr.io/v2/homebrew/core/librdkafka/manifests/1.9.0
Already downloaded: /Users/carlostangerino/Library/Caches/Homebrew/downloads/d7e9a712f8f7e97e8509a83f497d9e884db3bc48c7f98be129f912ce3be53901--librdkafka-1.9.0.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/librdkafka/blobs/sha256:bbe7efc09d58c2fd56280eb749883b330d7f066735be640911cea5e008bab6fa
Already downloaded: /Users/carlostangerino/Library/Caches/Homebrew/downloads/8353d49b621f5600c8f626c36681967098a4120527697f559c25b796e909a394--librdkafka--1.9.0.arm64_monterey.bottle.tar.gz
==> Reinstalling librdkafka
==> Pouring librdkafka--1.9.0.arm64_monterey.bottle.tar.gz
🍺 /opt/homebrew/Cellar/librdkafka/1.9.0: 38 files, 7.6MB
==> Running `brew cleanup librdkafka`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`). C_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/1.9.0/include
LIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/1.9.0/lib ❯ pip3 install confluent-kafka
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Collecting confluent-kafka
Using cached confluent-kafka-1.9.0.tar.gz (110 kB)
Preparing metadata (setup.py) ... done
Building wheels for collected packages: confluent-kafka
Building wheel for confluent-kafka (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [117 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-12-arm64-cpython-39
creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka
copying src/confluent_kafka/error.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka
copying src/confluent_kafka/serializing_producer.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka
copying src/confluent_kafka/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka
copying src/confluent_kafka/deserializing_consumer.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka
creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/avro.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/error.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/json_schema.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/schema_registry_client.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/protobuf.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/serialization
copying src/confluent_kafka/serialization/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/serialization
creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/admin
copying src/confluent_kafka/admin/_acl.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/admin
copying src/confluent_kafka/admin/_resource.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/admin
copying src/confluent_kafka/admin/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/admin
copying src/confluent_kafka/admin/_config.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/admin
creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro
copying src/confluent_kafka/avro/error.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro
copying src/confluent_kafka/avro/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro
copying src/confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro
copying src/confluent_kafka/avro/load.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro
creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/kafkatest
copying src/confluent_kafka/kafkatest/verifiable_client.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/kafkatest
copying src/confluent_kafka/kafkatest/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/kafkatest
copying src/confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/kafkatest
copying src/confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/kafkatest
creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro/serializer
copying src/confluent_kafka/avro/serializer/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro/serializer
copying src/confluent_kafka/avro/serializer/message_serializer.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro/serializer
running build_ext
building 'confluent_kafka.cimpl' extension
creating build/temp.macosx-12-arm64-cpython-39
creating build/temp.macosx-12-arm64-cpython-39/private
creating build/temp.macosx-12-arm64-cpython-39/private/var
creating build/temp.macosx-12-arm64-cpython-39/private/var/folders
creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr
creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn
creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T
creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3
creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98
creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src
creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka
creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/opt/homebrew/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c -o build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.o
In file included from /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:17:
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/confluent_kafka.h:64:2: error: "confluent-kafka-python requires librdkafka v1.9.0 or later. Install the latest version of librdkafka from Homebrew by running `brew install librdkafka` or `brew upgrade librdkafka`"
#error "confluent-kafka-python requires librdkafka v1.9.0 or later. Install the latest version of librdkafka from Homebrew by running `brew install librdkafka` or `brew upgrade librdkafka`"
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:156:8: error: unknown type name 'rd_kafka_AclBinding_t'
static rd_kafka_AclBinding_t *
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:162:9: error: use of undeclared identifier 'rd_kafka_AclBinding_t'
rd_kafka_AclBinding_t *ret = NULL;
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:162:32: error: use of undeclared identifier 'ret'
rd_kafka_AclBinding_t *ret = NULL;
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:172:21: error: use of undeclared identifier 'ret'
ret = rd_kafka_AclBinding_new(restype, resname, \
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:172:27: error: implicit declaration of function 'rd_kafka_AclBinding_new' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
ret = rd_kafka_AclBinding_new(restype, resname, \
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:179:16: error: use of undeclared identifier 'ret'
return ret;
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:185:8: error: unknown type name 'rd_kafka_AclBindingFilter_t'
static rd_kafka_AclBindingFilter_t*
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:192:9: error: use of undeclared identifier 'rd_kafka_AclBindingFilter_t'
rd_kafka_AclBindingFilter_t* ret = NULL;
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:192:38: error: use of undeclared identifier 'ret'
rd_kafka_AclBindingFilter_t* ret = NULL;
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:201:21: error: use of undeclared identifier 'ret'
ret = rd_kafka_AclBindingFilter_new(restype, resname, \
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:201:27: error: implicit declaration of function 'rd_kafka_AclBindingFilter_new' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
ret = rd_kafka_AclBindingFilter_new(restype, resname, \
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:208:16: error: use of undeclared identifier 'ret'
return ret;
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1045:9: error: use of undeclared identifier 'rd_kafka_AclBinding_t'
rd_kafka_AclBinding_t **c_objs = NULL;
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1045:33: error: use of undeclared identifier 'c_objs'
rd_kafka_AclBinding_t **c_objs = NULL;
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1081:46: error: use of undeclared identifier 'RD_KAFKA_ADMIN_OP_CREATEACLS'; did you mean 'RD_KAFKA_ADMIN_OP_CREATETOPICS'?
c_options = Admin_options_to_c(self, RD_KAFKA_ADMIN_OP_CREATEACLS,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~
RD_KAFKA_ADMIN_OP_CREATETOPICS
/usr/local/include/librdkafka/rdkafka.h:6117:9: note: 'RD_KAFKA_ADMIN_OP_CREATETOPICS' declared here
RD_KAFKA_ADMIN_OP_CREATETOPICS, /**< CreateTopics */
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1095:9: error: use of undeclared identifier 'c_objs'
c_objs = malloc(sizeof(*c_objs) * cnt);
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1095:33: error: use of undeclared identifier 'c_objs'
c_objs = malloc(sizeof(*c_objs) * cnt);
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1112:17: error: use of undeclared identifier 'c_objs'
c_objs[i] = Admin_py_to_c_AclBinding(res, errstr, sizeof(errstr));
^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
error: command '/usr/bin/clang' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for confluent-kafka
Running setup.py clean for confluent-kafka
Failed to build confluent-kafka
Installing collected packages: confluent-kafka
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Running setup.py install for confluent-kafka ... error
error: subprocess-exited-with-error
× Running setup.py install for confluent-kafka did not run successfully.
│ exit code: 1
╰─> [119 lines of output]
running install
/opt/homebrew/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_py
creating build
creating build/lib.macosx-12-arm64-cpython-39
creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka
copying src/confluent_kafka/error.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka
copying src/confluent_kafka/serializing_producer.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka
copying src/confluent_kafka/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka
copying src/confluent_kafka/deserializing_consumer.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka
creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/avro.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/error.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/json_schema.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/schema_registry_client.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/protobuf.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/serialization
copying src/confluent_kafka/serialization/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/serialization
creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/admin
copying src/confluent_kafka/admin/_acl.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/admin
copying src/confluent_kafka/admin/_resource.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/admin
copying src/confluent_kafka/admin/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/admin
copying src/confluent_kafka/admin/_config.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/admin
creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro
copying src/confluent_kafka/avro/error.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro
copying src/confluent_kafka/avro/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro
copying src/confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro
copying src/confluent_kafka/avro/load.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro
creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/kafkatest
copying src/confluent_kafka/kafkatest/verifiable_client.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/kafkatest
copying src/confluent_kafka/kafkatest/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/kafkatest
copying src/confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/kafkatest
copying src/confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/kafkatest
creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro/serializer
copying src/confluent_kafka/avro/serializer/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro/serializer
copying src/confluent_kafka/avro/serializer/message_serializer.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro/serializer
running build_ext
building 'confluent_kafka.cimpl' extension
creating build/temp.macosx-12-arm64-cpython-39
creating build/temp.macosx-12-arm64-cpython-39/private
creating build/temp.macosx-12-arm64-cpython-39/private/var
creating build/temp.macosx-12-arm64-cpython-39/private/var/folders
creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr
creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn
creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T
creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3
creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98
creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src
creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka
creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/opt/homebrew/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c -o build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.o
In file included from /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:17:
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/confluent_kafka.h:64:2: error: "confluent-kafka-python requires librdkafka v1.9.0 or later. Install the latest version of librdkafka from Homebrew by running `brew install librdkafka` or `brew upgrade librdkafka`"
#error "confluent-kafka-python requires librdkafka v1.9.0 or later. Install the latest version of librdkafka from Homebrew by running `brew install librdkafka` or `brew upgrade librdkafka`"
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:156:8: error: unknown type name 'rd_kafka_AclBinding_t'
static rd_kafka_AclBinding_t *
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:162:9: error: use of undeclared identifier 'rd_kafka_AclBinding_t'
rd_kafka_AclBinding_t *ret = NULL;
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:162:32: error: use of undeclared identifier 'ret'
rd_kafka_AclBinding_t *ret = NULL;
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:172:21: error: use of undeclared identifier 'ret'
ret = rd_kafka_AclBinding_new(restype, resname, \
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:172:27: error: implicit declaration of function 'rd_kafka_AclBinding_new' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
ret = rd_kafka_AclBinding_new(restype, resname, \
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:179:16: error: use of undeclared identifier 'ret'
return ret;
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:185:8: error: unknown type name 'rd_kafka_AclBindingFilter_t'
static rd_kafka_AclBindingFilter_t*
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:192:9: error: use of undeclared identifier 'rd_kafka_AclBindingFilter_t'
rd_kafka_AclBindingFilter_t* ret = NULL;
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:192:38: error: use of undeclared identifier 'ret'
rd_kafka_AclBindingFilter_t* ret = NULL;
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:201:21: error: use of undeclared identifier 'ret'
ret = rd_kafka_AclBindingFilter_new(restype, resname, \
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:201:27: error: implicit declaration of function 'rd_kafka_AclBindingFilter_new' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
ret = rd_kafka_AclBindingFilter_new(restype, resname, \
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:208:16: error: use of undeclared identifier 'ret'
return ret;
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1045:9: error: use of undeclared identifier 'rd_kafka_AclBinding_t'
rd_kafka_AclBinding_t **c_objs = NULL;
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1045:33: error: use of undeclared identifier 'c_objs'
rd_kafka_AclBinding_t **c_objs = NULL;
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1081:46: error: use of undeclared identifier 'RD_KAFKA_ADMIN_OP_CREATEACLS'; did you mean 'RD_KAFKA_ADMIN_OP_CREATETOPICS'?
c_options = Admin_options_to_c(self, RD_KAFKA_ADMIN_OP_CREATEACLS,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~
RD_KAFKA_ADMIN_OP_CREATETOPICS
/usr/local/include/librdkafka/rdkafka.h:6117:9: note: 'RD_KAFKA_ADMIN_OP_CREATETOPICS' declared here
RD_KAFKA_ADMIN_OP_CREATETOPICS, /**< CreateTopics */
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1095:9: error: use of undeclared identifier 'c_objs'
c_objs = malloc(sizeof(*c_objs) * cnt);
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1095:33: error: use of undeclared identifier 'c_objs'
c_objs = malloc(sizeof(*c_objs) * cnt);
^
/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1112:17: error: use of undeclared identifier 'c_objs'
c_objs[i] = Admin_py_to_c_AclBinding(res, errstr, sizeof(errstr));
^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
error: command '/usr/bin/clang' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure
× Encountered error while trying to install package.
╰─> confluent-kafka
note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
WARNING: There was an error checking the latest version of pip. |
Thanks for the suggestion, @armenzg. It does find the librdkafka by doing this; However, I have got an subsequent error
env setup:
Any suggestion? |
This worked for me! |
Take care, sometimes librdkafka and confluent-kafka-python versions don't match.
Once I @ddbhuang maybe this addresses your issue? |
Apparently,
Just make sure to change your |
Same error for me. I'll abandon this library :( |
In order this to work properly version of the library must be the same as in these vars. The newest version is 1.9.1 so this won't work. Run |
Most likely your library and cpath are not set for brew, where you probably installed
|
Confirming working steps in an M1 machine running macOS 12.4 and Python 3.10.5 with brew installed librdkafka. brew upgrade
brew install librdkafka
export C_INCLUDE_PATH=$(brew --prefix)/include
export LIBRARY_PATH=$(brew --prefix)/lib
pip install confluent-kafka Thanks everyone for the help! |
Thanks for all the great answers. Closing this issue as wheels for MacOS arm64 were added as part of v1.9.2 release and linux arm64 wheels were added in v2.1.0 release. |
Description
I cannot install confluent-kafka on Apple's M1 host with neither Python 3.8 nor Python 3.9 with the provided instructions.
Workaround
This is a combination of what @evgmoskalenko posted here and what @emrahgunduz posted here:
Once the above steps were taken I was able to install the Python package
confluent-kafka
on Python 3.8.10.How to reproduce
Versions:
Checklist
Please provide the following information:
confluent_kafka.version()
andconfluent_kafka.libversion()
):{...}
'debug': '..'
as necessary)Log:
The text was updated successfully, but these errors were encountered: