added Avro support for KafkaAdapter #645

mildek · 2025-12-09T12:41:15Z

No description provided.

Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

…rotocol Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

conda/dev-environment-unix.yml

cpp/cmake/modules/FindDepsKafkaAdapter.cmake

Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

The avro-cpp library added in this PR uses fmt, which requires the /utf-8 compiler flag on Windows MSVC to handle Unicode support properly. Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

…ndows. The avro-cpp library from conda-forge defines fmt::formatter<avro::Name> with a non-const format() method, but fmt v12 requires it to be const. This commit adds a const-correct version of the formatter that is used on Windows (MSVC) to prevent compilation errors. Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

The avro-cpp library is not yet compatible with fmt v12. The library defines fmt::formatter<avro::Name> with a non-const format() method, but fmt v12 requires it to be const. Homebrew and other package managers use fmt 11.2.0 with avro-cpp, so we pin fmt<12 in the Windows conda environment to avoid compilation errors on Windows CI. Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

vcpkg.json

Instead of pinning to an older fmt version (which creates version mismatch between platforms), we implement a workaround that provides a const-correct fmt::formatter<avro::Name> specialization on Windows. conda-forge's avro-cpp has an outdated formatter without const, while fmt v11+ requires const. We define the correct version first and suppress the redefinition warning, ensuring compatibility with latest fmt across all platforms. Also regenerated NOTICE file to include avro-cpp license. Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

conda-forge's avro-cpp defines fmt::formatter with non-const format() method, but fmt v11+ requires const. Create a centralized AvroIncludes.h wrapper that: 1. Temporarily sets FMT_VERSION to 80000 (below 90000 threshold) to prevent avro from defining its broken formatters 2. Includes all avro headers 3. Restores FMT_VERSION and defines const-correct formatters for both avro::Name and avro::Type This can be removed once conda-forge updates avro-cpp. Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

…ty on Windows Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

all good now

AdamGlustein · 2026-02-02T21:02:15Z

cpp/cmake/modules/FindAvro.cmake

+# conda-forge's avro-cpp has fmt::formatter specializations with non-const
+# format() methods, but fmt v12+ requires const. This causes MSVC error C2766.
+#
+# Solution: On Windows, we check if the avro headers have this bug, and if so,


Can you not just check for the bug by looking at what version of avro-cpp you are building against?

I am also not a fan of this patch, if the Windows version is too old to be compatible with the current env then I'd rather just skip it.

I still see us checking the code directly and not the avro-cpp version

Done. Earlier I wasn't 100% sure if we could reliably extract the version, but turns out we can. Also updated the vcpkg submodule to 1.12.1 which apparently has the fmt fix upstream, so we don't need the patch anymore.

Cool, so we can also get rid of the version logic in the FindAvro.cmake too then right.

AdamGlustein · 2026-02-02T21:08:25Z

cpp/csp/adapters/kafka/CMakeLists.txt


+# Link Avro library
+if(CSP_USE_VCPKG)
+  target_link_libraries(csp_kafka_adapter PUBLIC unofficial::avro-cpp::avrocpp)


The overall state of avro packaging (both in conda and vcpkg) really gives me hesitation on adding it as a build-time dependency of csp. I'd much prefer if we could split this out into it's own project, but not sure how feasible that is given our adapter ABI is still a WIP.

The Kafka adapter with Avro support is already fully optional (CSP_BUILD_KAFKA_ADAPTER=ON/OFF). With this PR, on Windows conda-forge builds where avro-cpp is incompatible, it automatically disables itself with a clear warning - no build failures, no patches, just graceful degradation. I agree the avro packaging situation is frustrating. Once the adapter ABI stabilizes, splitting this into a separate project would make sense. For now, keeping it optional within CSP seems like the pragmatic path - users who need Kafka/Avro can enable it, others aren't affected.

Yes, but I assume we are going to build CSP (for distribution) with Avro enabled, just like we build with Kafka enabled currently. So it's not optional from the perspective of the userm who installs pre-built csp from conda/pypi

should we wait then for ABI to stabilize before merging it?

We can merge it as an experimental feature and if avro becomes a nuisance in our build pipeline we can always re-evaluate/remove it. @timkpaine what do you think?

we will eventually support separate adapters but for now i think its better to support everything we support on every platform, and not do platform-specific stuff. There should be no incompatible platform as this has caused problems in the past and makes life annoying for users.

cpp/csp/adapters/utils/AvroMessageStructConverter.cpp

cpp/csp/adapters/utils/CMakeLists.txt

cpp/csp/adapters/utils/AvroMessageWriter.h

csp/tests/adapters/test_kafka.py

docs/wiki/api-references/Input-Output-Adapters-API.md

CMakeLists.txt

NOTICE

- Define CSP_AVRO_TARGET cmake variable to avoid repeating vcpkg checks - Remove braces from one-line loops (style) - Remove assert_values_equal helper - Avro preserves exact float values - Simplify FindAvro.cmake - disable Kafka adapter when avro-cpp has fmt bug - Make DepsKafkaAdapter optional in root CMakeLists.txt Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

- Skip Kafka adapter on Windows if avro-cpp < 1.12.1 - Update vcpkg submodule to 1.12.1 (has upstream fmt fix) - Rename CSP_AVRO_TARGET to AVRO_LIBRARIES Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

- Extract version from library soname on conda-forge - Skip Kafka adapter on Windows if avro-cpp < 1.12.1 or version unknown - Make Avro compilation conditional with CSP_USE_AVRO define - Add vcpkg.json override to use avro-cpp 1.12.1 without full submodule update - Rename CSP_AVRO_TARGET to AVRO_LIBRARIES for consistency Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

- Update vcpkg submodule to f9d8eecab9 which has avro-cpp 1.12.1 - Update vcpkg.json baseline to match - Update setup.py VCPKG_SHA to match - Add version check in FindDepsKafkaAdapter.cmake to gracefully skip Kafka adapter on Windows if avro-cpp < 1.12.1 Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

mildek requested review from AdamGlustein, alexddobkin, ptomecek, robambalu, svatasoiu and timkpaine as code owners December 9, 2025 12:41

added Avro support for KafkaAdapter

e3ea18f

Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

mildek force-pushed the feat/add-kafka-avro-protocol branch from b76ceb9 to e3ea18f Compare December 9, 2025 12:44

Merge remote-tracking branch 'origin/main' into feat/add-kafka-avro-p…

8ce67ec

…rotocol Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

mildek force-pushed the feat/add-kafka-avro-protocol branch from 1238242 to 8ce67ec Compare December 9, 2025 12:47

ruffed scripts

7396b22

Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

mildek force-pushed the feat/add-kafka-avro-protocol branch 2 times, most recently from 61090fb to e6af9f8 Compare December 11, 2025 09:48

conditionally skip avro installation for arm macos

aa8a87b

Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

mildek force-pushed the feat/add-kafka-avro-protocol branch from e6af9f8 to aa8a87b Compare December 11, 2025 09:49

timkpaine reviewed Dec 11, 2025

View reviewed changes

conda/dev-environment-unix.yml Show resolved Hide resolved

timkpaine reviewed Dec 11, 2025

View reviewed changes

cpp/cmake/modules/FindDepsKafkaAdapter.cmake Outdated Show resolved Hide resolved

mildek marked this pull request as draft December 11, 2025 15:23

fixed bugs, enabled osx-arm wheel

7eaf32a

Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

mildek force-pushed the feat/add-kafka-avro-protocol branch 2 times, most recently from ee61ade to 2b8dfca Compare January 14, 2026 13:46

fix: linted docs

1f4ab48

Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

mildek force-pushed the feat/add-kafka-avro-protocol branch from 2b8dfca to 1f4ab48 Compare January 14, 2026 13:50

mildek marked this pull request as ready for review January 14, 2026 14:45

mildek added 3 commits January 14, 2026 10:29

fix: add /utf-8 flag for Windows MSVC builds

26b6b99

The avro-cpp library added in this PR uses fmt, which requires the /utf-8 compiler flag on Windows MSVC to handle Unicode support properly. Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

timkpaine previously requested changes Jan 14, 2026

View reviewed changes

vcpkg.json Show resolved Hide resolved

mildek added 2 commits January 15, 2026 06:18

mildek force-pushed the feat/add-kafka-avro-protocol branch from 9632b13 to e9d65cb Compare January 15, 2026 11:52

fix: patch avro-cpp headers at configure time for fmt v12 compatibili…

acd75a1

…ty on Windows Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

mildek force-pushed the feat/add-kafka-avro-protocol branch from 29a3554 to acd75a1 Compare January 15, 2026 12:52

mildek requested a review from timkpaine January 15, 2026 13:16

timkpaine added type: enhancement Issues and PRs related to improvements to existing features adapter: kafka Issues and PRs related to the Apache Kafka adapter labels Jan 15, 2026

AdamGlustein requested changes Feb 2, 2026

View reviewed changes

mildek requested a review from AdamGlustein February 3, 2026 15:24

mildek added 2 commits February 5, 2026 07:20

- Extract version from library soname on conda-forge

ec0d6b7

- Skip Kafka adapter on Windows if avro-cpp < 1.12.1 - Update vcpkg submodule to 1.12.1 (has upstream fmt fix) - Rename CSP_AVRO_TARGET to AVRO_LIBRARIES Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

updated vcpkg sha in setup.py

d2e8cf7

Signed-off-by: Krzysztof Milde <Krzysztof.Milde@Point72.com>

mildek force-pushed the feat/add-kafka-avro-protocol branch 3 times, most recently from 9a57ddb to e4fc53c Compare February 5, 2026 14:09

mildek force-pushed the feat/add-kafka-avro-protocol branch 5 times, most recently from 376e271 to b9a26a8 Compare February 5, 2026 16:45

mildek force-pushed the feat/add-kafka-avro-protocol branch from b9a26a8 to 3ece184 Compare February 5, 2026 17:01

added Avro support for KafkaAdapter #645

Are you sure you want to change the base?

added Avro support for KafkaAdapter #645

Uh oh!

Conversation

mildek commented Dec 9, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants