CloudEvents with Avro payloads, on Kafka #238
Description
My team is looking at sending Avro-encoded cloud events over Kafka, often to or from Apache Camel-K. We’ve done some prototyping in which we’ve had to combine the Cloud Events Java SDK with the io.confluent.kafka Serde library. This leaves message receivers having to choose between the CloudEvent and Confluent deserializers since the former unpacks the headers nicely but produces an encoded payload, and the latter decodes the payload but leaves you to manage the headers yourself.
This is different from the Avro format for CloudEvents which describes a CloudEvent in Avro - we want to do the opposite, sending a binary, Avro-encoded payload with headers as per the Kafka Protocol Binding for CloudEvents.
In this Slack thread I wondered,
(For example, one could pass
io.confluent.kafka.serializers.KafkaAvroDeserializertoio.cloudevents.kafka.CloudEventDeserializeras something to delegate body-deserialization to, perhaps.)
@slinkydeveloper stated,
That sounds like an interesting problem, there is no official answer from the sdk to the problem "body decoding". This was left out from the sdk design by purpose, in order to avoid making complex abstractions on top of marshallers/unmarshallers and/or opinionated choices. [...] Your solution of "delegation" for the kafka deserializer looks like an interesting choice. However, It solves the particular problem only for kafka: I would love to find a solution that can be applied in all the other reader/writers.
He also mentioned Issue 196 as being somewhat related to this question.
I would like to provide a nice way of handling messages with as much CloudEvents, schema management and de/serialization support built in as possible. Currently our prototype code has two layers: one using the CloudEvent APIs for handling the envelope then a second using Confluent APIs for handling the payload. Can anyone see a better way of doing this? Many thanks in advance for any suggestions!