Provide a catalog of potential configuration properties specific to Camel Kafka Connector

[apupier]

additionally to Camel main catalog https://github.com/apache/camel/blob/camel-3.5.0/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/main/camel-main-configuration-metadata.json , there are specific Camel Kafka Connector configuration properties, for instance camel.source.* and camel.sink.* (are there others?)

providing a catalog would allow to have completion and hover provided by the Camel Language Server, see FUSETOOLS2-626

nota: a second step could be to provide validation too

[orpiske]

additionally to Camel main catalog https://github.com/apache/camel/blob/camel-3.5.0/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/main/camel-main-configuration-metadata.json , there are specific Camel Kafka Connector configuration properties, for instance camel.source.* and camel.sink.* (are there others?)

Sounds interesting! I think we would also need to provide the catalog for the Kafka Connect properties, right? Otherwise it would only complete the Camel-specific ones, but not the Kafka Connect ones we also need to use.

providing a catalog would allow to have completion and hover provided by the Camel Language Server, see FUSETOOLS2-626

nota: a second step could be to provide validation too

[oscerd]

In reality from the camel side we are just reusing the original Camel catalog and add the sink and source prefixes. So I don't know how much it makes sense to repackage another catalog just for adding the same set of information with prefixes in the name.

[apupier]

I think we would also need to provide the catalog for the Kafka Connect properties, right? Otherwise it would only complete the Camel-specific ones, but not the Kafka Connect ones we also need to use.

i would expect kafka tooling not Camel specific to provide it. But if it does not exists, we can also work on that.

[apupier]

In reality from the camel side we are just reusing the original Camel catalog and add the sink and source prefixes. So I don't know how much it makes sense to repackage another catalog just for adding the same set of information with prefixes in the name.

that would be easier.

For instance, from where maxPollDuration should be taken from?

camel-kafka-connector/examples/CamelAWSS3SourceConnector.properties

Line 25 in 15f7fb1

camel.source.maxPollDuration=10000

i don't find it in the Camel main catalog

isn't all properties here Camel Kafka Connector specific? https://github.com/apache/camel-kafka-connector/blob/81edd4b9a424fe4a06e6719e6d08e037c87c08c4/core/src/main/java/org/apache/camel/kafkaconnector/CamelSourceConnectorConfig.java

[oscerd]

There is a mixture of kafka-connect specific option, camel options and camel component/endpoint options.

The camel component/endpoint options are exactly the same of the camel main catalog plus prefixes. The maxPollDuration is a specific option of a polling consumer and it's named in that way in ckc. The options in sink and source connector (generic classes) are not generated, so what you see in CamelSourceConnectorConfig and CamelSinkConnectorConfig have been defined without taking into account the catalog content.

[apupier]

The camel component/endpoint options are exactly the same of the camel main catalog plus prefixes.

Can you precise which prefixes and points to an example?
Based on examples I found, it sounds like exactly the same than normal catalog, so writing the same than for Properties file for Camel main, for instance camel.component.amqp.includeAmqpAnnotations=true doesn't contain anything specific to Camel Kafka Connector.

what you see in CamelSourceConnectorConfig and CamelSinkConnectorConfig have been defined without taking into account the catalog content.

Do you mean that these configurations should be modified to reuse existing catalog content?
I thought it was specific Camel Kafka Connector properties, so having a catalog for what is defined inside these 2 classes would help providing completion.

[oscerd]

https://camel.apache.org/camel-kafka-connector/latest/connectors/camel-activemq-kafka-sink-connector.html

All the camel.sink.* stuff or camel.source.* stuff are camel catalog options renamed with sink or source in the name

For the second point, no, they don't need to be renamed. It was just to explain the only options not generated from catalog

[apupier]

https://camel.apache.org/camel-kafka-connector/latest/connectors/camel-activemq-kafka-sink-connector.html

All the camel.sink.* stuff or camel.source.* stuff are camel catalog options renamed with sink or source in the name

ok, just discovered for other options of camel.sink.* and camel.source.* (apart from camel.source.url and camel.sink.url).
It means that the potential properties depends on the value of the connector.class that has been provided? How to do the mapping between the connector.class value and the corresponding component in the normal Camel Catalog entry which is grouped by component id? Can camel.sink.url and using list of properties used at the same time?
(might be easier to continue this part of the conversation here https://issues.redhat.com/browse/FUSETOOLS2-670 )

For the second point, no, they don't need to be renamed. It was just to explain the only options not generated from catalog

If I understand well, these ones are really specific to Camel Kafka Connector AND cannot be derived from normal catalog. Providing the catalog for them will help.

[apupier]

I think the catalog will be useful for several cases:

• properties specific to Camel Kafka Connector, the ones that are in CamelSourceConnectorConfig and CamelSinkConnectorConfig. Can be leveraged then in Camel Language Server FUSETOOLS2-626
• provide the mapping between the connector.class and the Camel Connector id to be able to know which Camel Connector catalog to use for camel.sink.* and camel.source.* properties. Can be leveraged then in Camel Language Server [FUSETOOLS2-670]
• provide the available Converters. Can be leveraged then in Camel Language Server FUSETOOLS-686
• provide the available Transformers Can be leveraged then in Camel Language Server FUSETOOLS-687
• provide the available Aggregation Strategies Can be leveraged then in Camel Language Server FUSETOOLS-688
• have the list of supported Camel connector. As there are some "new" connectors marked in What's new 0.5.0, I'm guessing that there is not a 100% match with Camel main catalog.

I think that most of them can be implemented incrementally and separately.

[valdar]

@apupier sorry to jump this late in the conversation.

To summarize to see if I have understood correctly, you are proposing to create a new camel catalog specific for camel-kafka-connector containing all the information you mentioned?

As has been said that would be more or less a mapping of the original camel catalog with some prefixes for now... at the moment the other info you are listing (converters, transformers, aggregation strategies) are not present.

It remains the problem relative to the properties coming directly from kafka connect, did you investigate if there is a way to gathering those from somewhere in kafka?

[apupier]

To summarize to see if I have understood correctly, you are proposing to create a new camel catalog specific for camel-kafka-connector containing all the information you mentioned?

As has been said that would be more or less a mapping of the original camel catalog with some prefixes for now... at the moment the other info you are listing (converters, transformers, aggregation strategies) are not present.

not exactly, i'm proposing to create the Catalog for everything that is not coming from the original Camel catalog.
Currently, we cannot provide anything for Camel Kafka Connector (apart from harcoding it but I really don't think that itis a good idea to follow this path). Even for the part that can be "mapped from the original camel catalog", there is missing information in the Properties file to know which Camel connector id needs to be used. See https://issues.redhat.com/browse/FUSETOOLS2-670

It remains the problem relative to the properties coming directly from kafka connect, did you investigate if there is a way to gathering those from somewhere in kafka?

no. I'm awaiting feedback from Developer Tooling team that should come with plans for Kafka tooling in general. I hope it would include the kafka /kafka Connect properties (but I have absolutely no visibility so far on that, they just told me to wait).
This is a very separate problem than this issue anyway. EDIT: created https://issues.redhat.com/browse/FUSETOOLS2-699 to track it

[oscerd]

Personally I think repackaging a catalog could be an important improvement, but not critical currently. We have a lot of more crucial features to do. So we can leave this open and re-start the discussion once we have free cycle for this.

[janstey]

@oscerd @valdar would it be a big job to create this catalog that @apupier is requesting? I think we will need tooling for CKC sooner than later :-)

[oscerd]

It's for sure some work.