Define how to package the connectors

[oscerd]

We should do a zip for each connector, we can take as example this https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/1.1.0.Final/debezium-connector-postgres-1.1.0.Final-plugin.tar.gz

Lets collect ideas here.

ref: https://issues.redhat.com/browse/DBZ-1656

[omarsmak]

+1. I think each connector should independent as much as possible

[gunnarmorling]

👍 In Debezium, we create both, *.zip and *.tar.gz, so people can choose whichever they prefer. They are self-contained and can be extracted into the KC plug-in path.

[orpiske]

+1 for creating separate zip and tar.gz files.

I think it would be great if we could package them in a way that we can just drop the connector code into the plugin path and the connector dependencies in whatever is loaded into the KC classpath. I'm thinking about making it easier - for the user - to avoid potential class path issues and that annoying DelegatingClassLoader error.

[jpechane]

@orpiske +1
In fact it would be doable even now just with Maven assembly plug-in as all deps could be downloaded automatically.

The only issue I see now is that the dependencies are not exactly correct.

We have an example here - https://github.com/debezium/debezium-examples/tree/master/camel-kafka-connect it is a demo of Debezium to CoAP server. The connector distro is build using https://github.com/debezium/debezium-examples/tree/master/camel-kafka-connect/coap-connector-dist

If you look at the deps list

mvn dependency:tree
[INFO] Scanning for projects...
[INFO] 
[INFO] -----------< io.debezium.examples.camel:coap-connector-dist >-----------
[INFO] Building CoAP Server 1.0.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ coap-connector-dist ---
[INFO] io.debezium.examples.camel:coap-connector-dist:jar:1.0.0-SNAPSHOT
[INFO] \- org.apache.camel.kafkaconnector:camel-coap-kafka-connector:jar:0.0.1-SNAPSHOT:compile
[INFO]    +- org.apache.camel:camel-coap:jar:3.0.1:compile
[INFO]    |  +- org.apache.camel:camel-support:jar:3.0.1:compile
[INFO]    |  |  +- org.apache.camel:camel-api:jar:3.0.1:compile
[INFO]    |  |  +- org.apache.camel:camel-management-api:jar:3.0.1:compile
[INFO]    |  |  +- org.apache.camel:camel-util:jar:3.0.1:compile
[INFO]    |  |  +- org.apache.camel:camel-util-json:jar:3.0.1:compile
[INFO]    |  |  \- org.slf4j:slf4j-api:jar:1.7.29:compile
[INFO]    |  +- org.eclipse.californium:californium-core:jar:2.0.0-M16:compile
[INFO]    |  |  +- org.eclipse.californium:californium-legal:jar:2.0.0-M16:runtime
[INFO]    |  |  +- org.eclipse.californium:element-connector:jar:2.0.0-M16:compile
[INFO]    |  |  \- org.eclipse.californium:element-connector:test-jar:tests:2.0.0-M16:compile
[INFO]    |  +- org.eclipse.californium:scandium:jar:2.0.0-M16:compile
[INFO]    |  \- org.eclipse.californium:element-connector-tcp:jar:2.0.0-M16:compile
[INFO]    |     +- io.netty:netty-handler:jar:4.1.25.Final:compile
[INFO]    |     |  +- io.netty:netty-buffer:jar:4.1.25.Final:compile
[INFO]    |     |  |  \- io.netty:netty-common:jar:4.1.25.Final:compile
[INFO]    |     |  \- io.netty:netty-codec:jar:4.1.25.Final:compile
[INFO]    |     \- io.netty:netty-transport:jar:4.1.25.Final:compile
[INFO]    |        \- io.netty:netty-resolver:jar:4.1.25.Final:compile
[INFO]    \- org.apache.camel.kafkaconnector:camel-kafka-connector:jar:0.0.1-SNAPSHOT:compile
[INFO]       +- org.apache.camel:camel-core:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-core-engine:jar:3.0.1:compile
[INFO]       |  |  +- org.apache.camel:spi-annotations:jar:3.0.1:compile
[INFO]       |  |  +- org.apache.camel:camel-base:jar:3.0.1:compile
[INFO]       |  |  \- org.apache.camel:camel-jaxp:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-bean:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-browse:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-caffeine-lrucache:jar:3.0.1:compile
[INFO]       |  |  \- com.github.ben-manes.caffeine:caffeine:jar:2.8.0:compile
[INFO]       |  +- org.apache.camel:camel-controlbus:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-dataformat:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-dataset:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-direct:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-directvm:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-file:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-language:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-log:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-mock:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-ref:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-rest:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-saga:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-scheduler:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-seda:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-stub:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-timer:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-validator:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-vm:jar:3.0.1:compile
[INFO]       |  +- org.apache.camel:camel-xpath:jar:3.0.1:compile
[INFO]       |  \- org.apache.camel:camel-xslt:jar:3.0.1:compile
[INFO]       +- org.apache.camel:camel-main:jar:3.0.1:compile
[INFO]       \- org.apache.camel:camel-elasticsearch-rest:jar:3.0.1:compile
[INFO]          +- org.elasticsearch.client:elasticsearch-rest-high-level-client:jar:7.3.2:compile
[INFO]          |  +- org.elasticsearch:elasticsearch:jar:7.3.2:compile
[INFO]          |  |  +- org.elasticsearch:elasticsearch-core:jar:7.3.2:compile
[INFO]          |  |  +- org.elasticsearch:elasticsearch-secure-sm:jar:7.3.2:compile
[INFO]          |  |  +- org.elasticsearch:elasticsearch-x-content:jar:7.3.2:compile
[INFO]          |  |  |  +- org.yaml:snakeyaml:jar:1.17:compile
[INFO]          |  |  |  +- com.fasterxml.jackson.dataformat:jackson-dataformat-smile:jar:2.8.11:compile
[INFO]          |  |  |  +- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.8.11:compile
[INFO]          |  |  |  \- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.8.11:compile
[INFO]          |  |  +- org.elasticsearch:elasticsearch-geo:jar:7.3.2:compile
[INFO]          |  |  +- org.apache.lucene:lucene-core:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-analyzers-common:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-backward-codecs:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-grouping:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-highlighter:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-join:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-memory:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-misc:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-queries:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-queryparser:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-sandbox:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-spatial:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-spatial-extras:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-spatial3d:jar:8.1.0:compile
[INFO]          |  |  +- org.apache.lucene:lucene-suggest:jar:8.1.0:compile
[INFO]          |  |  +- org.elasticsearch:elasticsearch-cli:jar:7.3.2:compile
[INFO]          |  |  |  \- net.sf.jopt-simple:jopt-simple:jar:5.0.2:compile
[INFO]          |  |  +- com.carrotsearch:hppc:jar:0.8.1:compile
[INFO]          |  |  +- joda-time:joda-time:jar:2.10.2:compile
[INFO]          |  |  +- com.tdunning:t-digest:jar:3.2:compile
[INFO]          |  |  +- org.hdrhistogram:HdrHistogram:jar:2.1.9:compile
[INFO]          |  |  +- org.apache.logging.log4j:log4j-api:jar:2.11.1:compile
[INFO]          |  |  \- org.elasticsearch:jna:jar:4.5.1:compile
[INFO]          |  +- org.elasticsearch.client:elasticsearch-rest-client:jar:7.3.2:compile
[INFO]          |  |  +- org.apache.httpcomponents:httpasyncclient:jar:4.1.4:compile
[INFO]          |  |  \- org.apache.httpcomponents:httpcore-nio:jar:4.4.11:compile
[INFO]          |  +- org.elasticsearch.plugin:parent-join-client:jar:7.3.2:compile
[INFO]          |  +- org.elasticsearch.plugin:aggs-matrix-stats-client:jar:7.3.2:compile
[INFO]          |  +- org.elasticsearch.plugin:rank-eval-client:jar:7.3.2:compile
[INFO]          |  \- org.elasticsearch.plugin:lang-mustache-client:jar:7.3.2:compile
[INFO]          |     \- com.github.spullara.mustache.java:compiler:jar:0.9.3:compile
[INFO]          +- org.elasticsearch.client:elasticsearch-rest-client-sniffer:jar:7.3.2:compile
[INFO]          |  +- org.apache.httpcomponents:httpclient:jar:4.5.8:compile
[INFO]          |  +- org.apache.httpcomponents:httpcore:jar:4.4.11:compile
[INFO]          |  +- commons-codec:commons-codec:jar:1.11:compile
[INFO]          |  +- commons-logging:commons-logging:jar:1.1.3:compile
[INFO]          |  \- com.fasterxml.jackson.core:jackson-core:jar:2.8.11:compile
[INFO]          \- com.fasterxml.jackson.core:jackson-databind:jar:2.10.1:compile
[INFO]             \- com.fasterxml.jackson.core:jackson-annotations:jar:2.10.1:compile
[INFO] ------------------------------------------------------------------------

there are two majo issues - ES introduced via `org.apache.camel.kafkaconnector:camel-kafka-connector` and the dependency on `org.apache.camel:camel-core`, maybe `org.apache.camel:camel-core-engine` would be enough or many of the compoenents that are dependency of the `camel-core` should be excluded.
When done you'd have a nicely trimmed per-connector installation package.

[oscerd]

The elasticsearch stuff is part of the camel component, it cannot be excluded directly, if you exclude elasticsearch, you'll need to declare it explicitly, with the exact same version used by the component.

[oscerd]

ah, no, it's in all connectors. This must be fixed.

[oscerd]

camel-elasticsearch-rest is not there anymore.