How install camel-jira-kafka-connector

[tstuber]

Hi guys

I am struggling with camel-jira-kafka-connector. My goal is use it as a source connector for new comments. I followed the provided documentation and took the configuration base from the projects example.

However, I always run into the following error (Cannot find endpoint with scheme jira):

[2020-07-07 00:12:11,684] ERROR WorkerSourceTask{id=CamelJiraSourceConnector-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:186)
org.apache.kafka.connect.errors.ConnectException: Failed to create and start Camel context
        at org.apache.camel.kafkaconnector.CamelSourceTask.start(CamelSourceTask.java:103)
        at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:213)
        at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)
        at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalArgumentException: Cannot find endpoint with scheme jira
        at org.apache.camel.catalog.impl.AbstractCamelCatalog.doAsEndpointUri(AbstractCamelCatalog.java:684)
        at org.apache.camel.catalog.impl.AbstractCamelCatalog.asEndpointUri(AbstractCamelCatalog.java:673)
        at org.apache.camel.kafkaconnector.utils.TaskHelper.buildUrl(TaskHelper.java:44)
        at org.apache.camel.kafkaconnector.CamelSourceTask.start(CamelSourceTask.java:89)
        ... 8 more

I used the following configuration:

name=CamelJiraSourceConnector
connector.class=org.apache.camel.kafkaconnector.jira.CamelJiraSourceConnector
tasks.max=1

key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter

camel.component.jira.jiraUrl=http://localhost:8080
camel.component.jira.password=xxx
camel.component.jira.username=user
camel.source.endpoint.jiraUrl=http://localhost:8080
camel.source.endpoint.username=user
camel.source.endpoint.password=xxx

camel.source.path.type=newComments
topics=mytopic

I specified the JIRA endpoint according the documentation. I also tried Jira online, where I received the same result. Also, I tried OAuth - same result. What am I doing wrong?

It is also not clear to me why there are configurations for components and source.endpoints. What is the difference?

Thanks for your support

[oscerd]

Did you unzip the zip package in your kafka connect plugin path?

[tstuber]

Yes, it is unpacked.

[oscerd]

Let me try

[oscerd]

For me this is working fine.

[2020-07-07 11:03:21,543] INFO WorkerSourceTask{id=CamelJiraSourceConnector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:416) [2020-07-07 11:03:21,543] INFO WorkerSourceTask{id=CamelJiraSourceConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:433) [2020-07-07 11:03:21,543] ERROR WorkerSourceTask{id=CamelJiraSourceConnector-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:179) org.apache.kafka.connect.errors.ConnectException: Failed to create and start Camel context at org.apache.camel.kafkaconnector.CamelSourceTask.start(CamelSourceTask.java:103) at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:208) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: java.net.ConnectException: Connection refused at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.lambda$doExecute$4(ApacheAsyncHttpClient.java:345) at io.atlassian.util.concurrent.Promises.lambda$biFunction$7(Promises.java:424) at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836) at java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:848) at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2168) at io.atlassian.util.concurrent.Promises$OfStage.lambda$newPromise$5(Promises.java:369) at io.atlassian.util.concurrent.Promises$OfStage.fold(Promises.java:337) at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.doExecute(ApacheAsyncHttpClient.java:339) at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.execute(ApacheAsyncHttpClient.java:292) at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.execute(DefaultRequest.java:258) at org.apache.camel.component.jira.oauth.OAuthHttpClientDecorator$OAuthAuthenticatedRequestBuilder.execute(OAuthHttpClientDecorator.java:115) at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.get(DefaultRequest.java:221) at com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient.getAndParse(AbstractAsynchronousRestClient.java:69) at com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJqlImplGet(AsynchronousSearchRestClient.java:109) at com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJql(AsynchronousSearchRestClient.java:94) at org.apache.camel.component.jira.consumer.AbstractJiraConsumer.getIssues(AbstractJiraConsumer.java:55) at org.apache.camel.component.jira.consumer.AbstractJiraConsumer.getIssues(AbstractJiraConsumer.java:45) at org.apache.camel.component.jira.consumer.NewCommentsConsumer.getComments(NewCommentsConsumer.java:74) at org.apache.camel.component.jira.consumer.NewCommentsConsumer.doStart(NewCommentsConsumer.java:66) at org.apache.camel.support.service.BaseService.start(BaseService.java:115) at org.apache.camel.impl.engine.AbstractCamelContext.startService(AbstractCamelContext.java:3075) at org.apache.camel.impl.engine.AbstractCamelContext.doStartOrResumeRouteConsumers(AbstractCamelContext.java:3413) at org.apache.camel.impl.engine.AbstractCamelContext.doStartRouteConsumers(AbstractCamelContext.java:3344) at org.apache.camel.impl.engine.AbstractCamelContext.safelyStartRouteServices(AbstractCamelContext.java:3249) at org.apache.camel.impl.engine.AbstractCamelContext.doStartOrResumeRoutes(AbstractCamelContext.java:3011) at org.apache.camel.impl.engine.AbstractCamelContext.doStartCamel(AbstractCamelContext.java:2809) at org.apache.camel.impl.engine.AbstractCamelContext.doStartContext(AbstractCamelContext.java:2662) at org.apache.camel.impl.engine.AbstractCamelContext.doStart(AbstractCamelContext.java:2625) at org.apache.camel.support.service.BaseService.start(BaseService.java:115) at org.apache.camel.impl.engine.AbstractCamelContext.start(AbstractCamelContext.java:2467) at org.apache.camel.main.Main.doStart(Main.java:118) at org.apache.camel.support.service.BaseService.start(BaseService.java:115) at org.apache.camel.main.MainSupport.run(MainSupport.java:83) at org.apache.camel.kafkaconnector.utils.CamelMainSupport$CamelContextStarter.run(CamelMainSupport.java:234) ... 3 more Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714) at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174) at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ... 1 more

It fails because there is no JIRA there in my local env, but the scheme has been found.

How did you define the plugin.path property and where did you unpack the connector?

[tstuber]

I had unpacked plugins as well as the camel-kafka-connector project in /home/user/connectors.

-rw-r--r-- 1 user user     1637 Jul  7 00:12 CamelJiraSourceConnector.properties
-rw-r--r-- 1 user user     1119 Jul  6 21:54 CamelSinkConnector.properties
drwxr-xr-x 1 user user     4096 Jul  6 20:51 camel-jira-kafka-connector/
-rw-r--r-- 1 user user 14884835 Jul  6 21:50 camel-jira-kafka-connector-0.4.0-SNAPSHOT-package.zip
drwxr-xr-x 1 user user     4096 Jul  6 20:07 camel-kafka-connector/         
drwxr-xr-x 1 user user     4096 Jul  6 20:57 camel-log-kafka-connector/
-rw-r--r-- 1 user user  3759902 Jul  6 21:50 camel-log-kafka-connector-0.4.0-SNAPSHOT-package.zip

I moved the unpacked folders another, empty plugin location and pointed the plugin-dir to that path. Now it works. Thanks a lot :-)

The only question left is about the configuration of that connector. Why there are similar/identical configurations for components and source.endpoints?

[oscerd]

In Camel when you set something at component level you share that among all the endpoints the component will create, while setting something on the endpoint means it will be set only on that specific endpoint. Some of the components in Camel have both the parameters at component and endpoint level. So you should specify only at component or endpoint level.

Also you don't need camel-kafka-connector directory in your plugin path, the zipped content of a connector will be enough.

[tstuber]

Thanks a lot @oscerd