HDFS tests running suspiciously flaky on Github actions

[orpiske]

More than once the HDFS tests have caused the github actions to hang and run for a very long time. This could be caused by docker pull limits, but also could be caused by some resource usage limits on Github itself - this test is heavy.

[orpiske]

Likely caused by a conflict with Kafka 2.7.0:

021-01-18 18:06:58,937 [tp1105063017-88] WARN org.eclipse.jetty.server.HttpChannel - null
java.lang.NoSuchMethodError: 'org.eclipse.jetty.util.Attributes org.eclipse.jetty.util.Attributes.unwrap(org.eclipse.jetty.util.Attributes)'
	at org.eclipse.jetty.server.Request.recycle(Request.java:1891) ~[jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
	at org.eclipse.jetty.server.HttpChannel.recycle(HttpChannel.java:309) [jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
	at org.eclipse.jetty.server.HttpChannelOverHttp.recycle(HttpChannelOverHttp.java:81) [jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
	at org.eclipse.jetty.server.HttpConnection.onCompleted(HttpConnection.java:436) ~[jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
	at org.eclipse.jetty.server.HttpChannel.onCompleted(HttpChannel.java:743) ~[jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:363) [jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) [jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [jetty-io-9.4.33.v20201020.jar:9.4.33.v20201020]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [jetty-io-9.4.33.v20201020.jar:9.4.33.v20201020]
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [jetty-io-9.4.33.v20201020.jar:9.4.33.v20201020]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) [jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) [jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) [jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) [jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:367) [jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782) [jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918) [jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813]
	at java.lang.Thread.run(Thread.java:834) [?:?]
2021-01-18 18:06:58,937 [tp1105063017-88] WARN org.eclipse.jetty.util.thread.strategy.EatWhatYouKill - 
java.lang.IllegalStateException: s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0
	at org.eclipse.jetty.server.HttpChannelState.onError(HttpChannelState.java:763) ~[jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
	at org.eclipse.jetty.server.HttpChannel.handleException(HttpChannel.java:606) ~[jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
	at org.eclipse.jetty.server.HttpChannelOverHttp.handleException(HttpChannelOverHttp.java:488) ~[jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:535) ~[jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) ~[jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.33.v20201020.jar:9.4.33.v20201020]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.33.v20201020.jar:9.4.33.v20201020]
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[jetty-io-9.4.33.v20201020.jar:9.4.33.v20201020]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) [jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) [jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) [jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) [jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:367) [jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782) [jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918) [jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813]
	at java.lang.Thread.run(Thread.java:834) [?:?]

[oscerd]

https://github.com/apache/kafka/blob/2.7/gradle/dependencies.gradle#L72

while hadoop-common is still on 9.4.20

https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common/3.3.0

Probably we should try to exclude the jetty deps coming with camel-hdfs and see what happens with 9.4.33

[orpiske]

hadoop-common

Yeah, I think that's the way. It's a bit of a mine field, but we are already doing the same for jersey-core and jersey-server.

[orpiske]

Fixed the test code. The connector seems fine, so not change needed there at the moment.