Devservices Keycloak breaks integration tests

[constantin-ungureanu-github]

Describe the bug

With the quarkus-oidc in place, the restriction to the REST API are added.
The @QuarkusTest run okay. all of them.
The @QuarkusIntegrationTest do no run okay.

So adding quarkus-oidc to a quarkus project, adding few REST API restrictions (as in security-keycloak-authorization-quickstart demo), and @QuarkusTest runs fine, both with/withouth oidc restrictions, also healthchecks.

Same tests run as @quarkusIntegrationTest fail. (build native but is not related to the build as I understand)
The difference observed is that --hostname-port is sent when running the integration tests. Please remove this because was deprecated (25) and with keycloak 26+ was removed and now the keycloak is failing to start.

Workaround found for this was to:
quarkus.keycloak.devservices.service-name=keycloak
quarkus.keycloak.devservices.start-command=start --http-enabled=true --hostname=keycloak --hostname-strict=false --spi-user-profile-declarative-user-profile-config-file=/opt/keycloak/upconfig.json --features=hostname:v1

However, this works only with quarkus less than 3.18 (CR1), and it fails still in case of healthchecks (integration tests only). Sure, the keyloack can be downgraded but that still doesn't completely fix the issues.
For keycloak the v1 is replaced with v2 (--features=hostname:v2) . However, that does not work anymore, container fails to start.
The 3.17.7 with the workaround fails for the API that don't include oidc restrictions and also for the healthcheck IT tests.

This is what I get from the keycloak container (when running IT tests that fail):

Changes detected in configuration. Updating the server image.
Updating the configuration and installing your custom providers, if any. Please wait.
2025-01-22 12:56:41,951 INFO [io.qua.dep.QuarkusAugmentor] (main) Quarkus augmentation completed in 6640ms
Server configuration updated and persisted. Run the following command to review the configuration:

    kc.sh show-config

Next time you run the server, just run:

� kc.sh start --http-enabled=true --hostname-debug=true --hostname=localhost --spi-user-profile-declarative-user-profile-config-file=/opt/keycloak/upconfig.json --hostname-port=53483 --optimized

Unknown option: '--hostname-port'
Possible solutions: --hostname, --hostname-admin, --hostname-backchannel-dynamic, --hostname-strict, --hostname-debug
Try 'kc.sh start --help' for more information on the available options.

Expected behavior

All test run nicely.

Actual behavior

Integration tests fails

• in 3.18 CR1, no integration test runs and because you updated the Keycloak to verison 26.0.0.7 there's no workaround but to downgrade this version
• in version less than 3.18 (ex. 3.17.7) integration tests are failing as well, but for some there are ways to mitigate, see in the description.

I see there is a difference that the --hostname-port option has been removed

How to Reproduce?

Run integration tests involving devservices keycloak and this pops up immediately.

Also test this when endpoints are healtchecks and there are no restrictions and when the management port is used (with healthchecks on that port).

Output of uname -a or ver

No response

Output of java -version

java 21

Quarkus version or git rev

3.18 CR1 and also lower (3.17)

Build tool (ie. output of mvnw --version or gradlew --version)

maven 3.9.9

Additional information

I see for the version 3.18 there were numerous improvements related to OIDC, so would be nice to have the fix for quarkus-oidc running IT tests in 3.18. If needed, more details can be provided, but if you just run IT tests with OIDC this should pop-up.

[quarkus-bot]

/cc @geoand (devservices), @pedroigor (keycloak), @sberyozkin (keycloak), @stuartwdouglas (devservices)

[sberyozkin]

@constantin-ungureanu-github Can you please create a basic reproducer ? There are a few integration tests running in Quarkus with the Keycloak devservices

[constantin-ungureanu-github]

https://github.com/constantin-ungureanu-github/quarkus-oidc-example

Let me know if you need anything more.

[sberyozkin]

Thanks @constantin-ungureanu-github, that is helpful, it is a shared container case. #42509 related to it but as it happens, #42509 is currently blocked, hopefully not for too long.

[constantin-ungureanu-github]

I wonder if the shared container is the root cause. It fails regardless if it is shared or not (quarkus.keycloak.devservices.shared=true/false).

The cause seems to be that when running IT tests, it sets the '--hostname-port' that with Keycloak version 26+ is no longer available (has been removed completely), and that Keycloak container fails to start.

However, a more serious issue, that it seems it is a different problem, is that the management port and quarkus-oidc don't work together. This is not regarding only the tests, but the resulting container fails to start in a Kubernetes environment where the health probes are used. Also tested to directly access the health checks when the management port is used, they just don't work anymore. The sample code provided is built in this way and can be verified.

Please have these issues fixed. Let me know if you need more details.

[constantin-ungureanu-github]

I made some updates to the repository, to add running capabilities. Just run docker compose up or podman compose up and the container should pop-up, alongside a preconfigured Keycloak.

I found the root cause for the management port not working, that is only when "quarkus.http.non-application-root-path=/". The default is /q and I wanted to remove that altogether. Commenting out this change and I have back the metrics and the health checks.
One notice is that if I put the "quarkus.http.non-application-root-path=/", the image size is much smaller than when I take this out.
So the workaround to bypass this issue is to just not touch the quarkus.http.non-application-root-path altogether.

Now the "quarkus.http.non-application-root-path=/" works fine without using quarkus-oidc, so the issue resides in the combination of using all those in this configuration.

Please fix all these issues.

[jedla97]

Hi @constantin-ungureanu-github I look at your reproducer

For Keycloak issue I come up with some solution. Will discribe them later in the text.

For liveness check. Why test are not passing is because the quarkus test framework not exposing the management port. As workaround you can use -Dquarkus.test.container.additional-exposed-ports."9000"=9000 as addition to your mvn command. I'll try to look at this weekend or next week and raise PR for exposing the management port when running docker integration tests.

When running your docker compose I'm able to see http://localhost:9000/q/health/live.
If you want it to be pointed to http://localhost:9000/health/live you need to set the quarkus.management.root-path=/.
From quarkus.http.non-application-root-path description

If the management interface is enabled, the root path for the endpoints exposed on the management interface is configured using the quarkus.management.root-path property instead of this property.

Now for Keycloak. We need to definitely remove " --hostname-port=" + fixedExposedPort.getAsInt() from Keycloak start command. Looking at migration guide I don't see any example how the --hostname-port be migrated

There are two option and I'm not sure which is correct so here is needed input of @sberyozkin as he have much more knowledge about how thing work with Quarkus. Also I didn't encounter the blocking issue as mentioned in #42509 as blocking issue seems only affecting windows not linux.

1st option is to replace " --hostname-port=" + fixedExposedPort.getAsInt() with " --hostname-backchannel-dynamic true" and change KEYCLOAK_QUARKUS_HOSTNAME from "localhost" to "http://localhost"

The keycloak command would be start --http-enabled=true --spi-user-profile-declarative-user-profile-config-file=/opt/keycloak/upconfig.json --hostname=http://localhost --hostname-backchannel-dynamic true

2nd option it replace KEYCLOAK_QUARKUS_HOSTNAME value from "localhost" to "http://" + hostName + ":" + KEYCLOAK_PORT this will expose the generated Keycloak hostname (http://keycloak-vu5vi:8080)

Maybe there is some other option. I come with these option when playing with it. Here is a command which is executed when the integration test are executed

Executing "docker run --name quarkus-integration-test-KXPdv -i --rm -p 8081:8081 -p 8444:8444 -p 9000:9000
--net=e227ae9ccbe1166628c9d9f786458e8c654d568492ea5721027b18caaed17c31 
--env QUARKUS_LOG_CATEGORY__IO_QUARKUS__LEVEL=INFO 
--env QUARKUS_HTTP_PORT=8081 
--env QUARKUS_HTTP_SSL_PORT=8444 
--env TEST_URL=http://localhost:8081 
--env QUARKUS_PROFILE=prod 
--env QUARKUS_OIDC_CLIENT_ID=quarkus-app 
--env CLIENT_QUARKUS_OIDC_AUTH_SERVER_URL=http://localhost:34667/realms/quarkus 
--env KEYCLOAK_URL=http://keycloak-vu5vi:8080 
--env QUARKUS_OIDC_AUTH_SERVER_URL=http://keycloak-vu5vi:8080/realms/quarkus 
--env KEYCLOAK_REALMS=quarkus 
--env QUARKUS_CONTAINER_IMAGE_BUILD=true 
--env QUARKUS_TEST_CONTAINER_ADDITIONAL_EXPOSED_PORTS_9000=9000 
--env QUARKUS_NATIVE_CONTAINER_BUILD=true 
--env QUARKUS_DEVSERVICES_TIMEOUT=360s 
--env OIDC_USERS=alice=alice,bob=bob 
--env QUARKUS_OIDC_APPLICATION_TYPE=service 
--env QUARKUS_OIDC_CREDENTIALS_SECRET=secret quarkus-oidc-example:1.0.0-SNAPSHOT"

I checked with 3.8 which didn't have hostname v2 by default and the http://keycloak-vu5vi:8080 is not available from host so probably proposed 2nd option is wrong as that url should not be accesible from host machine.

We cant just remove the --hostname-port and add it to the hostname like this addEnv(KEYCLOAK_QUARKUS_HOSTNAME, "http://localhost:" + fixedExposedPort.getAsInt()); it will throw this error:

2025-01-25 17:04:19,119 WARNING [io.qua.ope.run.exp.otl.sen.VertxGrpcSender] (vert.x-eventloop-thread-3) Failed to export TraceRequestMarshalers. The request could not be executed. Full error message: Connection refused: localhost/127.0.0.1:4317
2025-01-25 17:04:19,123 ERROR [io.qua.run.Application] (main) Failed to start application: java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
	at io.quarkus.runtime.Application.start(Application.java:101)
	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:121)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:77)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:48)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:137)
	at io.quarkus.runner.GeneratedMain.main(Unknown Source)
Caused by: io.smallrye.mutiny.CompositeException: Multiple exceptions caught:
	[Exception 0] io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:35271
	[Exception 1] io.quarkus.oidc.OIDCException
	at io.smallrye.mutiny.operators.uni.UniOnFailureFlatMap$UniOnFailureFlatMapProcessor.performInnerSubscription(UniOnFailureFlatMap.java:94)
	at io.smallrye.mutiny.operators.uni.UniOnFailureFlatMap$UniOnFailureFlatMapProcessor.dispatch(UniOnFailureFlatMap.java:83)
	at io.smallrye.mutiny.operators.uni.UniOnFailureFlatMap$UniOnFailureFlatMapProcessor.onFailure(UniOnFailureFlatMap.java:60)
	at io.smallrye.mutiny.operators.uni.UniOperatorProcessor.onFailure(UniOperatorProcessor.java:55)
	at io.smallrye.mutiny.operators.uni.UniOperatorProcessor.onFailure(UniOperatorProcessor.java:55)
	at io.smallrye.mutiny.operators.uni.UniOperatorProcessor.onFailure(UniOperatorProcessor.java:55)
	at io.smallrye.mutiny.operators.uni.UniOnFailureFlatMap$UniOnFailureFlatMapProcessor.dispatch(UniOnFailureFlatMap.java:85)
	at io.smallrye.mutiny.operators.uni.UniOnFailureFlatMap$UniOnFailureFlatMapProcessor.onFailure(UniOnFailureFlatMap.java:60)
	at io.smallrye.mutiny.operators.uni.UniOperatorProcessor.onFailure(UniOperatorProcessor.java:55)
	at io.smallrye.mutiny.vertx.AsyncResultUni.lambda$subscribe$1(AsyncResultUni.java:37)
	at io.smallrye.mutiny.vertx.DelegatingHandler.handle(DelegatingHandler.java:25)
	at io.vertx.ext.web.client.impl.HttpContext.handleFailure(HttpContext.java:398)
	at io.vertx.ext.web.client.impl.HttpContext.execute(HttpContext.java:387)
	at io.vertx.ext.web.client.impl.HttpContext.next(HttpContext.java:362)
	at io.vertx.ext.web.client.impl.HttpContext.fire(HttpContext.java:329)
	at io.vertx.ext.web.client.impl.HttpContext.fail(HttpContext.java:310)
	at io.vertx.ext.web.client.impl.HttpContext.lambda$handleCreateRequest$6(HttpContext.java:498)
	Suppressed: io.quarkus.oidc.OIDCException
		at io.quarkus.oidc.runtime.TenantContextFactory$2.apply(TenantContextFactory.java:135)
		at io.quarkus.oidc.runtime.TenantContextFactory$2.apply(TenantContextFactory.java:116)
		at io.smallrye.context.impl.wrappers.SlowContextualFunction.apply(SlowContextualFunction.java:21)
		at io.smallrye.mutiny.groups.UniOnFailure.lambda$recoverWithItem$8(UniOnFailure.java:190)
		at io.smallrye.mutiny.operators.uni.UniOnFailureFlatMap$UniOnFailureFlatMapProcessor.performInnerSubscription(UniOnFailureFlatMap.java:92)
		... 71 more
	Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:35271
	Caused by: java.net.ConnectException: Connection refused
		at java.base@21.0.5/sun.nio.ch.Net.pollConnect(Native Method)
		at java.base@21.0.5/sun.nio.ch.Net.pollConnectNow(Net.java:682)
		at java.base@21.0.5/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973)
		at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:336)
		at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339)
		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		at java.base@21.0.5/java.lang.Thread.runWith(Thread.java:1596)
		at java.base@21.0.5/java.lang.Thread.run(Thread.java:1583)
		at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:896)
		at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:872)
Caused by: [CIRCULAR REFERENCE: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:35271]

[constantin-ungureanu-github]

Hi @jedla97 ! I updated the repository https://github.com/constantin-ungureanu-github/quarkus-oidc-example as per your suggestions. Instead of using quarkus.http.non-application-root-path I use quarkus.management.root-path and it is indeed working as expected !!

For the original issue, of the integration tests failing due to usage of quarkus-oidc, you guys for sure know better.
It maybe helpful to know that keycloak has a discovery option that should be disabled, as can be seen in the docker-compose.yml file. Maybe you could set up the Keycloak devservice similarly, passing the proper configuration.

I hope to see this fixed soon. Thanks for your replies!

[sberyozkin]

Hi @jedla97, many thanks for your investigation...

What I do recall is that when mvn clean install -Pnative -Dquarkus.native.container-build=true -Dquarkus.container-image.build=true command is used, the shared network is required (sorry, I mistakenly referred to it as a shared container property as I suggested to @constantin-ungureanu-github above), so it applies only to the case of the shared network, and your 2nd option looks fine to me.

Awhile back we had a dedicated integration test, created by @geoand, I was periodically doing some minor tweaks to get it passing, and then later it was removed due to some cleanup, and unfortunately I never got to restoring it, as far as I recall @iocanel was suggesting we did not need to have an image generated to test this shared network case...

Would you like to have a look at restoring that test in the next couple of weeks and CC @geoand, @iocanel and myself whenever you have it working ?

[jedla97]

Hi @sberyozkin I can look at restoring them. Thanks for the links!

For this issue are we want to provide the fix now so we have at least working solution (maybe not for all case but at least fix the startup of Keycloak) or are we want fixt it with test?
From my POV the fix it now make sense as we have already 2 issues and when the restored test found issue it can modified.

[iocanel]

I don't think that the removed tests are related this issue. The removed tests were maven invoker tests, which pretty much tested the image building.

If the removed modules included Quarkus integration tests that we need, then we could propably restore them, but please not as invoker tests.