Quarkus Maven Plugin behaviour in Gitlab CI - Gitlab CI breaks the Quarkus Maven Plugin?

[markusjnagel]

Hi all,

I have searched quite a bit, but haven't found an answer, so I have to ask here.

TL;DR: Gitlab CI / gitlab runner breaks the quarkus maven plugin... how can that be?

Here's the situation - we're building (trying to build) an application using GitLab CI with the gitlab runner on Kubernetes (OpenShift 4.8).
The code builds and packages fine locally, but when running via a gitlab runner, it gives the weirdest errors that might be related to how the container is spun up.

The .gitlab-ci.yml of the relevant step looks like this - and the same step works for a SpringBoot application

build:
  image: registry.access.redhat.com/ubi8/openjdk-11:1.11-2
  needs:
    - environment
  artifacts:
    reports:
      dotenv: $VARIABLES_FILE
    paths:
      - target/site/jacoco/index.html
      - $APP_NAME.tar
    expire_in: 30 days
  # tags:
  #   - openshift-dev
  script:
    - echo APP_VERSION=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout) >> $VARIABLES_FILE
    - ./mvnw package -B -e --settings .m2/settings.xml

So, it spins up a container with image: registry.access.redhat.com/ubi8/openjdk-11:1.11-2 and runs ./mvnw package -B -e --settings .m2/settings.xml inside it.
And yes, when I say it builds (and runs) locally, I mean including the maven settings from our repo (--settings .m2/settings.xml), not the local user maven settings.

The weird first error we hit was

[ERROR] Failed to execute goal io.quarkus.platform:quarkus-maven-plugin:2.7.1.Final:build (default) on project signingstatusupdate: Failed to build quarkus application: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[ERROR] 	[error]: Build step io.quarkus.deployment.steps.BannerProcessor#recordBanner threw an exception: java.lang.StringIndexOutOfBoundsException: begin 0, end -1, length 64
[ERROR] 	at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3319)
[ERROR] 	at java.base/java.lang.String.substring(String.java:1874)
[ERROR] 	at io.quarkus.deployment.steps.BannerProcessor.isQuarkusCoreBanner(BannerProcessor.java:112)
[ERROR] 	at io.quarkus.deployment.steps.BannerProcessor.getBanner(BannerProcessor.java:93)

and trying to switch off the banner (why would it complain about this in the first place) via quarkus.banner.enabled=false didn't solve it.
After we created an arbitrary default_banner.txt in the classpath, we passed this one.......... weird, huh? Shouldn't be necessary and I've never seen anything like it....

Now we're hitting the next one:

[ERROR] Failed to execute goal io.quarkus.platform:quarkus-maven-plugin:2.7.1.Final:build (default) on project signingstatusupdate: Failed to build quarkus application: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[ERROR] 	[error]: Build step io.quarkus.vertx.http.deployment.StaticResourcesProcessor#collectStaticResources threw an exception: java.lang.IllegalArgumentException: URI has a query component
[ERROR] 	at java.base/sun.nio.fs.UnixUriUtils.fromUri(UnixUriUtils.java:57)
[ERROR] 	at java.base/sun.nio.fs.UnixFileSystemProvider.getPath(UnixFileSystemProvider.java:103)
[ERROR] 	at java.base/java.nio.file.Path.of(Path.java:203)
[ERROR] 	at java.base/java.nio.file.Paths.get(Paths.java:97)
[ERROR] 	at io.quarkus.runtime.util.ClassPathUtils.toLocalPath(ClassPathUtils.java:207)

If you look at the stacktraces, it seems to be somehow file access related...

• first it wouldn't find its default banner
• then it seems it can't get some path

We're a bit at a loss here - one of our colleagues also tried to run the maven command in a container created from this image and there it built as it should....
We also tried the registry.redhat.io/openjdk/openjdk-11-rhel7:1.11-1 instead of ubi8, but (as expected) that didn't change anything.

Anyone seen anything like this before?

Thanks,

Markus

[quarkus-bot[bot]]

/cc @geoand, @iocanel, @quarkusio/devtools

[cescoffier]

Can you check the permissions on the project directory and on /tmp

[ckavili]

Here is an output from ubi8/openjdk-11:1.11 image. GitLab CI creates a directory under /builds, clones the repo there and runs the build command in that directory.

Initialized empty Git repository in /builds/wxPwJyfK/0/group/subgroup/repo/.git/
 
$ id
uid=1000710000 gid=0(root) groups=0(root),1000710000

$ cat /etc/passwd
...
jboss:x:185:0:JBoss user:/home/jboss:/sbin/nologin
1000710000:x:1000710000:0:1000710000 user:/home/jboss:/sbin/nologin
...

sh-4.4$ env |grep -i home
JAVA_HOME=/usr/lib/jvm/java-11
NSS_WRAPPER_PASSWD=/home/jboss/passwd
HOME=/home/jboss
sh-4.4$ 

sh-4.4$ ls -altr
total 56
drwxrwsrwx. 4 1000710000 1000710000    60 Feb 14 21:35  ..
-rw-rw-rw-. 1 1000710000 1000710000   334 Feb 14 21:35  .gitignore
-rw-rw-rw-. 1 1000710000 1000710000    75 Feb 14 21:35  .dockerignore
-rw-rw-rw-. 1 1000710000 1000710000  1835 Feb 14 21:35  README.md
-rw-rw-rw-. 1 1000710000 1000710000    88 Feb 14 21:35  Dockerfile
drwxrwsrwx. 3 1000710000 1000710000    21 Feb 14 21:35  .mvn
drwxrwsrwx. 2 1000710000 1000710000    26 Feb 14 21:35  .m2
-rw-rw-rw-. 1 1000710000 1000710000  5654 Feb 14 21:35  .gitlab-ci.yml
-rw-rw-rw-. 1 1000710000 1000710000  6719 Feb 14 21:35  pom.xml
-rwxrwxrwx. 1 1000710000 1000710000  6734 Feb 14 21:35  mvnw.cmd
-rwxrwxrwx. 1 1000710000 1000710000 10284 Feb 14 21:35  mvnw
drwxrwsrwx. 3 1000710000 1000710000    60 Feb 14 21:35  chart
drwxrwsrwx. 4 1000710000 1000710000    30 Feb 14 21:35  src
drwxrwsrwx. 6 1000710000 1000710000   128 Feb 14 21:35  .git
-rw-r--r--. 1 1000710000 1000710000    18 Feb 14 21:36  variables.env
drwxrwsrwx. 3 1000710000 1000710000    17 Feb 14 22:24 '?' <-- this one seems strange as $HOME is set 🤔
drwxrwsrwx. 9 1000710000 1000710000   248 Feb 14 22:25  .
drwxrwsrwx. 8 1000710000 1000710000   175 Feb 14 22:25  target


sh-4.4$ ls -altr \?/.m2/repository/
total 8
drwxrwsrwx.  3 1000710000 1000710000   24 Feb 14 22:24 ..
drwxrwsrwx.  3 1000710000 1000710000   24 Feb 14 22:24 commons-io
drwxrwsrwx.  3 1000710000 1000710000   25 Feb 14 22:24 aopalliance
drwxrwsrwx.  3 1000710000 1000710000   25 Feb 14 22:24 commons-cli
drwxrwsrwx.  3 1000710000 1000710000   27 Feb 14 22:24 commons-codec
...


$ ls -ld /tmp
drwxrwxrwt. 1 root root 29 Jan 19 14:53 /tmp

When container starts with jboss user, it works fine but maybe there is something 1000710000 user can't get that causes said error.

[eformat]

i had a chat with @ckavili and i think this is environment/settings.xml related

i can build quarkus app ok using a gitlab runner. i installed the gitlab-runner operator into a 4.9 cluster, then configured the runner

oc -n ateam-ci-cd apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: gitlab-runner-secret
type: Opaque
stringData:
  runner-registration-token: <redacted>
EOF

oc -n ateam-ci-cd apply -f - <<EOF
apiVersion: apps.gitlab.com/v1beta2
kind: Runner
metadata:
  name: gitlab-runner
spec:
  gitlabUrl: https://gitlab-ce.apps.openshift-<redacted>.redhatlabs.dev
  buildImage: registry.access.redhat.com/ubi8/ubi:latest
  token: gitlab-runner-secret
EOF

my simple build runs OK against https://github.com/rht-labs/pet-battle-api

not i'm using the mvn script from the image (not mvnw) as well as not overriding the settings.xml

my pipeline:

build:
  stage: build
  image: registry.access.redhat.com/ubi8/openjdk-11:1.11-2
  artifacts:
    paths:
      - $APP_NAME.tar
  script:
    - mvn package

[stuartwdouglas]

So it looks like the maven repo is somehow ending up in the '?' directory, which will cause the problems as some parts of Quarkus choke on paths with a question mark.

[stuartwdouglas]

I can reproduce the 'URI has a query component' part by building on a path with a '?' in it.

[stuartwdouglas]

I think you could reproduce the banner part as well if if the maven repo (quarkus jars) were also in a path with a '?' in it.

[eformat]

nice one @stuartwdouglas - #23696
i debugged further. i think this happens because the gitlab runner fires up with a certain securityContext which end up as

$ cat /etc/passwd
1000690000:x:1000690000:0:1000690000 user:/home/jboss:/sbin/nologin

in the pod. when i run this in the runner pod, you can see home is "?"

sh-4.4$ echo $HOME; mvn clean -X|grep setting 
/home/jboss
[DEBUG]   Imported: org.apache.maven.settings < plexus.core
[DEBUG] Reading global settings from /usr/share/maven/conf/settings.xml
[DEBUG] Reading user settings from ?/.m2/settings.xml

the reason we dont see this on other java apps (like the one i tried above) is because the io.quarkus.vertx.http.deployment.StaticResourcesProcessor#collectStaticResources is not triggered in that case just due to different dependencies etc.

the other workaround is to set the mvn option -Duser.home=/home/jboss which then does not generate the "?" folder and does not trip into this bug

[Jiehong]

Same issue encountered on the company's Jenkins building in a container, and setting -Duser.home is a working workaround.

This workaround was not needed on 2.6.1, but started being needed on 2.7.0, 2.7.1 and 2.7.2 so far.

So #23696 did not fix it in 2.7.2.Final (just tried).