Creating a new Thorntail project fails - build failure due to quota being exceeded

[ldimaggi]

Issue Overview

Creating a new project with the Thorntail runtime fails with build error.

Expected Behaviour

It should be possible for users to create new projects configured with the Thorntail runtime.

Current Behaviour

See build log fragment below for error.

Steps To Reproduce

1. Create new space and new project
2. For the project, select Mission = REST, and runtime = Thorntail
3. Observe build failure

Additional Information

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running io.openshift.booster.GreetingServiceTest
Resolving 395 out of 658 artifacts
Picked up _JAVA_OPTIONS: -Duser.home=/home/jenkins -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Xms10m -Xmx192m
Thu Nov 15 01:55:25 UTC 2018 INFO [org.wildfly.swarm.bootstrap] (main) Dependencies not bundled; resolving from M2REPO.
�[0m2018-11-15 01:55:42,757 INFO  [org.wildfly.swarm] (main) THORN0013: Installed fraction:                   JAX-RS - STABLE          io.thorntail:jaxrs:2.2.0.Final
�[0m�[0m2018-11-15 01:55:42,832 INFO  [org.wildfly.swarm] (main) THORN0013: Installed fraction:                  Elytron - STABLE          io.thorntail:elytron:2.2.0.Final
�[0m�[0m2018-11-15 01:55:42,833 INFO  [org.wildfly.swarm] (main) THORN0013: Installed fraction:      MicroProfile Health - STABLE          io.thorntail:microprofile-health:2.2.0.Final
�[0m�[0m2018-11-15 01:55:42,833 INFO  [org.wildfly.swarm] (main) THORN0013: Installed fraction:                 Undertow - STABLE          io.thorntail:undertow:2.2.0.Final
�[0m�[0m2018-11-15 01:55:42,833 INFO  [org.wildfly.swarm] (main) THORN0013: Installed fraction:               Management - STABLE          io.thorntail:management:2.2.0.Final
�[0m�[0m2018-11-15 01:55:42,833 INFO  [org.wildfly.swarm] (main) THORN0013: Installed fraction:             Transactions - STABLE          io.thorntail:transactions:2.2.0.Final
�[0m�[0m2018-11-15 01:55:42,833 INFO  [org.wildfly.swarm] (main) THORN0013: Installed fraction:                      CDI - STABLE          io.thorntail:cdi:2.2.0.Final
�[0m�[0m2018-11-15 01:55:42,834 INFO  [org.wildfly.swarm] (main) THORN0013: Installed fraction:                  Logging - STABLE          io.thorntail:logging:2.2.0.Final
�[0m�[0m2018-11-15 01:55:42,834 INFO  [org.wildfly.swarm] (main) THORN0013: Installed fraction:      MicroProfile Config - STABLE          io.thorntail:microprofile-config:2.2.0.Final
�[0m�[0m2018-11-15 01:55:42,834 INFO  [org.wildfly.swarm] (main) THORN0013: Installed fraction:          Bean Validation - STABLE          io.thorntail:bean-validation:2.2.0.Final
�[0m�[0m2018-11-15 01:55:42,834 INFO  [org.wildfly.swarm] (main) THORN0013: Installed fraction:               Arquillian - STABLE          io.thorntail:arquillian:2.2.0.Final
�[0m�[0m2018-11-15 01:55:42,834 INFO  [org.wildfly.swarm] (main) THORN0013: Installed fraction:        CDI Configuration - STABLE          io.thorntail:cdi-config:2.2.0.Final
/opt/rh/rh-maven33/root/usr/bin/mvn: line 9:   158 Killed                  $M2_HOME/bin/mvn "$@"
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline

GitHub has been notified of this commit’s build result

ERROR: script returned exit code 137
Finished: FAILURE

[ldimaggi]

This is in production - cannot yet recreate on prod-preview as env reset results in Jenkins pod not starting.

[ldimaggi]

Seeing this on prod-preview too:

[chmouel]

You seem to have run with an infamous OOM which basically mean thorntail is consuming too much memory to run on starter's OSIO restricted environment,

if you can it would be nice if you can copy and paste the oc get ev from the -jenkins namespace to confirm this,

[ldimaggi]

Your guess is correct - see log fragment below.

Note that the first step in the test was to reset the user's OpenShift.io env - no other spaces or projects were present when the test was run:

18m        18m         1         jenkins-1                   ReplicationController                                 
Warning   FailedCreate                  {replication-controller }                               
Error creating: pods "jenkins-1-crd5p" is forbidden: exceeded quota: compute-resources, 
requested: limits.cpu=2,limits.memory=1Gi, used: limits.cpu=2,limits.memory=1Gi, 
limited: limits.cpu=2,limits.memory=1Gi
18m 

[sthaha]

@ldimaggi if a project build fails due to OOM, I guess there is nothing the build can do about it. The only way I see is to request for more resources; probably something that will be coved in OSD.

So I am going to close it as "Not a Bug"

[alexeykazakov]

If some booster project can’t be built on OSO then it should be removed from the wizard. This is something for the getting started team.

I’m reopening the issue.

[bartoszmajsak]

Is removal the right thing to do? How about we try to optimize the booster itself? @slemeur any comment on that? Let's keep in mind it was working before - did the runtime version change? Booster was modified?

@ldimaggi I have few questions here

• is that a permanent issue?
• is it the only affected booster for this runtime?
• last but not least - is that really SEV-1?

[slemeur]

Yes, if it was working before, it's a regression and we need to look at that.

[GeorgeActon]

@bartoszmajsak Who will look at the booster.

[bartoszmajsak]

@GeorgeActon App Creation aka Getting Started with help of booster owner if our initial analysis will lead to a conclusion that we have to improve booster to make it running in our env

[GeorgeActon]

@sbose78 @tinakurian All indications at this point are to look at the Booster. Please sync with @chmouel and figure next steps.

[ldimaggi]

Yes - this is a regression. It is 100% reproducible. And it blocks the user from creating the booster. As there is no workaround I think that it is a SEV1. Thx!

[chmouel]

I think the first thing would be to look at the booster and see if there was any update for the last couple of weeks and track down the regression between the versions, I am not sure what is the girhub repo but assuming is this one there was a version update thorntail-examples/health-check@6291567

[GeorgeActon]

It fails for either REST API and for Health check. tried both this morning.

[bartoszmajsak]

This is how boosters are mapped:

• defined in the catalog (and enabled for OSIO) https://github.com/fabric8-launcher/launcher-booster-catalog/blob/master/thorntail/community/rest-http/booster.yaml#L10
• pointing to version 15 (tag) https://github.com/thorntail-examples/rest-http/tree/15 (which has been updated month ago)

[sbose78]

I agree @bartoszmajsak . @corinnekrych , let's take a look at possibly fixing the booster.

Btw, If that doesn't happen soon enough, we should temporarily take the booster out and then put it back only after its fixed. You OK with that, @slemeur ?

[slemeur]

Yes that would be fine with me @sbose78

[ldimaggi]

+1 to removing the booster until it works - anything visible should be working.

[corinnekrych]

Let's start with disabling it see PR.
@bartoszmajsak do you ping dev directly or automatic reviewer launcher-devs is enough?

[bartoszmajsak]

@bartoszmajsak do you ping dev directly or automatic reviewer launcher-devs is enough?

It's enough. @gastaldi will instantly look at it anyway. He's like a bot ;)

[sbose78]

FYI: Figuring out why something which was working is now breaking, and not owned by us, will take a while. The sev1 might take longer to resolve.

[chmouel]

We should get this booster only available when building on a OSD cluster.

Currently it's failing due of resources limitation for OSD, it may be possible to fit it to run on OSO but the users would never being able to use it :(

This may get better tho when we move over the new osio-pipeline library, since we will not be building anymore in the -jenkins namespace but in user namespace, so we may have more resources available for building as we are not sharing them with the jenkins master, but this may going to take a while until we can migrate to it,

@slemeur i think it's something that should be discussed

[corinnekrych]

@ldimaggi I've tested v11 (last updated 24th July) and v12 (last updated 16th October) of the thorntail health-check app and both fails on my prod-preview => which confirm that this is not a regression in booster themselves but indeed as @chmouel mention in the ticket an issue with build.

@shoubhikbose I'll remove my name from the ticket as I think this ticket will be better handle in build team. Let me know if you need more help on my side.

[sbose78]

Thanks for the investigation @corinnekrych and @chmouel .

This may get better tho when we move over the new osio-pipeline library, since we will not be building anymore in the -jenkins namespace but in user namespace,

I'll give this a shot.

[chmouel]

Thanks @sbose78 this would be a nice use case for osio-pipeline,

[sbose78]

Both the health-check and rest-http boosters work with the new pipeline library since the build happens in the user namespace.

Example:
thorntail-examples/rest-http#53
fabric8io/fabric8-jenkinsfile-library#44

Once we finish migrating our Java boosters to use the new pipeline library, then we can update the Jenkinsfiles and this should just work.

@rupalibehera let's work together on getting the migration done.

[sthaha]

@sbose78

@rupalibehera let's work together on getting the migration done.

#4535 would need to be done before we can start using the new pipeline for java projects.

[chmouel]

Raised #4535 as P1 since we are depending on this to fix this issue,

[gorkem]

#4535 is not a blocker for migrating only thorntail, please go ahead and enable thorntail without waiting for it. #4535 is a blocker for migrating all other boosters.