Intermittent issue - application is not available on run environment

[ljelinkova]

The E2E tests fail intermittently because the application is not available on run environment.

http://artifacts.ci.centos.org/devtools/e2e/devtools-test-e2e-openshift.io-smoketest-us-east-2a-released/322/07-02-run.html

The E2E workflow is:

1. Create space
2. Create Vert.x with REST API and Rollout to Run strategy
3. Waits until the pipeline is finished and promotes to Run
4. Checks the stage application
5. Checks the run application - this step fails

This is the Jenkins log
http://artifacts.ci.centos.org/devtools/e2e/devtools-test-e2e-openshift.io-smoketest-us-east-2a-released/322/05-01-jenkins-log.html

This is output of a script that lists Jenkins pods, we can add any oc command you might need to debug this issue

http://artifacts.ci.centos.org/devtools/e2e/devtools-test-e2e-openshift.io-smoketest-us-east-2a-released/322/oc-logs-output.txt

[ldimaggi]

Is there any possibility that this is a timing issue? I have seen random instances of the deployment to run requiring a long time to complete.

Or, might this be a resources issue? Do you see anything in the logs related to a quota being reached? Maybe the env reset is not removing any existing deployments to run?

[hrishin]

@ljelinkova did you see resource quota? sometimes quota gets full for target namespace. OS accepts the new DC request successfully but not able to deploy it if the quota is reached.

[ljelinkova]

@hrishin The resource quota should not be the problem since we reset the whole environment after each test.

@ldimaggi It might be the timing issue, maybe if the tests waited for some time the app would start.

But the main question is - how does random user know that the deployment failed? Or that the deployment is finished? Should not this be part of "Pipeline"? I simple assumed that once the Pipeline is finished I can assume all is set and ready and I can start using the application.

[ppitonak]

I agree that it should be part of the pipeline, i.e. "Rollout to Run" step should be marked as successful only when the app was deployed successfully. WDYT @openshiftio/uxd-team @catrobson

[catrobson]

@ppitonak Agree we would only mark that step successful when the app was deployed successfully.