test (e2e) : Add E2E test for java app deploy workflow using Eclipse JKube (#4397) #4414

rohanKanojia · 2024-10-24T10:01:49Z

Description

Fix #4397

Added new test application_deployment.feature that would deploy a simple Quarkus application onto existing CRC cluster and verify that application is accessible via endpoint.
Test would be performing a java maven based build inside a container within a pod inside CRC VM. At the moment, we're building the container image and running the pod at the same time.

Signed-off-by: Rohan Kumar rohaan@redhat.com

Type of change

Bug fix (non-breaking change which fixes an issue)
Feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change
Chore (non-breaking change which doesn't affect codebase;
test, version modification, documentation, etc.)

Checklist

Fixes: Issue #4397

Relates to: Issue #4397, Issue https://github.com/eclipse-jkube/team/issues/32

Solution/Idea

I added an E2E test that would clone the Eclipse JKube repository and deploy one of the quickstart jkube@quickstarts/maven/quarkus onto the deployed CRC cluster. I've used a simple Quarkus REST application, but it can be changed to something else as well.

Proposed changes

Add a new E2E test that would verify deployment workflow of a simple Java application using Eclipse JKube

Testing

This PR only adds an E2E test, in order to run it. Use the following command:

# Modify BUNDLE_LOCATION, PULL_SECRET_PATH and CRC_BINARY variables as per your environment
make e2e GODOG_OPTS="--godog.tags='@story_application_deployment'" BUNDLE_LOCATION="--bundle-location=xxxx" PULL_SECRET_PATH="--pull-secret-path=xxxx" CRC_BINARY="--crc-binary=xxxx"

Running e2e test in: /home/rokumar/go/src/github.com/crc-org/crc/test/e2e/out/test-run
Working directory set to: /home/rokumar/go/src/github.com/crc-org/crc/test/e2e/out/test-run
Expecting the bundle provided by the user
Using existing bundle: /home/rokumar/.crc/cache/crc_libvirt_4.17.0_amd64.crcbundle
Feature: Application Deployment Test
  User deploys a basic java application onto CRC cluster and expects that it's
  deployed successfully and is accessible via route

  Background:
    Given ensuring CRC cluster is running                                                                                                                             # testsuite.go:792 -> github.com/crc-org/crc/v2/test/e2e/testsuite.EnsureCRCIsRunning
    And ensuring oc command is available                                                                                                                              # testsuite.go:875 -> github.com/crc-org/crc/v2/test/e2e/testsuite.EnsureOCCommandIsAvailable
    And ensuring user is logged in succeeds                                                                                                                           # testsuite.go:842 -> github.com/crc-org/crc/v2/test/e2e/testsuite.EnsureUserIsLoggedIntoClusterSucceedsOrFails

  Scenario: Deploy a java application using Eclipse JKube in pod and then verify it's health                                                                          # features/application_deployment.feature:13
    When executing "oc new-project jkube-quarkus-app-deploy-flow-test" succeeds                                                                                       # shell.go:245 -> github.com/crc-org/crc/v2/test/extended/util.ExecuteCommandSucceedsOrFails
    And executing "oc create -f ../../../testdata/jkube-kubernetes-build-resources.yaml" succeeds                                                                     # shell.go:245 -> github.com/crc-org/crc/v2/test/extended/util.ExecuteCommandSucceedsOrFails
    And executing "oc start-build jkube-application-deploy-buildconfig --follow" succeeds                                                                             # shell.go:245 -> github.com/crc-org/crc/v2/test/extended/util.ExecuteCommandSucceedsOrFails
    And executing "oc rollout status -w dc/jkube-application-deploy-test --timeout=600s" succeeds                                                                     # shell.go:245 -> github.com/crc-org/crc/v2/test/extended/util.ExecuteCommandSucceedsOrFails
    Then stdout should contain "successfully rolled out"                                                                                                              # shell.go:301 -> github.com/crc-org/crc/v2/test/extended/util.CommandReturnShouldContain
    And executing "oc logs -lapp=jkube-application-deploy-test -f" succeeds                                                                                           # shell.go:245 -> github.com/crc-org/crc/v2/test/extended/util.ExecuteCommandSucceedsOrFails
    And executing "oc rollout status -w dc/quarkus --namespace jkube-quarkus-app-deploy-flow-test --timeout=600s" succeeds                                            # shell.go:245 -> github.com/crc-org/crc/v2/test/extended/util.ExecuteCommandSucceedsOrFails
    Then stdout should contain "successfully rolled out"                                                                                                              # shell.go:301 -> github.com/crc-org/crc/v2/test/extended/util.CommandReturnShouldContain
    And executing "oc get build -lapp=quarkus --namespace jkube-quarkus-app-deploy-flow-test" succeeds                                                                # shell.go:245 -> github.com/crc-org/crc/v2/test/extended/util.ExecuteCommandSucceedsOrFails
    Then stdout should contain "quarkus-s2i"                                                                                                                          # shell.go:301 -> github.com/crc-org/crc/v2/test/extended/util.CommandReturnShouldContain
    And executing "oc get buildconfig -lapp=quarkus --namespace jkube-quarkus-app-deploy-flow-test" succeeds                                                          # shell.go:245 -> github.com/crc-org/crc/v2/test/extended/util.ExecuteCommandSucceedsOrFails
    Then stdout should contain "quarkus-s2i"                                                                                                                          # shell.go:301 -> github.com/crc-org/crc/v2/test/extended/util.CommandReturnShouldContain
    And executing "oc get imagestream -lapp=quarkus --namespace jkube-quarkus-app-deploy-flow-test" succeeds                                                          # shell.go:245 -> github.com/crc-org/crc/v2/test/extended/util.ExecuteCommandSucceedsOrFails
    Then stdout should contain "quarkus"                                                                                                                              # shell.go:301 -> github.com/crc-org/crc/v2/test/extended/util.CommandReturnShouldContain
    And executing "oc get pods -lapp=quarkus --namespace jkube-quarkus-app-deploy-flow-test" succeeds                                                                 # shell.go:245 -> github.com/crc-org/crc/v2/test/extended/util.ExecuteCommandSucceedsOrFails
    Then stdout should contain "quarkus"                                                                                                                              # shell.go:301 -> github.com/crc-org/crc/v2/test/extended/util.CommandReturnShouldContain
    And stdout should contain "1/1     Running"                                                                                                                       # shell.go:301 -> github.com/crc-org/crc/v2/test/extended/util.CommandReturnShouldContain
    And executing "oc get svc -lapp=quarkus --namespace jkube-quarkus-app-deploy-flow-test" succeeds                                                                  # shell.go:245 -> github.com/crc-org/crc/v2/test/extended/util.ExecuteCommandSucceedsOrFails
    Then stdout should contain "quarkus"                                                                                                                              # shell.go:301 -> github.com/crc-org/crc/v2/test/extended/util.CommandReturnShouldContain
    And executing "oc get routes -lapp=quarkus --namespace jkube-quarkus-app-deploy-flow-test" succeeds                                                               # shell.go:245 -> github.com/crc-org/crc/v2/test/extended/util.ExecuteCommandSucceedsOrFails
    Then stdout should contain "quarkus"                                                                                                                              # shell.go:301 -> github.com/crc-org/crc/v2/test/extended/util.CommandReturnShouldContain
    And with up to "4" retries with wait period of "1m" http response from "http://quarkus-jkube-quarkus-app-deploy-flow-test.apps-crc.testing" has status code "200" # testsuite.go:560 -> github.com/crc-org/crc/v2/test/e2e/testsuite.CheckHTTPResponseWithRetry
    Then executing "curl -s http://quarkus-jkube-quarkus-app-deploy-flow-test.apps-crc.testing" succeeds                                                              # shell.go:245 -> github.com/crc-org/crc/v2/test/extended/util.ExecuteCommandSucceedsOrFails
    And stdout should contain "{"applicationName":"JKube","message":"Subatomic JKube really whips the llama's ass!"}"                                                 # shell.go:301 -> github.com/crc-org/crc/v2/test/extended/util.CommandReturnShouldContain
    When executing "oc delete project jkube-quarkus-app-deploy-flow-test" succeeds                                                                                    # shell.go:245 -> github.com/crc-org/crc/v2/test/extended/util.ExecuteCommandSucceedsOrFails
Deleted CRC instance (if one existed).

1 scenarios (1 passed)
28 steps (28 passed)
33m33.762301105s
ok      github.com/crc-org/crc/v2/test/e2e      2013.779s

openshift-ci · 2024-10-24T10:01:56Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign cfergeau for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

test/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2024-10-24T10:02:01Z

Hi @rohanKanojia. Thanks for your PR.

I'm waiting for a crc-org member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

praveenkumar · 2024-10-24T13:24:47Z

/ok-to-test

praveenkumar · 2024-10-24T13:26:30Z

test/e2e/features/application_deployment.feature

+
+    @linux @windows @darwin
+    Scenario: Deploy a java application using Eclipse JKube in podman container and then verify it's health
+        When executing "podman build --tag localhost/redhat-ubi-openjdk17-with-oc:latest -f ../../../testdata/JKubeAppDeployment_Containerfile   ." succeeds


do you still need ../../../testdata/JKube... as full path because it should directly work

When I only specify file name, I get this error:

--- Failed steps: Scenario: Deploy a java application using Eclipse JKube in podman container and then verify it's health # features/application_deployment.feature:13 When executing "podman build --tag localhost/redhat-ubi-openjdk17-with-oc:latest -f JKubeAppDeployment_Containerfile ." succeeds # features/application_deployment.feature:14 Error: command 'podman build --tag localhost/redhat-ubi-openjdk17-with-oc:latest -f JKubeAppDeployment_Containerfile .', expected to succeed, exited with exit code: 125 Command stdout: Command stderr: Error: the specified Containerfile or Dockerfile does not exist, JKubeAppDeployment_Containerfile: no such file or directory

If I don't specify any source file override for podman build, I see that podman is trying to look into this directory:

--- Failed steps: Scenario: Deploy a java application using Eclipse JKube in podman container and then verify it's health # features/application_deployment.feature:13 When executing "podman build --tag localhost/redhat-ubi-openjdk17-with-oc:latest ." succeeds # features/application_deployment.feature:14 Error: command 'podman build --tag localhost/redhat-ubi-openjdk17-with-oc:latest .', expected to succeed, exited with exit code: 125 Command stdout: Command stderr: Error: no Containerfile or Dockerfile specified or found in context directory, /home/rokumar/go/src/github.com/crc-org/crc/test/e2e/out/test-run: no such file or directory

@adrianriobo do you know where we put the file to use it without absolute path?

@praveenkumar : I was missing @testdata annotation in test scenario. It should be fixed now.

praveenkumar · 2024-10-24T13:27:07Z

test/e2e/features/application_deployment.feature

+    @linux @windows @darwin
+    Scenario: Deploy a java application using Eclipse JKube in podman container and then verify it's health
+        When executing "podman build --tag localhost/redhat-ubi-openjdk17-with-oc:latest -f ../../../testdata/JKubeAppDeployment_Containerfile   ." succeeds
+        And executing "podman run --rm --dns=8.8.8.8 --name application-deployment-build-e2e --add-host=oauth-openshift.apps-crc.testing:host-gateway localhost/redhat-ubi-openjdk17-with-oc:latest" succeeds


why we need extra dns=8.8.8.8 because by default it is not working?

Not 100% sure whether this is due to some misconfiguration on my system or an actual issue. But When I was not using --dns=8.8.8.8 maven dependency download was failing with this intermittent error :

[INFO] Scanning for projects... Downloading from central: https://repo1.maven.org/maven2/io/quarkus/quarkus-bom/3.9.1/quarkus-bom-3.9.1.pom [ERROR] [ERROR] Some problems were encountered while processing the POMs: [ERROR] Non-resolvable import POM: Could not transfer artifact io.quarkus:quarkus-bom:pom:3.9.1 from/to central (https://repo1.maven.org/maven2): transfer failed for https://repo1.maven.org/maven2/io/quarkus/quarkus-bom/3.9.1/quarkus-bom-3.9.1.pom @ line 41, column 19 @ [ERROR] The build could not read 1 project -> [Help 1] [ERROR] [ERROR] The project org.eclipse.jkube.quickstarts.maven:quarkus:1.17.0 (/home/default/jkube/quickstarts/maven/quarkus/pom.xml) has 1 error [ERROR] Non-resolvable import POM: Could not transfer artifact io.quarkus:quarkus-bom:pom:3.9.1 from/to central (https://repo1.maven.org/maven2): transfer failed for https://repo1.maven.org/maven2/io/quarkus/quarkus-bom/3.9.1/quarkus-bom-3.9.1.pom @ line 41, column 19: Connect to repo1.maven.org:443 [repo1.maven.org/151.101.36.209] failed: Connection timed out -> [Help 2] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException [ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException

However, when I run the same command directly against CRC's podman daemon I don't see this issue.

This is mostly your network which might blocking maven dep target.

okay, I will remove --dns override from podman command

This should no longer be required due to changed approach.

praveenkumar · 2024-10-24T14:34:47Z

test/testdata/JKubeAppDeployment_Containerfile

+  oc get pods
+  cd jkube/quickstarts/maven/quarkus
+  # When
+  mvn -U package oc:build oc:resource oc:apply


So whole point of we are creating this containerfile is to clone the repo and have mvn plugin installed ?

Eclipse JKube requires these things to be installed to work:

Java 1.8+ Runtime (provided by registry.access.redhat.com/ubi9/openjdk-17:1.20-2.1726695177)

Apache Maven/Gradle (provided by registry.access.redhat.com/ubi9/openjdk-17:1.20-2.1726695177)

We can move the clone part to the script. Or maybe don't clone any repository, I can add a simple Quarkus project here in test folder.

As per discussion in #4397 (comment) we though it's better to not add extra dependencies and do java based build inside a container.

As per discussion in #4397 (comment) we though it's better to not add extra dependencies and do java based build inside a container.

Yes, that is what we want. Have we tried using this container file directly using oc command instead building it with podman and then running with podman? I think if we do it as part of oc then build happen in the cluster and we don't need the login part.

Do you mean to execute it in a Pod instead? It would use the mounted ServiceAccount's token for authentication (we might need to give it appropriate permissions for that)

Yes, we can create a service account first with required permission and then add that service account to the pod which would do required stuff?

Thanks, Let me update this PR to use this approach instead.

Can we keep it simple for now? The containerfile seems to be a situation that is easy to test with. The suggestion about building in a pod, and therefore involve S2I sounds like an improvement.

The first step was to make sure the cluster is operational to run an application.

The next step would be to make sure the expected functionality of the cluster is available/operational.

The third step would be to load test, memory test, etc with an application we understand and 'own', to ensure the cluster keeps operating under conditions like stress, long use, etc.

With current flow this looks like as part of containerfile we are embedding logic to login to cluster and then deploy it to cluster which seems overkill and have hardcoded cluster login info instead what we should've this containerfile generic enough to target any cluster.

I have modified this PR to perform jkube build in a pod instead.

praveenkumar · 2024-11-05T05:34:12Z

/ok-to-test

praveenkumar · 2024-11-11T06:55:49Z

@rohanKanojia check the e2e failure, looks like it is failing with basic test because status of crc is running which is not expected.

rohanKanojia · 2024-11-11T10:32:20Z

@praveenkumar : I tried reproducing it locally but couldn't reproduce it. Could it be a flaky failure?

rohanKanojia · 2024-11-13T14:51:58Z

@praveenkumar : I have added @cleanup annotation that shall perform additional cleanup after test execution.

Now the only failing e2e test I see is of cert rotation:

  Scenario: Start CRC "in the future" and clean up # test/e2e/features/cert_rotation.feature:9
    When starting CRC with default bundle along with stopped network time synchronization succeeds # test/e2e/features/cert_rotation.feature:11
      Error: command 'CRC_DISABLE_UPDATE_CHECK=true CRC_DEBUG_ENABLE_STOP_NTP=true crc start -p '/home/packer/pull-secret' -b /home/packer/crc_libvirt_4.17.1_amd64.crcbundle', expected to succeed, exited with exit code: 1

I think it was failing before too.

praveenkumar · 2024-11-14T08:14:52Z

@rohanKanojia squash the commits because second commit is making changes what you did in first one. other than that looks good to me.

…JKube (crc-org#4397) + Added new test `application_deployment.feature` that would deploy a simple Quarkus application onto existing CRC cluster and verify that application is accessible via endpoint. + Perform JKube application deploy step in a Pod instead of podman container in order to skip oc login Signed-off-by: Rohan Kumar <rohaan@redhat.com>

openshift-ci · 2024-11-14T10:26:27Z

@rohanKanojia: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/integration-crc	`8e6e2f6`	link	true	`/test integration-crc`
ci/prow/e2e-crc	`8e6e2f6`	link	true	`/test e2e-crc`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci bot added the do-not-merge/work-in-progress label Oct 24, 2024

openshift-ci bot added the needs-ok-to-test label Oct 24, 2024

openshift-ci bot added ok-to-test and removed needs-ok-to-test labels Oct 24, 2024

praveenkumar reviewed Oct 24, 2024

View reviewed changes

rohanKanojia mentioned this pull request Oct 25, 2024

fix (test/extended/util) : shell doesn't parse command output ending without newline (#4416) #4417

Merged

15 tasks

rohanKanojia force-pushed the pr/issue4397 branch from 1694b71 to 822ab7b Compare October 25, 2024 16:04

rohanKanojia force-pushed the pr/issue4397 branch 2 times, most recently from d1d5f41 to e0902c5 Compare November 4, 2024 15:46

rohanKanojia force-pushed the pr/issue4397 branch 2 times, most recently from 9e7d7b1 to 705f351 Compare November 6, 2024 14:55

rohanKanojia marked this pull request as ready for review November 7, 2024 08:44

openshift-ci bot removed the do-not-merge/work-in-progress label Nov 7, 2024

openshift-ci bot requested review from adrianriobo and praveenkumar November 7, 2024 08:45

rohanKanojia force-pushed the pr/issue4397 branch from 705f351 to 4bc5be3 Compare November 11, 2024 10:29

rohanKanojia force-pushed the pr/issue4397 branch from 4bc5be3 to 7f67ea7 Compare November 13, 2024 08:14

rohanKanojia force-pushed the pr/issue4397 branch 2 times, most recently from 2d1e43b to c17f5bb Compare November 13, 2024 20:28

rohanKanojia force-pushed the pr/issue4397 branch from c17f5bb to 8e6e2f6 Compare November 14, 2024 08:28

praveenkumar merged commit d485151 into crc-org:main Nov 14, 2024
26 of 32 checks passed

rohanKanojia deleted the pr/issue4397 branch November 14, 2024 12:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test (e2e) : Add E2E test for java app deploy workflow using Eclipse JKube (#4397) #4414

test (e2e) : Add E2E test for java app deploy workflow using Eclipse JKube (#4397) #4414

rohanKanojia commented Oct 24, 2024 •

edited

Loading

openshift-ci bot commented Oct 24, 2024

openshift-ci bot commented Oct 24, 2024

praveenkumar commented Oct 24, 2024

praveenkumar Oct 24, 2024

rohanKanojia Oct 24, 2024

praveenkumar Oct 29, 2024

rohanKanojia Nov 6, 2024

praveenkumar Oct 24, 2024

rohanKanojia Oct 24, 2024

praveenkumar Oct 29, 2024

rohanKanojia Oct 29, 2024

rohanKanojia Nov 5, 2024

praveenkumar Oct 24, 2024

rohanKanojia Oct 24, 2024 •

edited

Loading

praveenkumar Oct 24, 2024

rohanKanojia Oct 24, 2024 •

edited

Loading

praveenkumar Oct 29, 2024

rohanKanojia Oct 29, 2024

gbraad Nov 4, 2024

praveenkumar Nov 4, 2024

rohanKanojia Nov 5, 2024

praveenkumar commented Nov 5, 2024

praveenkumar commented Nov 11, 2024

rohanKanojia commented Nov 11, 2024

rohanKanojia commented Nov 13, 2024

praveenkumar commented Nov 14, 2024

openshift-ci bot commented Nov 14, 2024

test (e2e) : Add E2E test for java app deploy workflow using Eclipse JKube (#4397) #4414

test (e2e) : Add E2E test for java app deploy workflow using Eclipse JKube (#4397) #4414

Conversation

rohanKanojia commented Oct 24, 2024 • edited Loading

Description

Type of change

Checklist

Solution/Idea

Proposed changes

Testing

openshift-ci bot commented Oct 24, 2024

openshift-ci bot commented Oct 24, 2024

praveenkumar commented Oct 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rohanKanojia Oct 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rohanKanojia Oct 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

praveenkumar commented Nov 5, 2024

praveenkumar commented Nov 11, 2024

rohanKanojia commented Nov 11, 2024

rohanKanojia commented Nov 13, 2024

praveenkumar commented Nov 14, 2024

openshift-ci bot commented Nov 14, 2024

rohanKanojia commented Oct 24, 2024 •

edited

Loading

rohanKanojia Oct 24, 2024 •

edited

Loading

rohanKanojia Oct 24, 2024 •

edited

Loading