Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[testing] Fix e2e tests timed out, but there's empty test log #2810

Merged
merged 6 commits into from
Jan 10, 2020
Merged

[testing] Fix e2e tests timed out, but there's empty test log #2810

merged 6 commits into from
Jan 10, 2020

Conversation

Bobgy
Copy link
Contributor

@Bobgy Bobgy commented Jan 7, 2020 •
edited
Loading

Recent e2e tests usually fail with

Argo workflow timed out

e.g. in PR: #2603

It seems 30 minutes limit is no longer enough. I will investigate while time increases.
EDIT: something hangs the go test command, but log is not streamed out. So we should first make test log stream during testing.

/priority p0
/area testing
/kind bug

This change is Reviewable

@Bobgy
Copy link
Contributor Author

Bobgy commented Jan 7, 2020

api integration tests seem to be the one that doesn't finish: https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubeflow_pipelines/2603/kubeflow-pipeline-e2e-test/1214467257921441792/artifacts/pods_info/integration-test-h7lf7-3930211570.txt
in https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubeflow_pipelines/2603/kubeflow-pipeline-e2e-test/1214467257921441792

@Bobgy Bobgy changed the title Wait longer for e2e tests [testing] E2e tests timed out Jan 7, 2020
@Bobgy
Copy link
Contributor Author

Bobgy commented Jan 7, 2020 •
edited
Loading

Looked into the issue together with @jingzhang36, log for the running test is strange:
https://console.cloud.google.com/logs/viewer?project=ml-pipeline-test&advancedFilter=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22ml-pipeline-test%22%0Aresource.labels.location%3D%22us-east1-b%22%0Aresource.labels.cluster_name%3D%22e2e-ce0de86-26641%22%0Aresource.labels.namespace_name%3D%22kubeflow%22%0Aresource.labels.pod_name%3D%22integration-test-h7lf7-3930211570%22

It is almost empty. There are no logs after the following one.

go test -v ./... -namespace kubeflow -args -runIntegrationTests=true

Comparing with a succeeded test: https://console.cloud.google.com/logs/viewer?project=ml-pipeline-test&advancedFilter=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22ml-pipeline-test%22%0Aresource.labels.location%3D%22us-east1-b%22%0Aresource.labels.cluster_name%3D%22e2e-ce0de86-26641%22%0Aresource.labels.namespace_name%3D%22kubeflow%22%0Aresource.labels.pod_name%3D%22integration-test-h7lf7-3930211570%22

It seems the problem is:

pipelines/test/api-integration-test/run_test.sh

Line 67 in 07efee5

TEST_RESULT=`go test -v ./... -namespace ${NAMESPACE} -args -runIntegrationTests=true 2>&1`

The test command doesn't print anything if it hangs somewhere.
We need to first fix this issue to see what went wrong in the test.

@Bobgy Bobgy changed the title [testing] E2e tests timed out [testing] E2e tests timed out, but there's empty error log Jan 7, 2020
@Bobgy Bobgy changed the title [testing] E2e tests timed out, but there's empty error log [testing] e2e tests timed out, but there's empty error log Jan 7, 2020
@Bobgy Bobgy changed the title [testing] e2e tests timed out, but there's empty error log [testing] Fix e2e tests timed out, but there's empty error log Jan 7, 2020
@Bobgy Bobgy changed the title [testing] Fix e2e tests timed out, but there's empty error log [testing] Fix e2e tests timed out, but there's empty test log Jan 7, 2020
@Bobgy
Copy link
Contributor Author

Bobgy commented Jan 7, 2020

Error in cloudbuild:

Step #1 - "scheduledworkflow": go: gopkg.in/tomb.v1@v1.0.0-20141024135613-dd632973f1e7: unrecognized import path "gopkg.in/tomb.v1" (parse https://gopkg.in/tomb.v1?go-get=1: no go-import meta tags ())

Example 2

Step #0 - "persistenceagent": go: gopkg.in/airbrake/gobrake.v2@v2.0.9: unrecognized import path "gopkg.in/airbrake/gobrake.v2" (parse https://gopkg.in/airbrake/gobrake.v2?go-get=1: no go-import meta tags ())

It seems the problem is gopkg website having higher network failure rate now than usual. Any network failure will fail go module requirements from finishing.

@Bobgy
Copy link
Contributor Author

Bobgy commented Jan 7, 2020

/retest

@Bobgy
Copy link
Contributor Author

Bobgy commented Jan 7, 2020 •
edited
Loading

golang/go#28194 should be the best way to reduce flakiness caused by unstable import websites, but there's been no progress on that issue at all.

We can add some retry to hopefully mitigate flakiness in our presubmit tests.

@Bobgy
Copy link
Contributor Author

Bobgy commented Jan 7, 2020

/assign @Ark-kun
Can you help me review?

@Bobgy
Copy link
Contributor Author

Bobgy commented Jan 7, 2020

/test kubeflow-pipeline-e2e-test

@Bobgy
Copy link
Contributor Author

Bobgy commented Jan 7, 2020

Take a look at succeeded pod log, this fix is correct: https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubeflow_pipelines/2810/kubeflow-pipeline-e2e-test/1214503372829233152/artifacts/pods_info/integration-test-mlpb7-1035667546.txt

@Bobgy
Copy link
Contributor Author

Bobgy commented Jan 8, 2020

/assign @jingzhang36
Can you help me review too?

Bobgy
Bobgy commented Jan 9, 2020
View reviewed changes
Copy link
Contributor Author

@Bobgy Bobgy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Ark-kun Thanks for the review! I've addressed all the comments. Would you mind taking a look again?

test/api-integration-test/run_test.sh Show resolved Hide resolved
@Ark-kun
Copy link
Contributor

Ark-kun commented Jan 9, 2020

/lgtm

@Bobgy
Copy link
Contributor Author

Bobgy commented Jan 10, 2020

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Bobgy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 493c3d4 into master Jan 10, 2020
@Bobgy Bobgy deleted the Bobgy-patch-3 branch January 10, 2020 06:53
rui5i pushed a commit to rui5i/pipelines that referenced this pull request Jan 16, 2020
@Bobgy @Ark-kun @rui5i
[testing] Fix e2e tests timed out, but there's empty test log (kubefl…
b384b33 
…ow#2810)

* Wait longer for e2e tests

* Let api integration test stream log output while testing

* Also fix initialization test

* Fix exit code on error

* Update test/initialization-test/run_test.sh

Co-Authored-By: Alexey Volkov <avolkov@google.com>

* Update according to CR

Co-authored-by: Alexey Volkov <alexey.volkov@ark-kun.com>
Jeffwan pushed a commit to Jeffwan/pipelines that referenced this pull request Dec 9, 2020
@Bobgy @Ark-kun @Jeffwan
[testing] Fix e2e tests timed out, but there's empty test log (kubefl…
5be0346 
…ow#2810)

* Wait longer for e2e tests

* Let api integration test stream log output while testing

* Also fix initialization test

* Fix exit code on error

* Update test/initialization-test/run_test.sh

Co-Authored-By: Alexey Volkov <avolkov@google.com>

* Update according to CR

Co-authored-by: Alexey Volkov <alexey.volkov@ark-kun.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Ark-kun Ark-kun Ark-kun left review comments

@IronPan IronPan Awaiting requested review from IronPan

Assignees

@Ark-kun Ark-kun

@jingzhang36 jingzhang36

Labels
approved area/testing kind/bug lgtm priority/p1 size/S
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Bobgy @Ark-kun @k8s-ci-robot @jingzhang36