Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sample test improvement - using python fire to launch sample test #1897

Merged
merged 81 commits into from
Aug 23, 2019
Merged
Changes from 1 commit
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
e0c7132
Remove redundant import.
numerology Jul 23, 2019
70972db
Simplify sample_test.yaml by using withItem syntax.
Jul 31, 2019
deb6cc0
Merge branch 'master' of https://github.com/kubeflow/pipelines
Jul 31, 2019
da9a42a
Simplify sample_test.yaml by using withItem syntax.
Jul 31, 2019
a203039
Merge branch 'master' of https://github.com/numerology/pipelines
Jul 31, 2019
81ccbf4
Change dict to str in withItems.
Jul 31, 2019
d94a945
Merge branch 'master' of https://github.com/kubeflow/pipelines
Jul 31, 2019
ee38eb1
Merge branch 'master' of https://github.com/kubeflow/pipelines
Aug 7, 2019
9b3ad8e
Merge branch 'master' of https://github.com/kubeflow/pipelines
Aug 8, 2019
4ba8f85
Merge branch 'master' of https://github.com/kubeflow/pipelines
Aug 8, 2019
2b39278
Merge branch 'master' of https://github.com/kubeflow/pipelines
Aug 9, 2019
3061791
Merge branch 'master' of https://github.com/kubeflow/pipelines
Aug 9, 2019
1cc64ec
Merge remote-tracking branch 'origin/master'
Aug 11, 2019
a42ac40
Merge branch 'master' of https://github.com/kubeflow/pipelines
Aug 12, 2019
2398017
Merge remote-tracking branch 'origin/master'
Aug 12, 2019
a0bd790
Merge branch 'master' of https://github.com/kubeflow/pipelines
Aug 13, 2019
433180c
Merge branch 'master' of https://github.com/kubeflow/pipelines
Aug 15, 2019
fcabe77
Merge branch 'master' of https://github.com/kubeflow/pipelines
Aug 15, 2019
dfc0b16
Add back coveralls.
Aug 15, 2019
fb734de
Squashed commit of the following:
Aug 16, 2019
2b6907e
WIP: rewrite sample test infra
Aug 16, 2019
55a8667
WIP: rewrite sample test infra
Aug 16, 2019
e063bc2
WIP: rewrite sample test infra
Aug 16, 2019
17348a8
Fixing
Aug 16, 2019
b77703b
WIP: add injection function in python launcher.
Aug 19, 2019
797b618
Refactor the logic of run_test.sh into sample_test_launcher.py
Aug 20, 2019
dcdefd3
Fix exit_code
Aug 20, 2019
aa12ff1
Fix imports and add todo
Aug 20, 2019
f85e10a
Squashed commit of the following:
Aug 20, 2019
34d25d4
Merge branch 'master' of https://github.com/kubeflow/pipelines into t…
Aug 20, 2019
de74804
Clean up.
Aug 20, 2019
b685c62
Fix launcher flag.
Aug 20, 2019
d1994d6
switch to sys.execxutable.
Aug 20, 2019
57763d0
Fix bugs
Aug 20, 2019
d3db0bc
Add copy blob util function
Aug 20, 2019
da3a5dd
Fix default image prefix.
Aug 20, 2019
950035f
Fix dependencies.
Aug 20, 2019
905610b
Fix gs op
Aug 20, 2019
1dab733
Lint
Aug 20, 2019
8c814ba
Fix papermill exec
Aug 20, 2019
ff26024
Init for componentTest
Aug 21, 2019
bc0d319
Debugging.
Aug 21, 2019
5dcc495
Try to fix the issue.
Aug 21, 2019
fffee19
Fix exit code
Aug 21, 2019
748e9d9
Another fix.
Aug 21, 2019
5e7a8c5
Squashed commit of the following:
Aug 21, 2019
c902070
Fix working dir
Aug 21, 2019
8ef3fa1
Fix exit code format
Aug 21, 2019
3a33af2
Merge branch 'master' of https://github.com/kubeflow/pipelines into t…
Aug 21, 2019
68a4089
Remove kfp notebook sample using tfx:oss components.
Aug 21, 2019
3de8843
Refactor to reduce potential dup code.
Aug 21, 2019
ad90563
Clear unused const.
Aug 21, 2019
fc6301a
Fix redundant check
Aug 21, 2019
23349b1
Add image injection for component test.
Aug 21, 2019
ad01f7d
Fix unused import
Aug 21, 2019
73b1409
Merge branch 'master' of https://github.com/kubeflow/pipelines into t…
Aug 21, 2019
f5a857d
Squash and merge from master
Aug 21, 2019
bed1135
Squashed commit of the following:
Aug 22, 2019
14dfd4e
# Conflicts:
Aug 22, 2019
a08fb85
Merge branch 'master' of https://github.com/kubeflow/pipelines into t…
Aug 22, 2019
e9b0cd8
Fix version for papermill
Aug 22, 2019
04546c5
Merge remote-tracking branch 'origin/test-improvement' into test-impr…
Aug 22, 2019
35ae6f5
Squashed commit of the following:
Aug 22, 2019
6fc6252
Merge branch 'master' of https://github.com/kubeflow/pipelines into t…
Aug 22, 2019
c5640bf
Fix google-cloud-storage and google fire version info.
Aug 22, 2019
d6a29b9
Delete run_test.sh
Aug 22, 2019
ad177e5
Merge from kfp master
Aug 23, 2019
0104c43
Squashed commit of the following:
Aug 23, 2019
364af5c
Merge branch 'master' of https://github.com/kubeflow/pipelines into t…
Aug 23, 2019
58937af
Fix check_notebook_results args.
Aug 23, 2019
8f21a36
Fix.
Aug 23, 2019
85f5391
Merge branch 'master' of https://github.com/kubeflow/pipelines into t…
Aug 23, 2019
e8694ab
Fix work dir path
Aug 23, 2019
9aed040
Fix global const naming convention
Aug 23, 2019
b8129b4
Remove redundant check.
Aug 23, 2019
7e627aa
Extract project name from result gcs dir.
Aug 23, 2019
0b3c323
Add error catching
Aug 23, 2019
2c07792
Name change.
Aug 23, 2019
911ceb3
Improve error catching.
Aug 23, 2019
55533a2
Improve error catching.
Aug 23, 2019
ee66780
naming correction.
Aug 23, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions test/sample-test/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the Licens

import google
import os
import re
import subprocess
Expand Down Expand Up @@ -88,15 +89,17 @@ def file_injection(file_in, tmp_file_out, subs):

def upload_blob(bucket_name, source_file_name, destination_blob_name):
"""Uploads a file to the bucket."""
storage_client = storage.Client()
try:
storage_client = storage.Client()
bucket = storage_client.get_bucket(bucket_name)
blob = bucket.blob(destination_blob_name)
except google.cloud.exceptions.NotFound as e:
raise RuntimeError("Bucket not found: {}\n".format(str(e)))

blob = bucket.blob(destination_blob_name)
try:
blob.upload_from_filename(source_file_name)
except:
raise RuntimeError('Failure when uploading %s from %s to %s'
% (source_file_name, bucket_name, destination_blob_name))
except google.cloud.exceptions.GoogleCloudError as e:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, except's can be combined as such:
try:
except A:
except B:

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call. Done.

raise RuntimeError('Failure when uploading {}\n'.format(str(e)))

print('File {} uploaded to {}.'.format(
source_file_name,
Expand Down