Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to set up enviroment for Kubeflow/backend development? #3942

Closed
NikeNano opened this issue Jun 9, 2020 · 9 comments
Closed

How to set up enviroment for Kubeflow/backend development? #3942

NikeNano opened this issue Jun 9, 2020 · 9 comments
Labels
kind/bug

Comments

@NikeNano
Copy link
Member

NikeNano commented Jun 9, 2020 •
edited
Loading

What steps did you take:

I cloned the kubeflow/pipelines repo and tried to run some of the backend tests but get issues. I can't find any instructions related to how to set up enviroments to do development for kubeflow/backend and cant get it correct myself.

What happened:

When I try to run the kubeflow/pipelines/backend/src/apiserver/storage_test test: TestListJobs_TotalSizeWithFilter i get the following error:

# github.com/argoproj/argo/util/kubeconfig
/Users/niklashansson/Documents/go/pkg/mod/github.com/argoproj/argo@v2.5.2+incompatible/util/kubeconfig/kubeconfig.go:62:12: restConfig.BearerTokenFile undefined (type *rest.Config has no field or method BearerTokenFile)
/Users/niklashansson/Documents/go/pkg/mod/github.com/argoproj/argo@v2.5.2+incompatible/util/kubeconfig/kubeconfig.go:76:12: restConfig.BearerTokenFile undefined (type *rest.Config has no field or method BearerTokenFile)
# github.com/argoproj/argo/pkg/apis/workflow/v1alpha1
/Users/niklashansson/Documents/go/pkg/mod/github.com/argoproj/argo@v2.5.2+incompatible/pkg/apis/workflow/v1alpha1/generated.pb.go:33:11: undefined: proto.GoGoProtoPackageIsVersion3
/Users/niklashansson/Documents/go/pkg/mod/github.com/argoproj/argo@v2.5.2+incompatible/pkg/apis/workflow/v1alpha1/generated.pb.go:2290:33: m.PasswordSecret.MarshalToSizedBuffer undefined (type *"k8s.io/api/core/v1".SecretKeySelector has no field or method MarshalToSizedBuffer)
/Users/niklashansson/Documents/go/pkg/mod/github.com/argoproj/argo@v2.5.2+incompatible/pkg/apis/workflow/v1alpha1/generated.pb.go:2302:33: m.UsernameSecret.MarshalToSizedBuffer undefined (type *"k8s.io/api/core/v1".SecretKeySelector has no field or method MarshalToSizedBuffer)
/Users/niklashansson/Documents/go/pkg/mod/github.com/argoproj/argo@v2.5.2+incompatible/pkg/apis/workflow/v1alpha1/generated.pb.go:2431:28: m.ObjectMeta.MarshalToSizedBuffer undefined (type "k8s.io/apimachinery/pkg/apis/meta/v1".ObjectMeta has no field or method MarshalToSizedBuffer)
/Users/niklashansson/Documents/go/pkg/mod/github.com/argoproj/argo@v2.5.2+incompatible/pkg/apis/workflow/v1alpha1/generated.pb.go:2478:26: m.ListMeta.MarshalToSizedBuffer undefined (type "k8s.io/apimachinery/pkg/apis/meta/v1".ListMeta has no field or method MarshalToSizedBuffer)
/Users/niklashansson/Documents/go/pkg/mod/github.com/argoproj/argo@v2.5.2+incompatible/pkg/apis/workflow/v1alpha1/generated.pb.go:2583:36: m.LastScheduledTime.MarshalToSizedBuffer undefined (type *"k8s.io/apimachinery/pkg/apis/meta/v1".Time has no field or method MarshalToSizedBuffer)
/Users/niklashansson/Documents/go/pkg/mod/github.com/argoproj/argo@v2.5.2+incompatible/pkg/apis/workflow/v1alpha1/generated.pb.go:2596:33: m.Active[iNdEx].MarshalToSizedBuffer undefined (type "k8s.io/api/core/v1".ObjectReference has no field or method MarshalToSizedBuffer)
/Users/niklashansson/Documents/go/pkg/mod/github.com/argoproj/argo@v2.5.2+incompatible/pkg/apis/workflow/v1alpha1/generated.pb.go:2837:38: m.SSHPrivateKeySecret.MarshalToSizedBuffer undefined (type *"k8s.io/api/core/v1".SecretKeySelector has no field or method MarshalToSizedBuffer)
/Users/niklashansson/Documents/go/pkg/mod/github.com/argoproj/argo@v2.5.2+incompatible/pkg/apis/workflow/v1alpha1/generated.pb.go:2849:33: m.PasswordSecret.MarshalToSizedBuffer undefined (type *"k8s.io/api/core/v1".SecretKeySelector has no field or method MarshalToSizedBuffer)
/Users/niklashansson/Documents/go/pkg/mod/github.com/argoproj/argo@v2.5.2+incompatible/pkg/apis/workflow/v1alpha1/generated.pb.go:2861:33: m.UsernameSecret.MarshalToSizedBuffer undefined (type *"k8s.io/api/core/v1".SecretKeySelector has no field or method MarshalToSizedBuffer)
/Users/niklashansson/Documents/go/pkg/mod/github.com/argoproj/argo@v2.5.2+incompatible/pkg/apis/workflow/v1alpha1/generated.pb.go:2861:33: too many errors
# github.com/mattn/go-sqlite3

What did you expect to happen:

Expected the test to run successfully since I have not touched it.

Environment:

Mac, Let me know what more is needed.

How did you deploy Kubeflow Pipelines (KFP)?

KFP version:

KFP SDK version:

Anything else you would like to add:

[Miscellaneous information that will assist in solving the issue.]

/kind bug

Disclaimer: I am new to golang so this could be a super stupid question due to that I have not set up Go correctly ... :(
@numerology
Copy link

Hi @NikeNano thanks for trying on backend work!

One question, did you run the test using bazel test or something else?

@NikeNano
Copy link
Member Author

NikeNano commented Jun 10, 2020 •
edited
Loading

bazel test

Yes, I have tried to both do the build and run the tests. If I run
bazel test --action_env=PATH --define=grpc_no_ares=true //backend/...

Which returns:

Starting local Bazel server and connecting to it...
INFO: SHA256 (https://github.com/grpc-ecosystem/grpc-gateway/archive/v1.9.0.tar.gz) = bea3b9c3b8f979c448c935f6c5c2c06b4c0bac14300ceb0802cf3419bb92cba7
DEBUG: Rule 'com_github_grpc_ecosystem_grpc_gateway' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "bea3b9c3b8f979c448c935f6c5c2c06b4c0bac14300ceb0802cf3419bb92cba7"
DEBUG: Repository com_github_grpc_ecosystem_grpc_gateway instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_niklashansson/b4c0a60a6d8bdd6f03d97960bcedc1d2/external/bazel_tools/tools/build_defs/repo/http.bzl:336:16: in <toplevel>
INFO: SHA256 (https://codeload.github.com/googleapis/googleapis/zip/0ac60e21a1aa86c07c1836865b35308ba8178b05) = 78d08369bab24c1699b237a963da385d63fb0484f6a626de1ea3f23f7056d33b
INFO: Repository go_googleapis instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_niklashansson/b4c0a60a6d8bdd6f03d97960bcedc1d2/external/bazel_tools/tools/build_defs/repo/http.bzl:336:16: in <toplevel>
ERROR: An error occurred during the fetch of repository 'go_googleapis':
   Traceback (most recent call last):
	File "/private/var/tmp/_bazel_niklashansson/b4c0a60a6d8bdd6f03d97960bcedc1d2/external/bazel_tools/tools/build_defs/repo/http.bzl", line 121
		patch(ctx)
	File "/private/var/tmp/_bazel_niklashansson/b4c0a60a6d8bdd6f03d97960bcedc1d2/external/bazel_tools/tools/build_defs/repo/utils.bzl", line 135, in patch
		ctx.patch(patchfile, strip)
com.google.devtools.build.lib.syntax.EvalException: Error applying patch /private/var/tmp/_bazel_niklashansson/b4c0a60a6d8bdd6f03d97960bcedc1d2/external/io_bazel_rules_go/third_party/go_googleapis-directives.patch: Incorrect Chunk: the position of chunk > target size
**Original Position**: 26

**Original Content**:
# build_flags:
#  --experimental_enable_repo_mapping

workspace(name = "com_google_googleapis")

#
# grpc-java repository dependencies (required to by `java_grpc_library` bazel rule)

**Revised Content**:
# build_flags:
#  --experimental_enable_repo_mapping

workspace(name = "go_googleapis")

#
# grpc-java repository dependencies (required to by `java_grpc_library` bazel rule)
INFO: Repository com_google_protobuf instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_niklashansson/b4c0a60a6d8bdd6f03d97960bcedc1d2/external/bazel_tools/tools/build_defs/repo/http.bzl:336:16: in <toplevel>
INFO: Repository 'com_google_protobuf' used the following cache hits instead of downloading the corresponding file.
 * Hash '9510dd2afc29e7245e9e884336f848c8a6600a14ae726adb6befdb4f786f0be2' for https://github.com/protocolbuffers/protobuf/archive/v3.6.1.3.zip
If the definition of 'com_google_protobuf' was updated, verify that the hashes were also updated.
ERROR: /Users/niklashansson/Documents/go/src/github.com/kubeflow/pipelines/backend/api/BUILD.bazel:5:1: //backend/api:go_client_proto depends on @go_googleapis//google/api:annotations_proto in repository @go_googleapis which failed to fetch. no such package '@go_googleapis//google/api': Traceback (most recent call last):
	File "/private/var/tmp/_bazel_niklashansson/b4c0a60a6d8bdd6f03d97960bcedc1d2/external/bazel_tools/tools/build_defs/repo/http.bzl", line 121
		patch(ctx)
	File "/private/var/tmp/_bazel_niklashansson/b4c0a60a6d8bdd6f03d97960bcedc1d2/external/bazel_tools/tools/build_defs/repo/utils.bzl", line 135, in patch
		ctx.patch(patchfile, strip)
com.google.devtools.build.lib.syntax.EvalException: Error applying patch /private/var/tmp/_bazel_niklashansson/b4c0a60a6d8bdd6f03d97960bcedc1d2/external/io_bazel_rules_go/third_party/go_googleapis-directives.patch: Incorrect Chunk: the position of chunk > target size
**Original Position**: 26

**Original Content**:
# build_flags:
#  --experimental_enable_repo_mapping

workspace(name = "com_google_googleapis")

#
# grpc-java repository dependencies (required to by `java_grpc_library` bazel rule)

**Revised Content**:
# build_flags:
#  --experimental_enable_repo_mapping

workspace(name = "go_googleapis")

#
# grpc-java repository dependencies (required to by `java_grpc_library` bazel rule)
ERROR: Analysis of target '//backend/api:go_client_proto' failed; build aborted: no such package '@go_googleapis//google/api': Traceback (most recent call last):
	File "/private/var/tmp/_bazel_niklashansson/b4c0a60a6d8bdd6f03d97960bcedc1d2/external/bazel_tools/tools/build_defs/repo/http.bzl", line 121
		patch(ctx)
	File "/private/var/tmp/_bazel_niklashansson/b4c0a60a6d8bdd6f03d97960bcedc1d2/external/bazel_tools/tools/build_defs/repo/utils.bzl", line 135, in patch
		ctx.patch(patchfile, strip)
com.google.devtools.build.lib.syntax.EvalException: Error applying patch /private/var/tmp/_bazel_niklashansson/b4c0a60a6d8bdd6f03d97960bcedc1d2/external/io_bazel_rules_go/third_party/go_googleapis-directives.patch: Incorrect Chunk: the position of chunk > target size
**Original Position**: 26

**Original Content**:
# build_flags:
#  --experimental_enable_repo_mapping

workspace(name = "com_google_googleapis")

#
# grpc-java repository dependencies (required to by `java_grpc_library` bazel rule)

**Revised Content**:
# build_flags:
#  --experimental_enable_repo_mapping

workspace(name = "go_googleapis")

#
# grpc-java repository dependencies (required to by `java_grpc_library` bazel rule)
INFO: Elapsed time: 8.224s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (72 packages loaded, 12 targets configured)
FAILED: Build did NOT complete successfully (72 packages loaded, 12 targets configured)
    Fetching @local_config_cc_toolchains; fetching
    Fetching @go_sdk; fetching
    Fetching ...otobuf; Extracting /private/var/tmp/_bazel_niklashansson/b4c0a60a6d8bdd6f03d97960bcedc1d2/exter\
nal/com_google_protobuf/v3.6.1.3.zip

Thanks for the help @numerology!

@NikeNano
Copy link
Member Author

NikeNano commented Jun 11, 2020 •
edited
Loading

I managed to the environment and run the test on vs code inside a container. However I don't get the Bazel test to work ...

Starting local Bazel server and connecting to it...
INFO: SHA256 (https://github.com/grpc-ecosystem/grpc-gateway/archive/v1.9.0.tar.gz) = bea3b9c3b8f979c448c935f6c5c2c06b4c0bac14300ceb0802cf3419bb92cba7
DEBUG: Rule 'com_github_grpc_ecosystem_grpc_gateway' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "bea3b9c3b8f979c448c935f6c5c2c06b4c0bac14300ceb0802cf3419bb92cba7"
DEBUG: Repository com_github_grpc_ecosystem_grpc_gateway instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule http_archive defined at:
  /root/.cache/bazel/_bazel_root/595e728d121f60f59a14887cb2b2325a/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
INFO: SHA256 (https://codeload.github.com/googleapis/googleapis/zip/0ac60e21a1aa86c07c1836865b35308ba8178b05) = 78d08369bab24c1699b237a963da385d63fb0484f6a626de1ea3f23f7056d33b
INFO: Repository go_googleapis instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule http_archive defined at:
  /root/.cache/bazel/_bazel_root/595e728d121f60f59a14887cb2b2325a/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
ERROR: An error occurred during the fetch of repository 'go_googleapis':
   Traceback (most recent call last):
        File "/root/.cache/bazel/_bazel_root/595e728d121f60f59a14887cb2b2325a/external/bazel_tools/tools/build_defs/repo/http.bzl", line 121
                patch(ctx)
        File "/root/.cache/bazel/_bazel_root/595e728d121f60f59a14887cb2b2325a/external/bazel_tools/tools/build_defs/repo/utils.bzl", line 135, in patch
                ctx.patch(patchfile, strip)
com.google.devtools.build.lib.syntax.EvalException: Error applying patch /root/.cache/bazel/_bazel_root/595e728d121f60f59a14887cb2b2325a/external/io_bazel_rules_go/third_party/go_googleapis-directives.patch: Incorrect Chunk: the position of chunk > target size
**Original Position**: 26

**Original Content**:
# build_flags:
#  --experimental_enable_repo_mapping

workspace(name = "com_google_googleapis")

#
# grpc-java repository dependencies (required to by `java_grpc_library` bazel rule)

**Revised Content**:
# build_flags:
#  --experimental_enable_repo_mapping

workspace(name = "go_googleapis")

#
# grpc-java repository dependencies (required to by `java_grpc_library` bazel rule)
INFO: Repository go_sdk instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule _go_download_sdk defined at:
  /root/.cache/bazel/_bazel_root/595e728d121f60f59a14887cb2b2325a/external/io_bazel_rules_go/go/private/sdk.bzl:53:35: in <toplevel>
INFO: Repository com_google_protobuf instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule http_archive defined at:
  /root/.cache/bazel/_bazel_root/595e728d121f60f59a14887cb2b2325a/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
ERROR: /workspaces/pipelines/backend/api/BUILD.bazel:5:14: //backend/api:go_client_proto depends on @go_googleapis//google/api:annotations_proto in repository @go_googleapis which failed to fetch. no such package '@go_googleapis//google/api': Traceback (most recent call last):
        File "/root/.cache/bazel/_bazel_root/595e728d121f60f59a14887cb2b2325a/external/bazel_tools/tools/build_defs/repo/http.bzl", line 121
                patch(ctx)
        File "/root/.cache/bazel/_bazel_root/595e728d121f60f59a14887cb2b2325a/external/bazel_tools/tools/build_defs/repo/utils.bzl", line 135, in patch
                ctx.patch(patchfile, strip)
com.google.devtools.build.lib.syntax.EvalException: Error applying patch /root/.cache/bazel/_bazel_root/595e728d121f60f59a14887cb2b2325a/external/io_bazel_rules_go/third_party/go_googleapis-directives.patch: Incorrect Chunk: the position of chunk > target size
**Original Position**: 26

**Original Content**:
# build_flags:
#  --experimental_enable_repo_mapping

workspace(name = "com_google_googleapis")

#
# grpc-java repository dependencies (required to by `java_grpc_library` bazel rule)

**Revised Content**:
# build_flags:
#  --experimental_enable_repo_mapping

workspace(name = "go_googleapis")

#
# grpc-java repository dependencies (required to by `java_grpc_library` bazel rule)
ERROR: Analysis of target '//backend/api:go_client_proto' failed; build aborted: Analysis failed
INFO: Elapsed time: 13.458s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (73 packages loaded, 12 targets configured)
FAILED: Build did NOT complete successfully (73 packages loaded, 12 targets configured)

@numerology
Copy link

Perhaps I'm not a bazel expert. Do you mind sharing your setup info like Bazel version etc.?

Also /cc @neuromage

@alexlatchford
Copy link
Contributor

I can confirm this issue exists with both Bazel v2.2.0 and v3.2.0 (just tried both and get the same error as above).

@numerology
Copy link

I can confirm this issue exists with both Bazel v2.2.0 and v3.2.0 (just tried both and get the same error as above).

Thanks.

My experience is that using a pretty old bazel version should work (I am using 0.23.1 IIRC).

@Gsantomaggio
Copy link
Member

Gsantomaggio commented Jun 11, 2020 •
edited
Loading

Yes, you need to use an old version:
I just compiled it using:

$ bazel version
Build label: 0.24.1

here:

bazel test --action_env=PATH --define=grpc_no_ares=true //backend/...                                                    
...
INFO: Analysed 89 targets (506 packages loaded, 11336 targets configured).
INFO: Found 71 targets and 18 test targets...
INFO: Elapsed time: 14.919s, Critical Path: 4.59s
INFO: 36 processes: 36 linux-sandbox.
INFO: Build completed successfully, 19 total actions
//backend/src/agent/persistence/worker:go_default_test                   PASSED in 2.1s
//backend/src/apiserver/client:go_default_test                           PASSED in 0.1s
//backend/src/apiserver/common:go_default_test                           PASSED in 0.1s
//backend/src/apiserver/filter:go_default_test                           PASSED in 0.1s
//backend/src/apiserver/list:go_default_test                             PASSED in 0.1s
//backend/src/apiserver/model:go_default_test                            PASSED in 0.1s
//backend/src/apiserver/resource:go_default_test                         PASSED in 0.5s
//backend/src/apiserver/server:go_default_test                           PASSED in 0.6s
//backend/src/apiserver/storage:go_default_test                          PASSED in 0.5s
//backend/src/cache/client:go_default_test                               PASSED in 0.1s
//backend/src/cache/server:go_default_test                               PASSED in 0.1s
//backend/src/cache/storage:go_default_test                              PASSED in 0.1s
//backend/src/common/util:go_default_test                                PASSED in 0.5s
//backend/src/crd/controller/scheduledworkflow/client:go_default_test    PASSED in 0.1s
//backend/src/crd/controller/scheduledworkflow/util:go_default_test      PASSED in 0.1s
//backend/src/crd/controller/viewer/reconciler:go_default_test           PASSED in 0.1s
//backend/test/initialization:go_default_test                            PASSED in 0.1s
//backend/test/integration:go_default_test                               PASSED in 0.1s

Executed 18 out of 18 tests: 18 tests pass.
INFO: Build completed successfully, 19 total actions

see also: #3061 and #3461

@alexlatchford
Copy link
Contributor

Yep makes sense, can confirm it works with 0.23.1 after downgrading :) Thanks for the help!

@NikeNano NikeNano mentioned this issue Jun 11, 2020
Merged
@NikeNano
Copy link
Member Author

Thanks for the help @alexlatchford, @Gsantomaggio and @numerology it indeed seems to the the version. I also update the docs with it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@Gsantomaggio @alexlatchford @numerology @k8s-ci-robot @NikeNano