Runfiles for go_proto_compiler plugins

[fischor]

What type of PR is this?
Bug fix
Feature

What does this PR do? Why is it needed?

Allow protoc plugins to work with runfiles.

Which issues(s) does this PR fix?

This is for #2704. However, it does not fix it yet.

Other notes for review

Firstly, I wanted to fix #2704.
As state there, plugins do not have access to their runfiles when the compile action of go_proto_compiler runs.
Here is a failing test: fischor@5a67a43

I've tried to pass the plugin as a tool to the compile action: fischor@459d7fd

Still, compiling the tests/core/go_proto_compiler/foo.proto file with the plugin does not work.
The runfiles can not be located.
The bazel.Runfile function returns an error: could not locate runfiles directory.

b test //tests/core/go_proto_compiler:executable_runfiles_test --sandbox_debug

DEBUG: /private/var/tmp/_bazel_rtf/981b61ef1c88e8af3a753ebe44c59757/external/bazel_toolchains/rules/rbe_repo/version_check.bzl:68:14:
Current running Bazel is ahead of bazel-toolchains repo. Please update your pin to bazel-toolchains repo in your WORKSPACE file.
DEBUG: /private/var/tmp/_bazel_rtf/981b61ef1c88e8af3a753ebe44c59757/external/bazel_toolchains/rules/rbe_repo/checked_in.bzl:125:14: buildkite_config not using checked in configs; Bazel version 4.1.0 was picked/selected but no checked in config was found in map {"0.20.0": ["8.0.0"], "0.21.0": ["8.0.0"], "0.22.0": ["8.0.0", "9.0.0"], "0.23.0": ["8.0.0", "9.0.0"], "0.23.1": ["8.0.0", "9.0.0"], "0.23.2": ["9.0.0"], "0.24.0": ["9.0.0"], "0.24.1": ["9.0.0"], "0.25.0": ["9.0.0"], "0.25.1": ["9.0.0"], "0.25.2": ["9.0.0"], "0.26.0": ["9.0.0"], "0.26.1": ["9.0.0"], "0.27.0": ["9.0.0"], "0.27.1": ["9.0.0"], "0.28.0": ["9.0.0"], "0.28.1": ["9.0.0"], "0.29.0": ["9.0.0"], "0.29.1": ["9.0.0", "10.0.0"], "1.0.0": ["9.0.0", "10.0.0"], "1.0.1": ["10.0.0"], "1.1.0": ["10.0.0"], "1.2.0": ["10.0.0"], "1.2.1": ["10.0.0"], "2.0.0": ["10.0.0"], "2.1.0": ["10.0.0"], "2.1.1": ["10.0.0", "11.0.0"], "2.2.0": ["11.0.0"], "3.0.0": ["11.0.0"], "3.1.0": ["11.0.0"], "3.2.0": ["11.0.0"], "3.3.0": ["11.0.0"], "3.3.1": ["11.0.0"], "3.4.1": ["11.0.0"], "3.5.0": ["11.0.0"], "3.5.1": ["11.0.0"], "3.6.0": ["11.0.0"], "3.7.0": ["11.0.0"], "3.7.1": ["11.0.0"], "3.7.2": ["11.0.0"], "4.0.0": ["11.0.0"]}
INFO: Analyzed target //tests/core/go_proto_compiler:executable_runfiles_test (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
ERROR: /Users/rtf/Github/fischor/rules_go/tests/core/go_proto_compiler/BUILD.bazel:33:17: Generating into bazel-out/darwin-fastbuild/bin/tests/core/go_proto_compiler/foo_go_proto_/github.com/bazelbuild/rules_go/tests/core/go_proto_compiler/foo failed: (Exit 1): sandbox-exec failed: error executing command
  (cd /private/var/tmp/_bazel_rtf/981b61ef1c88e8af3a753ebe44c59757/sandbox/darwin-sandbox/436/execroot/io_bazel_rules_go && \
  exec env - \
    APPLE_SDK_PLATFORM=MacOSX \
    APPLE_SDK_VERSION_OVERRIDE=11.3 \
    CGO_ENABLED=1 \
    DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer \
    GOARCH=amd64 \
    GOOS=darwin \
    GOPATH='' \
    GOROOT=external/go_sdk \
    GOROOT_FINAL=GOROOT \
    PATH=external/local_config_cc:/bin:/usr/bin \
    SDKROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk \
    TMPDIR=/var/folders/9b/ljy4fm6n5bl1jcvfm_v0cql80000gn/T/ \
    XCODE_VERSION_OVERRIDE=12.5.0.12E262 \
  /usr/bin/sandbox-exec -f /private/var/tmp/_bazel_rtf/981b61ef1c88e8af3a753ebe44c59757/sandbox/darwin-sandbox/436/sandbox.sb /var/tmp/_bazel_rtf/install/d07238c5957d0addf241b6be07f3c14b/process-wrapper '--timeout=0' '--kill_delay=15' bazel-out/darwin-opt-exec-2B5CBBC6/bin/go/tools/builders/go-protoc/go-protoc-bin -protoc bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/protoc -importpath github.com/bazelbuild/rules_go/tests/core/go_proto_compiler/foo -out_path bazel-out/darwin-fastbuild/bin/tests/core/go_proto_compiler/foo_go_proto_/ -plugin bazel-out/darwin-opt-exec-2B5CBBC6/bin/tests/core/go_proto_compiler/executable_runfiles_reset_plugin_/protoc-gen-executable_runfiles -descriptor_set bazel-out/darwin-fastbuild/bin/tests/core/go_proto_compiler/foo_proto-descriptor-set.proto.bin -expected bazel-out/darwin-fastbuild/bin/tests/core/go_proto_compiler/foo_go_proto_/github.com/bazelbuild/rules_go/tests/core/go_proto_compiler/foo/foo.executable_runfiles.pb.go -import 'tests/core/go_proto_compiler/foo.proto=github.com/bazelbuild/rules_go/tests/core/go_proto_compiler/foo' tests/core/go_proto_compiler/foo.proto)
panic: could not locate runfiles directory

goroutine 1 [running]:
main.main()
	tests/core/go_proto_compiler/executable_runfiles_plugin.go:18 +0x11e
--executable_runfiles_out: protoc-gen-executable_runfiles: Plugin failed with status code 2.
2021/06/17 12:18:40 error running protoc: exit status 1
Target //tests/core/go_proto_compiler:executable_runfiles_test failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.216s, Critical Path: 0.04s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully
//tests/core/go_proto_compiler:executable_runfiles_test         FAILED TO BUILD

FAILED: Build did NOT complete successfully

I can not see any special environment variable that is used by bazel.Runfiles to locate runfiles in the above command.
Also, when inspecting the sandbox environment of the above command, I can not locate the executable_runfiles.conf file.

cd /private/var/tmp/_bazel_rtf/981b61ef1c88e8af3a753ebe44c59757/sandbox/darwin-sandbox/436/execroot/io_bazel_rules_go
tree

.
└── bazel-out
    ├── darwin-fastbuild
    │   └── bin
    │       └── tests
    │           └── core
    │               └── go_proto_compiler
    │                   ├── foo_go_proto_
    │                   │   └── github.com
    │                   │       └── bazelbuild
    │                   │           └── rules_go
    │                   │               └── tests
    │                   │                   └── core
    │                   │                       └── go_proto_compiler
    │                   │                           └── foo
    │                   └── foo_proto-descriptor-set.proto.bin -> /private/var/tmp/_bazel_rtf/981b61ef1c88e8af3a753ebe44c59757/execroot/io_bazel_rules_go/bazel-out/darwin-fastbuild/bin/tests/core/go_proto_compiler/foo_proto-descriptor-set.proto.bin
    └── darwin-opt-exec-2B5CBBC6
        └── bin
            ├── external
            │   └── com_google_protobuf
            │       └── protoc -> /private/var/tmp/_bazel_rtf/981b61ef1c88e8af3a753ebe44c59757/execroot/io_bazel_rules_go/bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/protoc
            ├── go
            │   └── tools
            │       └── builders
            │           └── go-protoc
            │               └── go-protoc-bin -> /private/var/tmp/_bazel_rtf/981b61ef1c88e8af3a753ebe44c59757/execroot/io_bazel_rules_go/bazel-out/darwin-opt-exec-2B5CBBC6/bin/go/tools/builders/go-protoc/go-protoc-bin
            └── tests
                └── core
                    └── go_proto_compiler
                        └── executable_runfiles_reset_plugin_
                            └── protoc-gen-executable_runfiles -> /private/var/tmp/_bazel_rtf/981b61ef1c88e8af3a753ebe44c59757/execroot/io_bazel_rules_go/bazel-out/darwin-opt-exec-2B5CBBC6/bin/tests/core/go_proto_compiler/executable_runfiles_reset_plugin_/protoc-gen-executable_runfiles

26 directories, 4 files

You can modify the tests/core/go_proto_compiler/executable_runfiles_plugin.go to print out the content of the runfile.
When running the go_binary target, the runfile can be located.

diff --git a/tests/core/go_proto_compiler/executable_runfiles_plugin.go b/tests/core/go_proto_compiler/executable_runfiles_plugin.go
index b04372e0..afd2e9ea 100644
--- a/tests/core/go_proto_compiler/executable_runfiles_plugin.go
+++ b/tests/core/go_proto_compiler/executable_runfiles_plugin.go
@@ -22,6 +22,7 @@ func main() {
        if err != nil {
                panic(err)
        }
+       print(string(config))

        protogen.Options{}.Run(func(gen *protogen.Plugin) error {
                for _, f := range gen.Files {

b run //tests/core/go_proto_compiler:protoc-gen-executable_runfiles                                                                                                                                                                                          // ...
INFO: Analyzed target //tests/core/go_proto_compiler:protoc-gen-executable_runfiles (1 packages loaded, 3 targets configured).
INFO: Found 1 target...
Target //tests/core/go_proto_compiler:protoc-gen-executable_runfiles up-to-date:
  bazel-bin/tests/core/go_proto_compiler/protoc-gen-executable_runfiles_/protoc-gen-executable_runfiles
INFO: Elapsed time: 0.653s, Critical Path: 0.45s
INFO: 4 processes: 2 internal, 2 darwin-sandbox.
INFO: Build completed successfully, 4 total actions
INFO: Build completed successfully, 4 total actions
Hello World!

So I am assuming that the plugin works correctly, unless I miss something.
Can you help me out here? Why can't it locate the runfile when the go_proto_compile action is run?

---

I've also added a data attribute to the go_proto_compiler rule: fischor@00e36e5
That allows files to be passed to the plugin. When specifing the path additionally in options, the plugin can access the file.
I think that would be a better solution to provide configuration files to a protoc plugin.
Since when simply relying of runfiles of the executable, the executable has to be rebuild i.e. a new go_binary target has to be created for every compiler that uses a different configuration file.

go_proto_compiler(
    name = "data",
    data = ["data.conf"],    
    options = ["config=tests/core/go_proto_compiler/data.conf"],    
    plugin = ":protoc-gen-data",
    suffix = ".data.pb.go",
)

That works and should not break anything.
Test case for that: b test //tests/core/go_proto_compiler:data_test.

One thing that would be nice is if one could use the "Predefined label variables" for the paths specified in options.
But the Bazel docs also state that,

All referenced labels must appear in the consuming rule's srcs, output files, or deps. Otherwise the build fails. C++ rules can also reference labels in data.

So there is no way to use them with files specified in a data attribute? Or can I make the label variables work somehow?

[fischor]

The buildkite logs show that both relevant tests pass on Windows

//tests/core/go_proto_compiler:data_test                                 PASSED in 0.1s
//tests/core/go_proto_compiler:executable_runfiles_test                  PASSED in 0.1s

The //tests/core/go_proto_compiler:executable_runfiles_tests fails on Linux:
When trying to access its runfile (passed via go_library data attribute)

https://github.com/bazelbuild/rules_go/blob/6841be07fba973f0bd270ed9d40fa25238fd94ff/tests/core/go_proto_compiler/executable_runfiles_plugin.go#L17-L20

the plugin panics with could not locate runfiles directory.

[fischor]

Also there is now a failing test, this PR is no regression. That test would have failed before also.
Please review @jayconrod @ianthehat

[fmeum]

@fischor What you do in this PR looks reasonable, it's quite possible that bazel.Runfile just didn't behave correctly. We have replaced it with //go/runfiles, which mimics what the runfiles library for native runfiles does. Please take a look if you still need the functionality added by this PR.

[fischor]

@fmeum Thanks for the note. I rebased and updated to use //go/runfiles. The tests still fail with "no runfiles found":

ERROR: /home/rtf/github.com/fischor/rules_go/tests/core/go_proto_compiler/BUILD.bazel:33:17: Generating into bazel-out/k8-fastbuild/bin/tests/core/go_proto_compiler/foo_go_proto_/github.com/bazelbuild/rules_go/tests/core/go_proto_compiler/foo failed: (Exit 1): go-protoc-bin failed: error executing command bazel-out/k8-opt-exec-2B5CBBC6/bin/go/tools/builders/go-protoc/go-protoc-bin -protoc bazel-out/k8-opt-exec-2B5CBBC6/bin/proto/protoc/protoc -importpath ... (remaining 12 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
panic: runfiles: no runfiles found

goroutine 1 [running]:
main.main()
	tests/core/go_proto_compiler/executable_runfiles_plugin.go:19 +0xb6
--executable_runfiles_out: protoc-gen-executable_runfiles: Plugin failed with status code 2.
2022/11/27 21:40:21 error running protoc: exit status 1
ERROR: /home/rtf/github.com/fischor/rules_go/tests/core/go_proto_compiler/BUILD.bazel:75:17: Generating into bazel-out/k8-fastbuild/bin/tests/core/go_proto_compiler/foo_go_proto_with_data_/github.com/bazelbuild/rules_go/tests/core/go_proto_compiler/foo failed: (Exit 1): go-protoc-bin failed: error executing command bazel-out/k8-opt-exec-2B5CBBC6/bin/go/tools/builders/go-protoc/go-protoc-bin -protoc bazel-out/k8-opt-exec-2B5CBBC6/bin/proto/protoc/protoc -importpath ... (remaining 14 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
panic: runfiles: no runfiles found

goroutine 1 [running]:
main.main.func1(0xc000127220)
	tests/core/go_proto_compiler/data_plugin.go:24 +0x15e
google.golang.org/protobuf/compiler/protogen.run({0xc00001e5e0?, 0x0?}, 0xc000067f60)
	external/org_golang_google_protobuf/compiler/protogen/protogen.go:74 +0x142
google.golang.org/protobuf/compiler/protogen.Options.Run({0xc00001e5e0?, 0x0?}, 0x6?)
	external/org_golang_google_protobuf/compiler/protogen/protogen.go:52 +0x2a
main.main()
	tests/core/go_proto_compiler/data_plugin.go:18 +0xb1
--data_out: protoc-gen-data: Plugin failed with status code 2.
2022/11/27 21:40:21 error running protoc: exit status 1
INFO: Elapsed time: 18.724s, Critical Path: 9.11s
INFO: 197 processes: 7 internal, 190 linux-sandbox.
FAILED: Build did NOT complete successfully
//tests/core/go_proto_compiler:data_test                              NO STATUS
//tests/core/go_proto_compiler:executable_runfiles_test               NO STATUS

FAILED: Build did NOT complete successfully

I am currently not interested in pushing forward this PR. If you or anyone else is interested, feel free to take it over.