Go container image build fails on macOS in 0.23.0 (worked in 0.22.0) due to toolchain resolution

[Xjs]

🐞 bug report

Affected Rule

The issue is caused by the rule: go_image

Is this a regression?

Yes, the previous version in which this bug was not present was: 0.22.0

Description

On macOS (I tested both macOS 12 on Apple Silicon and macOS 11 on Intel) go_image targets fail to build using rules_docker 0.23.0. The error message is:

ERROR: /private/var/tmp/_bazel_jannis/9dc304b8c991dee4d60914cc5c48b927/external/io_bazel_rules_go/BUILD.bazel:86:17: While resolving toolchains for target @io_bazel_rules_go//:cgo_context_data: No matching toolchains found for types @bazel_tools//tools/cpp:toolchain_type. Maybe --incompatible_use_cc_configure_from_rules_cc has been flipped and there is no default C++ toolchain added in the WORKSPACE file? See https://github.com/bazelbuild/bazel/issues/10134 for details and migration instructions.

(which is actually quite misleading). After some search I was able to pinpoint this regression to the rules_docker 0.23.0 release.

This appears to be a similar, but not exactly the same, issue as #2009.

🔬 Minimal Reproduction

The bug can be reproduced with https://github.com/Xjs/rules-docker-macos-regression .

🔥 Exception or Error

~/src/stuff/rules-docker-macos-regression> bazel build ... --define=VERBOSE_LOGS=1
WARNING: Download from https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.31.0/rules_go-v0.31.0.zip failed: class java.io.FileNotFoundException GET returned 404 Not Found
DEBUG: Rule 'io_bazel_rules_docker' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "85ffff62a4c22a74dbd98d05da6cf40f497344b3dbf1e1ab0a37ab2a1a6ca014"
DEBUG: Repository io_bazel_rules_docker instantiated at:
  /Users/jannis/src/stuff/rules-docker-macos-regression/WORKSPACE:32:13: in 
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_jannis/9dc304b8c991dee4d60914cc5c48b927/external/bazel_tools/tools/build_defs/repo/http.bzl:353:31: in 
INFO: Build option --define has changed, discarding analysis cache.
DEBUG: Rule 'go_static' indicated that a canonical reproducible form can be obtained by modifying arguments digest = "sha256:709eb1a2c147fd064900a5e8a3ff1f03b201a36f1f2954bb9e60f4953c5f641a"
DEBUG: Repository go_static instantiated at:
  /Users/jannis/src/stuff/rules-docker-macos-regression/WORKSPACE:62:15: in 
Repository rule container_pull defined at:
  /private/var/tmp/_bazel_jannis/9dc304b8c991dee4d60914cc5c48b927/external/io_bazel_rules_docker/container/pull.bzl:278:33: in 
ERROR: /private/var/tmp/_bazel_jannis/9dc304b8c991dee4d60914cc5c48b927/external/io_bazel_rules_go/BUILD.bazel:86:17: While resolving toolchains for target @io_bazel_rules_go//:cgo_context_data: No matching toolchains found for types @bazel_tools//tools/cpp:toolchain_type. Maybe --incompatible_use_cc_configure_from_rules_cc has been flipped and there is no default C++ toolchain added in the WORKSPACE file? See https://github.com/bazelbuild/bazel/issues/10134 for details and migration instructions.
ERROR: Analysis of target '//:layer' failed; build aborted: 
INFO: Elapsed time: 3.307s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 9868 targets configured)
    Fetching @local_config_cc; Building xcode-locator
    Fetching @com_github_pkg_errors; Restarting.
    Fetching @com_github_google_go_containerregistry; Restarting.
    Fetching @bazel_gazelle_go_repository_config; Restarting.
    Fetching @bazel_gazelle_go_repository_tools; fetching
Exception: bazel exited with 1
[tty 13], line 1: bazel build ... --define=VERBOSE_LOGS=1

🌍 Your Environment

Operating System:

  
macOS 11.6.4 (on Intel Core M)
macOS 12.3 (on Apple M1)
  

Output of bazel version:

On both machines:

  
Build label: 5.1.0
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Mar 24 14:10:21 2022 (1648131021)
Build timestamp: 1648131021
Build timestamp as int: 1648131021
  

Rules_docker version:

  
0.23.0
  

[jbgosselin]

Got same problem, did a git bisect between v0.23.0 and v0.22.0 and found that 76c708f was the commit where it brokes.

[linzhp]

container_image files too with Go target in it:

With the following workspace, bazel build --platforms @io_bazel_rules_go//go/toolchain:linux_amd64 //:image fails:

-- WORKSPACE --
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "io_bazel_rules_go",
    sha256 = "f2dcd210c7095febe54b804bb1cd3a58fe8435a909db2ec04e31542631cf715c",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.31.0/rules_go-v0.31.0.zip",
        "https://github.com/bazelbuild/rules_go/releases/download/v0.31.0/rules_go-v0.31.0.zip",
    ],
)

load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")

go_rules_dependencies()

go_register_toolchains(version = "1.18")

http_archive(
    name = "io_bazel_rules_docker",
    sha256 = "27d53c1d646fc9537a70427ad7b034734d08a9c38924cc6357cc973fed300820",
    strip_prefix = "rules_docker-0.24.0",
    urls = ["https://github.com/bazelbuild/rules_docker/releases/download/v0.24.0/rules_docker-v0.24.0.tar.gz"],
)

load(
    "@io_bazel_rules_docker//repositories:repositories.bzl",
    container_repositories = "repositories",
)
container_repositories()

load("@io_bazel_rules_docker//repositories:deps.bzl", container_deps = "deps")

container_deps()

load("@io_bazel_rules_docker//container:container.bzl", "container_pull")

container_pull(
    name = "distroless_base",
    registry = "gcr.io",
    repository = "distroless/base",
    digest = "sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5",
)
-- BUILD.bazel --
load("@io_bazel_rules_go//go:def.bzl", "go_binary")
load("@bazel_tools//tools/build_defs/pkg:pkg.bzl", "pkg_tar")
load("@io_bazel_rules_docker//container:container.bzl", "container_image")

go_binary(
    name = "hello",
    srcs = ["main.go"],
    visibility = ["//visibility:public"],
)

pkg_tar(
    name = "binaries_tar",
    srcs = [":hello"],
    mode = "0o755",
    package_dir = "/usr/bin",
)


container_image(
    name = "image",
    base = "@distroless_base//image",
    entrypoint = ["/usr/bin/hello"],
    tars = [
        ":binaries_tar",
    ],
    user = "root",
)
-- main.go --
package main

import "fmt"

func main() {
	fmt.Println("hello world")
}

[barm]

I have been running into the same issue starting with the same rules_docker version, but with the java_image rule.

I just figured out that I could work around the issue by passing the following argument:

--@io_bazel_rules_docker//transitions:enable=false

Our use-case is we generate our images with our CI running on x86 but we use Mac dev machines and want to be able to sanity check things locally as necessary, which means being able to run java_image locally.

We were holding back from doing an upgrade until this was fixed, so we've been pinned on 0.22.0. We're now able to upgrade to 0.25.0. I'm running Bazel version 5.2.0.

[barm]

The bisected commit mentioned above #2052 (comment) also touches the transitions feature.

[barm]

Okay did some more reading and it looks like this flag was introduced specifically to disable the feature added in this commit. This looks to be the recommended way of handling this behavior for now.

#1963 (comment)

Also worth mentioning, I've added this line to my bazelrc so I don't have to specify it explicitly:

build:macos --@io_bazel_rules_docker//transitions:enable=false

[barm]

@uhthomas fwiw I tried to use Bazel 6-pre in order to test out the optional toolchain support. Is there something specific I need to do to opt into that behavior?

[uhthomas]

@uhthomas fwiw I tried to use Bazel 6-pre in order to test out the optional toolchain support. Is there something specific I need to do to opt into that behavior?

What specifically are you trying to do? rules_docker and any rulesets which are not using optional toolchains will need to be updated to use them.

https://bazel.build/docs/toolchains#optional-toolchains

[nickgooding]

For anyone still struggling with this, I think I've found a (not so elegant) workaround that seems to do the trick without disabling transitions. This is great for us as it means we no longer have to supply --platforms when building container images.

Set up a "dummy" C++ toolchain:

# build/toolchains/dummy_toolchain.bzl
def _dummy_toolchain_impl(ctx):
    """Implementation of the dummy_toolchain rule."""
    return [
        platform_common.ToolchainInfo(),
    ]

dummy_toolchain = rule(
    implementation = _dummy_toolchain_impl,
    doc = """
A rule that can be used to create dummy toolchains, which can be useful when a
toolchain is only optionally required and setting up a real toolchain is hard.
""",
    provides = [platform_common.ToolchainInfo],
)

# build/toolchains/BUILD.bazel
load(":dummy_toolchain.bzl", "dummy_toolchain")

dummy_toolchain(
    name = "dummy_toolchain",
)

toolchain(
    name = "macos_dummy_cpp_toolchain",
    exec_compatible_with = [
        "@platforms//os:macos",
    ],
    target_compatible_with = [
        "@platforms//os:linux",
        "@platforms//cpu:x86_64",
    ],
    toolchain = ":dummy_toolchain",
    toolchain_type = "@bazel_tools//tools/cpp:toolchain_type",
)

# WORKSPACE
register_toolchains(
    # A dummy toolchain that enables compiling Go on MacOS for Linux without
    # setting up a real C++ toolchain. This should no longer be required once
    # rules_go supports optional toolchains, which are to be added in Bazel v6.
    "//build/toolchains:macos_dummy_cpp_toolchain",
)

then add to .bazelrc:

# To avoid setting up proper cross-compiling, just disable cgo
build --@io_bazel_rules_go//go/config:pure

Obviously this won't work at all if you actually do need cgo, but it seems to do the job for us.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_docker!

[kylekurz]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_docker!

Not stale, this still needs addressed.

[fishy]

I'm seeing the same issue with rules_go v0.38.1 and rules_docker v0.25.0. @nickgooding's workaround works for my case as well.

The fact that cgo is not used but got involved into the toolchain dependency tree, sounds like a bug to me.

[Anmol1696]

Any progress on this. Even with the latest version of Bazel, rules_docker facing the same issue.
Although --@io_bazel_rules_docker//transitions:enable=false does work, but I am curious to know if there is a better way.

[dwin]

Same issue.