Expose main_output on cc_common.link

[keith]

This field is useful for many situations, such as when you want to
create a native python extension, which doesn't follow the standard
library naming conventions.

[keith]

I didn't notice anything in the code about issues that need to be solved before this makes sense to be public, so I was hoping this PR could serve as the discussion point for that, cc @comius

[comius]

There's a lot of tech debt around this code, so I wouldn't expose it. It's mostly related to Apple and Android rules.

There's a design doc about the ideal end state: https://docs.google.com/document/d/1zm1UOftT2xHQiNNxNO7XU_BOn2KrXjFlx5tl4QBVjV0/edit#heading=h.5mcn15i0e1ch

I have an idea how to makes things nicer even without it, but I'll only be able to do it once Starlark cc_common.link lands. Currently link is feature complete, passing Hyrum depo-wide testing, but it has a large regression, which I just didn't have time to look into.

[keith]

thanks for the context. AFAICT in the open source world, rules_apple doesn't ever use main_output https://github.com/bazelbuild/rules_apple/blob/b4a1d1e71ee655ff6d6b51b5845757377febf321/apple/internal/linking_support.bzl#L430-L439 and neither does rules_swift https://github.com/bazelbuild/rules_swift/blob/1aec64c218fc057c2a836e67bd55bc514e0ef8bb/swift/internal/linking.bzl#L612-L624

This must have diverged internally? (even the upstream branches don't use these either)

Also worth noting that it looks like internally google has solved the python case that I am trying to solve using this API

bazel/src/main/starlark/builtins_bzl/common/python/py_executable.bzl

Line 518 in 99434b1

main_output = linked_lib,

, but externally this code is a no-op

bazel/src/main/starlark/builtins_bzl/common/python/py_executable.bzl

Lines 472 to 473 in 99434b1

	if not semantics.should_build_native_deps_dso(ctx):
	return struct(dso = None, runfiles = ctx.runfiles())

[keith]

@comius since it's been a while has there been any movement on your end around this stuff?

Looks to me like rules_python is still relying on these private APIs for the same reason, and now they're exposed through a new python specific shim for that reasons bazel-contrib/rules_python@6607a4a#diff-c93321fbbb76c6d72593e7c24e6b7d460e7a92dace1a7fd73123232e7a590bc0R525-R529

I still have this case internally but also have a new case where rules_mojo would like to provide a mojo_shared_library rule (ideally not a macro wrapping a cc_shared_library), and in order to support creating python extensions we also need this behavior. I imagine the same is true for rules_go and rules_rust but I guess not enough folks are creating native python extensions for them to have hit that case yet.

For other folks hitting this issue you can workaround the private API checks like this:

def _foo(rctx):
    rctx.file("BUILD.bazel", "")
    rctx.file("foo.bzl", """\
def link(**kwargs):
    return cc_common.link(**kwargs)
""")

foo = repository_rule(
    implementation = _foo,
)

# MODULE.bazel
foo = use_repo_rule("//:foo.bzl", "foo")

foo(name = "build_bazel_rules_android")

[keith]

past 26760a0 you need a new hack, instead of using a separate repository you need the bzl file calling the private API to be at a specific path.

doing tools/build_defs/android/link_hack.bzl works, example:

modular/rules_mojo@a972803

[keith]

bazelbuild/rules_cc#578

[aherrmann]

doing tools/build_defs/android/link_hack.bzl works

I tried that approach in aherrmann/rules_zig@5b5dfc6. It does indeed seem to work for targets within the same module, but I'm still seeing failures in the e2e tests that load rules_zig as an external module.

(10:31:52) ERROR: /home/runner/work/rules_zig/rules_zig/e2e/workspace/configure-use_cc_common_link/shared-library/BUILD.bazel:22:11: in zig_binary rule //configure-use_cc_common_link/shared-library:binary: 
Traceback (most recent call last):
	File "/home/runner/.cache/bazel-repo/bazel/_bazel_runner/bf578624e6d2c856d2af70bd6da49baf/external/rules_zig+/zig/private/zig_binary.bzl", line 51, column 41, in _zig_binary_impl
		build, build_groups = zig_build_impl(ctx, kind = "zig_binary")
	File "/home/runner/.cache/bazel-repo/bazel/_bazel_runner/bf578624e6d2c856d2af70bd6da49baf/external/rules_zig+/zig/private/common/zig_build.bzl", line 463, column 42, in zig_build_impl
		link_outputs = cc_common_link(
	File "/home/runner/.cache/bazel-repo/bazel/_bazel_runner/bf578624e6d2c856d2af70bd6da49baf/external/rules_zig+/tools/build_defs/android/cc_common_link.bzl", line 9, column 26, in cc_common_link
		return cc_common.link(*args, **kwargs)
	File "/home/runner/.cache/bazel-repo/bazel/_bazel_runner/bf578624e6d2c856d2af70bd6da49baf/external/rules_cc+/cc/private/cc_common.bzl", line 111, column 39, in _link
		_cc_internal.check_private_api(allowlist = _PRIVATE_STARLARKIFICATION_ALLOWLIST)
Error in check_private_api: file '@@rules_zig+//tools/build_defs/android:cc_common_link.bzl' cannot use private API

Am I missing something?

@keith have you seen that new link-hack approach work for use-sites of rules_mojo, i.e. other modules that depend on rules_mojo, or only in rules_mojo itself?

[keith]

I poked around the new change for a while and couldn't find a workaround. I think our best bet is to try and get the rules_cc change merged and released. In rules_mojo the feature is somewhat optional so I put the rules_cc patch there and downstream users would have to apply it.

[fmeum]

It wasn't my intention, but I also don't see a workaround that applies past my commit.

[aherrmann]

I've opened a PR to add rules_zig to the allowlist bazelbuild/rules_cc#589.