exec_properties and remote execution platforms in Bazel 1.0

[eytankidron]

This is not exactly an issue as such. It is a description of how to use exec_properties in Bazel 1.0 and a doc for migrating from the old way of handling multiple remote execution platforms (used in Bazel versions before 1.0) to the new way of handling such platforms (used in Bazel versions 1.0 and up).

Background

Bazel 1.0 (and to a limited extend Bazel 0.29) introduces the exec_properties attribute which may be assigned to a platform or to a target. exec_properties in a platform is basically just a replacement of the deprecated remote_execution_properties. However, the ability to set exec_properties in a target is new.

If you have multiple platforms that only differ in execution properties, you can simplify your configuration. You would set exec_properties on the target and replace the multiple platforms by a single one.
Note that this simplification only applies if your targets do not need to select different toolchains.

Is this migration for me?

In order to figure out whether this migration would benefit your work space, let’s break the migration down to two parts:

• Part 1: Move platforms from remote_execution_properties to exec_properties.
• Part 2: Remove unnecessary platforms by replacing them with a single platform; replace exec_compatible_with attribute with the exec_properties attribute in the targets.

Part 1

If you have any platforms using the deprecated remote_execution_properties attribute or if you call rbe_autoconfig without setting the attribute use_legacy_platform_definition = False, then you probably want to do part 1 of the migration.
This migration allows platforms to be used by targets that populate their exec_properties attribute.

Part 2

The goal of part 2 of the migration is to simplify your setup and reduce the number of platforms. It is relevant for setups that have at least two platforms that only differ in remote execution properties. In this case, the execution properties can be set on the target and the extra platforms can be removed.

These platforms would typically have different constraints. This is how targets choose one platform over the other. If these constraints exist for the sole purpose of distinguishing between these platforms and do not affect toolchains in any way, this is a good indication that the platforms are good candidates for this collapsing.

Here is an example of such platforms. In this example we have two platforms, one for running on n1-standard-2 machines and one for running on n1-standard-8 machines (this example uses the remote_execution_properties attribute, which means it hasn’t yet undergone part 1 of the migration).

constraint_setting(name = "machine_size")

constraint_value(
    name = "large_machine",
    constraint_setting = ":machine_size",
)
constraint_value(
    name = "small_machine",
    constraint_setting = ":machine_size",
)

platform(	
    name = "small_machine_platform",	
    parents = ["@rbe_default//config:platform"],
    constraint_values = [
        ":small_machine",	
    ],	
    remote_execution_properties = """	
        {PARENT_REMOTE_EXECUTION_PROPERTIES}	
        properties: {	
          name: "gceMachineType"
          value: "n1-standard-2"
        }
        """,
)

platform(
    name = "large_machine_platform",
    parents = ["@rbe_default//config:platform"],
    constraint_values = [
        ":large_machine",
    ],
    remote_execution_properties = """	
        {PARENT_REMOTE_EXECUTION_PROPERTIES}
        properties: {
          name: "gceMachineType”
          value: "n1-standard-8"
        }
        """,
)

Another example of platforms that may be collapsed is the following pair of platforms:

• A platform created by rbe_autoconfig (e.g. @rbe_default//config:platform), and
• A platform that inherits from that platform and adds remote execution properties.

The reason why this collapsing of platforms is now possible is because rbe_autoconfig now accepts remote execution properties directly, which it did not in earlier versions.

Migration steps

Let’s break up the migration into two parts as outlined above:

Part 1

rbe_autoconfig

If you are calling rbe_autoconfig from your WORKSPACE file, add use_legacy_platform_definition = False as follows:

rbe_autoconfig(
    name = "rbe_default",
    use_legacy_platform_definition = False,
)

This will cause the underlying platform @rbe_default//config:platform to use exec_properties rather than remote_execution_properties.

platforms

In every platform in your work space replace remote_execution_properties with exec_properties.
Note that while remote_execution_properties receives a protobuf-as-string value, exec_properties receives a dict of string->string. If you are using RBE, do not create that dict manually. Instead use create_rbe_exec_properties_dict.

For example, this platform:

platform(
    # [...]
    remote_execution_properties = """	
        {PARENT_REMOTE_EXECUTION_PROPERTIES}
        properties: {
          name: "gceMachineType”
          value: "n1-standard-8"
        }
        """,
)

Would be replaced by this platform:

platform(
    # [...]
    exec_properties = create_rbe_exec_properties_dict(
        gce_machine_type = "n1-standard-8",
    ),
)

Deal with inter-repo dependency of platforms

Note that updating all the platforms in your repo from using remote_execution_properties to using exec_properties might not be sufficient if any of your platforms inherit from platforms from other repos. And vice versa; if there are platforms in other repos that inherit from your platforms, they might be broken by this change.
The reason for this is that platforms must either use remote_execution_properties or exec_properties. They cannot use a combination of both.

Part 2

Part 2 of the migration will assume that part 1 has completed and no platform is using the deprecated remote_execution_properties.

Identify platforms to be collapsed

Identify the platforms that can be collapsed into a single platform.
For these platforms, define:

• On what execution property (or in some cases, properties) these platforms differ,
• For each such property, what is the default value and what is the specialized value. The specialized value is the value that will need to be specified explicitly on the target.
  • Normally there will be a default platform that goes first in the order of platforms. The value of execution properties on this platform will be your default value.
  • Targets will use constraints to select other, non-default platforms. Look at the differences in the execution properties of the default and non-default platforms. These differences are the specialized constraints that need to be specified explicitly on the target.

Some examples are:

• Property: gceMachineType; Default: n1-standard-2; Specialized: n1-standard-8, or
• Property: dockerNetwork; Default: off; Specialized: standard.

Declare property values in WORKSPACE

When referencing an execution property value it is best to do so through a constant defined in the WORKSPACE file via a repository rule that creates a local rep. See GitHub for more about why this is considered best practice. It is typically enough to just define the specialized values as constants but for the sake of completeness, this migration process includes defining both the specialized values and the default values.

In order to define these constants we should distinguish between standard remote execution values and non-standard remote execution values.

• Standard execution values are defined using rbe_exec_properties. To see a full list of these, see the STANDARD_PROPERTY_SETS in exec_properties.bzl.
• Non-standard execution values are defined using custom_exec_properties.

The following code snippet, which would live in the WORKSPACE file, demonstrates both types:

load("@bazel_toolchains//rules/exec_properties:exec_properties.bzl",
        "create_rbe_exec_properties_dict",
        "custom_exec_properties",
        "rbe_exec_properties")


# rbe_exec_properties defines a set of standard execution properties. Among them
# are NETWORK_ON and NETWORK_OFF.
# They are then accessed using the following import command:
# load("@exec_properties//:constants.bzl", "NETWORK_ON", "NETWORK_OFF")
rbe_exec_properties(
    name = "exec_properties",
)

# custom_exec_properties defines a custom set of execution properties.
# In this case we define SMALL_MACHINE and LARGE_MACHINE.
# They are then accessed using the following import command:
# load("@my_repo_custom_exec_properties//:constants.bzl", "SMALL_MACHINE", "LARGE_MACHINE")
custom_exec_properties(
    name = "my_repo_custom_exec_properties",
    constants = {
        "SMALL_MACHINE": create_rbe_exec_properties_dict(
            gce_machine_type = "n1-standard-2",
        ),
        "LARGE_MACHINE": create_rbe_exec_properties_dict(
            gce_machine_type = "n1-standard-8",
        ),
    },
)

Update rbe_autoconfig with default values

The default remote execution property values are the remote execution property values that will apply to targets that do not choose to override them. These values should be set directly on your platform. If you are using rbe_autoconfig, you can pass them as an attribute to the rule.

The following is an example that creates, in the WORKSPACE file, an underlying platform with SMALL_MACHINE and NETWORK_OFF:

load("@exec_properties//:constants.bzl", "NETWORK_OFF")
load("@my_repo_custom_exec_properties//:constants.bzl", "SMALL_MACHINE")
load("@bazel_skylib//lib:dicts.bzl", "dicts")

rbe_autoconfig(
    name = "rbe_default",
    use_legacy_platform_definition = False,
    exec_properties = dicts.add(SMALL_MACHINE, NETWORK_OFF),
)

In the example above, specifying NETWORK_OFF is not actually necessary since off is the default value for dockerNetwork. It was added here just for clarity and symmetry (and to demonstrate the use of dicts.add). Setting exec_properties = SMALL_MACHINE would have been sufficient and equivalent in this case.

Update targets

Identify all the targets that need the specialized remote execution value. To find them, look for targets that use exec_compatible_with field to select non-default platforms. Remove these constraints and populate the exec_property field with the corresponding constant.

load("@exec_properties//:constants.bzl", "NETWORK_ON")
load("@my_repo_custom_exec_properties//:constants.bzl", "LARGE_MACHINE")

cc_test(
    "run_on_a_large_machine_test"
    …
    exec_property = LARGE_MACHINE,
)

cc_test(
    "run_with_networking_test"
    …
    exec_property = NETWORK_ON,
)

cc_test(
    "run_with_platforms_defaults_test"
    …
    # No need to specify exec_property. 
)

Clean up constraints and local platforms

At this point you should probably no longer be using the constraint settings and constraint values that were used to distinguish between platforms, and which were previously set in the exec_compatible_with attributes of the targets that you just updated.

You can now remove them.

Hopefully, the only place that these constraints are still used are in platforms, which can now be removed as well.

You may have local platforms which inherit from a base local platform, but add the just-deleted constraint. These platforms were necessary in order to locally run targets that were compatible with that constraint but are no longer necessary once the constraint is no longer being used.

Removing these platforms also entails registering other platforms (presumably just @rbe_default//config:platform) in their place. This registration can be in the WORKSPACE or through command line flags or bazelrc.

[buchgr]

@eytankidron you might want to publish this as a blog post?

[philwo]

Thanks for writing these great docs!

For future readers: It seems like the exec_property attribute was renamed to exec_properties for rules like cc_test in the meantime.

[sventiffe]

@juliexxia is this related to your work exec groups
@eytankidron any thoughts on Jakob's suggestion from last year?

[eytankidron]

@eytankidron any thoughts on Jakob's suggestion from last year?
Honestly, I haven't been working on bazel lately and I don't think I'll bother writing a blog post. But if anyone else is interested in writing one and linking to this issue, I would have no objections.