Remote execution of integration tests

[Eric-Arellano]

#7649 and #8051 added support for remote execution of unit tests in CI. Now, we want to have remote execution of integration tests, which is where we spend the bulk of our CI time.

This will be harder because integration tests (naively) depend on the entire pants.pex and not just individual files like we have for unit tests.

One of the main decisions is whether we have every integration tests depend on the entire Pants codebase, or only the actual parts possible. Likely in the beginning, we will take the naive approach of depending on the entire pants.pex as a first prototype. The downside is that anytime anyone makes a change to any file, the cache will be invalidated for all integration tests. As a followup, we could de-monotholize Pants to work around this.

Possible steps to land this:

• Run all tests with --chroot, which will catch several issues with implicit dependencies on being in the actual buildroot and better reflects how V2 works. migrate python tests in the pants repo to work with --chroot, --no-fast #7281 and Run contrib unit tests with --chroot #8084.
• Figure out how to get integration tests to depend on having pants.pex available to them. This may be tricky because that is a generated file. Some prior art is how we depend on native_engine.so, which is also a generated file:

  pants/src/python/pants/engine/BUILD

  Lines 210 to 212 in a996b57

  	resources(
  	name='native_engine_shared_library',
  	sources=['native_engine.so']

  • How should we ensure that this file is not out of date? Especially when testing locally, this is a likely issue, that the integration test is using an outdated version of pants.pex. This may also make local integration testing much happier if it's going to require regenerating the PEX every time!
• Get integration tests working for tests/python/pants_test/help:help_integration, both locally with V2 and with remoting.

We will almost certainly use a whitelist approach to remoting tests in CI. Even if we can only remote 1-5 integration tests in CI at first, we should land that because it establishes the infrastructure needed to port more tests.

[stuhood]

Likely in the beginning, we will take the naive approach of depending on the entire pants.pex as a first prototype.

Yea, we should. As tempted as I am to suggest that we should change this at the same time, let's try to avoid thinking about it, and just swap from throwing money at Travis to throwing it at RBE (where at least we think that we can get more parallelism). We can worry about reducing the amount of money next.

[stuhood]

We will almost certainly use a whitelist approach to remoting tests in CI. Even if we can only remote 1-5 integration tests in CI at first, we should land that because it establishes the infrastructure needed to port more tests.

So, in theory there is no need to have separate integration and unit test shards. And retries should be (relatively) cheap, because retrying should mostly hit the RBE/remoting cache.