Add support for multi-region sweepers

[shuyama1]

What kind of contribution is this issue about?

Other (specify in details)

Details

Currently, all sweepers are running only in us-central1. However, it does not meet our requirement anymore, some services need to create resource in multiple regions in the tests, especially for regional resources.

References

No response

[SarahFrench]

Some context:

The sweepers in TeamCity run this command:

go test -v "%PACKAGE_PATH%" -sweep="%SWEEPER_REGIONS%" -sweep-allow-failures -sweep-run="%SWEEP_RUN%" -timeout 30m"

Where these values are configuration parameters set on the builds:

• PACKAGE_PATH
• SWEEPER_REGIONS
• SWEEP_RUN

The PACKAGE_PATH parameter is always set to ./google/sweeper or ./google-beta/sweeper as that's the entry point for sweepers.

SWEEPER_REGIONS is currently hardcoded to always be "us-central1", which I believe was carried over from the settings in the old, on premise TeamCity config for sweepers.

SWEEP_RUN determines what sweeper(s) run. It is set to "" in the service sweeper builds as this results in all sweepers running (except project sweeper, which needs an ENV to be set to run without skipping). In the project sweeper builds we provide the specific name of the project sweeper.

[SarahFrench]

Note from triage:

The solution for this is likely to be making sweepers stop using region values passed to sweepers from the go test command and instead make sweepers be templated using a user-defined list of regions that are relevant for a given resource/service.

E.g. this sweeper receives a region argument and it's used in the code, which corresponds to the SWEEPER_REGIONS value above. Instead the sweeper should know which region it should sweep.

TeamCity should be updated to provide no region value, as any value it provides will be unused.

[melinath]

TestAccComputeRegionSecurityPolicyRule_regionSecurityPolicyRuleNetworkMatchUpdate is an example of a test that provisions resources in a region not covered by the related sweeper.

[SarahFrench]

Another thing to bear in mind is that some sweepers are zonal but are being passed regions.

There could be a solution where we define the list of inputs (not regions per se) per resource, and TeamCity triggers multiple sweeper processes (within a single build or across multiple) for a given resource using that input.

[melinath]

#18909 is also impacted by the need for sweeping multiple regions.