Retry requests in tests where we run out of admin request quota

[skuruppu]

We get flaky test failure notifications regularly due to the following error (#944 and issues referencing it):

8 RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'Administrative requests' and limit 'Administrative requests per minute' of service 'spanner.googleapis.com' for consumer 'project_number:<project>'.
Error: 8 RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'Administrative requests' and limit 'Administrative requests per minute' of service 'spanner.googleapis.com' for consumer 'project_number:<project>'.

I requested a quota increase for the project but the Spanner team recommended that we "implement exponential back-off with extended total timeout and number of re-tries for admin operations".

I think this is reasonable to do in the system and samples tests, starting with the create instance and database requests.

[skuruppu]

@AVaksman if you can please get someone in your team to work on this, that would be great.

[olavloite]

@AVaksman One additional possibility is to (also) implement a client side rate limiter for the administrative requests. This still doesn't guarantee that the rate of administrative requests is exceeded, as it is only on a per-client basis, but it does reduce the possibility that it will happen.

The Java client library includes this, so it could be used as an example:

1. The rate limiter is defined here: https://github.com/googleapis/java-spanner/blob/2d19c25ba32847d116194565e67e1b1276fcb9f8/google-cloud-spanner/src/main/java/com/google/cloud/spanner/spi/v1/GapicSpannerRpc.java#L228
2. The methods that call the administrative RPCs must first acquire the rate limiter before they are allowed to proceed. See example here: https://github.com/googleapis/java-spanner/blob/2d19c25ba32847d116194565e67e1b1276fcb9f8/google-cloud-spanner/src/main/java/com/google/cloud/spanner/spi/v1/GapicSpannerRpc.java#L746

As mentioned above, this still doesn't guarantee that the rate will never be exceeded, but it should guarantee that a single client won't exceed the rate, which is normally what happens in system tests.

The rate that we are talking about here is the one mentioned here: https://cloud.google.com/spanner/quotas#administrative_limits. As noted in the footer of that documentation, this rate also applies to operations like getting or polling a long-running operation.

[bcoe]

👋 just happened to see this in triage. If you do happen to go the retry route, note that mocha has this.retries built in to the test runner these days:

googleapis/nodejs-monitoring#402

[AVaksman]

Currently offering a PR #959 that, among other improvements, refactors system-tests to reduce the number of database and table creations and re-use already created resources.

[AVaksman]

Another clarification, @skuruppu, is the quota for Administrative requests per project or per instance?
Prior to PR #855 samples-test used to run on a separate instance

[olavloite]

@AVaksman The Administrative requests limit is per project (or actually: per user per project, but as the tests all use the same user, it is effectively per project). The limit is max 5 per second averaged over 100 seconds.

[skuruppu]

This is becoming high priority since we're currently not able to run a couple of presubmits concurrently without buildcop filing dozens of issues for failing tests.