org.elasticsearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT failures

[jkakavas]

This failed in a PR build https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request-1/11310/console. I won't be muting it as it doesn't seem to reproduce locally.

https://gradle.com/s/mc2fttwxkboa2
https://gradle.com/s/of3fjbh6gv4kq

REPRODUCE WITH: ./gradlew :plugins:repository-s3:integTestMinioRunner \
-Dtests.seed=1C404B332F7967AD \
-Dtests.class=org.elasticsearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT \
-Dtests.method="test {yaml=repository_s3/20_repository_permanent_credentials/Register a repository with a non existing client}" -Dtests.security.manager=true -Dtests.locale=ar-LB -Dtests.timezone=America/Chihuahua -Dcompiler.java=12 -Druntime.java=8 -Dtests.rest.blacklist=repository_s3/30_repository_temporary_credentials/*,repository_s3/40_repository_ec2_credentials/*,repository_s3/50_repository_ecs_credentials/*

java.lang.AssertionError: Failure at [repository_s3/20_repository_permanent_credentials:21]: expected [2xx] status code but api [snapshot.delete] returned [429 Too Many Requests] [{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [498234232/475.1mb], which is larger than the limit of [493030604/470.1mb], real usage: [498234232/475.1mb], new bytes reserved: [0/0b]","bytes_wanted":498234232,"bytes_limit":493030604,"durability":"TRANSIENT","stack_trace":"CircuitBreakingException[[parent] Data too large, data for [<http_request>] would be [498234232/475.1mb], which is larger than the limit of [493030604/470.1mb], real usage: [498234232/475.1mb], new bytes reserved: [0/0b]]

Looks like a request to delete a snapshot returned a very large response that tripped the circuit breaker. There seems to have been quite a few failures in this class last week but not this one exactly

[elasticmachine]

Pinging @elastic/es-distributed

[original-brownbear]

Seems we just tripped the parent circuit breaker here (so it may not even be a huge request in the snapshot delete?) and it only happened this once. The other failures I see in the build stats were a result of an issue with the third party tests fixed in #40588. Maybe we just saw some random slowness here an it tripped the circuit breaker?

[original-brownbear]

I haven't seen this again after this issue was opened 3 weeks ago and I think it's likely not a specific S3 plugin issue as explained above -> closing here.