[CI] IngestGeoIpClientYamlTestSuiteIT class failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #4557 / ubuntu-2004-aarch64_checkrestcompat_platform-support-arm
• elasticsearch-periodic-platform-support #4517 / ubuntu-2004-aarch64_checkrestcompat_platform-support-arm
• elasticsearch-periodic-platform-support #4492 / ubuntu-2004-aarch64_checkrestcompat_platform-support-arm
• elasticsearch-periodic-platform-support #4476 / ubuntu-2004-aarch64_checkrestcompat_platform-support-arm
• elasticsearch-periodic-platform-support #4471 / amazonlinux-2_platform-support-aws

Reproduction Line:

./gradlew ':modules:ingest-geoip:yamlRestTestV7CompatTest' --tests "org.elasticsearch.ingest.geoip.IngestGeoIpClientYamlTestSuiteIT.test {yaml=ingest_geoip/20_geoip_processor/Test geoip processor with different database file - GeoLite2-Country}" -Dtests.seed=B8B3FDF7ACC10DE2 -Dtests.locale=en-SI -Dtests.timezone=Europe/Vaduz -Druntime.java=22

Applicable branches:
8.15

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: 
Expected: <4>
     but: was <3>

Issue Reasons:

• [8.15] 5 failures in class org.elasticsearch.ingest.geoip.IngestGeoIpClientYamlTestSuiteIT (0.7% fail rate in 739 executions)
• [8.15] 4 failures in step ubuntu-2004-aarch64_checkrestcompat_platform-support-arm (23.5% fail rate in 17 executions)
• [8.15] 5 failures in pipeline elasticsearch-periodic-platform-support (27.8% fail rate in 18 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[masseyke]

Here's the relevant log for at least one of these:

[2024-08-02T04:31:27,297][INFO ][o.e.c.r.a.AllocationService] [test-cluster-0] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.geoip_databases][0]]])." previous.health="YELLOW" reason="shards started [[.geoip_databases][0]]"	
[2024-08-02T04:31:30,921][INFO ][o.e.i.g.GeoIpDownloader  ] [test-cluster-0] successfully downloaded geoip database [GeoLite2-City.mmdb]	
[2024-08-02T04:31:31,027][INFO ][o.e.i.g.DatabaseNodeService] [test-cluster-0] successfully loaded geoip database file [GeoLite2-City.mmdb]	
[2024-08-02T04:31:32,183][INFO ][o.e.i.g.IngestGeoIpClientYamlTestSuiteIT] [test] Waiting for all cluster updates up to this moment to be processed	
[2024-08-02T04:31:32,595][INFO ][o.e.i.g.GeoIpDownloader  ] [test-cluster-0] successfully downloaded geoip database [GeoLite2-ASN.mmdb]	
[2024-08-02T04:31:32,605][INFO ][o.e.i.g.DatabaseNodeService] [test-cluster-0] successfully loaded geoip database file [GeoLite2-ASN.mmdb]	
[2024-08-02T04:31:32,623][INFO ][o.e.c.m.MetadataDeleteIndexService] [test-cluster-0] [.geoip_databases/6aJkWw-TQ2qmKr2gpD0qCg] deleting index	
[2024-08-02T04:31:32,664][ERROR][o.e.i.g.GeoIpDownloader  ] [test-cluster-0] error downloading geoip database [GeoLite2-Country.mmdb] [.geoip_databases] org.elasticsearch.index.IndexNotFoundException: no such index [.geoip_databases]	
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.notFoundException(IndexNameExpressionResolver.java:554)	
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$ExplicitResourceNameFilter.ensureAliasOrIndexExists(IndexNameExpressionResolver.java:1716)	
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$ExplicitResourceNameFilter.filterUnavailable(IndexNameExpressionResolver.java:1696)	
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.resolveExpressions(IndexNameExpressionResolver.java:265)	
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:340)	
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:331)	
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:90)	
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.replication.TransportBroadcastReplicationAction.shards(TransportBroadcastReplicationAction.java:183)	
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.replication.TransportBroadcastReplicationAction$1.accept(TransportBroadcastReplicationAction.java:94)	
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.replication.TransportBroadcastReplicationAction$1.accept(TransportBroadcastReplicationAction.java:83)	
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:100)	
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)	
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)	
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)	
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)	
	at java.base/java.lang.Thread.run(Thread.java:1570)

That's the first error described at #92888.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in class org.elasticsearch.ingest.geoip.IngestGeoIpClientYamlTestSuiteIT (0.8% fail rate in 240 executions)

Build Scans:

• elasticsearch-periodic-platform-support #3944 / ubuntu-2004-aarch64_checkrestcompat_platform-support-arm
• elasticsearch-pull-request #30090 / rest-compatibility

[samxbr]

This test suite was unmuted as part making GeoIP project aware to ensure they are not broken by the change, the tests seem to be passing. The tests were failing from yamlRestTestV7CompatTest, it's possible that it's no longer relevant. In the worst case the tests will fail be muted again.