FieldSortIT.testCastDate fails reproducibly

[pgomulka]

Build scan:
https://gradle-enterprise.elastic.co/s/jtddpibu72kzs
Repro line:
./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.search.sort.FieldSortIT.testCastDate" -Dtests.seed=6F58A9AF4FD8005F -Dtests.security.manager=true -Dtests.locale=zh -Dtests.timezone=Asia/Qyzylorda -Druntime.java=11 -Dtests.fips.enabled=true
Reproduces locally?:
yes
Applicable branches:
master (have not checked others)
Failure history:

Failure excerpt:
Caused by: java.lang.IllegalArgumentException: date[+54280952-03-25T07:12:55.807Z] is after 2262-04-11T23:47:16.854775807 and cannot be stored in nanosecond resolution
07:27:18 at org.elasticsearch.common.time.DateUtils.toLong(DateUtils.java:224)
07:27:18 at org.elasticsearch.index.mapper.DateFieldMapper$Resolution$2.convert(DateFieldMapper.java:107)
07:27:18 at org.elasticsearch.index.mapper.DateFieldMapper$DateFieldType.parseToLong(DateFieldMapper.java:431)
07:27:18 at org.elasticsearch.index.mapper.DateFieldMapper$DateFieldType.isFieldWithinQuery(DateFieldMapper.java:462)
07:27:18 at org.elasticsearch.search.sort.FieldSortBuilder.isBottomSortShardDisjoint(FieldSortBuilder.java:403)
07:27:18 at org.elasticsearch.search.internal.ShardSearchRequest$RequestRewritable.rewrite(ShardSearchRequest.java:448)
07:27:18 at org.elasticsearch.search.internal.ShardSearchRequest$RequestRewritable.rewrite(ShardSearchRequest.java:429)
07:27:18 at org.elasticsearch.index.query.Rewriteable.rewrite(Rewriteable.java:68)
07:27:18 at org.elasticsearch.search.SearchService.canMatch(SearchService.java:1183)
07:27:18 at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:392)
07:27:18 at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:383)
07:27:18 at org.elasticsearch.search.SearchService.lambda$rewriteAndFetchShardRequest$7(Sea

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[pgomulka]

I have not investigated this from core/infra perspective yet, but looks like
assertEquals(1712879236854775807L, hits.getAt(0).getSortValues()[0]);
is expected.
That epoch milli is equal to +54280952-03-25T07:12:55.807Z

[pgomulka]

also the assertion seems to be almost right

            assertThat(exc.toString(), containsString("are after 2262"));

somehow it is singular in the exception thrown

 is after 2262

[cbuescher]

Just see another one today:
https://gradle-enterprise.elastic.co/s/7xvxfqm7dglsc

gradlew ':server:internalClusterTest' --tests "org.elasticsearch.search.sort.FieldSortIT.testCastDate"
-Dtests.seed=D6B25B5CCFB81908
-Dtests.security.manager=true
-Dtests.locale=fr
-Dtests.timezone=Etc/GMT+5
-Druntime.java=11

java.lang.AssertionError: 
Expected: a string containing "are before the epoch in 1970"
     but: was "Failed to execute phase [query], Partial shards failure; shardFailures {[X_u_7958Stitjf_XrN6GaQ][index_date_nanos][0]: org.elasticsearch.transport.RemoteTransportException: [node_s0][127.0.0.1:53643][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: date[+54280952-03-25T07:12:55.807Z] is after 2262-04-11T23:47:16.854775807 and cannot be stored in nanosecond resolution
	at org.elasticsearch.common.time.DateUtils.toLong(DateUtils.java:224)
	at org.elasticsearch.index.mapper.DateFieldMapper$Resolution$2.convert(DateFieldMapper.java:107)
	at org.elasticsearch.index.mapper.DateFieldMapper$DateFieldType.parseToLong(DateFieldMapper.java:431)
	at org.elasticsearch.index.mapper.DateFieldMapper$DateFieldType.isFieldWithinQuery(DateFieldMapper.java:462)
	at org.elasticsearch.search.sort.FieldSortBuilder.isBottomSortShardDisjoint(FieldSortBuilder.java:403)
	at org.elasticsearch.search.internal.ShardSearchRequest$RequestRewritable.rewrite(ShardSearchRequest.java:448)
	at org.elasticsearch.search.internal.ShardSearchRequest$RequestRewritable.rewrite(ShardSearchRequest.java:429)
	at org.elasticsearch.index.query.Rewriteable.rewrite(Rewriteable.java:68)
	at org.elasticsearch.search.SearchService.canMatch(SearchService.java:1183)
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:392)
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:383)

[not-napoleon]

I saw this fail on a community PR I'm reviewing:
https://gradle-enterprise.elastic.co/s/dbjq6ynusikvs

 ./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.search.sort.FieldSortIT.testCastDate"
 -Dtests.seed=3B88E397C70F0E35
 -Dtests.security.manager=true
 -Dtests.locale=en
 -Dtests.timezone=America/Caracas
 -Druntime.java=11

It did not reproduce locally against master for me with that command.

[nik9000]

There seems to be a timing issue here. If I add the seed to the test and debug into the method and take my time when I step over stuff it works. But if I just run it then it doesn't.

[nik9000]

I think this comes because the "bottom" sort values from other shards are in millis and the field that has nanos gets confused.