[CI] Failure in BWC on debian 8 tests with `_cat/nodes`. Error: Values less than -1 bytes are not supported

[benwtrent]

Multiple BWC tests fail in similar ways.

Values less than -1 bytes are not supported: -44383772672b

This occurs in BWC tests with 7.7.

Some failures:

https://gradle-enterprise.elastic.co/s/rpa3bpzgm7bkg

https://gradle-enterprise.elastic.co/s/or7qqkbuuazto

Here is one of the many stack traces:

Full stack trace

org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:38624], URI [/_cat/nodes?h=master%2Cversion&error_trace=true], status line [HTTP/1.1 400 Bad Request]
    {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -46886875136b","stack_trace":"[Values less than -1 bytes are not supported: -46886875136b]; nested: IllegalArgumentException[Values less than -1 bytes are not supported: -46886875136b];
 org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:644)
 org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:572)
 org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:138)
 org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:96)
 org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:91)
 org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:58)
 org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:49)
 org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:70)
 org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)
 org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:236)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:213)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:142)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:191)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:183)
 org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1129)
 org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:222)
 org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
 org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
 org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:214)
 org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:139)
 org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:103)
 org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:676)
 org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
 io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470)
 io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219)
 io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266)
 io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
 io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
 io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
 io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
 io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
 io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)
 io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)
 io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
 io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
 io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
 java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.lang.IllegalArgumentException: Values less than -1 bytes are not supported: -46886875136b
 org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:80)
 org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:75)
 org.elasticsearch.monitor.os.OsStats$Mem.getUsed(OsStats.java:256)
 org.elasticsearch.rest.action.cat.RestNodesAction.buildTable(RestNodesAction.java:316)
 org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:116)
 org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:113)
 org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37)
 org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47)
\t... 50 more
"}],"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -46886875136b","stack_trace":"java.lang.IllegalArgumentException: Values less than -1 bytes are not supported: -46886875136b
 org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:80)
 org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:75)
 org.elasticsearch.monitor.os.OsStats$Mem.getUsed(OsStats.java:256)
 org.elasticsearch.rest.action.cat.RestNodesAction.buildTable(RestNodesAction.java:316)
 org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:116)
 org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:113)
 org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37)
 org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47)
 org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:70)
 org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)
 org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:236)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:213)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:142)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:191)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:183)
 org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1129)
 org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:222)
 org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
 org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
 org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:214)
 org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:139)
 org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:103)
 org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:676)
 org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
 io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470)
 io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219)
 io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266)
 io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
 io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
 io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
 io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
 io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
 io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)
 io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)
 io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
 io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
 io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
 java.base/java.lang.Thread.run(Thread.java:832)
"},"status":400}
        at __randomizedtesting.SeedInfo.seed([30DEEF650CBC9654:34BFE11467792139]:0)
        at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:283)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:261)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
        at org.elasticsearch.xpack.test.rest.XPackRestTestHelper.lambda$waitForTemplates$0(XPackRestTestHelper.java:48)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:905)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:890)
        at org.elasticsearch.xpack.test.rest.XPackRestTestHelper.waitForTemplates(XPackRestTestHelper.java:44)

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/CAT APIs)

[benwtrent]

Here are some nifty server logs:
659.tar.gz

[benwtrent]

Related to PR: #54259

[dakrone]

@gwbrown I think we may need a way to be more lenient for BWC when receiving values from other nodes, as otherwise users may run into this problem with mixed version clusters, what do you think?

[dakrone]

Oh nevermind! sorry @gwbrown, I read this as relating to the TimeValue change but this is bytes, so ignore this!

[gwbrown]

Ha, I made the same mistake at first, then started wondering if I'd done some work on preventing negative byte values and just forgotten. Happy to help if you want assistance, of course!

[ywangd]

Another failure today:
https://gradle-enterprise.elastic.co/s/wthzhjyetha7c

Seems to be specific 7.x and debian08

https://gradle-enterprise.elastic.co/scans/tests?list.size=50&list.sortColumn=startTime&list.sortOrder=desc&search.buildToolType=gradle&search.buildToolType=maven&search.startTimeMax=1585699386158&search.startTimeMin=1585094586151&search.tags=CI&search.tzOffset=660&tests.container=org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT&tests.sortField=FAILED&tests.test=test%20%7Bp0%3Dold_cluster/10_basic/Create%20a%20task%20result%20record%20in%20the%20old%20cluster%7D&tests.unstableOnly=true

[williamrandolph]

It also appears that the affected tests are UpgradeClusterClientYamlTestSuiteIT and MlConfigIndexMappingsFullClusterRestartIT#testMlConfigIndexMappingsAfterMigration. \

Now that we have an error trace on the response, there's a stack trace in the failing response for the ML test that we didn't have when this ticket was filed:

java.lang.IllegalArgumentException: Values less than -1 bytes are not supported: -46886875136b
	at org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:80)
	at org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:75)
	at org.elasticsearch.monitor.os.OsStats$Mem.getUsed(OsStats.java:256)
	at org.elasticsearch.rest.action.cat.RestNodesAction.buildTable(RestNodesAction.java:316)
	at org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:116)
	at org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:113)
	at org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37)
	at org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47)
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:70)
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
	at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:236)
	at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:213)
	at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:142)
	at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:191)
	at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:183)
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1129)
	at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:222)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
	at org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:214)
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:139)
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:103)
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:676)
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.base/java.lang.Thread.run(Thread.java:832)

It appears that this error happens when OsStats.Mem has a larger value for free memory than for total memory:

        public ByteSizeValue getUsed() {
            return new ByteSizeValue(total - free);
        }

The first failure was on March 25th, not long after we cut the 7.7 branch. There have been consistent failures in the twice-daily bwc run for debian 8. Chart: gradle enterprise

[mayya-sharipova]

Recent failures for today on debian-8 and 7.x:

Log: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-unix-compatibility/os=debian-8&&immutable/667/console
Build scan: https://gradle-enterprise.elastic.co/s/zgglcpzvad2m6

Failing tests:
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=old_cluster/10_basic/Create a task result record in the old cluster}
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=old_cluster/10_basic/Create things in the cluster state that we'll validate are there after the upgrade}
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=old_cluster/10_basic/Index percolator queries and use the percolate query in old cluster}
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=old_cluster/20_date_range/Create index with java style index in 6}
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=old_cluster/20_date_range/Create index with joda style index that is incompatible with java.time (>6.1)}
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=old_cluster/20_date_range/Create index with joda style index that is incompatible with java.time. (6.0)}
org.elasticsearch.xpack.restart.MlConfigIndexMappingsFullClusterRestartIT testMlConfigIndexMappingsAfterMigration

The same error message:
method [GET], host [http://127.0.0.1:44582], URI [/_cat/nodes?h=version%2Cmaster], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -48852353024b"}]

[dimitris-athanasiou]

Another failure: Jenkins, Build scan

[romseygeek]

And another: https://gradle-enterprise.elastic.co/s/oauuxgrtd6i5e/console-log?task=:x-pack:qa:full-cluster-restart:v7.7.0%23oldClusterTest

[cbuescher]

Another instance of the the MlConfigIndexMappingsFullClusterRestartIT.testMlConfigIndexMappingsAfterMigration failure on 7.x:
https://gradle-enterprise.elastic.co/s/i73lmau2ago5w

rg.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:59558], URI [/_cat/nodes?h=master%2Cversion&error_trace=true], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -43149541376b","stack_trace":"[Values less than -1 bytes are not supported: -43149541376b]; nested: IllegalArgumentException[Values less than -1 bytes are not supported: -43149541376b];
at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:644)
at org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:572)
at org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:138)
at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:96)
at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:91)\n\tat