[Bug] BWC Rolling Upgrade Tests are failing on neural search due to changes in MLPredictionTaskRequest Object

[vibrantvarun]

What is the bug?
The BWC Rolling upgrade tests are failing in the neural search because of recent changes in ML-Commons.
So the scenario is when the bwc version is 2.10 and current version is 2.12.0-SNAPSHOT

./gradlew :qa:rolling-upgrade:testAgainstOneThirdUpgradedCluster -Dbwc.version=2.10.0 

The tests fails with a reason

org.opensearch.transport.RemoteTransportException: [neuralSearchBwcCluster-rolling-1][127.0.0.1:35231][cluster:admin/opensearch/ml/predict]
Caused by: java.lang.IllegalStateException: unexpected byte [0x48]
	at org.opensearch.core.common.io.stream.StreamInput.readBoolean(StreamInput.java:592) ~[opensearch-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at org.opensearch.core.common.io.stream.StreamInput.readBoolean(StreamInput.java:582) ~[opensearch-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at org.opensearch.ml.common.transport.prediction.MLPredictionTaskRequest.<init>(MLPredictionTaskRequest.java:56) ~[?:?]
	at org.opensearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:85) ~[opensearch-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at org.opensearch.transport.InboundHandler.newRequest(InboundHandler.java:283) [opensearch-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:243) [opensearch-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:133) [opensearch-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:115) [opensearch-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:767) [opensearch-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) [opensearch-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) [opensearch-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) [opensearch-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.101.Final.jar:4.1.101.Final]

It is because there is one more constructor for MLPredictionTaskRequest has been added in the PR mentioned above.

In BWC tests when cluster has 3 nodes and 2 of them are running on 2.10 and one node gets upgraded to 2.12.0-SNAPSHOT. Now
when neural search calls predict api then while initializing the MLPredictionTaskRequest here it fails at this line with error mentioned above.

It is happening because it is either getting more parameters or no parameters for boolean while executing in.readBoolean().

The same is happening when the BWC tests are executed for 2.9 OS version.

The tests are running fine for 2.11 OS version.
But it makes this tests execution flaky.
Because if Out of 3 nodes if it uses the node which is running 2.11 version for calling predict api rather than 2.12.0-SNAPSHOT then it will face the same issue.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Clone the neural search project
2. checkout on 2.x branch
3. Run ./gradlew :qa:rolling-upgrade:testAgainstOneThirdUpgradedCluster -Dbwc.version=2.10.0
4. See error

What is the expected behavior?
The BWC rolling upgrade tests should run with all previous version with no error.

What is your host/environment?

• OS: linux
• Version
• Plugins Neural Search

[vibrantvarun]

I have discussed the issue with @reta .

[b4sjoo]

I am still writing the test cases now, since in previous we only used built in function to test and we need to test neural models this time it’s actually like writing from very beginning.

[b4sjoo]

Can confirm bug reproduced in manually setup cluster when sending predict request to 2.11 cluster with 2.9 master node:

POST /_plugins/_ml/models/bsBlQI0Bqvn5_05_or5u/_predict
{
  "text_docs":[ "today is sunny"],
  "return_number": true,
  "target_response": ["sentence_embedding"]
}

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_state_exception",
        "reason": "unexpected byte [0x73]"
      }
    ],
    "type": "illegal_state_exception",
    "reason": "unexpected byte [0x73]"
  },
  "status": 500
}

[b4sjoo]

In the meantime we noticed that pretrained model registration tasks also failed from 2.9 to 2.11

Caused by: java.lang.IllegalStateException: Message fully read (request) but more data is expected for requestId [1421], action [cluster:admin/opensearch/mlinternal/forward]; resetting
	at org.opensearch.transport.InboundHandler.newRequest(InboundHandler.java:304) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:255) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:140) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:123) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:770) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-2.9.0.jar:2.9.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.94.Final.jar:4.1.94.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.io.EOFException
	at org.opensearch.core.common.io.stream.InputStreamStreamInput.readByte(InputStreamStreamInput.java:74) ~[?:?]
	at org.opensearch.core.common.io.stream.FilterStreamInput.readByte(FilterStreamInput.java:55) ~[?:?]
	at org.opensearch.core.common.io.stream.StreamInput.readOptionalBoolean(StreamInput.java:598) ~[?:?]
	at org.opensearch.ml.common.transport.register.MLRegisterModelInput.<init>(MLRegisterModelInput.java:163) ~[?:?]
	at org.opensearch.ml.common.transport.forward.MLForwardInput.<init>(MLForwardInput.java:63) ~[?:?]
	at org.opensearch.ml.common.transport.forward.MLForwardRequest.<init>(MLForwardRequest.java:43) ~[?:?]
	at org.opensearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:85) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.newRequest(InboundHandler.java:300) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:255) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:140) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:123) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:770) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) ~[transport-netty4-client-2.9.0.jar:2.9.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]

[b4sjoo]

Stack trace for the error:

[2024-01-25T12:21:09,697][ERROR][o.o.m.c.MLCommonsClassLoader] [data1] Failed to init instance for type TEXT_DOCS
java.lang.reflect.InvocationTargetException: null
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
"mlplugin-bwc.log" 971L, 138488B                                                    341,86        34%
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.IllegalStateException: unexpected byte [0x64]
        at org.opensearch.core.common.io.stream.StreamInput.readBoolean(StreamInput.java:592) ~[opensearch-core-2.11.0.jar:2.11.0]
        at org.opensearch.core.common.io.stream.StreamInput.readBoolean(StreamInput.java:582) ~[opensearch-core-2.11.0.jar:2.11.0]
        at org.opensearch.core.common.io.stream.StreamInput.readOptionalString(StreamInput.java:372) ~[opensearch-core-2.11.0.jar:2.11.0]
        at org.opensearch.ml.common.dataset.TextDocsInputDataSet.<init>(TextDocsInputDataSet.java:47) ~[?:?]
        ... 52 more

[b4sjoo]

After fixing, the predict still return the exception:

org.opensearch.transport.TransportSerializationException: Failed to deserialize response from handler [org.opensearch.transport.TransportService$ContextRestoreResponseHandler/org.opensearch.action.ActionListenerResponseHandler@32734b9b/org.opensearch.action.ActionListener$5@129fd05a]
	at org.opensearch.transport.InboundHandler.handleResponse(InboundHandler.java:375) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:160) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:114) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:769) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:94) [transport-netty4-client-2.9.0.jar:2.9.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.94.Final.jar:4.1.94.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.IllegalStateException: Message not fully read (response) for requestId [472], handler [org.opensearch.transport.TransportService$ContextRestoreResponseHandler/org.opensearch.action.ActionListenerResponseHandler@32734b9b/org.opensearch.action.ActionListener$5@129fd05a], error [false]; resetting
	at org.opensearch.transport.InboundHandler.checkStreamIsFullyConsumed(InboundHandler.java:341) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.handleResponse(InboundHandler.java:373) ~[opensearch-2.9.0.jar:2.9.0]
	... 26 more
[2024-01-27T00:03:25,979][WARN ][r.suppressed             ] [master1] path: /_plugins/_ml/models/2Ws3SI0Btppe6bLUosJi/_predict, params: {model_id=2Ws3SI0Btppe6bLUosJi}
org.opensearch.transport.RemoteTransportException: [Failed to deserialize response from handler [org.opensearch.transport.TransportService$ContextRestoreResponseHandler/org.opensearch.action.ActionListenerResponseHandler@32734b9b/org.opensearch.action.ActionListener$5@129fd05a]]
Caused by: org.opensearch.transport.TransportSerializationException: Failed to deserialize response from handler [org.opensearch.transport.TransportService$ContextRestoreResponseHandler/org.opensearch.action.ActionListenerResponseHandler@32734b9b/org.opensearch.action.ActionListener$5@129fd05a]
	at org.opensearch.transport.InboundHandler.handleResponse(InboundHandler.java:375) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:160) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:114) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:769) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:94) [transport-netty4-client-2.9.0.jar:2.9.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.94.Final.jar:4.1.94.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.IllegalStateException: Message not fully read (response) for requestId [472], handler [org.opensearch.transport.TransportService$ContextRestoreResponseHandler/org.opensearch.action.ActionListenerResponseHandler@32734b9b/org.opensearch.action.ActionListener$5@129fd05a], error [false]; resetting
	at org.opensearch.transport.InboundHandler.checkStreamIsFullyConsumed(InboundHandler.java:341) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.handleResponse(InboundHandler.java:373) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:160) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:114) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:769) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:94) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) **~[?:?]**

[b4sjoo]

It seems this is a change from 2.9.0 to 2.9.1 (Not release) at this line: