[BUG] Receiving EOFException[tried to read: 137217827 bytes but this stream is limited to: 398] on node start-up

[cwperks]

What is the bug?

After merging the latest changes of opensearch:main into opensearch:feature/identity I am now receiving the following exception when trying to startup a node using ./bin/opensearch after creating a localDistro, setting the feature flag to true and creating extensions.yml based on the latest in the developer guide of the sdk to include minimumCompatibleVersion. To ensure it wasn't related to any changes in opensearch:feature/identity I also created a localDistro based off of opensearch:main and obtained the same error message.

RemoteTransportException[[hello-world][127.0.0.1:4532][internal:discovery/extensions]]; nested: IllegalStateException[Message fully read (request) but more data is expected for requestId [3], action [internal:discovery/extensions]; resetting]; nested: EOFException[tried to read: 137217827 bytes but this stream is limited to: 398];
Likely root cause: java.io.EOFException: tried to read: 137217827 bytes but this stream is limited to: 398
	at org.opensearch.common.io.stream.InputStreamStreamInput.ensureCanReadBytes(InputStreamStreamInput.java:137)
	at org.opensearch.common.io.stream.FilterStreamInput.ensureCanReadBytes(FilterStreamInput.java:110)
	at org.opensearch.common.io.stream.StreamInput.readArraySize(StreamInput.java:1310)
	at org.opensearch.common.io.stream.StreamInput.readString(StreamInput.java:420)
	at org.opensearch.plugins.PluginInfo.<init>(PluginInfo.java:154)
	at org.opensearch.extensions.DiscoveryExtensionNode.<init>(DiscoveryExtensionNode.java:73)
	at org.opensearch.discovery.InitializeExtensionRequest.<init>(InitializeExtensionRequest.java:37)
	at org.opensearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:85)
	at org.opensearch.transport.InboundHandler.newRequest(InboundHandler.java:282)
	at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:242)
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:132)
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:114)
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:769)
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175)
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150)
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115)
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:94)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	<<<truncated>>>

How can one reproduce the bug?

1. Create a local distribution of OpenSearch: ./gradlew localDistro
2. Navigate to the local distro. For me its cd ./distribution/archives/darwin-tar/build/install/opensearch-3.0.0-SNAPSHOT
3. Enable extensions feature flag by adding opensearch.experimental.feature.extensions.enabled: true to config/opensearch.yml (Wait for Initialize feature flags before extensions manager OpenSearch#6082 to be merged to fix a bug in reading from opensearch.yml)
4. Create extensions/extensions.yml
5. Start the node using ./bin/opensearch

What is the expected behavior?

Node should start-up without error.

What is your host/environment?

Mac OS X

[saratvemulapalli]

Thanks for opening up the issue @cwperks.
DiscoveryExtensionNode data structures have changed in PR: opensearch-project/OpenSearch#6003
Could you rebase changes in identity and SDK?

[cwperks]

@saratvemulapalli Closing this issue, I had forgotten to perform ./gradlew publishToMavenLocal on the core side before running ./gradlew helloWorld. feature/identity has been synchronized recently.