[Triple][3.3.6] First client-streaming call after consumer restart may fail with malformed gRPC frame and IllegalReferenceCountException

[1536191986zh-cloud]

Pre-check

• I am sure that all the content I provide is in English.

Search before asking

• I had searched in the issues and found no similar issues.

Apache Dubbo Component

Java SDK (apache/dubbo)

Dubbo Version

Apache Dubbo：3.3.6
JDK：17

Steps to reproduce this issue

1. Start a Triple provider in one JVM with a client-streaming RPC:
   rpc Upload(stream UploadFrame) returns (UploadResult);

2. Start a new consumer JVM and immediately invoke Upload:
   send the metadata frame first, then send one or more file-content frames.

3. The first call after the consumer restart may fail. Subsequent calls usually succeed without restarting either side.

4. The issue is reproducible with both:

   • a single Dubbo port with ext-protocol: tri
   • native tri protocol

5. The payload can be very small (for example, 16 KB); it is not dependent on a large gRPC frame.

Actual result

The consumer receives:

org.apache.dubbo.rpc.StatusRpcException: UNKNOWN : gRPC frame header malformed:
reserved bits not zero
    at org.apache.dubbo.rpc.TriRpcStatus.asException(TriRpcStatus.java:260)
    at org.apache.dubbo.rpc.protocol.tri.call.ObserverToClientCallListenerAdapter.onClose(
        ObserverToClientCallListenerAdapter.java:52)
    at org.apache.dubbo.rpc.protocol.tri.call.TripleClientCall.onComplete(
        TripleClientCall.java:126)
    at org.apache.dubbo.rpc.protocol.tri.stream.AbstractTripleClientStream$
        ClientTransportListener.finishProcess(AbstractTripleClientStream.java:224)


The provider 

<!-- Failed to upload "dubbo-triple-first-call-repro-standalone.zip" -->

receives:

[dubbo-triple-first-call-repro-standalone.zip](https://github.com/user-attachments/files/30777940/dubbo-triple-first-call-repro-standalone.zip)

io.netty.util.IllegalReferenceCountException: refCnt: 0
    at io.netty.buffer.AbstractByteBuf.ensureAccessible(AbstractByteBuf.java:1454)
    at io.netty.buffer.ByteBufInputStream.read(ByteBufInputStream.java:183)
    at org.apache.dubbo.remoting.http12.CompositeInputStream.read(
        CompositeInputStream.java:82)
    at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder
        .processHeader(LengthFieldStreamingDecoder.java:147)
    at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder
        .deliver(LengthFieldStreamingDecoder.java:117)
    at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder
        .decode(LengthFieldStreamingDecoder.java:69)
    at org.apache.dubbo.rpc.protocol.tri.h12.http2.
        GenericHttp2ServerTransportListener$StreamingHttpMessageListener.onMessage(
        GenericHttp2ServerTransportListener.java:220)

### What you expected to happen

The first client-streaming Triple call after a consumer restart should be handled
the same as subsequent calls.

The provider should retain the received buffers until
`LengthFieldStreamingDecoder` finishes decoding them, so no malformed gRPC frame
is reported and no `IllegalReferenceCountException: refCnt: 0` occurs.

If a transport failure does occur, it should be propagated through the normal
stream error callback so that application resources can be released safely.

### Anything else

- Apache Dubbo: 3.3.6
- JDK: 17
- Transport: Triple / HTTP2
- Provider and consumer run in separate JVM processes.
- The failure also occurs with a 16 KB payload, so it does not appear to be
  caused by exceeding the default gRPC frame-size limit.
- We reproduced it with both native `tri` and single-port `dubbo` plus
  `ext-protocol: tri`; therefore it does not appear to be specific to the
  single-port configuration.
- The minimal two-process demo is nondeterministic, but the production symptom
  consistently occurs on the first call after the consumer restarts. We can
  provide the demo if needed.

### Do you have a (mini) reproduction demo?

- [x] Yes, I have a minimal reproduction demo to help resolve this issue more effectively!

### Are you willing to submit a pull request to fix on your own?

- [x] Yes I am willing to submit a pull request on my own!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

[LI123456mo]

Hi @1536191986zh-cloud , let me work on this

[Anon2Tokyo]

hi, I can take this issue.

[LI123456mo]

hi, I can take this issue.

You can work on it

[Anon2Tokyo]

@1536191986zh-cloud
I investigated this issue and found that it is caused by decoder state loss in the no-stub gRPC provider path.

Root cause

This only affects the no-stub provider path, where GrpcHttp2ServerTransportListener first uses LazyFindMethodListener to parse the initial client-streaming message and then switches to the real method listener. The problematic case is when the first HTTP/2 DATA frame contains more than one gRPC message fragment, for example:

[complete message #1][beginning of message #2]

In the old implementation, LazyFindMethodListener created a temporary GrpcStreamingDecoder to decode the first message. If the same DATA frame also contained the beginning of the second message, that partial second message was buffered inside the temporary decoder.

After the listener switched to the real method listener, the real streaming decoder did not have that buffered state. As a result, decoding continued from an incorrect message boundary, which could lead to protobuf parse errors, invalid compression/reserved-bit errors, or Netty buffer reference-count errors.

This is why the issue is hard to reproduce locally: it depends on HTTP/2 DATA frame splitting/coalescing timing. It is more likely with:

• no-stub provider
• client streaming
• many fast consecutive onNext calls
• a very small first message
• larger following messages, for example 128KB
• no artificial delay or logging inside the sending loop

Fix

The fix is to make LazyFindMethodListener use the transport's actual streaming decoder instead of creating a temporary decoder.

So instead of decoding the first message with a separate new GrpcStreamingDecoder(), it now reuses getStreamingDecoder().

This preserves any buffered partial gRPC message across the transition from LazyFindMethodListener to the real method listener.

Verification

I added a regression test that simulates this exact shape:

DATA frame = [full first gRPC message][partial second gRPC message]

The test verifies that after the listener switch, the remaining bytes of the second message are still decoded correctly.

I also reproduced the issue against Dubbo 3.3.6 using a no-stub provider and client-streaming calls. With 20 messages of 128KB sent quickly, one run failed at iteration 17 with server-side decode errors including LengthFieldStreamingDecoder, InvalidProtocolBufferException, and IllegalReferenceCountException: refCnt: 0.

provider log：

2026-08-08 02:34:09.244 ERROR 10324 --- [0051-thread-200] .r.p.t.h.AbstractServerTransportListener :  [DUBBO] An error occurred while processing the http request with GrpcHttp2ServerTransportListener, Http2MetadataFrame{method='POST', path='/repro.api.UploadFacade/upload', contentType='application/grpc+proto', streamId=3, endStream=false}, method=repro.api.UploadFacade.upload(StreamObserver), dubbo version: 3.3.6, current host: 192.168.0.115, error code: 99-0. This may be caused by , go to https://dubbo.apache.org/faq/99/0 to find instructions. 

org.apache.dubbo.remoting.http12.exception.DecodeException: Internal Server Error
	at org.apache.dubbo.rpc.protocol.tri.h12.grpc.GrpcCompositeCodec.decode(GrpcCompositeCodec.java:105) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.protocol.tri.h12.grpc.GrpcCompositeCodec.decode(GrpcCompositeCodec.java:111) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.remoting.http12.message.HttpMessageDecoder.decode(HttpMessageDecoder.java:56) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.remoting.http12.message.DefaultListeningDecoder.decode(DefaultListeningDecoder.java:41) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.remoting.http12.message.StreamingDecoder$DefaultFragmentListener.onFragmentMessage(StreamingDecoder.java:55) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.invokeListener(LengthFieldStreamingDecoder.java:180) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.processBody(LengthFieldStreamingDecoder.java:172) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.deliver(LengthFieldStreamingDecoder.java:121) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.decode(LengthFieldStreamingDecoder.java:69) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.protocol.tri.h12.http2.GenericHttp2ServerTransportListener$StreamingHttpMessageListener.onMessage(GenericHttp2ServerTransportListener.java:220) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.protocol.tri.h12.AbstractServerTransportListener.doOnData(AbstractServerTransportListener.java:183) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.protocol.tri.h12.AbstractServerTransportListener.lambda$onData$1(AbstractServerTransportListener.java:168) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.common.threadpool.serial.SerializingExecutor.run(SerializingExecutor.java:111) ~[dubbo-3.3.6.jar:3.3.6]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
	at org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:39) ~[dubbo-3.3.6.jar:3.3.6]
	at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
Caused by: java.io.IOException: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either that the input has been truncated or that an embedded message misreported its own length.
	at org.apache.dubbo.rpc.protocol.tri.SingleProtobufUtils.deserialize(SingleProtobufUtils.java:104) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.protocol.tri.PbUnpack.unpack(PbUnpack.java:35) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.model.PackableMethod.parseRequest(PackableMethod.java:26) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.protocol.tri.h12.grpc.GrpcCompositeCodec.decode(GrpcCompositeCodec.java:101) ~[dubbo-3.3.6.jar:3.3.6]
	... 16 common frames omitted
Caused by: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either that the input has been truncated or that an embedded message misreported its own length.
	at com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:92) ~[protobuf-java-3.25.5.jar:na]
	at com.google.protobuf.CodedInputStream$StreamDecoder.readRawBytesSlowPathRemainingChunks(CodedInputStream.java:2950) ~[protobuf-java-3.25.5.jar:na]
	at com.google.protobuf.CodedInputStream$StreamDecoder.readBytesSlowPath(CodedInputStream.java:2987) ~[protobuf-java-3.25.5.jar:na]
	at com.google.protobuf.CodedInputStream$StreamDecoder.readBytes(CodedInputStream.java:2373) ~[protobuf-java-3.25.5.jar:na]
	at repro.upload.UploadFrame$Builder.mergeFrom(UploadFrame.java:452) ~[classes/:na]
	at repro.upload.UploadFrame$1.parsePartialFrom(UploadFrame.java:648) ~[classes/:na]
	at repro.upload.UploadFrame$1.parsePartialFrom(UploadFrame.java:640) ~[classes/:na]
	at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:192) ~[protobuf-java-3.25.5.jar:na]
	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:209) ~[protobuf-java-3.25.5.jar:na]
	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:25) ~[protobuf-java-3.25.5.jar:na]
	at org.apache.dubbo.rpc.protocol.tri.SingleProtobufUtils$SingleMessageMarshaller.parse(SingleProtobufUtils.java:132) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.protocol.tri.SingleProtobufUtils.deserialize(SingleProtobufUtils.java:102) ~[dubbo-3.3.6.jar:3.3.6]
	... 19 common frames omitted

2026-08-08 02:34:09.253 ERROR 10324 --- [0051-thread-200] .r.p.t.h.AbstractServerTransportListener :  [DUBBO] An error occurred while processing the http request with GrpcHttp2ServerTransportListener, Http2MetadataFrame{method='POST', path='/repro.api.UploadFacade/upload', contentType='application/grpc+proto', streamId=3, endStream=false}, method=repro.api.UploadFacade.upload(StreamObserver), dubbo version: 3.3.6, current host: 192.168.0.115, error code: 99-0. This may be caused by , go to https://dubbo.apache.org/faq/99/0 to find instructions. 

io.netty.util.IllegalReferenceCountException: refCnt: 0
	at io.netty.buffer.AbstractByteBuf.ensureAccessible(AbstractByteBuf.java:1454) ~[netty-buffer-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1440) ~[netty-buffer-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.buffer.AbstractByteBuf.checkReadableBytes(AbstractByteBuf.java:1428) ~[netty-buffer-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:895) ~[netty-buffer-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.buffer.ByteBufInputStream.read(ByteBufInputStream.java:183) ~[netty-buffer-4.1.101.Final.jar:4.1.101.Final]
	at org.apache.dubbo.remoting.http12.CompositeInputStream.read(CompositeInputStream.java:82) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.readRawMessage(LengthFieldStreamingDecoder.java:185) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.protocol.tri.h12.grpc.GrpcStreamingDecoder.readRawMessage(GrpcStreamingDecoder.java:54) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.processBody(LengthFieldStreamingDecoder.java:170) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.deliver(LengthFieldStreamingDecoder.java:121) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.decode(LengthFieldStreamingDecoder.java:69) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.protocol.tri.h12.http2.GenericHttp2ServerTransportListener$StreamingHttpMessageListener.onMessage(GenericHttp2ServerTransportListener.java:220) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.protocol.tri.h12.AbstractServerTransportListener.doOnData(AbstractServerTransportListener.java:183) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.protocol.tri.h12.AbstractServerTransportListener.lambda$onData$1(AbstractServerTransportListener.java:168) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.common.threadpool.serial.SerializingExecutor.run(SerializingExecutor.java:111) ~[dubbo-3.3.6.jar:3.3.6]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
	at org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:39) ~[dubbo-3.3.6.jar:3.3.6]
	at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]

2026-08-08 02:34:09.281  INFO 10324 --- [rverWorker-3-21] o.a.d.r.t.netty4.NettyServerHandler      :  [DUBBO] The connection [id: 0xf714b008, L:/192.168.0.115:50051 ! R:/192.168.0.115:56989] of 192.168.0.115:56989 -> 192.168.0.115:50051 is disconnected., dubbo version: 3.3.6, current host: 192.168.0.115
2026-08-08 02:34:09.282  INFO 10324 --- [rverWorker-3-21] o.a.d.r.t.n.NettyPortUnificationServer   :  [DUBBO] All clients has disconnected from /192.168.0.115:50051. You can graceful shutdown now., dubbo version: 3.3.6, current host: 192.168.0.115
2026-08-08 02:34:09.282  INFO 10324 --- [rverWorker-3-21] o.a.d.r.t.netty4.NettyChannelHandler     :  [DUBBO] The connection [id: 0xf714b008, L:/192.168.0.115:50051 ! R:/192.168.0.115:56989] of 192.168.0.115:56989 -> 192.168.0.115:50051 is disconnected., dubbo version: 3.3.6, current host: 192.168.0.115

[Anon2Tokyo]

I will submit a PR.

[LI123456mo]

@1536191986zh-cloud I investigated this issue and found that it is caused by decoder state loss in the no-stub gRPC provider path.

Root cause

This only affects the no-stub provider path, where GrpcHttp2ServerTransportListener first uses LazyFindMethodListener to parse the initial client-streaming message and then switches to the real method listener. The problematic case is when the first HTTP/2 DATA frame contains more than one gRPC message fragment, for example:

[complete message #1][beginning of message #2]

In the old implementation, LazyFindMethodListener created a temporary GrpcStreamingDecoder to decode the first message. If the same DATA frame also contained the beginning of the second message, that partial second message was buffered inside the temporary decoder.

After the listener switched to the real method listener, the real streaming decoder did not have that buffered state. As a result, decoding continued from an incorrect message boundary, which could lead to protobuf parse errors, invalid compression/reserved-bit errors, or Netty buffer reference-count errors.

This is why the issue is hard to reproduce locally: it depends on HTTP/2 DATA frame splitting/coalescing timing. It is more likely with:

• no-stub provider
• client streaming
• many fast consecutive onNext calls
• a very small first message
• larger following messages, for example 128KB
• no artificial delay or logging inside the sending loop

Fix

The fix is to make LazyFindMethodListener use the transport's actual streaming decoder instead of creating a temporary decoder.

So instead of decoding the first message with a separate new GrpcStreamingDecoder(), it now reuses getStreamingDecoder().

This preserves any buffered partial gRPC message across the transition from LazyFindMethodListener to the real method listener.

Verification

I added a regression test that simulates this exact shape:

DATA frame = [full first gRPC message][partial second gRPC message]

The test verifies that after the listener switch, the remaining bytes of the second message are still decoded correctly.

I also reproduced the issue against Dubbo 3.3.6 using a no-stub provider and client-streaming calls. With 20 messages of 128KB sent quickly, one run failed at iteration 17 with server-side decode errors including LengthFieldStreamingDecoder, InvalidProtocolBufferException, and IllegalReferenceCountException: refCnt: 0.

provider log：

2026-08-08 02:34:09.244 ERROR 10324 --- [0051-thread-200] .r.p.t.h.AbstractServerTransportListener :  [DUBBO] An error occurred while processing the http request with GrpcHttp2ServerTransportListener, Http2MetadataFrame{method='POST', path='/repro.api.UploadFacade/upload', contentType='application/grpc+proto', streamId=3, endStream=false}, method=repro.api.UploadFacade.upload(StreamObserver), dubbo version: 3.3.6, current host: 192.168.0.115, error code: 99-0. This may be caused by , go to https://dubbo.apache.org/faq/99/0 to find instructions. 

org.apache.dubbo.remoting.http12.exception.DecodeException: Internal Server Error
	at org.apache.dubbo.rpc.protocol.tri.h12.grpc.GrpcCompositeCodec.decode(GrpcCompositeCodec.java:105) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.protocol.tri.h12.grpc.GrpcCompositeCodec.decode(GrpcCompositeCodec.java:111) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.remoting.http12.message.HttpMessageDecoder.decode(HttpMessageDecoder.java:56) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.remoting.http12.message.DefaultListeningDecoder.decode(DefaultListeningDecoder.java:41) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.remoting.http12.message.StreamingDecoder$DefaultFragmentListener.onFragmentMessage(StreamingDecoder.java:55) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.invokeListener(LengthFieldStreamingDecoder.java:180) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.processBody(LengthFieldStreamingDecoder.java:172) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.deliver(LengthFieldStreamingDecoder.java:121) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.decode(LengthFieldStreamingDecoder.java:69) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.protocol.tri.h12.http2.GenericHttp2ServerTransportListener$StreamingHttpMessageListener.onMessage(GenericHttp2ServerTransportListener.java:220) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.protocol.tri.h12.AbstractServerTransportListener.doOnData(AbstractServerTransportListener.java:183) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.protocol.tri.h12.AbstractServerTransportListener.lambda$onData$1(AbstractServerTransportListener.java:168) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.common.threadpool.serial.SerializingExecutor.run(SerializingExecutor.java:111) ~[dubbo-3.3.6.jar:3.3.6]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
	at org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:39) ~[dubbo-3.3.6.jar:3.3.6]
	at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
Caused by: java.io.IOException: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either that the input has been truncated or that an embedded message misreported its own length.
	at org.apache.dubbo.rpc.protocol.tri.SingleProtobufUtils.deserialize(SingleProtobufUtils.java:104) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.protocol.tri.PbUnpack.unpack(PbUnpack.java:35) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.model.PackableMethod.parseRequest(PackableMethod.java:26) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.protocol.tri.h12.grpc.GrpcCompositeCodec.decode(GrpcCompositeCodec.java:101) ~[dubbo-3.3.6.jar:3.3.6]
	... 16 common frames omitted
Caused by: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either that the input has been truncated or that an embedded message misreported its own length.
	at com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:92) ~[protobuf-java-3.25.5.jar:na]
	at com.google.protobuf.CodedInputStream$StreamDecoder.readRawBytesSlowPathRemainingChunks(CodedInputStream.java:2950) ~[protobuf-java-3.25.5.jar:na]
	at com.google.protobuf.CodedInputStream$StreamDecoder.readBytesSlowPath(CodedInputStream.java:2987) ~[protobuf-java-3.25.5.jar:na]
	at com.google.protobuf.CodedInputStream$StreamDecoder.readBytes(CodedInputStream.java:2373) ~[protobuf-java-3.25.5.jar:na]
	at repro.upload.UploadFrame$Builder.mergeFrom(UploadFrame.java:452) ~[classes/:na]
	at repro.upload.UploadFrame$1.parsePartialFrom(UploadFrame.java:648) ~[classes/:na]
	at repro.upload.UploadFrame$1.parsePartialFrom(UploadFrame.java:640) ~[classes/:na]
	at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:192) ~[protobuf-java-3.25.5.jar:na]
	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:209) ~[protobuf-java-3.25.5.jar:na]
	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:25) ~[protobuf-java-3.25.5.jar:na]
	at org.apache.dubbo.rpc.protocol.tri.SingleProtobufUtils$SingleMessageMarshaller.parse(SingleProtobufUtils.java:132) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.protocol.tri.SingleProtobufUtils.deserialize(SingleProtobufUtils.java:102) ~[dubbo-3.3.6.jar:3.3.6]
	... 19 common frames omitted

2026-08-08 02:34:09.253 ERROR 10324 --- [0051-thread-200] .r.p.t.h.AbstractServerTransportListener :  [DUBBO] An error occurred while processing the http request with GrpcHttp2ServerTransportListener, Http2MetadataFrame{method='POST', path='/repro.api.UploadFacade/upload', contentType='application/grpc+proto', streamId=3, endStream=false}, method=repro.api.UploadFacade.upload(StreamObserver), dubbo version: 3.3.6, current host: 192.168.0.115, error code: 99-0. This may be caused by , go to https://dubbo.apache.org/faq/99/0 to find instructions. 

io.netty.util.IllegalReferenceCountException: refCnt: 0
	at io.netty.buffer.AbstractByteBuf.ensureAccessible(AbstractByteBuf.java:1454) ~[netty-buffer-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1440) ~[netty-buffer-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.buffer.AbstractByteBuf.checkReadableBytes(AbstractByteBuf.java:1428) ~[netty-buffer-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:895) ~[netty-buffer-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.buffer.ByteBufInputStream.read(ByteBufInputStream.java:183) ~[netty-buffer-4.1.101.Final.jar:4.1.101.Final]
	at org.apache.dubbo.remoting.http12.CompositeInputStream.read(CompositeInputStream.java:82) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.readRawMessage(LengthFieldStreamingDecoder.java:185) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.protocol.tri.h12.grpc.GrpcStreamingDecoder.readRawMessage(GrpcStreamingDecoder.java:54) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.processBody(LengthFieldStreamingDecoder.java:170) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.deliver(LengthFieldStreamingDecoder.java:121) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.remoting.http12.message.LengthFieldStreamingDecoder.decode(LengthFieldStreamingDecoder.java:69) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.protocol.tri.h12.http2.GenericHttp2ServerTransportListener$StreamingHttpMessageListener.onMessage(GenericHttp2ServerTransportListener.java:220) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.protocol.tri.h12.AbstractServerTransportListener.doOnData(AbstractServerTransportListener.java:183) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.rpc.protocol.tri.h12.AbstractServerTransportListener.lambda$onData$1(AbstractServerTransportListener.java:168) ~[dubbo-3.3.6.jar:3.3.6]
	at org.apache.dubbo.common.threadpool.serial.SerializingExecutor.run(SerializingExecutor.java:111) ~[dubbo-3.3.6.jar:3.3.6]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
	at org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:39) ~[dubbo-3.3.6.jar:3.3.6]
	at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]

2026-08-08 02:34:09.281  INFO 10324 --- [rverWorker-3-21] o.a.d.r.t.netty4.NettyServerHandler      :  [DUBBO] The connection [id: 0xf714b008, L:/192.168.0.115:50051 ! R:/192.168.0.115:56989] of 192.168.0.115:56989 -> 192.168.0.115:50051 is disconnected., dubbo version: 3.3.6, current host: 192.168.0.115
2026-08-08 02:34:09.282  INFO 10324 --- [rverWorker-3-21] o.a.d.r.t.n.NettyPortUnificationServer   :  [DUBBO] All clients has disconnected from /192.168.0.115:50051. You can graceful shutdown now., dubbo version: 3.3.6, current host: 192.168.0.115
2026-08-08 02:34:09.282  INFO 10324 --- [rverWorker-3-21] o.a.d.r.t.netty4.NettyChannelHandler     :  [DUBBO] The connection [id: 0xf714b008, L:/192.168.0.115:50051 ! R:/192.168.0.115:56989] of 192.168.0.115:56989 -> 192.168.0.115:50051 is disconnected., dubbo version: 3.3.6, current host: 192.168.0.115

Hello @Anon2Tokyo , can you give the steps to run this to also affirm the same failure.

[Anon2Tokyo]

@LI123456mo Sure, I already upload the demo to this branch. Do you think this approach is convenient?
https://github.com/Anon2Tokyo/dubbo/tree/issue16414-test

[Anon2Tokyo]

STEP：

The runtime reproduction is timing-sensitive because it depends on how HTTP/2 DATA frames are coalesced, but the scenario I used is:

• Dubbo 3.3.6
• Triple provider without generated stub inheritance
• client-streaming RPC
• the client sends many small messages quickly, for example 2000 messages with 8 bytes each

The important part is that the provider should expose a plain Java service interface instead of extending the generated Triple stub, so the request goes through the no-stub method-discovery path.

Example run shape:

# build the repro app
mvn.cmd -q -DskipTests package
mvn.cmd -q dependency:build-classpath "-Dmdep.outputFile=target\classpath.txt"

# start provider
.\run-provider.ps1 -ConfigName application-provider -LogName provider-tri -ClasspathFile target\classpath.txt

# run consumers repeatedly
.\run-consumers.ps1 -ConfigName application-consumer -RunName native-tri -Iterations 30 -Chunks 2000 -ChunkBytes 8 -ClasspathFile target\classpath.txt

A more aggressive run is:

.\run-consumers.ps1 -ConfigName application-consumer -RunName native-tri-stress -Iterations 60 -Chunks 10000 -ChunkBytes 1 -ClasspathFile target\classpath.txt

Expected failure signatures are usually one of:

reserved bits not zero
IllegalReferenceCountException
LengthFieldStreamingDecoder
Upload failed

The underlying frame layout that triggers the bug is:

DATA frame 1: [complete message 1][prefix of message 2]
DATA frame 2: [remaining bytes of message 2]

With the old implementation, the prefix of message 2 can be buffered in the temporary GrpcStreamingDecoder used during method discovery, then lost when the stream switches to the actual decoder. This PR avoids that by reusing the stream's existing decoder during method discovery.

One note: I do not think this is 100% deterministic as a black-box runtime repro, since it depends on local scheduling and HTTP/2 frame packing. The regression test in this PR directly covers the problematic decoder state transition.

[LI123456mo]

@LI123456mo Sure, I already upload the demo to this branch. Do you think this approach is convenient? https://github.com/Anon2Tokyo/dubbo/tree/issue16414-test

Sure , i'll try to check on it then i'll inform you