Sending large binary data (Or doing custom marshalling/framing)

[tjad]

I have to integrate with a gRPC endpoint that takes large binary data. I have no control over the endpoint - I don't define how it works.

My solution is to do my own framing - almost.

I noticed there is a PreparedMsg, I want to make use of this, so that I can do the encoding step manually. This is so that I encode the protobuf request, and modify where the binary data should be.
Basically i would
1.) encode a stubbed protobuf request (which includes a placeholder for binary input)
2.) split the encoded protobuf into 2 parts (before and after where the binary data should be)
3.) modify the size indication within the protobuf (so that it matches the size of my binary data)
4.) prepare a message with PreparedMsg.Encode() for the first part of the protobuf (everything before binary data)
5.) SendMsg with the clientstream
6.) using an io.Reader, loop, populating mem.BufferSlice at a fixed size (and yes, respect that I may not modify this slice as per internals), and SendMsg ffor each mem.BufferSlice containing a portion of the binary data
7.) prepare the final part of the protobuf in a PreparedMsg and send it

Currently the problem is that there is no type check here for mem.BufferSlice , so it returns an error, telling me the encoder is expecting a proto.Message (i.e it still tries to encode the already encoded message)
https://github.com/grpc/grpc-go/blob/master/rpc_util.go#L690

Something like this

	msg_buf, ok := msg.(mem.BufferSlice)

	if ok {
		return msg_buf, nil
	}

I have tested this, when adding this check, the rest of the process seems to go normally. The server receives my data and responds correctly.

I am certain there would be a better way of doing this,, but currently there is no indication of being able to support for streaming data from an io.Reader. This seems like a big flaw in design, or at least it is a massive limitation.

I need to do this in order to not have to load up the entire data in memory, encode it as protobuff, and then have gRPC do its thing. The size of my data can be as small as 1MB and span all the way up to 200MB (or more). This is a massive constraint on scalability of my client service where it is constrained to memory.

I think ideally, there should be a way to "StreamMsg" (like SendMsg), where I can pass in an io.Reader, and the Transport layer can handle it appropriately. The io.Reader would output marshalled protobuf.

This way we would be able to effectively write our own custom data processing, and the gRPC Transport( and http2.Framing) could still be harnessed via gRPC

[tjad]

In my scenario above, the server is acting upon the entire binary data immediately anyway - so it doesn't matter for the server to load up all the data at once, but obviously if they could do similar io.Reader streaming of binary data out of the protobuf message, that would be more ideal for them - I think anyway.

I think in the general case, it is not always possible to stream parts of the data, say I were sending RGB image data for example and pushing it into an AI model where the model uses all 3 layers of data as a single sample for input into the model.

I really think that gRPC (for golang anway) would benefit from this - it would remove the only constraint for TRUE streaming.

[purnesh42H]

@tjad thanks for the question. Any specific reason why you are using PreparedMsg? It is a special message type that represents a pre-serialized Protobuf message. It essentially allows you to bypass gRPC's internal marshaling step if you've already serialized your message manually.

I think in your case, you probably just need to implement your own codec since you don't want dependency on proto https://github.com/grpc/grpc-go/blob/master/Documentation/encoding.md#implementing-a-codec

And then you can create a client stream and use stream.Send() to send your raw binary data (preferably in chunks).

[tjad]

Thank you for responding @purnesh42H
The encoder would need to encapsulate the above logic yes?
It would then mean that I would return a mem.BufferSlice holding 3 Buffers
1.) the head (first part of the modified protobuf - as per above)
2.) an io.Reader (this returns the actual binary data)
3.) the tail (last part of the protobuf - as per above).

Does this sound doable?

I would only call stream.Send() once - as it is a single gRPC request for the method. The API does not permit sending "multiple Requests" with separate payloads and having them treated as a single request for processing.

I can't serialize all the data at once into a single protobuf binary - as it would consume too much memory, resulting in an application that needs to be scaled by memory, and the memory would be depleted quickly given the size of data I'm dealing with 1MB upto 200MB or greater (we haven't started processing larger data yet).

The binary data is originally streamed from another data source (i/o bound)

[tjad]

It essentially allows you to bypass gRPC's internal marshaling step if you've already serialized your message manually.

This does not actually work in the current state of code - as per my indication above. Even though I serialized the protobuf already and then sent it as a PreparedMsg, I got an error telling me it was expecting protobuf message (unserialized)

Currently the problem is that there is no type check here for mem.BufferSlice , so it returns an error, telling me the encoder is expecting a proto.Message (i.e it still tries to encode the already encoded message) https://github.com/grpc/grpc-go/blob/master/rpc_util.go#L690

Something like this

	msg_buf, ok := msg.(mem.BufferSlice)

	if ok {
		return msg_buf, nil
	}

You will notice that encode is just encoding, and not doing any checks currently

Using a codec would get around this problem.

[purnesh42H]

The encoder would need to encapsulate the above logic yes?
It would then mean that I would return a mem.BufferSlice holding 3 Buffers
1.) the head (first part of the modified protobuf - as per above)
2.) an io.Reader (this returns the actual binary data)
3.) the tail (last part of the protobuf - as per above).

Does this sound doable?

If you want to do manual encoding, there is no other way but to define a custom codec. You need to implement the Codec or CodecV2 interface which will have its own init method to register itself as a codec and a Name() method to return the name of the codec. Codecs are registered by name into a global registry maintained in the encoding package. The Marshal and Unmarshal methods of CodecV2 use mem.BufferSlice, which offers a means to represent data that spans one or more Buffer instances so that might suit your use case better. The Codec, however, just works on the single byte slice. CodecV2 was released in grpc-go version 1.66.0 though.

You can refer to the grpc's implementation of the proto codec to get an idea https://github.com/grpc/grpc-go/blob/master/encoding/proto/proto.go.

I would only call stream.Send() once - as it is a single gRPC request for the method. The API does not permit sending "multiple Requests" with separate payloads and having them treated as a single request for processing.

One thing to call out here is that codecs provide the wire format of the request data. It is not recommended to send more than few KBs of data on the wire. If you are using ClientStream to send data, you will be sending the individual messages/chunks over the network and gRPC will receive the chunks from the network and makes them available to the server-side streaming RPC method. It will be part of the same streaming call. If not already done, take a look at client streaming example here https://grpc.io/docs/languages/go/basics/#client-side-streaming-rpc-1. Are you saying the endpoint won't process the chunks even if they are in correct order?

I can't serialize all the data at once into a single protobuf binary - as it would consume too much memory, resulting in an application that needs to be scaled by memory, and the memory would be depleted quickly given the size of data I'm dealing with 1MB upto 200MB or greater (we haven't started processing larger data yet).

The binary data is originally streamed from another data source (i/o bound)

After starting the client stream, you will have to chunk at the application level and custom codec will be used for each chunk sent using stream.Send(). See how to use the custom codec https://github.com/grpc/grpc-go/blob/master/Documentation/encoding.md#using-a-codec

[purnesh42H]

This does not actually work in the current state of code - as per my indication above. Even though I serialized the protobuf already and then sent it as a PreparedMsg, I got an error telling me it was expecting protobuf message (unserialized)

If you have already serialized your message, you don't need PreparedMsg. Just send the serialized bytes directly (if your API supports it) or encapsulate them in a suitable protobuf message with a bytes field and use regular stream.Send().

PreparedMsg.Encode() still expects the unserialized protobuf message as input. The Encode() method uses the codec associated with the RPC stream to perform the serialization. When you call Encode(), it internally serializes the message you provide using the configured codec. It does not simply take the bytes you've already serialized.

The main benefit of using PreparedMsg is the encoding happens before the sending path. So, if you have many messages to send you can encode them separately before calling stream.Send()

preparedMsgs := make([]*grpc.PreparedMsg, len(messages))
for i, msg := range messages {
    preparedMsgs[i] = &grpc.PreparedMsg{}
    preparedMsgs[i].Encode(stream, msg) // Encode concurrently
}

for _, pMsg := range preparedMsgs {
    stream.SendMsg(pMsg) // Fast, just sends the pre-encoded data
}

[github-actions]

This issue is labeled as requiring an update from the reporter, and no update has been received after 6 days. If no update is provided in the next 7 days, this issue will be automatically closed.

[tjad]

@purnesh42H I think you're not fully comprehending my use case provided above. I have tried to explain very elaborately what needs to be achieved, and why this library's implementation is the only thing preventing or limited in providing support for such requirements.

I have taken a lot of time to review the internals of this gRPC client, and it certainly does not permit fully usage of 2GB as per protobuf limitation of message sizes efficiently. HTTP2's Dataframes is just a transport, it does not limit anything other than sending packets in 16kB parts. This library is memory bound.

Please try to have someone who is very experienced with the internals try to assist - or at least try to comprehend fully what I am trying to achieve, I have provided enough information.

In short, I want to have an io.Reader be read by the underlying layers and transport. Encoding is only 1 part of the problem, compression is another layer - which "materializes" the whole mem.BufferSlice into memory, which is yet another problem. Why is an io.Reader not used through the layers for pipelining data through the gRPC layers ? All applications are memory bound due to this and memory is not cheap.

The internals need to be reworked - I'm not saying it's a quick thing to do, but it is certainly a limitation and oversight in terms of efficiency of the technologies brought together (Golang, Protobuf, gRPC).

Do I need to open a feature request, or assist in fixing the internals with a PR etc ? @dfawley

[tjad]

And FYI, I have built a custom serializer and attached/registered it to the encoding registry, this alone does not resolve the problem.

[tjad]

I honestly don't understand how "mem buffers" were seen as a means for efficiently transferring data between layers, mem.BufferSlice is an interesting idea, but loading data into memory should not have been seen as the only possible kind of data being sent over gRPC. Loading data into memory should be a last resort. I should only load data into memory if I am acting upon that data. When I have binary data, I don't particularly want to act on the data, and there is no need to require a 2nd means/technology for transferring large binary. The mere fact that I can swap out protobuf for different message framing means that I can send > 2GB anyway, if I really need to (that's really the ultimate limitation when using protobuf)

It doesn't have to be limited to binary data either, imagine a large database query, I could potentially stream data out of the database with database streaming as a different kind of use case.

Nothing is made perfect, it's an iterative process, and we can only make progress when we acknowledge where there is room for improvement.