[sc-25841] Remove 4MB limit for gRPC message payloads

[justinGilmer]

Currently our logic chunks data to send over gRPC according to numbers of rows of data, this led to large streamsets in the multivalue api erroring out while trying to recv this data on the client side.

This PR sets the limit for the client to receive from the server to be unlimited for the time being. This will allow arbitrarily-sized streamsets to have successful multivalue queries.

In the future we should update our logic to better handle these size limits when sending from the server, but this is a patch fix for now.

I tested this for 5000 streams on ni4ai, with a sampling_frequency of 30 for 100seconds of data, this would error out on the master branch, but successfully returns the data on with this PR.

MWE

import btrdb

conn = btrdb.connect(profile='CLUSTER_WITH_MANY_STREAMS')
streams = conn.streams_in_collection(*conn.list_collections(), is_collection_prefix=False)

end = btrdb.utils.timez.currently_as_ns()
start = end - btrdb.utils.timez.ns_delta(seconds=100)

strm_set = btrdb.stream.StreamSet(streams).filter(start=start, end=end, sampling_frequency=30)

table = strm_set.arrow_values()

[justinGilmer]

trying to link ticket

[sc-25841]

[shortcut-integration]

This pull request has been linked to Shortcut Story #25841: Update grpc message size on the python client to prevent 4MB size limit for large streamset queries.

[jleifnf]

LGTM.
Wonder how the no limit on the size to send would effect for the local querying of the data.

[justinGilmer]

I think there will still be a bit of a limit:

But, there will still be a upper bound since we currently enforce only 5000 rows to be sent at a time per message, so we can technically have a message size of: 5000points * 8bytes/point (assuming 64bit) * N streams + 1 time stream
My rough estimates (13K streamset ~500MB, 2.5K streamset ~100MB)

We have a similar limit on the send to the server side when inserting data as well.

[andrewchambers]

Our server limits things itself, if we trust our server then it shouldn't be a problem.