Bigtable Python Client Max Row size is 4mb

[brandon-white]

On the big table documents, it says the max cell size is 100mb. However, when I try to read a row with a cell of size 10mb using the Bigtable Python client, I get the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/happybase/table.py", line 190, in row
    row, filter_=filter_)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/bigtable/table.py", line 234, in read_row
    rows_data.consume_all()
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/bigtable/row_data.py", line 323, in consume_all
    self.consume_next()
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/bigtable/row_data.py", line 261, in consume_next
    response = six.next(self._response_iterator)
  File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 344, in next
    return self._next()
  File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 335, in _next
    raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.INTERNAL, Max message size exceeded)>

This max size seems to be hard coded in the grpc library. Has anybody been able to read large rows using the Bigtable Python client? Any idea for workarounds or how I can set the max size?

[dhermes]

@nathanielmanistaatgoogle WDYT?

[nathanielmanistaatgoogle]

A few gRPC users have had this kind of problem and we've started a discussion in grpc/grpc.github.io issue 371.

For now you can cross your fingers and see what happens when you pass a channel options value like options=(('grpc.max_message_length', 100 * 1024 * 1024),) to grpc.insecure_channel or grpc.secure_channel (know that grpc.max_message_length is broken up into grpc.max_send_message_length and grpc.max_receive_message_length in gRPC Python 1.1-and-later). It might work.

If it doesn't work, then you've hit some limit inside gRPC that isn't overridable and you'll have to break up the large message into a stream of smaller ones.

[dhermes]

@nathanielmanistaatgoogle The "you" in "when you pass a channel options value" here is the library maintainers, yes? As a user, @brandon-white isn't directly creating any gRPC object, he is dealing with our Bigtable classes (i.e. the nouns) and calling the API via their instance methods (i.e. the verbs).

[brandon-white]

Thanks @nathanielmanistaatgoogle I'll try this but, ideally, I would prefer a solution where I do not edit the Bigtable Python client.

For the Google Bigtable Client folks, have you encountered any use cases where the rows and cells are large? If so, how do you break up these large rows and cells into streams? Are large rows and cells supported through the Python Client?

[dhermes]

@mbrukman Have you run into this in other clients?

[mbrukman]

While there's a per-response limit in gRPC, it is possible to retrieve larger cell values via streaming responses and reconstructing them client-side. ReadRowsResponse includes a list of CellChunks which are the pieces of cells that can be reconstructed to form the larger values.

/cc: @sduskis, @garye

[brandon-white]

Thank you @mbrukman ! I don't see these calls in the BigTable Python Client. Does this mean I need to write my own client if I want to consume these large cell values? Are there any plans to incorporate this into existing clients?

[sduskis]

The python client already deals with ReadRowResponse under the covers. This seems like something we need to change in the service.

[sduskis]

I'll take that back. This seems like a client setting. In the java client, we set the value on the netty channel (maxMessageSize) to be 256MB. Is that a setting that the python client controls?

[brandon-white]

@sduskis I looked for that but didn't see any option to set that in the Python client. So it seems like we need to add this max_message_size config to the BigTable Python Client?

[sduskis]

@dhermes, do you have any idea where grpc settings are set, and if max_message_size is a settable property?

[dhermes]

I do not, though @nathanielmanistaatgoogle would be a good person to ask. I'm not 100% clear on if "grpc settings" are a global thing or a per-call setting? The options= described above is how we'd set gRPC metadata on a per-call basis.

[gamorris]

It looks like the channel options get set in core/google/cloud/_helpers.py:489. I don't see a way at the moment to pass options specifically for Bigtable.

I think grpc.max_receive_message_length would be the option to set.

We might ultimately want to set the max send message length for Bigtable also. It looks like it is currently unlimited, but there is a TODO there to change that.

[brandon-white]

So do I need to make the PR to this or do any of the committers have bandwidth to take a look at this setting? What is the process for getting fixes like this out?