Binary data in BigTable column

[arbesfeld]

Hi, I am trying to store an encode protobuf in BigTable and I'm not able to insert binary data (represented as a Buffer) into a BigTable column. I have tried the following ways:

this.table.insert([
  key: `foo`,
  data: {
    event: new Buffer(),
  },
]);

this.table.insert([
  key: `foo`,
  data: {
    event: {
     data: new Buffer(),
    }
  },
]);

It seems to get stringified somewhere in the client library, which is making the data invalid.

[callmehiphop]

Hi @arbesfeld, we don't stringify any data before trying to insert it. However, we do transform any data you pass in into Buffers. Have you tried passing your data in as a string/number?

e.g.

this.table.insert([
  {
    key: 'foo',
    data: {
      familyName: {
        columnName: 'data'
      }
    }
  }
]);

[callmehiphop]

@arbesfeld can you give me an example of what kind of data you're trying to save? (Aside from Node buffers)

[callmehiphop]

@arbesfeld we've just merged some changes for the Bigtable API. Assuming what you're passing in is a Buffer, then it should pass through just fine to the API. When retrieving the data, if you want a Buffer returned to you, you'll need to set the decode: false option like so

table.getRows({ decode: false }, function(err, rows) {});

[arbesfeld]

Hey @callmehiphop, I'm setting decode: false and getting seemingly all base64 encoded buffers when I call .getRows(). I passed in a binary-encoded Buffer. Is this the expected behavior?

[stephenplusplus]

@callmehiphop I think we should always create the Buffer from the response base64 value: https://github.com/GoogleCloudPlatform/google-cloud-node/blob/3cbc93b9d770dd25aa5c6ed1f215e01518e42a03/packages/bigtable/src/row.js#L163

    if (qualifier && chunk.value) {
      var value = Mutation.convertFromBytes(chunk.value, options);

      qualifier.push({
        value: value,
        labels: chunk.labels,
        timestamp: chunk.timestampMicros,
        size: chunk.valueSize
      });
    }

and do something like this in Mutation.convertFromBytes:

  var buf = new Buffer(bytes, 'base64');
  var num = new Int64(buf).toNumber();

  if (!isNaN(num) && isFinite(num)) {
    return num;
  }

  if (options && options.decode === false) {
    return buf;
  }

  return buf.toString();