google.protobuf.Struct always empty

[CaioF]

Hey ts-proto mantainers 👋
I have an issue with google.protobuf values always being empty.

I have a nestJS monorepo with a gRPC microservice,
common.proto

syntax = "proto3";

import "google/protobuf/struct.proto";

service EventsService {
    rpc HandleEvent (Event) returns (Event) {}
}

message Event {
    string name = 1;
    google.protobuf.Struct data = 2;
    string correlationId = 3;
}

Using npx protoc --plugin=./node_modules/.bin/protoc-gen_ts_proto --ts_proto_out=./ --ts_proto_opt=snakeToCamel=false --ts_proto_opt=env=node --ts_proto_opt=nestJs=true ./proto/common.proto
This builds into common.ts:

//   protoc-gen-ts_proto  v2.1.0
//   protoc               v3.20.3

import { wrappers } from "protobufjs";
import { Struct } from "../../../../google/protobuf/struct";

export interface Event {
  name: string;
  data: { [key: string]: any } | undefined;
  correlationId: string;
}

export interface EventsServiceClient {
  handleEvent(request: Event): Observable<Event>;
}

wrappers[".google.protobuf.Struct"] = { fromObject: Struct.wrap, toObject: Struct.unwrap } as any;

And then, using the nest microservice package to create a client the following payload being sent:

const event: Event = {
      name: 'Workflow',
      correlationId: '48f60676-c03d-477d-a20f-c42e05086350',
      data: { hello: 'world' },
    };

The message is received by the gRPC server, however the data parameter is empty, on the logs of the gRPC server it shows:

[Nest] 275137 - 09/07/2024, 9:34:30 PM DEBUG [EventsService] handleEvent called.
Input: {"name":"Workflow","data":{},"correlationId":"48f60676-c03d-477d-a20f-c42e05086350"}

[stephenh]

Hi @CaioF ! I believe you are running into #852 , where NestJS's grpc integration does not successfully go through ts-proto's blessed encode / decode methods.

The wrappers[".google.protobuf.Struct"] line that we output is supposed to fix this, but again I believe #852 has found that it's not correctly invoked...

It's been awhile since I've dug into that issue, but I'm 90% sure that's what you're running into.

Fwiw a big part of our ts-proto 2.x release was moving to @bufbuild/protobuf precisely because of long-standing bugs like this in protobufjs, but unfortunately ts-proto moving to @bufbuild/protobuf doesn't mean the NestJS stack has moved over.

I'd love to have you/someone dig into #852 , and finally fix that for NestJS users, as it's been a pita for awhile. :-/

Thanks!

[CaioF]

@stephenh thanks for pointing me to the issue, I will take a deeper look at it!

[eladhaim]

@CaioF actually what we did is writing a "custom grpc-loader" that return something like:

export const loadSync = (
  filename: string | string[],
  options?: Options,
): PackageDefinition => {
  if (!options?.serviceDefinition) {
    throw new Error('serviceDefinition is required');
  } else {
    return options?.serviceDefinition;
  }
};

where serviceDefinition come from the loaderOptions of nest:

interface ServiceDefinition {
   [serviceWithPackageName: string]: generatedService
}

Where serviceWithPackageName is ${packageName}.${serviceName} and generatedService is actually the grpc-js service generated by ts-proto

[CaioF]

@CaioF actually what we did is writing a "custom grpc-loader" that return something like:

     [`${packageName}.${serviceName}`]: service

Where packageName and serviceName and the service are generating by ts-proto

I see, so that is just for loading not for encoding. Did you face the issue with using google.protobuf types in ts-proto with NestJS? The one in the link I sent you. Basically I can't find any way to use Google's well known types with ts-proto and nestJS because nestJS just does not use the encoders from ts-proto.

[eladhaim]

we actually use google wrappers and it seralize/deserialez it perfectly. for example StringValue will be get deserialized to:

property: string | undefined;

[CaioF]

we actually use google wrappers and it seralize/deserialez it perfectly. for example StringValue will be get deserialized to:

property: string | undefined;

How do you do that? Because whenever I use the types from ts-proto (i.e. Struct) they always fail to get serialized and come up as empty objects in the server. See the example I linked #1106

[CaioF]

@eladhaim if you want I can setup an example repo, let me know if that would be helpful.

[eladhaim]

let's say we have a service with package name:
foo
and the service name is bar

our loader will return

{
   ["foo.bar"]: barService <----- generated with ts-proto with --ts_proto_opt=outputServices=grpc-js
}

the generated service includes the seralization/desarialization details.