reflect/protoreflect: add MessageDescriptor to MessageType examples

[DazWilkin]

I'm struggling to grok APIv2 and apologize if this is a stupid question and the wrong place to ask.

Background

I'm toying with a solution in which some client may submit requests to a (remote) server that invokes functions on the server. While the client and server may use gRPC and protobufs, the functions do not implement protobufs, may be hosted by different server runtimes but, in Go are of the type func f([]byte) ([]byte,error) and may be invoked as result,err := host.invoke(f,params). The requirement will be that the incoming params will be a []interface{} and parseable by the function. I'm thinking to envelope the functions' types as protobuf services (==functions) and messages (*Request==params; *Response==result).

Implementation

• protoc generates descriptor set
• descriptor set parsed into protoregistry.Files
• MessageDescriptors gleaned from FileDescriptors in Files Find*ByName and Find*ByPath
• Marshaled into anypb.Any using dynamicpb.NewMessage() and proto.Unmarshal()
• Shipped to the server

After much reading of the Godocs and struggling to grasp the API, this appears to work.

Challenge

I'm challenged to perform the reverse.

The package structure suggests that, whereas I used protoregistry.Files on the way in, I will use protoregistry.Types on the way out... With anypb.Any I should have as TypeURL that I could FindMessageBy* to get a MessageType (Go type) rather than the MessageDescriptor (protobuf).

The server is able to invoke the desired function, it is able to determine which function to invoke and the types of the request and response from the incoming request message's envelope metadata and it has a concrete anypb.Any message corresponding to the parameters.

But, I need but don't have MessageType to populate protoregistry.Types

IIUC MessageType correspond to Go concrete types. I'm OK to use Golang for this implementation but I'd like the possibility that the server be implemented in e.g. Rust too. So, I think I'd prefer to have messages convey protobuf types rather than language-specific representations.

If I had `protoregistry.Types, this #1085 issue helps get me where I need to (I think).

Questions

1. Is there example code for constructing protoregistry.Types?
2. Is there example code for moving to|from anypb.Any using TypeURL as the key?
3. If my explanation is understandable, am I on the right path?

Looking through other issues, other developers appear to be less challenged by this API than me. But assuming that I'm representative of the other half the normal distribution, please blog more describing use-cases and best-practices with this API.

[dsnet]

Is there example code for constructing protoregistry.Types?
Is there example code for moving to|from anypb.Any using TypeURL as the key?

No and no. Examples and documentation for all the new features are unfortunately lacking at the moment.

MessageDescriptor --> MessageType?

The protoreflect package currently mentions the use of dynamicpb. Was there someplace else you were looking during your journey that similar things should have been documented there as well?

But, I need but don't have MessageType to populate protoregistry.Types

Does dynamicpb.NewMessageType do what you want?

Looking through other issues, other developers appear to be less challenged by this API than me.

It seems to me that you're doing some pretty advanced things dynamically with protobufs, so it's unsurprising that you're treading on waters that most people haven't need to traverse. It does amuse me that almost everything you're describing would have been impossible to do with the github.com/golang/protobuf, but is at least doable with the google.golang.org/protobuf module.

[DazWilkin]

Thank you for such a prompt and helpful response!

I missed The "google.golang.org/protobuf/types/dynamicpb" package can be used to create Go type descriptors from protobuf descriptors.. I'll have a look dynamicpb.NewMessageType. Thank you!

Reading through the Overview and the very interesting Redaction example, some of the motivations for API bump change are explained. And, the overview does hint at the possibilities.

Respectfully, I think the reasoning would benefit from more explanations and examples showing what's possible|easier with APv2.

I stumbled upon APIv2 when I began receiving deprecated warnings from APIv1 but these warnings didn't guide me to APIv2 and, in truth, it took me some time to find it and some time to understand what's going on.

Thank you for this helpful reply. I'll take a look at dynamicpb.MessageType and update|close this.

[DazWilkin]

@dsnet I've made some progress but once more stuck, asking in this thread as it's related to my original comment.

My test proto:

service Adder {
    rpc add(AdderRequest) returns (AdderResponse) {};
}
message AdderRequest {
    int32 a = 1;
    int32 b = 2;
}
message AdderResponse {
    int32 result = 1;
}

I'm able to find the descriptor, create the type (thanks -- your previous guidance) and a new up a ProtoMessage which will receive Unmarshal'd data.

In the following, the output matches AdderRequest (as expected):

md := f.filedescriptor.Messages().ByName("AdderRequest")
mt := dynamicpb.NewMessageType(md)
pm := mt.New().Interface()
if err := proto.Unmarshal(params.GetValue(), pm); err != nil {
	return nil, err
}

log.Printf("%+v", pm)

For testing (only!), I've implemented the services by implementing the grpc stubs.

NOTE In the solution I'm planning, these services will not implement the stubs and I will not need to assert them as concrete (Go) types but be able to range over a ProtoMessage's FieldDescriptors and values.

For testing, I have a ProtoMessage and in this case, I need a concrete AdderRequest type.

resp, err := x.Add(context.TODO(), &pm)

Which fails with:

cannot use ... (type *protoreflect.ProtoMessage) as type *protos.AdderRequest ...

It's also not possible to assert into AdderRequest

rqst, ok := pm.(*pb.AdderRequest)
if !ok {
	log.Print("type assertion to `AdderRequest` failed")
}

Since I know the concrete type, I could manually construct (e.g. &AdderRequest{...}) from pm but it feels as though this should be automatic.

Can I do this?

[dsnet]

If you have direct access to AdderRequest, is there a reason to not use that concrete type instead of creating a dynamic message?

pm := new(protos.AdderRequest)
if err := proto.Unmarshal(params.GetValue(), pm); err != nil {
	return nil, err
}
...

[dsnet]

Fundamentally, the generated grpc stub expects a concrete message type in its API. Thus you either have to:

• Determine the concrete message type from the grpc API (either through Go reflection) or through reflection API provided by grpc if one exists (I'm not an expert on the grpc API), create an instance of that message type and unmarshal into it, OR
• Construct something through dynamic message types, but prior to passing to the grpc API, you need to convert the dynamic message into a concrete message. You can do so by merging the dynamic message into a newly created concrete message:

md := f.filedescriptor.Messages().ByName("AdderRequest")
mt := dynamicpb.NewMessageType(md)
pm := mt.New().Interface()
if err := proto.Unmarshal(params.GetValue(), pm); err != nil {
	return nil, err
}
...
m := new(protos.AdderRequest)
proto.Merge(m, pm)
...
resp, err := x.Add(context.TODO(), m)

[DazWilkin]

Very helpful, thank you.

[DazWilkin]

@dsnet -- an update to confirm that I've got everything working. Thanks for your help!

[dsnet]

Great to hear! Let's keep this issue as I agree that the documentation can be better improved.