Add support for Flatbuffers

[mustiikhalil]

Good afternoon,
Hope are are all doing well. I want to add swift-gRPC to Flatbuffers, however swift-gRPC is heavily coupled to swift-protobuf. Most of the methods are using protocol Message, and I would love to know the following:

• If there is a way to get to the underlying data that is storing the actual buffer? and would it be safe to access, or how can we just get an array of [UInt8]?
• Is there a way to bypass the following function decodeMessage<D: SwiftProtobuf.Decoder>(decoder: inout D) throws just to return our Buffer? or is there simply a way to just get the buffer from Message
  note: in flatbuffers, it would be nice to get either the Data, or [UInt8]

[glbrntt]

Hi @mustiikhalil,

Thanks for reaching out: it'd be great to have Flatbuffers support! Just so we're on the same page all my comments below refer to the nio branch:

You're right, the API is pretty tightly coupled with SwiftProtobuf at the moment, however I don't think that changing that would require a significant amount of work. I'd need to spend some more time looking into this to validate that claim though.

• If there is a way to get to the underlying data that is storing the actual buffer? and would it be safe to access, or how can we just get an array of [UInt8]?

This isn't possible for the caller at the moment, but we've had requests to allow callers to bypass the encoding/decoding so that they can deal with bytes directly.

• Is there a way to bypass the following function decodeMessage<D: SwiftProtobuf.Decoder>(decoder: inout D) throws just to return our Buffer? or is there simply a way to just get the buffer from Message

Not at the moment, I think the better approach here would be supporting different message encoders/decoders in the generated API and having that as the entry point for direct-byte access.

As far as I understand flatc doesn't have support for plugins, so the code generation would have to live in https://github.com/google/flatbuffers, is that right? And would it have to be C++..?

[mustiikhalil]

Thanks for your reply!
Yes, we are talking about the same branch nio. I took the liberty to create a playground for it and test things out, My finds are the following:

• If a protocol is created that would give us a certain level of access that would be nice, since Message is just a protocol that's found in SwiftProtobuf, so build a protocol called Payload that has a decode and encode functions that would allow/give us the freedom to deal with data that would be amazing! since currently when decoding we call merge in the Message+Binary.swift in SwiftProtobuf. cause I already tried bypassing that and it didn't work.

• Yes, the code will have to live in the FlatBuffers repository, hopefully that won't be a huge issue.

Is there anyway I can help? I can help migrate it
playground
FlatBuffers issue

[glbrntt]

Yes, we are talking about the same branch nio.

Awesome, worth checking just in case 🤪

• If a protocol is created that would give us a certain level of access that would be nice, since Message is just a protocol that's found in SwiftProtobuf, so build a protocol called Payload that has a decode and encode functions that would allow/give us the freedom to deal with data that would be amazing!

That's more or less what I came up with as well. I think we need to be careful here such that the protocol is flexible enough to enable different encoding/decoding without incurring extra copies. To that end I think the protocol would probably look something like:

protocol Payload {
  init(serializedByteBuffer: inout NIO.ByteBuffer) throws
  mutating func serialize(into buffer: inout NIO.ByteBuffer) throws
}

• Yes, the code will have to live in the FlatBuffers repository, hopefully that won't be a huge issue.

Copying the source there shouldn't be an issue. My major concern here is that our code generator is written in Swift, not C++.

Is there anyway I can help? I can help migrate it

I think the first step here is attempting to switch the current implementation to use a protocol similar to the one above. You'll no doubt have to trace through a number of layers in the code: I'd probably start with LengthPrefixedMessage{Reader,Writer} and work back from there.

[mustiikhalil]

Sounds good! I'll start working on that then and see if it we would be able to make this happen!

[mustiikhalil]

the issue's implementation for FlatBuffers will be in the FlatBuffers repository since currently FlatBuffers doesn't support plugins

@glbrntt Thanks for you help 🎉🎉