Discussion: package pubsub

[peterbourgon]

I'd like to start thinking about a design for package pubsub/msgbus/...

Important use cases:

• An async transport for existing/typical RPC messaging
• A foundation for event sourcing
• A way to manage streaming
• Others?

Would like support for:

• Kafka
• RabbitMQ
• NATS
• NSQ
• AWS SQS
• GCP PubSub
• Weave Mesh
• Others?

Prior art:

• NYTimes/gizmo/pubsub
• micro/go-micro/broker
• Micro on NATS
• Others?

Open questions:

• What are the core abstractions and interfaces?
• Is streaming significantly different than event sourcing and transport use cases?
• What is the lowest common denominator for serialization?
• Others?

[basvanbeek]

Shall we start a document for this or are you going to edit the above message to keep track of state?

Peter,

Task-oriented message passing architectures are great for flexibility, wiring together functional blocks or micro-services through subscriptions. These tasks send/route messages of a negotiated format with identified payloads through a mesh of subscriptions, adding value as the information flows and routing onward to downstream subscribers. We’d love to use Go Kit to do this, else we’ll write our own, it’s not too hard really. We’ve been doing this successfully for 16 years. Tasks/services have a collection of subscribers and a subscription object is relatively simple, upon which events are raised containing the transmitted message with its envelope information and payload. Tasks, or a supervisory agent, can place their subscription object into the collection of other objects and begin receiving. They can unsubscribe by removing the subscription object from the message sender’s collection.

I would avoid a “bus” concept (centralized pipe) but use independent synchronous / asynchronous connections between tasks or services.

I would seek to keep the pub/sub as a transport that is independent of what is being exchanged as much as possible. What the message is, it’s meaning, its format is the snarly part, so I’d avoid mingling it into the transport. I think it should support fan-out (multiple subscribers) at a minimum and fan-in is sometimes needed too (use case: synchronizing data sets coming from different sources).

You may also need different pub/sub “channels” (may or may not be in the golang sense) as data may be flowing at extremely high rates through one channel and you’ll want to put “command and control” messages in another channel (Prepare, Execute, Pause, Stop/Halt, Do This, Do That J, etc.).

My 2 cents. Opinions will vary. J

Thanks,

Carl

Carl Cook

President / CTO

IntelliDynamics

http://www.intellidynamics.net/ www.intellidynamics.net

---

IntelliDynamics is a brand and division of BioComp Systems, Inc.

From: Peter Bourgon [mailto:notifications@github.com]
Sent: Wednesday, June 15, 2016 8:52 AM
To: go-kit/kit
Subject: [go-kit/kit] Discussion: package pubsub (#295)

I'd like to start thinking about a design for package pubsub/msgbus/...

Important use cases:

• An async transport for existing/typical RPC messaging
• A foundation for event sourcing http://microservices.io/patterns/data/event-sourcing.html
• A way to manage streaming What about streaming API #213
• Others?

Would like support for:

• Kafka
• RabbitMQ
• NATS
• NSQ
• AWS SQS
• GCP PubSub
• Others?

Prior art:

• NYTimes/gizmo/pubsub https://godoc.org/github.com/NYTimes/gizmo/pubsub
• micro/go-micro/broker https://godoc.org/github.com/micro/go-micro/broker
• Micro on NATS https://blog.micro.mu/2016/04/11/micro-on-nats.html
• Others?

Open questions:

• What are the core abstractions and interfaces?
• Is streaming significantly different than event sourcing and transport use cases?
• What is the lowest common denominator for serialization?
• Others?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub #295 , or mute the thread https://github.com/notifications/unsubscribe/AJ2SvFoXjVH4bk_BhOUMjWl0FuRAPtptks5qMAN2gaJpZM4I2YN6 . https://github.com/notifications/beacon/AJ2SvODan_e_8AIiCmI7G9wjOWzPpyQrks5qMAN2gaJpZM4I2YN6.gif

[jmank88]

RFC 7464 - JSON Text Sequences could be a good candidate for the JSON streaming transport. Some of the general streaming protocol concepts discussed here are relevant as well.

[ChrisHines]

Streaming RPC makes me think of WebSockets and gRPC streaming. Go examples for streaming gRPC are here: http://www.grpc.io/docs/tutorials/basic/go.html

[peterbourgon]

@basvanbeek

Shall we start a document for this or are you going to edit the above message to keep track of state?

I'd like to keep discussion in the issue for now. I'll update the OP as necessary.

—

@intellidynamics

I would avoid a “bus” concept (centralized pipe) but use independent synchronous / asynchronous connections between tasks or services.

For event sourcing (typical pub/sub) the logical domain language is absolutely in terms of abstract topics, publishers, and consumers. But I think the nature of the beast requires us to manage an actual bus, n.b. connections, state management, etc. Details varying based on implementation.

Otherwise thanks for the great feedback. I don't think we need to rule any of your ideas out at this stage.

—

@jmank88

RFC 7464 - JSON Text Sequences could be a good candidate for the JSON streaming transport.

Great, thanks for the link, I've added it to Prior Art (above) and will review. In general I'd expect details like JSON to be implemented by user-provided serialization modules; I wouldn't want to enforce anything at that level in Go kit's opinions. It's also reminded me of EventSourcing and server-sent-events, also added to Prior Art.

—

@ChrisHines

Streaming RPC makes me think of WebSockets and gRPC streaming.

Agreed; I think gRPC streaming is probably the benchmark here, and (at least at the moment) I'd expect any Go kit streaming API to very closely mirror the gRPC model. It's also very close to how Asim modeled things in https://github.com/micro/go-micro.

[peterbourgon]

Starting with pubsub, as it's easiest and we have a lot of prior art. Here's my first draft of a set of interfaces, basically lifted verbatim from basvanbeek/pubsub via gizmo/pubsub.

// Publisher is a minimal interface for publishing messages to a pool of
// subscribers. Publishers are probably (but not necessarily) sending to a
// message bus.
//
// Most paramaterization of the publisher (topology restrictions like a topic,
// exchange, or specific message type; queue or buffer sizes; etc.) should be
// done in the concrete constructor.
type Publisher interface {
    // Publish a single message, described by an io.Reader, to the given key.
    Publish(key string, r io.Reader) error

    // Stop the publisher.
    Stop() error
}

// Subscriber is a minimal interface for subscribing to published messages.
// Subscribers are probably (but not necessarily) receiving from a message bus.
//
// Most paramaterization of the subscriber (topology restrictions like a topic,
// exchange, or specific message type; queue or buffer sizes; etc.) should be
// done in the concrete constructor.
type Subscriber interface {
    // Start returns a channel of messages that the caller should consume.
    // Failure to keep up with the incoming messages will have different
    // consequences depending on the concrete implementation of the subscriber.
    //
    // The channel will be closed when the subscriber encounters an error, or
    // when the caller invokes Stop, whichever comes first.
    Start() <-chan Message

    // Err returns the error that was responsible for closing the channel of
    // incoming messages.
    Err() error

    // Stop the subscriber, closing the channel that was returned by Start.
    Stop() error
}

// Message is a minimal interface to describe payloads received by subscribers.
// Clients may type-assert to more concrete types (e.g. pubsub/kafka.Message) to
// get access to more specific behaviors.
type Message interface {
    // Messages implement io.Reader to access the payload data.
    io.Reader

    // Done indicates the client is finished with the message, and the
    // underlying implementation may free its resources. Clients should ensure
    // to call Done for every received message.
    Done() error
}

Questions on my side:

• Is there a reason Publish can't take an io.Reader?
• Is there a reason Message can't simply implement io.Reader?
• For which use case(s) does Publish need its own key parameter?
• I can imagine a few ways to build an async (RPC) transport on top of this set of interfaces. Before I get too deep, does anyone know of any prior art?

Streaming interfaces to come...

[basvanbeek]

I can only talk about Kafka as I haven't seen other Go API's

• yes it could take an io.Reader. The most commonly used Go API for Kafka uses []byte for the message payload and it is what protobuf messages serialize to, so that's why I used it in my pubsub take.
• Key is not a broker generic item. In Kafka it is used as partitioning key for partitioned topics. I would like to be able to set a partitioning key for each message but it might make no sense for other broker back-ends.
• Async RPC and Prior Art... often built on top of JMS, AMQP, Stomp and involving some messageID, correlationID and a return path/topic.

[Ayiga]

I'm not sure about this current interface for Publisher. The way I've understood pubsub, the entity doing the publishing isn't necessarily sticking around after Publishing something. So I'm uncertain about the need to have a method called Stop on it, because what would it do? Stop everything you have published, stop future publishes? Is there some nuance with a potential implementation that may require this?

I'm going to try and mock this out with rabbitmq real quick to see how this might work. On the whole I'm having trouble visualizing the components that you've laid out, as you've laid them out. However, it's possible that I'm just missing something.

[Ayiga]

Ok, I've created an example, with some example programs, just to see if I could adhere to the interface that is laid out as-is. It turns out I can, and I can see the purpose of every piece so far.

If you'd like to see the implementation, for review or reference, you can find it here:
https://github.com/Ayiga/kit/tree/pubsub-rabbitmq/pubsub/rabbitmq

[peterbourgon]

So I'm uncertain about the need to have a method called Stop on it, because what would it do?

Effectively every implementation will require a stateful connection to the broker. Stop will gracefully terminate that connection, including (potentially) any in-flight requests. But, you got it in your example. Nice! Thanks. That's some validation.

[waigani]

This could potentially be a good base for an implementation: https://github.com/juju/pubsub.