pubsub: support providers that don't do acks

[ijt]

NATS and Redis pubsub don't have acking, and it doesn't make sense to try to shoehorn them in by making message acks a no-op. That would mean the application code would be misleading.

To support pubsub providers without acks, I propose to add the following new types:

pubsub.FAFSubscription, whose Receive method returns a pubsub.FAFMessage
pubsub.FAFMessage that is mostly like a pubsub.Message but has no Ack method or associated bookkeeping

and corresponding interfaces within pubsub/driver. FAF here stands for "Fire and Forget".

Drivers would either implement driver.Subscription or driver.FAFSubscription, but probably not both.

[derekcollison]

Not sure I am liking FAF or that a message changes due to its Ack methods. If the message is truly at most once than Ack would be a no-op.

[ijt]

Fair enough. Let's try your approach.

…

On Wed, Feb 20, 2019, 3:00 PM Derek Collison ***@***.***> wrote: Not sure I am liking FAF or that a message changes due to its Ack methods. If the message is truly at most once than Ack would be a no-op. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1368 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAA8qiofZSNv5QCoM5P7sJzx1hECWHjHks5vPdOIgaJpZM4bFo2J> .

[derekcollison]

I have implemented the driver and necessary interfaces but things of course do not pass, so will look to work through those tomorrow. Thx.

[ijt]

Great. Glad to hear about this progress.

…

On Wed, Feb 20, 2019, 7:58 PM Derek Collison ***@***.***> wrote: I have implemented the driver and necessary interfaces but things of course do not pass, so will look to work through those tomorrow. Thx. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1368 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAA8qqdjBlyufMDFUivNviwH1UISToymks5vPhlJgaJpZM4bFo2J> .

[derekcollison]

Maybe a message should have knowledge if it requires an ack. pubsub.Message.RequiresAck() bool.

[ijt]

I'm not sure how that would be an improvement over having just pubsub.Message.Ack() that does a no-op on some providers. I can see a downside though: having to check that before calling Ack all the time.

[derekcollison]

Possibly not, but if I was reading someone's code and just saw an Ack that seemed to return no error I would probably think it did something. That make sense? I would rather see something like..

if msg.RequiresAck() {
    msg.Ack()
}

[ijt]

I don't think that works well from the perspective of app developers. They would have to scratch their heads about whether to include that logic every time they write code that receives a message.

The evidence that I see so far points towards needing a separate Go CDK subpackage to support at-most-once providers. I filed an issue for that: #1378. Closing this issue as redundant with that one.

[jba]

Robert and I just revisited the ideas here. We now think it might work to support ack-less providers with the same API, like so:

• The driver interface will have a HasAck method (all names open to bikeshedding).
• The concrete Subscription type will also have a HasAck method that just delegates to the driver. (Note: the method is on Subscription, not Message.)
• Ack will panic if called when HasAck is false. This is harsh, but we need to be harsh so nobody ever thinks they are using an at-least-once system when they aren't; the failure mode in that case is severe (messages that should be processed never are).

Unfortunately, as Robert pointed out to me, there is another distinction between these two kinds of systems that is harder to capture in the API. If a subscriber disconnects from a system like NATS and then reconnects, it will miss any messages that are sent while it's away. That won't happen for, say, Google PubSub (unless it disconnects for a long time). Let's call this persistence. I don't know if I should be as worried about persistence as I am about Ack. I also don't know if there are systems that support one of these features but not the other; for example, persistence with at-most-once delivery.

@vsaroopchand, does Azure Service Bus with Receive-and-Delete qualify as an at-most-once system with persistence?

I don't think that works well from the perspective of app developers. They would have to scratch their heads about whether to include that logic every time they write code that receives a message.

That's a good point, but I don't think it will be a problem in practice. When you design a system, you have to decide up front whether you want at-most-once or at-least once, and you will code accordingly. I actually think code like

if msg.HasAck() {
    msg.Ack()
}

indicates a poor design, which is why I don't think HasAck should be on Message. I think the main use of HasAck should look something like:

func main() {
    sub := foopubsub.OpenSubscription(...)
    if !sub.HasAck() {
        log.Fatal("need Ack")
}

[ijt]

@jba, what you have there is better than checking before every ack, but as far as I can tell it does nothing to support transient topics or subscriptions. It sounds like making the existing API more complicated without the gain in functionality that we would have by having a specialized API for at-most-once providers.

Acklessness is only one aspect of these providers, and not a very important one relative to their greater flexibility.

[vangent]

support transient topics or subscription

What do think we need to do to support those? IIUC, if we provide OpenTopic and OpenSubscription constructors (like we do for other drivers), they will work. mempubsub does this today in fact.

[jba]

@ijt, what's the "gain in functionality"? It seems like an at-most-once API would have less functionality.