Overhaul event publication lifecycle

[odrotbohm]

The persistent structure of an event publication currently effectively represents two states. Their default state captures the fact that a transactional event listener will have to be invoked eventually. The publication also stays in that state while the listener processes the event. Once the listener succeeds, the event publication is marked as completed. If the listener fails, the event publication stays in its original state.

This basic lifecycle is easy to work with, but has a couple of downsides. First and foremost, we cannot differentiate between publications that are about to be processed, ones that are processed and ones that have failed. Especially the latter is problematic, as a primary use case supported by the registry is to be able to recover from erroneous situations by resubmitting failed event publications. Developers usually resort to rather fuzzy approaches like considering events that have not been completed in a given time frame to be incomplete.

To improve on this, we’d like to move to a more sophisticated event publication lifecycle that allows to detect failed ones easier. One possible way to achieve this would be to introduce a dedicated status field, or — consistent with the current approach of setting a completion date — a failed date field which would need to be set in case an event listener fails. That step, however, might fail as well, as the erroneous situation that leads to the event listener failing in the first place. That’s why it might make sense to introduce a duration configuration property, after which incomplete event publications might be considered incomplete as well.
The feature bears a bit of risk, as we will have to think about the upgrade process of Spring Modulith applications. Existing apps might still contain entries in the database of incomplete event publications.

Ideas / Action Items

• Expand database table with a failedDate column.
• Introduce a configuration property, after which duration to consider a publication failed.
• Queries would need to be augmented with publishedDate before now() minus duration.
• CompletionRegisteringMethodInterceptor would need to issue the marking as failed on exception.
• IncompleteEventPublications would have to get a getFailedPublications().

Related tickets

• Allow differentiating between failed event publications and ones that are currently processed #663
• Overhaul event publication lifecycle #796
• Event resending: Performance Issues with large number of publications (Batching, Indexing, OoM) #1148

[breun]

Does not being able to tell that an event is being processed also mean that currently multi-instance apps are not an option?

I’m not a database expert, but I believe at least PostgreSQL supports row-level locking, which would allow concurrent processing of events by multiple instances, unlike some leader election method.

[annnoo]

@breun
We are currently using spring-modulith and the event-publication in one of our projects. Multi-Instance apps are possible, but the misconception we had is that the whole event publication log is not a "Message Queue" - it is currently a publication log. The whole table just keeps track what event's have been sent and allows you to retry them on startup - and he it can't distinguish between currently being processed events and "stuck" events.

The whole processing (sending event -> handler -> mark as finished) is not done by "submitting" the event to the table and a "worker" picks it up. The processing happens always on the instance the Event has been sent in the first place.

The publication_log is just there to keep track of what events have been processed. And the only information you currently have is if there is a completion_date on the event and when it got published.

We've built our own retry mechanism around the log, in which we are retrying events that are at least n-minutes old, but because it is an "publication_log" we have the issue that sometimes events get processed multiple times, when an event takes a long time to be processed (either because one of the steps take a long time or when a lot of events get sent and they can't get processed because all threads in our thread pool are already busy).

And that's where this misconception comes into place. If you work around the fact that the table is not used for processing at all than you may get into these issues if you build your retry mechanism.

My thoughts around this topic

We would really wish for a way to distinguish between events that are currently being processed and events that have failed, but in all implementations there are edge cases which you may or may not support in spring-modulith.

If you have a dedicated status field (e.g. SUBMITTED, PROCESSING, FINSIHED, FAILED) you can easily find out which events to retry, based on the failed field and can skip the PROCESSING ones - unless you have events that are struck, because the instance went down when processing them. In order to identify them in a multi-instance setup you would have to keep track of which instance is currently active.

If you handle it via a failedDate column you have to identify the currently being processed ones via an offset (as described in the issue description) - but here you have to be careful with longer running tasks, because as i mentioned, it can happen that it takes a few minutes until an event is picked up (because of all threads are being utilized)

In that case it could make sense to also have an column for when the Event got picked up and the handler is being triggered....

Conclusion

Thinking more about it, a big problem with the event_publication table is the misconception I mentioned. For example I expected that the event_publication could be seen as a "light" version of an event-externalization, but it does definetely does not work that way (and probably shouldn't be used in that way). From my gut feeling (and talking with colleagues about that) it feels like that I am not the only one that stepped into that.

Maybe I am jumping a little bit to a different topic, where this issue isn't about, but I think it should be clearer from the docs that the current event_publication mechanism should not be seen as as an externalized event processing mechanism and should show up the limitations of that.

And regarding what the users are expecting (and what @breun even mentioned)
Could it make sense to have "some kind" of event-externalization for postgres or other databases? Or at least some functionality for putting the message processing more to the database? I know this is a lot of work (and definetely not part of the issue - I just wanted to mention it here) , but I have a feeling that that's what developers want and "see" in the event_publication, what it is not.

Edit:
Removed

(unless you use the event externalization, meaning that events are sent and handled via Kafka, SQS, SNS... etc. - haven't touched this to be honest)

I just took a look into the docs and Event-Externalization means that you just publish events to other systems so that other applications can get them - not that you consume them from these.

[aahlenst]

It is unclear to me what Modulith's responsibility should be: ensuring the delivery of events only or also dealing with problems.

To ensure the delivery of events, failedDate seems questionable. Either the event has been processed or not. Handling failures is the responsibility of the application code. The only failures the application cannot handle are failures caused by the event delivery mechanism (event can no longer be deserialized, …). But I have a hard time imagining a scenario where I might find failedDate alone useful. Either I need more diagnostics (see next paragraph), or I have to choose between dropping the event or retrying forever.

For handling problems, failedDate alone is inadequate. For effectively dealing with failed deliveries, I would have to distinguish whether the failures might be transient (think OptimisticLockingFailureException, some network problem) or permanent (object concerning the event has been deleted, …). Furthermore, I would want to keep track of the number of failures and stop retrying after some attempts. This means we're quickly getting into the territory of Spring Retry and friends.

Therefore, I think Modulith should focus on the delivery side of things, for example, by better tracking the status (event queued, event processing, …) and providing better docs, and perhaps some callbacks, on how to deal with event delivery problems, like event classes that have been removed or event listeners that no longer exist.

[ooraini]

Event publication in modulith is simple and I think(my opinion) that expanding it will get you into a scope creep and you will not know when to stop.

Thinking about the problem differently, instead of an event handler that is recorded(and committed) as part of a transaction, what about a job getting scheduled. Now you are solving a different problem, running background jobs, which we have many solutions for, and the typically solve:

• Locking (this issue)
• Failure handing and retries
• ...

I'm experimenting with db-scheduler and it does exactly that.

[breun]

In order to identify them in a multi-instance setup you would have to keep track of which instance is currently active.

Does this mean you assume that only one instance in a multi-instance is actively doing work?

I’m still trying to wrap my head around this concept and its implications. I have the feeling I don’t grasp it completely yet, but my main feeling is now that it feels ‘dangerous’ to use this for high-traffic multi-instance services, which is a typical use case for me.

I’ve seen most Spring Modulith talks promoting using events instead of direct method calls across module boundaries, and I see the benefits of decoupling via events, but I feel that this event publication mechanism could become a performance/availability risk when it becomes a dependency for every call to a high-traffic module method that publishes an event for other module’s to listen for. In that sense a direct method feels ‘safer’ than using events: no database that could have a full table, or otherwise impact performance or availability.

Is this an irrational fear I have of the Spring Modulith event publication mechanism? Should I just configure Modulith to auto-remove completed events in a high-traffic scenario to avoid the event log from growing indefinitely and not worry about it too much otherwise?

[ivangsa]

Including a failed_attempts column would help implement a max retries logic as well as "Some other node is working on it" logic when failed_attempts == 0

[odrotbohm]

I've played with a couple of ideas that created new questions (who would've thought!? :D).

Event Publication States

Fundamentally, I'd like to introduce new states for event publication. A basic new statemachine would look like this:

The core goal here would be to introduce the new processing state, so that we can clearly differentiate between publications currently worked on and ones that have failed. Resubmissions would flip the state from failed to published. The transition from published to processing would be triggered by the interceptor decorating the listeners.

An alternative state design could introduce a dedicated resubmitted state as follows:

This would have the advantage of being able to differentiate between "fresh" submissions and resubmitted ones based on the state alone without having to refer to further publication details (more on that below). I don't see the additional state having an impact on the complexity of the necessary database interactions. So I think it might be worth introducing it as making concepts explicit is a valuable thing in general. I'd love to hear your opinions on that.

Event Publication Details

As many of you have expressed in the discussion so far, it might be useful to capture additional information in the publications.

• status – obviously needed
• lastFailureDate, lastResubmissionDate – I think it would be helpful to identify publications that have failed a long time ago and not been resubmitted. Or even more precise: if the time span between the two dates grows beyond a certain threshold.
• numberOfCompletionAttemps / numberOfFailures – I am leaning towards going with the former as it would allow us to increment that number during the state transition from published / resubmitted to processing, i.e. before the event listener logic is triggered. That logic might be hazardous to the system and the system crashing entirely would still leave us with the resubmission attempt recorded. An update of numberOfFailures would have to be the response to a listener or system that has already seen some kind of failure and is less likey to succeed.

These additional properties would allow us to offer the following functionality out of the box:

• explicitly select only failed publications and ones in progress
• safely automatically resubmit publications in multi-instance scenarios (select … for update)
• automatically resubmit publications stuck in published / resubmitted in case the system crashed in between the commit of the original business transaction and the actual execution of the listener
• automatic resubmission upon a configurable threshold (“don't retry more than 5 times”)
• guard resubmission by the age of a publication (“don't retry if the publication has failed more than 5 hours ago”)

Call to action

I'd love to hear if you think anything obvious is missing or — alternatively — if you think that such an arrangement would allow you to achieve the things you want or currently do (albeit in a more complex way). For the new functionality to be added, I see that we'll have to discuss which of these should be exposed as simple configuration options and which need callback APIs being introduced, but I'd like to postpone that discussion for now.

[ghferrari]

@odrotbohm I think the general proposal to store information about the event lifecycle is a very good one. In particular, I think it brings a brilliant opportunity to solve some significant problems with the current version of Spring Modulith.

Based on your conversation with @breun above and other discussions (#402, #727), I've understood that:

• Spring Modulith events are (at least initially) intended to be published and processed within the same JVM instance.
• Spring Modulith serializes its events and stores them in your database.
• It is possible and desirable to configure a Spring application to republish incomplete events on startup. For example, if your Spring app gets restarted before an event can be completed, it can be configured to "republish" these events, which I take to mean those events will then get processed within that new Spring instance.

This situation causes several problems:

• It is more difficult to use Spring Modulith in situations where more than one instance of your Spring app is running (i.e. when your app is horizontally scaled). As mentioned in the other discussions, using Spring Modulith in a horizontally scaled context requires somehow arranging that only one instance of Spring is configured to republish events on startup. Otherwise, you could end up with multiple Spring instances processing the same event - with potentially catastrophic consequences.
• Even if you (somehow) arrange for only one Spring instance to be republishing events, that means that the number of Spring app instances processing republished events will only ever be a maximum of 1. That is a clear bottleneck in a horizontally scaled application.
• There's also a related bottleneck with events when they are published the first time. Suppose you have 5 instances of your Spring app running, and suppose one of your instances publishes 1000 events. There is currently no mechanism to share that workload over the five instances. Instead, these events will (in the first instance) only be attempted by the instance that published them.

On a side note, I should point out I found none of the above information in the current Spring Modulith documentation, and the discussions I mentioned above seem to show that others also did not realise that Spring Modulith worked like this. So, my first suggestion is that the current Modulith documentation really needs revising to add these details. Perhaps (just a guess) this stuff is obvious to some people (I don't know, maybe that's always how Spring Application Events have worked...) but it wasn't obvious to me and it's not spelled out in the documentation. Given how common horizontal scaling is, I would say the documentation needs updating urgently because otherwise the effects might be catastrophic.

But, it seems to me that adding state information about the lifecycle events provides just the opportunity you need to solve ALL of the above problems - and to do be able to do it out of the box, without needing anything like schedlock.

From @odrotbohm 's draft state diagrams above, we see that events arrive into the database in state published and then move into state processing. Well, in that case, it seems like it's going to be ultra-simple to ensure that events are processed only by one instance of a Spring application - simply wrap the transition from published to processing in an appropriate database transaction and voila, only one app instance is going to be able to start processing an event. In a single stroke, this solves all the problems above:

• You can use horizontally scaled Spring Modulith app out of the box without needing anything like Schedlock
• You can, if you wish, make republish events on startup the default. Or at least, users can configure that across all instances of their app.
• All the scaling issues are removed - events and republished events can be processed by any instance of your app.
• Any changes to the documentation can be minimal because most of the problems above are now avoided rather than left as an exercise to the reader. And because of that, you would no longer need to use an external event broker like Kafka to get around the current limitations.

@odrotbohm I'm sure I've oversimplified a little bit how easy this would be (e.g. maybe a Spring modulith app will need to poll the database at interval to see if there are any newly published events that it can start processing...), but hopefully not by much. Does it sound feasible?