Change default of outstanding event republication to `false`

[robbertvanwaveren]

It's not clear to me if the framework has any mechanism / support to prevent multiple application instances for handling the same event (multiple times)?

If not, are there any concrete plans to support this through locking / partition or hooking in a messaging brokers or something?

[odrotbohm]

Can you elaborate? For Spring Modulith, an event is a JVM-internal construct to begin with. In the scope of our support, the event would only be “seen” by the application instance creating it.

That said, we're looking into approaches of how to externalize some domain events published to external brokers easily. I guess you could already use Spring Kafka, Integration or Cloud Stream manually to set up an arrangement like this. Is it that you're interested in?

[yewton]

For example, you have 3 Spring Boot app instances A, B, C running behind a load balancer.

Then the instance A published an event and a listener failed to process the event.

When these instances restart, the failed event re-published to listeners of all 3 instances?

[robbertvanwaveren]

Ok, maybe I'm still missing something but I clearly see multiple event submodules that support writing events to databases? How does that relate to events being JVM-internal?

[odrotbohm]

It's not clear to me if the framework has any mechanism / support to prevent multiple application instances for handling the same event (multiple times)?

That line here read as if you were concerned about the handling of the original event publication. That publication gets registered in the log, but the listeners are only invoked within the instance originally producing the event.

However, you have a point regarding the republication of outstanding events on application bootstrap. I have introduced a property to disable that for #240. It currently leaves the default to enable the functionality. I assume that for production usage, disabling this by default and selectively enabling it for one instance or rather resorting to an approach similar to ShedLock to handle the concurring instances?

[robbertvanwaveren]

ShedLock would be typical solution but that still only works if the datastore is the only way to generate events from. The way you describe it still results in-JVM event consumption competing with datastore-driven event comsuption even if the datastore-driven comsuption is guarded.

i.e. if instance 1 is running normal and is pumping events around, when instance 2 restarts it might read and process some of the events from instance 1 as well. Even if instance 2 is using ShedLock or the only instance with the property enabled.

The way I see it there are two options:

1. use the datastore + locking/partition as only way to receive events (slow but realiable, with deserialization logic always firing instead of rarely)
2. use some kind of partitioning like k8s statefulset deployment where each instance only reads its own events during restart from the datastore (fast but harder to operate)

[odrotbohm]

Given the feedback here, I am in inclined to flip the default for automatic event resubmission. I guess the ultimate problem lies in the fact that we cannot distinguish events whose processing has really already resulted in a failure from ones currently in processing. While we could, of course, add a failed marker in case of an exception issued by the listener, we might miss event publications that are left uncompleted as the machine died while processing the event.

Until we get that thought about and straightened out, I'd suggest the following:

• I'll tweak the default of the event re-publication to false so that it doesn't cause havoc in case an instance of a cluster is restarted. This still leaves room to deliberately trigger the resubmission by enabling the flag during a particular restart of an instance.
• Resubmission would then have to be handled by user code calling findIncompleteEventPublications() behind some SchedLock'ed code. It would apply an application-specific filter to, for example, only resubmit incomplete publications that have exceeded a certain timing threshold (e.g., are older than 2 minutes).

Once we gain a bit more experience on the strategies that users deploy in that regard, we can think about providing more convenient support out of the box.

WDYT?