Eloquent events aware of database transactions.

[TitasGailius]

I has a situation where a database transaction fails but an eloquent event is still being dispatched.

To illustrate this problem take this test into consideration:

    /** @test */
    public function it_fires_an_eloquent_event_for_committed_transactions()
    {
        Event::fake();

        DB::transaction(function () {
            factory(User::class)->create();
        });

        Event::assertDispatched("eloquent.created: App\User");
    }

Everything works as expected. The user gets stored into the database and an event is fired.

Although if we manually fail the transaction, an event would still be fired:

    /** @test */
    public function it_fires_an_eloquent_event_for_committed_transactions()
    {
        Event::fake();

        DB::beginTransaction();
        factory(User::class)->create();
        DB::rollBack();

        Event::assertDispatched("eloquent.created: App\User");
    }

Problem:

The user is not stored into the database but an event is still being fired.

This creates situations where a specific action would be done even if they shouldn't. "Welcome email" is a perfect example, it would try to send the email even if the user is not there.

Suggestion:

Collect pending eloquent events and dispatch them only after the transaction is committed.

[TitasGailius]

Also in this example, an event is dispatched even if a user is not stored yet.

    /** @test */
    public function it_fires_an_eloquent_event_for_committed_transactions()
    {
        Event::fake();

        DB::transaction(function () {
            factory(User::class)->create();

            sleep(10);
        });

        Event::assertDispatched("eloquent.created: App\User");
    }

[mfn]

This generally a problem with all the events/listeners/dispatchers: nothing is aware of Transactions. I haven't come around an ActiveRecord system yet which doens't have this inherent problem.

My solution so far:

• built an abstraction around the transaction which covers the database and an in-memory queue of "collected events"
• every event I dispatch is first collected in aforementioned in-memory queue
• once the database transaction successfully finishes, all the jobs collected in-memory are dispatched

The "abstraction" I mentioned is a custom solution I built for that purpose: https://github.com/mfn/php-transaction-manager
You nest the transactions, the last/inner transaction is supposed to be Laravels database transaction and as it unravels the "success" (commit) phase, it first commits the inner database transaction and then the others which dispatches the collected jobs "for real".

Ruby has a similar open issue: rails/rails#26045

The also talk and consider another use-case which isn't "un-important": handling of such events when the Queue system is database based.

I personally don't use database-based queuing systems but they definitely have their use-case especially together with transactions.

TL;DR: innocent looking but complex topic 😄

[TitasGailius]

This generally a problem with all the events/listeners/dispatchers.

Well, I disagree about custom events because you can put these outside of a transaction manually, while eloquent is a different story. Also, there might be some events that you want to dispatch even if a transaction is not committed.

As a simple solution I think deferring only eloquent events would be fine.

[mfn]

As a simple solution I think deferring only eloquent events would be fine.

I guess it's context dependent (not disagreeing though).

I think the "perfect" solution would be an issue way to let the developer choose if and what events should honor transactions or not. Then again, maybe this is already the solution I had to implement: custom.

[deleugpn]

This is a classic problem from Message Driven systems. I had a few discussions about this at my workplace and the older coworker mentioned that this problem is at least 25 years old.

We pinned it down to a very subtle difference on the message sent to the Bus / Topic: A request for change vs A change happened. Pub/Sub systems rely heavily on the 2nd concept; when a message is published, it means something genuinely happened and the world has to deal with that however they see fit. The systems interested in that change will subscribe to the Message / Topic.
The 1st concept differs in the sense that a commitment hasn't been reached yet. In this case, you're being hostage of the 1st approach. You're putting a Listener / Subscriber in a Topic that only represents a Request For Change, not a genuine change. If you can rollback the user creation, then you should not listen to eloquent.created. You should dispatch your own custom event as soon as you commit. Only you know if the commit will actually happen or not.
By creating your custom event and dispatching it only after the commit takes effect, then your Topic / Event will actually carry the meaning of "A change has happened, do whatever you need to do about it".
The difference between these two are very subtle and lead to a lot of problems down the road because it's really easy to misuse a Topic / Event as an effective change when in fact it can simply be a request for a change.
One of the reasons why this is a delicate subject it's because it comes down to semantics. The semantic of a topic may change overtime without a clear reason.
Imagine you had a system that created a user and sent a welcome email without transactions. The eloquent topic / event was a genuine "change has happened". 3 months later your product requirements change and now the user creation is split into two steps and require a database transaction. At this moment, automatically, the semantic of the topic changed without any clear notice. As soon as you rollback because the 2nd step failed, the event dispatched no longer means "a change has happened". It instead changed to "A request for change" and the subscribers now either need to be rollbackable or move on to a secondary topic / event.

[matt-allan]

As a simple solution I think deferring only eloquent events would be fine.

The problem with this approach is sometimes you want the event to be handled inside of the transaction, and sometimes you don't.

For example, I might want to add an entry to the permissions_audit_log table every time someone changes something in the user_permissions table. I would want that listener to receive the event inside of the transaction, because I want to guarantee the audit log entry is written if the permissions write succeeds. If the event was received outside of the transaction there is no guarantee the audit log write would succeed.

On the other hand, if I am using an event to kick off a job I don't want to dispatch the job until the transaction commits. Otherwise the worker could run before the data it needs to use is visible.

I've sometimes seen these two types of events called 'domain events' and 'integration events'.

The way we solve this at work is by using a marker interface (Similar to ShouldBroadcast in Laravel) to denote events that are meant for other processes. If the event matches that interface and we are in a transaction, we delay firing it and queue it up in memory. Once the transaction commits we fire all of the queued events. If the transaction rolls back we discard all of the queued events.

It's possible to write an event subscriber to do this. You don't need to change how the Dispatcher works at all. Eloquent already emits transaction began, committed, and rolled back events that you can hook into. For example:

class TransactionHoldingSubscriber
{
    /**
     * @var bool
     */
    private $inTransaction = false;

    /**
     * @var array
     */
    private $queued = [];

    public function subscribe(Dispatcher $events): void
    {
        $events->listen(TransactionBeginning::class, function () {
            $this->inTransaction = true;
        });

        $events->listen(TransactionCommitted::class, function () use ($events) {
            $this->inTransaction = false;

            while ([$event, $payload] = array_pop($this->queued)) {
                $events->dispatch($event, $payload);
            }
        });

        $events->listen(TransactionRolledBack::class, function () {
            $this->inTransaction = false;
            $this->queued        = [];
        });

        $events->listen('*', function (string $event, array $payload) {
            if ($this->inTransaction) {
                $this->queued[] = [$event, $payload];
                return false;
            }
        });
    }
}

You can easily expand that example to only hold events matching an interface or pattern, i.e. if ($this->inTransaction && $event instanceof .... This also works well for queuing jobs.

This could be implemented as a package if someone feels inspired 😁

Note that there is a small risk the process might crash between the DB transaction committing and the events being fired. If that is unacceptable for you you can write events/jobs to the database first.

[mvanduijker]

Maybe we can take a look how Rails does it: https://guides.rubyonrails.org/active_record_callbacks.html#transaction-callbacks

[deleugpn]

Laravel already support Database events by emitting Illuminate\Database\Events\TransactionBeginning, Illuminate\Database\Events\TransactionCommitted and Illuminate\Database\Events\TransactionRolledBack, but that is not what this topic is about.

[mfn]

Also they are different from what Rails transaction callback does: Rails keeps track of the models mutated in the transaction and fires callbacks on a model basis.

Laravel only has the generic callback, void of any models.

[mvanduijker]

Just a brain dump by using a trait:

Trait:

trait TransactionalAwareEvents
{
    protected static $queuedTransactionalEvents;

    public static function bootTransactionalAwareEvents()
    {
        $events = static::getEventDispatcher();

        static::created(function (Model $model) {
            if (!DB::transactionLevel()) {
                return;
            }
            self::$queuedTransactionalEvents['created'][] = $model;
        });

        static::saved(function (Model $model) {
            if (!DB::transactionLevel()) {
                return;
            }
            self::$queuedTransactionalEvents['saved'][] = $model;
        });

        static::updated(function (Model $model) {
            if (!DB::transactionLevel()) {
                return;
            }
            self::$queuedTransactionalEvents['updated'][] = $model;
        });

        static::deleted(function (Model $model) {
            if (!DB::transactionLevel()) {
                return;
            }
            self::$queuedTransactionalEvents['deleted'][] = $model;
        });

        $events->listen(TransactionCommitted::class, function () {
            foreach (self::$queuedTransactionalEvents as $eventName => $models) {
                foreach ($models as $model) {
                    $model->fireModelEvent('after_commit_' . $eventName);
                }
            }
            self::$queuedTransactionalEvents = [];
        });

        $events->listen(TransactionRolledBack::class, function () {
            foreach (self::$queuedTransactionalEvents as $eventName => $models) {
                foreach ($models as $model) {
                    $model->fireModelEvent('after_rollback_' . $eventName);
                }
            }
            self::$queuedTransactionalEvents = [];
        });
    }

    public static function afterCommitCreated($callback)
    {
        static::registerModelEvent('after_commit_created', $callback);
    }

    public static function afterCommitSaved($callback)
    {
        static::registerModelEvent('after_commit_saved', $callback);
    }

    public static function afterCommitUpdated($callback)
    {
        static::registerModelEvent('after_commit_updated', $callback);
    }

    public static function afterCommitDeleted($callback)
    {
        static::registerModelEvent('after_commit_deleted', $callback);
    }

    public static function afterRollbackCreated($callback)
    {
        static::registerModelEvent('after_rollback_created', $callback);
    }

    public static function afterRollbackSaved($callback)
    {
        static::registerModelEvent('after_rollback_saved', $callback);
    }

    public static function afterRollbackUpdated($callback)
    {
        static::registerModelEvent('after_rollback_updated', $callback);
    }

    public static function afterRollbackDeleted($callback)
    {
        static::registerModelEvent('after_rollback_deleted', $callback);
    }
}

Model:

class MyModel extends Model
{
    use TransactionalAwareEvents;

    public static function boot()
    {
        parent::boot();

        static::afterRollbackCreated(function ($model) {
            dump("Created MyModel::{$model->id} is rolled back");
        });
    }
}

Tinker:

\DB::beginTransaction();
MyModel::create();
\DB::rollback();

[TitasGailius]

The problem with this solution is that event listeners defined in your trait must always be handled first. I solved my case by simply overriding fireModelEvent model method. It collects all model events and dispatches them only when the transaction is completed. It worked perfectly for my case and the solution is not too magical.

[mvanduijker]

@TitasGailius I understand your case. In this case we add extra events "after_commit_created", "after_commit_updated", etc and keep the working of the regular events in tact (they are performed within the transaction). You get the flexibility to act on events within the transaction (ie "created") and act on events after the transaction is committed (ie "after_commit_created") or rolled back (ie "after_rollback_created". Not that much different as in Rails. As far as I can see the ordering of the listeners doesn't matter. We just record what happened with the model and fire the correct event after the transaction is committed or rolled back.

[mvanduijker]

I created a package for transactional events https://github.com/mvanduijker/laravel-transactional-model-events

This will provide a trait you can put on your models and will fire extra events after commit or rollback. I am going to test it in one of my projects. If there is anyone willing to review / test it, I would really appreciate it.