Transactions - behaviour or implementation?

[GazEdge]

Wondering how transactional consistency is tested in true behaviour TDD.

I spent some time thinking about it and decided that the output of the behaviour has to be transactional i.e. we get all the behaviour or non of the behaviour. The issue is that transactional consistency depends on how you implement the solution. This then would force implementation details to leak into the tests (in the form of a failing mock) to test that either all the behaviour happens, or none of it happens.

Any thoughts?

[valentinajemuovic]

@GazEdge I was thinking about this too, this is where there's possible issues regarding impelmentation purity.

So first question is, how is someone doing transactions?

2 approaches:

1. Having Unit of Work interface, and having repositories. The commit method would be on the unit of work only, not repository. However, the DB might in any case be doing auto-commit even if no explicit transaction has been opened.
2. Without unit of work. Someone directly performs commit within repository method.

I'm currently doing option 1, so I have a unit of work interface and repository interfaces. In the use case handler, I inject the repository(ies) and the unit of work. Let's say this is an SubmitOrderUseCase, I would check that, after executing the use case, that the FakeOrderRepository contains the newly created order, and that the commit() method has been executed exactly once. For invalid (negative scenarios, I would check that the FakeOrder Repository is unchanged (i.e. it does not have the new order) and unit of work has not been called.

The unit test does have awareness of the unit of work interface, but not of the real DB implementation

[mica16]

transactionPerformer.perform(trx => myUseCaseCode);
And use trx (ideally through currying like this: myRepo.save(obj)(trx)) to link the transaction to all the queries involved by the use case.

TransactionPerformer is injected in the use case, with two distinct implementations (Dummy one, for unit testing that basically calls the use case code), and a Real One implemented by what your ORM/query builder has to offer.
The later surround the use case code call with a beginTransaction and a commit/rollback.

Your use case stays pure of implementation details and every one is happy;)

To TDD the fact that the transactionPeformer is used for the use case, I provide an implementation that ... does not call the use case code! So the context of my test is a valid scenario where I would have a side effect, and I ensure that side effect does not occur, revealing the presence of the transactionPerformer that "controls" the use case code.

[valentinajemuovic]

@mica16 thanks for describing this possibility!

I haven't heard of the word TransactionPerformer (I only came across terms Unit of Work, Unit of Work Factory, interceptors/decorators/AOP). From reading this, to me it looks like the interceptor/decorator/AOP approach - i.e. the TransactionPerformer is the interceptor/decorator, and thus the use case remains pure since it is not aware of the transactions.

Just wanted to check if that's the meaning (I'd also be interested in your feedback to what @scalasm since I see some similarities)

[mica16]

TransactionPerformer is just the name I’ve chosen.

Each use case starts by transactionPerformer.perform with its code within.
That call is possibly in an abstract class to avoid duplicates.

[anaelChardan]

@mica16 Thanks for this Idea ;) How do you type the trx argument, as you are using it inside the use case code? and what type is exposing your repository awaiting for the transaction?

[scalasm]

Sorry for jumping into this but I am curious about the topic too and how to model it :)

My perspective is that a use case represents the primary concern of the application (the core domain) while transactions represent a secondary concern (system). That is, one represents my business goal (e.g. placing an order) and the other one describes a non-functional requirement (e.g., ensure that data stay consistent). You can have multiple secondary concerns like security & audit too, of course.

Aspects (as in Aspect Oriented Programming) or their poor cousins, the interceptors, are good fit for modelling these secondary, cross-cutting concerns and can be modelled separately.

Therefore, I would :

• model the primary concern using OOP (classes & c.) and have TDD (like in BDD) testing for this - this is my primary "unit"
• model the secondary concern using AOP (aspects & c.) and have a mix of mockist + classical TDD approach (e.g., I can extract the transaction strategy into a class and test its correct behavior + I will probably need to ensure that the aspect/interceptor correctly weaves the logic into target use cases) - for this concern my primary unit is the system logic (e.g., the transactional strategy) + the glue for weaving it into business logic
• make the use case class "declare" the need for secondary concern using annotation (e.g., like the famous @transactional for Spring guys) - this should allow things to still separated while connected.

From there on I start getting "pragmatic" (shame on me, wannabe clean programmer) - I leverage existing frameworks / solutions to achieve this (e.g., for transactions I would use Spring built-in transaction support, same for authorization / security, ...). I won't re-invent the wheel unless I don't trust my framework (but this is another story) or I have to implement my own system logic (e.g., custom integration or rules that I have to enforce across different use cases).

I believe that by "wisely" adapting the test pyramid it is possible to test both primary and secondary concerns and the ensure that integration between them works and still have robust tests against structural refactoring.

My 2 cents.

Ps: testing asynchronous use cases (e.g., orchestrated sagas of microservices) is another interesting topic but I will leave it for another entry :D

[valentinajemuovic]

@scalasm excellent comment! Based on this, I'm extending my response above to include the option you mentioned:

Aspect-oriented way of modeling transactions, whereby we would separately test the interceptor (so it means we don't test the transactions through our use case, but instead testing the interceptor task)
3.a. Keeping it "pure" - Making our own interceptor annotation/class (like a wrapper) and use it for the use cases, whereas in the infrastructure layer we would implement it with spring built-in transaction support
3.b. Being "pragmatic" - Using the spring transactional annotation directly in the use case
3.c. Another way, which I hadn't tried in Java but did something similar in .NET is to do search via reflection (e.g. find all the use case handlers) and then annotate them via reflection.
Some questions that arise are:

Distinguishing between use cases where the whole use case handle method is transactional versus the case where only a part of the use case handle method is transactional. If the whole method is transactional, then AOP is fine. However, if we need granularity, then the only option I see is injecting unit of work.
Batch-based processing. If there aren't any batches, then AOP is fine, but if we need batches or more fine-grained control, in that case, we can't even inject unit of work but instead create multiple unit of works (e.g. in a loop)
The great point is that you mention how these questions arise not just for transactions, but also for other cross-cutting concerns, like authorization, audit trails, etc... The choice between AOP (so the interceptor is tested separately) versus injecting those dependencies (affecting the use case test).

I think this brings us to a series of options and trade-offs. Like a tradeoff matrix.

[mikewojtyna]

Unit of work is not easily applied to any type of storage. This usually works best when working with relation dbs and multiple repositories. I usually simply stick with repository implementation (and manually call save each time in the use case) and take care that each use case changes just a single aggregate. Synchronization between different aggregates is then performed asynchronously

[mikewojtyna]

Creation of aggregates is a different use case than executing commands, which is the topic of this discussion. In such case you can use any batch methods. Unit of work is also fine. If there's some domain logic around batch process, you can use domain factories or services.

Regarding loading aggregates: in most cases you need to employ either pessimistic or optimistic locking. Optimistic is usually preferred in most cases to avoid performance bottlenecks. Anyway, my point is: it depends on the isolation level. Even if you have an active transaction (which btw is not supported in many nosql storages) it doesn't protect you from lost updates if you don't serialize all transactions.

[marciovmartins]

I'm also curious about the problem and the need for a transaction to save multiple users (like you mentioned). I usually don´t need this kind of transaction, like @mikewojtyna said, also for me the majority of my use cases invoke only one command.

Also curious to see why eventual consistency do not work in this case. About database transaction the framework take care for me for free. But between different resources (like db and messaging), we have an TransactionManager implementation which do lots of coordination to hold the messages and only if the database was successful the messages are send and the transaction is commit.

[valentinajemuovic]

Regarding what @mica16 wrote above:

Furthermore, how do you handle the case of batching creation (not update) of aggregates in a same use case.
I want all of them or neither if failure.

Yes, I had the exact same situation - for example in the case of bulk import, and it wasn't just plain insert/update into the DB, but rather there were some calculations involved. So I used a transaction for the use case.

Also, having a transaction around the use case allows all read queries (in the write side) to be viewed in the transaction scope (findById, etc). Integrity saved.

That's another important point - which for me occurred on most write-side queries. A typical example is: ApproveOrderUseCaseHandler, which has order = orderRepository.findById(orderId), then order.Approve(), then orderRepository.update(order). In this case, the only way to ensure correctness in concurrency is by having a transaction around the use case. Because, between the moment findById and order.Approve() it is possible that another use case got activated, and that order is no longer in a valid state to be approved.

For this reasons, currently by write-side use cases all have transactions around the use case itself.

@mikewojtyna I'd be interested in hearing your thoughts about these aspects? e.g. how would you implement ApproveOrderUseCaseHandler if there was no unit of work around the use case itself?

[marciovmartins]

@valentinacupac , for this kind of situation from the findById to approve there are strategies to guarantee that the order did not change because of some race condition.

It is possible to have some versioning together with the order and if the version is different between the one returned with the orderRepository.findById and the one in the database when you do an orderRepository.update, this means that the order retrieved before is out-dated and the command fails and should be retried.

My point is that you can push this kind of complexity to the infrastructure.

[mikewojtyna]

@valentinacupac

Regarding importing aggregates
Importing aggregates is usually nothing more than data management. It's perfectly fine to use transaction if you use the same database. But, even if there's no logic behind it, then you might get into issues when sharding comes into place. In highly scalable systems aggregates are not usually stored in the same db on the same machine. And, as we know, ACID dbs scale poorly comparing to e.g. NoSql storages. However, if there's some business process behind the import, then you indeed need to use something more. It can be a UseCase with some logic inside it (as it was in your case) or, preferably, domain factory or domain service. However, you can also use a separate aggregate just to model the "import" process. Then, you can use similar mechanisms as it's used in process managers or sagas. In fact, saga can be tracked using an aggregate pattern. We could call it e.g. LegacyImportProcess. Then, you can execute the import in distributed systems (even more: heterogenous dbs) and apply compensating actions when something goes wrong.

I'm usually working with horizontally scalable systems. I don't have the comfort of using a single local transaction to import the aggregates. Sure, I do use them, but only as part of the bigger process.

Now, about the ApproveOrderUseCaseHandler

You can employ the optimistic locking. And that's how it's usually done in most systems which don't perform huge amounts of writes to the same aggregate instance. Please note, that data integrity provided by unit of work pattern works only when having required transaction isolation level to protect against the lost update phenomenon. Repeatable read at minimum. You can also use locks (for Update in SQL). BUT, there's one big shortcoming with this approach... Even when you can use transactions and unit of work, you still need to protect against lost updates. Why? Because you are usually not making decisions inside the transaction. That's the external user who makes a decision based on the read model. And, at the moment of fetching the read model and using it to render the UI, it's already stale. Users make decisions which command to invoke (and with which parameters) based on the view presented to them. Even if you could use transactions with at least Repeatable read isolation level, all you would achieve is that aggregate logic would validate the command to ensure internal data integrity and decide whether to execute the command or not. But, obviously, it cannot make decision on its own which command and with which parameters to invoke ;).