Transaction support

[Steveb-p]

Feature Request

Introduce transaction support on database level as MongoDB starts to allow since version 4.0.

Q	A
New Feature	yes
RFC	yes/no
BC Break	should not be a BC break

Summary

I'd like to ask for opinion and discuss future solutions regarding transactions in Doctrine ODM.

MongoDB documentation regarding transactions: https://docs.mongodb.com/manual/core/transactions/

There are a few relevant bits of information regarding this feature. Specifically:

In most cases, multi-document transaction incurs a greater performance cost over single document writes, and the availability of multi-document transaction should not be a replacement for effective schema design.

Multi-document transactions are available for replica sets only. Transactions for sharded clusters are scheduled for MongoDB 4.2

What is Doctrine ODM desired approach on this? Introducing beginTransaction(), transactional() methods into DocumentManager? Since MongoDB transactions incur performance penalty, they probably will not be enabled on flush by default?

[jmikola]

@alcaeus: FYI, we have a new session API in the specifications pipeline for MongoDB drivers, which is intended to also improve the API around transactions. ETA for those APIs will be for the driver versions supporting MongoDB 4.2. It may be far off, but it's is something we can keep in mind when brainstorming about how best to integrate transactions into ODM.

On a related note: https://github.com/p-mongo/tests/blob/master/docs/mongoid-sessions-transactions.md

[Ludo444]

If somebody passes around this, it's pretty easy to implement with ODM 2.0 by passing session into flush() method $options

$session = $dm->getClient()->startSession();
$session->startTransaction();

// persist some documents
$dm->flush(['session' => $session]);

$session->commitTransaction();
$session->endSession();

Ideally you can override DocumentManager and store session there.

[alcaeus]

I'll just note here that the above is not suggested: this will lead to the UnitOfWork assuming that all data was written successfully, while the commitTransaction call might still lead to errors. In that case, the UnitOfWork will be left in a very inconsistent state.

[Ludo444]

Fair point, in my case it's used with DEFERRED_EXPLICIT policy and in case of any error I am persisting all documents again.

[DavideTriso]

Hello, I would like to contribute and work on an implementation of this feature. Could you please provide some advice and tell me how the desired implementation should work?

[alcaeus]

@DavideTriso That is great news!

The first step would be basic transaction handling: have the UnitOfWork start a session (if the feature is enabled), do its usual commit operations, then commit the transaction and handle any errors. From then on, we can see what other features we want to add. Note that we can't make this an "always-on" feature, as transactions require replica sets which may not be available. Thus, I'd add this to the configuration and allow users to turn transactions on or off as necessary.

To get started, I recommend forking from the "master" branch, which always points to the upcoming minor release and creating a draft PR as a discussion point. If you need more guidance, also feel free to contact us in the Doctrine Slack via the #mongodb-odm channel and we can go from there. The resources @jcaillot are a good starting point for the non-technical side of things, so go ahead and check those out if you need help.

Thanks to both of you for moving this along!

[DavideTriso]

Tanks for your advice @alcaeus and @jcaillot.

I have given a nearest look at the mongodb-odm code before starting to work at the implementation
This is how I would implement this feature:

1. Add a new enableSession attribute to Doctrine\ODM\MongoDB\Configuration which defaults to false.
2. When enableSession is true, have the DocumentManager create a new session in the constructor and store it.
3. In DocumentManager::flush(), let the user decide if they want the operation to be performed in a multi-document transaction or not (maybe by adding a bool $transactional = false argument to the flush method?)
4. If the user requested the flush operation to be performed in a transaction, start a transaction and commit it in UnitOfWork::commit().

Please let me know your thoughts!

[Steveb-p]

2. When enableSession is true, have the DocumentManager create a new session in the constructor and store it.

Session should be "opened" as late as possible, possibly only before flush call or explicit DocumentManager::beginTransaction. They should be as short-lived as possible. I'm not an expert in MongoDB, but I'm pretty sure for high throughput multi-document transactions you would start getting into deadlocks.

3. In DocumentManager::flush(), let the user decide if they want the operation to be performed in a multi-document transaction or not (maybe by adding a bool $transactional = false argument to the flush method?)

I believe current approach is to leave flush method without arguments (just recently a single entity/document flush was removed - true that it was mostly because of issues with ORM/ODM database desync, but I think it's also against the purpose those tools are for).
Introducing one option argument would also cause issues in cases where another similar option is introduced as second and so on.

An argument can be made that if transactions are enabled in configuration, then it should become the default way of handling flush operations. In this case transaction should be explicitly disabled instead (mostly for batch or simple operations).

Just my two cents ;)

[alcaeus]

To elaborate on what @Steveb-p commented:

Session should be "opened" as late as possible, possibly only before flush call or explicit DocumentManager::beginTransaction.
I agree. If we were to start the transaction in the constructor, this would also directly open the connection to the server, which is not desired. Since there are no plans to use sessions on operations outside of flush, we shouldn't keep an open session around.

I believe current approach is to leave flush method without arguments [...]
There is an $options array, but I'd have to double-check to see if the contents of it are passed through to the persisters performing the individual write commands. If the configuration says to flush with transactions, we do so. If we want to be able to configure this on a per-call basis, I'd take care of this in a separate PR.

IMO, this should happen as follows:

1. If configuration has enableCommitTransaction false, don't change current behaviour
2. If configuration has enableCommitTransaction true, start a session, start a transaction, and make sure the session is used in all commands sent to the server during the commit.

From there, we can expand this. For example, we may want to let the user pass their own session into flush, in which case we start a transaction in the provided session instead of starting a new session. I don't see this as part of this scope though - the most important part is making sure that all commands run in a transaction.

As for testing this, just some notes that might make your life easier:

• Sessions and transactions require a replica set. Using the provided configurations (if you're using mongo-orchestration) or mtools can help you set up a cluster to run tests
• For tests, using command monitoring allows you to inspect the commands UoW generates and to ensure that they all contain an lsid field for the session in question.
• While transactions are available starting MongoDB 4.0, you need to run MongoDB 4.2 for them to work on sharded clusters. To detect what topology the tests are running on, you can copy over the helpers that we use in the PHP library: https://github.com/mongodb/mongo-php-library/blob/474ec83cbe7f9a464b5edfdc4d5ca2ac6fded2f0/tests/FunctionalTestCase.php#L304..L338.

[delta4op]

@DavideTriso Hey,
Any update on transactions support ?

I tried implementing this using this blog.
https://zgadzaj.com/development/mongodb/mongodb-multi-document-transactions-in-symfony-4-with-doctrine-and-mongodb-odm-bundle#symfony-doctrine

Though I get the following error while passing session to flush()
The \"writeConcern\" option cannot be specified within a transaction. Instead, specify it when starting the transaction.

I want to develop my new application based on this package but I am just unsure because of its lack of support for multi-document transactions.

[DavideTriso]

Hello @gitwithravish,

the error you have reported is related to how MongoDB works and not to the ODM itself.
When using multi-document transactions the write concern must be set on the whole transaction, which means you cannot set this on a single operation (e.g. when calling $dm->persist()).

Implementing multi-document transactions in ODM is currently not possible without BCs.
See #2220 for the details.

As far as I know, if you need multi-document transactions you have to handle them yourself, for example by wrapping the DocumentManager methods in another class which handles the transaction (start, commit, rollback). Also remember you cannot use the ODM events when doing this, because the ODM is not aware of the transaction and will fire the postUpdate, postRemove, postPersist events as soon as the DB command is executed, but if you are inside a transaction the operations can still be rolled back if an error occurs.

[delta4op]

@DavideTriso
Thanks for reverting back.

I did not pass session in $dm->persist() method. I passed it in ->flush() method as mentioned here. This implementation is very old and it was demonstrated with version 1 odm package and it worked as well. However its not working in odm 2.2.

Currently I have my entire application built upon doctrine-odm package. But wherever I am required to use transactions, I am using mongodb php driver functions directly to start a session, pass the session to all transactions and then commit/abort.

Issue that I have to use 2 database communication approaches. Some using docrine-odm and some using pure php mongodb driver. Can you suggest me when and where do I pass a session to doctrine-odm package and make use of existing functionalities ?

I understand that there is no straightforward way to this but since you have been onto this since a long time, you will be able to help me out with this i believe.

Can you brief me about the following statement ?

if you need multi-document transactions you have to handle them yourself, for example by wrapping the DocumentManager methods in another class which handles the transaction (start, commit, rollback)

This approach has also done something similar but it didnt do the job for me.

[DavideTriso]

@gitwithravish I will reach you on Slack for further details