Cross-platform distributed transactions

[roji]

#715 tracks porting the .NET Framework distributed transaction support over to modern .NET. Unfortunately, that support is tightly coupled to the Windows MSDTC service (and the OleTx protocol), so this works on Windows only. This issue tracks enabling distributed transactions in a cross-platform way. We are currently gauging user interest in this; if this is something you want, please upvote.

[paillave]

This feature is dramatically useful on a cross-platform target since docker takes more place every day in infrastructures.
At my side, I don't use WCF since ages, but TransactionScope remains extremely important when dealing with multiple Dbs, or simply to manage transaction in a simple and generic way.

[roji]

I've discussed this with various people, and there seems to be a trend away from 2-phase-commit distributed transactions in new applications utilizing cloud-native and microservice architectures. The alternatives are typically compensation-based patterns (e.g. Sagas), which allow some degree of eventual consistency in the system, but do not require a centralized, highly-available transaction manager, and do not introduce the additional dependencies and latency that 2PC does.

For more info on why 2PC may not be the right choice for the cloud, I recommend reading this. For a good architectural comparison of 2PC vs. compensation strategies, I recommend this post and this one (thanks to @scoriani for these).

Note that my intent isn't to start a big discussion on 2PC vs. Sagas on this issue - that's better done elsewhere. I do understand that Docker is a reason why cross-platform is more important; at this point we want to gauge how many users are interested in this, and why also why the alternatives (e.g. Sagas) aren't viable for their situation.

[ImGonaRot]

@roji I think you meant to say (#715 is for gauging that interest) since it was created back in 2019 and has lots of user comments versus #71769 which was just created 19 hours ago. I am a .NET contractor for several companies, one with a 100+ developers, that still use MSDTC for distributed transactions across multiple MS SQL servers. They do use some .NET Core for smaller projects but will be stuck with .NET 4.x framework for their main websites and EDI engines since those require heavy use of distributed transactions with MSDTC. As long as MS SQL server can use MSDTC for distributed transactions, I think you will find a lot of the bigger shops that use multiple MS SQL servers will sticking with good old ADO.NET / MSDTC / distributed transactions. I know it is Windows only but I'm not sure why MS would abandon a great key feature of MS SQL. That is just my opinion though. I do hope some day it is added to .NET 7+.

[roji]

I think you meant to say (#715 is for gauging that interest) since it was created back in 2019 and has lots of user comments versus #71769 which was just created 19 hours ago.

The problem is that #715 was generally about supporting distributed transactions in .NET Core; it's not clear whether people voting on it are interested in simply moving existing applications from .NET Framework to Core - on Windows (that's what we aim to enable in .NET 7.0), or whether they're looking for cross-platform support. I suspect that a lot of the interest is about moving away from .NET Framework, and much less is about being able to move from Windows to Linux/Mac. This is why this separate issue now explicitly tracks cross-platform support, as opposed to Windows-only support.

[paillave]

What I'm going to say may sound crazy, but I'll give it a try:
As Microsoft is a large powerful company, is it crazy to imagine that a new standard protocol is created for 2PC? This protocol would be simple enough to permit to any DB engine to be updated to support it. Then Microsoft could start to embed it in the DB engine it supports: SqlServer, CosmoDb, even Postgre...
I know this sounds like quite a journey ( 🤣 ) but this is a very useful and powerful pattern that needs an influent actor such as Microsoft to deal with this large scale problem.

[scoriani]

Let me clarify: Microsoft is not "abandoning" 2PC and MSDTC, period! This is and will be supported by SQL Server on both Windows and Linux deployments across the entire stack. That said, as the industry recognise (see the articles linked above), 2PC doesn't really cope with cloud native development and architectures so, for those scenarios, we're not really going to promote this approach. Hope this clarifies.

[roji]

Thanks for the clarification @scoriani. Yes; we're assuming that cross-platform supported would mainly be to support cloud-native deployments, but in those deployments 2PC is generally considered problematic (see the articles above). This is why we're looking to understand whether people are actually interested this. We're still committed to supporting 2PC and MSDTC on Windows, which is what we're targeting for .NET 7.0.

@paillave there's no need to invent a new protocol for 2PC; XA already exists and is well-supported across databases and transaction managers. If we did go ahead and implement cross-platform distributed transactions, that's what we'd build upon.

[rwasef1830]

Is the transaction scope API pluggable ? If not, can it at least be made with a provider model to allow others to implement an xa based transaction scope support ? The case I need is the simple synchronization between 2 postgres databases and two phase commit.

…

On Fri, 8 Jul 2022, 2:43 pm Shay Rojansky, ***@***.***> wrote: Thanks for the clarification @scoriani <https://github.com/scoriani>. Yes; we're assuming that cross-platform supported would mainly be to support cloud-native deployments, but in those deployments 2PC is generally considered problematic (see the articles above). This is why we're looking to understand whether people are actually interested this. We're still committed to supporting 2PC and MSDTC on Windows, which is what we're targeting for .NET 7.0. @paillave <https://github.com/paillave> there's no need to invent a new protocol for 2PC; XA already exists and is well-supported. If we did go ahead and implement cross-platform distributed transactions, that's what we'd build upon. — Reply to this email directly, view it on GitHub <#71769 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABDK5PNYJLN54VQADTK3NEDVTAPAZANCNFSM526BXHJQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[roji]

@rwasef1830 System.Transactions (TransactionScope) currently isn't pluggable in terms of the transaction manager - it's currently mostly hard-wired to using MSDTC. What you're say basically amounts to implementing cross-platform distributed transactions, as I've described above. It would roughly mean the following (this isn't exhaustive):

• Creating a new a transaction manager abstraction, to allow users to plug in 3rd-party TM (e.g. Redhat Narayana, Atomikos, Tuxedo, etc.
• Modifying Sys.Tx to allow users to provide their TM implementation (Globally? For each TransactionScope?)
• Modifying the interface between Sys.Tx and resource managers (i.e. database drivers) to allow supporting XA. For example, the recovery flow in Sys.Tx is currently geared around SQL Server/MSDTC, and isn't really compatible with XA. There are other incompatibilities.
• Modify all database drivers (e.g. SqlClient, Npgsql) to work with the modified Sys.Tx.
• As an alternative to retrofitting XA into Sys.Tx, we could create a separate, new API only for XA-based distributed transactions.

The above is, well, far from trivial, and most probably not something one can simply do as a 3rd-party thing.

Note that Npgsql did support distributed transactions for PostgreSQL on .NET Framework, via Sys.Tx. However, it did so as a volatile RM, meaning that recovery wasn't implemented; this was mainly because of the incompatible recovery flow mentioned above (see npgsql/npgsql#1378).

PS As a sort of hack, if you're just looking to do 2PC without recovery, you can manually send PostgreSQL PREPARE TRANSACTION and COMMIT PREPARED from your application. This should work, but if there's a crash in between, you'll have to somehow deal with it yourself.

[swh-cb]

@roji Do you know if it's technically possible to deploy MSDTC as a stand-alone service (i.e. without SQL Server) on platforms other than Windows in the future? If MSDTC would provide a new, platform-independent (HTTP?) API as an alternative to the existing COM interface for this stand-alone service, .NET could continue to use OleTx transactions with MSDTC for all platforms. This would have the advantage that the existing database drivers doesn't need to be changed. If an application wants to use a 3rd party XA transaction manager, .NET can still map the XA transaction to an internal OleTx transaction using the XA mapping functionality provided by MSDTC. For example the ODBC driver for SQL server can use a remote MSDTC (that is connected to the SQL server) to coordinate XA transactions platform-independently on Windows, Linux and macOS.

By the way, distributed transactions are already possible for cloud-native deployments if you deploy your apps to Azure with Azure SQL only. Azure SQL supports distributed transactions (called elastic transactions) in a limited way using a non-MSDTC transaction coordinator. For this to work in .NET, the SQL client performs a PSPE enlistment with a custom promoter type on Azure. Custom promoter types were introduced in .NET Framework 4.6.1 and ported to .NET Core to allow external non-MSDTC transaction managers in System.Transactions for delegated transactions only.