General reorg to improve content flow

Author: kbatuigas

modules/develop/pages/transactions.adoc

@@ -6,9 +6,11 @@
 
 Redpanda supports Apache Kafka®-compatible transaction semantics and APIs. For example, you can fetch messages starting from the last consumed offset and transactionally process them one by one, updating the last consumed offset and producing events at the same time.
 
+A transaction can span partitions from different topics, and a topic can be deleted while there are active transactions on one or more of its partitions. In-flight transactions can detect deletion events, remove the deleted partitions (and related messages) from the transaction scope, and commit changes to the remaining partitions.
+
 If a producer is sending multiple messages to the same or different partitions, and network connectivity or broker failure cause the transaction to fail, then it's guaranteed that either all messages are written to the partitions or none. This is important for applications that require strict guarantees, like financial services transactions.
 
-Transactions guarantee both exactly-once semantics (EOS) and atomicity.
+Transactions guarantee both exactly-once semantics (EOS) and atomicity:
 
 * EOS helps developers avoid the anomalies of at-most-once processing (with potential lost events) and at-least-once processing (with potential duplicated events). Redpanda supports EOS when transactions are used in combination with xref:develop:produce-data/idempotent-producers.adoc[idempotent producers].
 * Atomicity additionally commits a set of messages across partitions as a unit: either all messages are committed or none. Encapsulated data received or sent across multiple topics in a single operation can only succeed or fail globally.
@@ -23,31 +25,18 @@ endif::[]
 
 == Use transactions
 
-By default, the `enable_transactions` cluster configuration property is set to true. However, in the following use cases, clients must explicitly use the Transactions API to perform operations within a transaction.
-
-The required `transactional.id` property acts as a producer identity. It enables reliability semantics that span multiple producer sessions by allowing the client to guarantee that all transactions issued by the client with the same ID have completed prior to starting any new transactions.
-
-The two primary use cases for transactions are:
+By default, the `enable_transactions` cluster configuration property is set to true. However, in the following use cases, clients must explicitly use the Transactions API to perform operations within a transaction:
 
 * xref:develop:transactions.adoc#atomic-publishing-of-multiple-messages[Atomic (all or nothing) publishing of multiple messages]
 * xref:develop:transactions.adoc#exactly-once-stream-processing[Exactly-once stream processing]
 
-=== Transaction usage tips
-
-* A transaction can span partitions from different topics, and a topic can be deleted while there are active transactions on one or more of its partitions. In-flight transactions can detect deletion events, remove the deleted partitions (and related messages) from the transaction scope, and commit changes to the remaining partitions.
-* Ongoing transactions can prevent consumers from advancing. To avoid this, don't set transaction timeout (`transaction.timeout.ms` in Java client) to high values: the higher the timeout, the longer consumers may be blocked. By default, it's about a minute, but it's a client setting that depends on the client.
-ifndef::env-cloud[]
-* When running transactional workloads from clients, tune xref:reference:cluster-properties#max_transactions_per_coordinator[`max_transactions_per_coordinator`] to the number of active transactions that you expect your clients to run at any given time (if your client transaction IDs are not reused). The total number of transactions in the cluster at any one time is `max_transactions_per_coordinator * transaction_coordinator_partitions` (default is 50). When the threshold is exceeded, Redpanda terminates old sessions. If an idle producer corresponding to the terminated session wakes up and produces, its batches are rejected with the message `invalid producer epoch` or `invalid_producer_id_mapping`, depending on where it is in the transaction execution phase.
-Be aware that if you keep the default as 50 and your clients create a new ID for every transaction, the total continues to accumulate, which bloats memory.
-* When upgrading a self-managed deployment, make sure to use maintenance mode with a glossterm:rolling upgrade[].
-
-endif::[]
+The required `transactional.id` property acts as a producer identity. It enables reliability semantics that span multiple producer sessions by allowing the client to guarantee that all transactions issued by the client with the same ID have completed prior to starting any new transactions.
 
 === Atomic publishing of multiple messages
 
-With its event sourcing microservice architecture, a banking IT system illustrates the necessity for transactions well. A bank has multiple branches, and each branch is an independent microservice that manages its own non-intersecting set of accounts. Each branch keeps its own ledger, which is represented as a Redpanda partition. When a branch representing a microservice starts, it replays its ledger to reconstruct the actual state.
+A banking IT system with an event-sourcing microservice architecture illustrates the necessity for transactions. A bank has multiple branches, and each branch is an independent microservice that manages its own non-intersecting set of accounts. Each branch keeps its own ledger, represented as a Redpanda partition. When a branch (microservice) starts, it replays its ledger to reconstruct the actual state.
 
-Financial transactions (money transfers) require the following guarantees:
+Financial transactions such as money transfers require the following guarantees:
 
 * A sender can't withdraw more than the account withdrawal limit.
 * A recipient receives exactly the same amount sent.
@@ -61,6 +50,9 @@ Things get more complex with cross-branch financial transactions, because they i
 
 Redpanda natively supports transactions, so it's possible to atomically update several ledgers at the same time. For example:
 
+.Show multi-ledger transaction example:
+[%collapsible]
+====
 [,java]
 ----
 Properties props = new Properties();
@@ -131,6 +123,7 @@ while (true) {
     // TIP: notify the initiator of a transaction about the success
 }
 ----
+====
 
 When a transaction fails before a `commitTransaction` attempt completes, you can assume that it is not executed. When a transaction fails after a `commitTransaction` attempt completes, the true transaction status is unknown. Redpanda only guarantees that there isn't a partial result: either the transaction is committed and complete, or it is fully rolled back.
 
@@ -154,6 +147,9 @@ Postgresql -> topic(1) transform topic(2) -> warehouse
 
 The transformation reads a record from `topic(1)`, processes it, and writes it to `topic(2)`. Without transactions, an intermittent error can cause a message to be lost or processed several times. With transactions, Redpanda guarantees exactly-once semantics. For example:
 
+.Show exactly-once processing example:
+[%collapsible]
+====
 [,java]
 ----
 var source = "source-topic";
@@ -260,20 +256,40 @@ while (true) {
     }
 }
 ----
+====
 
-Different transactions require different approaches to handling failures within the application. Consider the approaches to failed or timed-out transactions in the provided use cases.
-
-* Publishing of multiple messages: The request came from outside the system, and it is the application's responsibility to discover the true status of a timed-out transaction. (This example doesn't use consumer groups to distribute partitions between consumers.)
-* Exactly-once streaming (consume-transform-loop): This is a closed system. Upon re-initialization of the consumer and producer, the system automatically discovers the moment it was interrupted and continues from that place. Additionally, this automatically scales by the number of partitions. Run another instance of the application, and it starts processing its share of partitions in the source topic.
+==== Exactly-once processing configuration requirements
 
-== Enabling exactly-once processing
-
-The default configuration of Redpanda supports exactly-once processing. Preserving this capability requires maintaining the following settings:
+Redpanda’s default configuration supports exactly-once processing. To preserve this capability, ensure the following settings are maintained:
 
 * `enable_idempotence = true`
 * `enable_transactions = true`
 * `transaction_coordinator_delete_retention_ms >= transactional_id_expiration_ms`
 
+== Best practices
+
+To help avoid common pitfalls and optimize performance, consider the following when configuring transactional workloads in Redpanda:
+
+* Ongoing transactions can prevent consumers from advancing. To avoid this, don't set transaction timeout (`transaction.timeout.ms` in Java client) to high values: the higher the timeout, the longer consumers may be blocked. By default, it's about a minute, but it's a client setting that depends on the client.
+ifndef::env-cloud[]
+* When running transactional workloads from clients, tune xref:reference:cluster-properties#max_transactions_per_coordinator[`max_transactions_per_coordinator`] to the number of active transactions that you expect your clients to run at any given time (if your client transaction IDs are not reused). 
++
+The total number of transactions in the cluster at any one time is `max_transactions_per_coordinator * transaction_coordinator_partitions` (default is 50). When the threshold is exceeded, Redpanda terminates old sessions. If an idle producer corresponding to the terminated session wakes up and produces, its batches are rejected with the message `invalid producer epoch` or `invalid_producer_id_mapping`, depending on where it is in the transaction execution phase.
++
+Be aware that if you keep the default as 50 and your clients create a new ID for every transaction, the total continues to accumulate, which bloats memory.
+* Transactional metadata is stored in the internal topic `kafka_internal/tx`. Over time, this topic can consume disk space. You can manage its disk usage by tuning the `transaction_coordinator_delete_retention_ms` and `transactional_id_expiration_ms` cluster properties.
++
+See also: xref:manage:cluster-maintenance/disk-utilization.adoc#manage_transaction_coordinator_disk_usage[Manage Disk Space]
+* When upgrading a self-managed deployment, make sure to use maintenance mode with a glossterm:rolling upgrade[].
+endif::[]
+
+== Handle transaction failures
+
+Different transactions require different approaches to handling failures within the application. Consider the approaches to failed or timed-out transactions in the provided use cases:
+
+* Publishing of multiple messages: The request came from outside the system, and it is the application's responsibility to discover the true status of a timed-out transaction. (This example doesn't use consumer groups to distribute partitions between consumers.)
+* Exactly-once streaming (consume-transform-loop): This is a closed system. Upon re-initialization of the consumer and producer, the system automatically discovers the moment it was interrupted and continues from that place. Additionally, this automatically scales by the number of partitions. Run another instance of the application, and it starts processing its share of partitions in the source topic.
+
 == Transactions with compacted segments
 
 Transactions are supported on topics with compaction configured. The compaction process removes the aborted transaction's data and all transactional control markers from the log. The resulting compacted segment contains only committed data batches (and potentially harmless gaps in the offsets due to skipped batches).