Update LDR docs for GA & remove validated #19626

katmayb · 2025-05-21T20:22:08Z

This PR:

Removes the preview label from LDR in v25.2 to bring into GA.
Removes validated mode from all versions — for now, removed the mode option and immediate value because this is the default and only option.
Adds a description of how unique secondary indexes interact with LDR.

Preview

Navigate to pages from v25.2 Setup tutorial.

netlify · 2025-05-21T20:22:30Z

✅ Deploy Preview for cockroachdb-interactivetutorials-docs canceled.

Name	Link
🔨 Latest commit	`b433e3d`
🔍 Latest deploy log	https://app.netlify.com/projects/cockroachdb-interactivetutorials-docs/deploys/6838aadf03591400087d88de

github-actions · 2025-05-21T20:22:36Z

Files changed:

netlify · 2025-05-21T20:22:46Z

✅ Deploy Preview for cockroachdb-api-docs canceled.

Name	Link
🔨 Latest commit	`b433e3d`
🔍 Latest deploy log	https://app.netlify.com/projects/cockroachdb-api-docs/deploys/6838aadf3cf42500084e3bfe

netlify · 2025-05-21T20:31:01Z

✅ Netlify Preview

Name	Link
🔨 Latest commit	`b433e3d`
🔍 Latest deploy log	https://app.netlify.com/projects/cockroachdb-docs/deploys/6838aadf84f3cf00087a0d76
😎 Deploy Preview	https://deploy-preview-19626--cockroachdb-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

alicia-l2 · 2025-05-23T13:56:38Z

src/current/v25.2/manage-logical-data-replication.md

@@ -31,21 +29,16 @@ Conflicts at the KV level are detected when there is either:

 ### SQL level conflicts

-In `validated` mode, when a conflict cannot apply due to violating [constraints]({% link {{ page.version.version }}/constraints.md %}), for example, a foreign key constraint or schema constraint, it will be retried for up to a minute and then put in the [DLQ](#dead-letter-queue-dlq) if it could not be resolved. 
+When a conflict cannot apply due to violating [constraints]({% link {{ page.version.version }}/constraints.md %}), for example, a foreign key constraint or schema constraint, LDR will send the row to the [DLQ](#dead-letter-queue-dlq).


I believe in immeidate mode we don't even allow foreign key constraints

I removed this reference.

alicia-l2 · 2025-05-23T14:05:41Z

src/current/v25.2/set-up-logical-data-replication.md


 For more details, refer to the LDR [Known limitations]({% link {{ page.version.version }}/logical-data-replication-overview.md %}#known-limitations).

-When you run LDR in [`immediate` mode](#modes), you cannot replicate a table with [foreign key constraints]({% link {{ page.version.version }}/foreign-key.md %}). In [`validated` mode](#modes), foreign key constraints **must** match. All constraints are enforced at the time of SQL/application write.
+#### Unique secondary indexes


looks great -- thank you!

msbutler · 2025-05-23T14:22:28Z

src/current/v24.3/manage-logical-data-replication.md


 ### Dead letter queue (DLQ)

-When the LDR job starts, it will create a DLQ table with each replicating table so that unresolved conflicts can be tracked. The DLQ will contain the writes that LDR cannot apply after the retry period, which could occur if:
+When the LDR job starts, it will create a DLQ table with each replicating table so that unresolved conflicts can be tracked. The DLQ will contain the writes that LDR cannot apply after the retry period of a minute, which could occur if:


I think this list could be updated. I think it could read:

Presence of Unique index on destination table (see section below)

(only for 24.3) Loss of Quorum of the underlying ranges of the destination table

we also prevent DROP table on LDR tables on all versions given cockroachdb/cockroach#136172

Is this what you were thinking for the list (24.3) below @msbutler? lmk if I mixed things up at all...

for 24.3:

Presence of Unique index on destination table (see section below)

Loss of Quorum of the underlying ranges of the destination table

for 25.1/2

Presence of Unique index on destination table (see section below)

ah, did not see the updated commit. you can remove "table schemas do not match"

msbutler

reviewed current. looks good!

src/current/v24.3/set-up-logical-data-replication.md

src/current/v25.1/manage-logical-data-replication.md

msbutler · 2025-05-23T15:59:56Z

src/current/v25.1/manage-logical-data-replication.md

- The destination table was dropped.
- The destination cluster is unavailable.
- Tables schemas do not match.
+- The destination table is unavailable.


we can remove the unavailable line and the table schemas line for 25.1 and 25.2

msbutler · 2025-05-23T18:26:15Z

src/current/v24.3/manage-logical-data-replication.md

@@ -39,7 +39,6 @@ When the LDR job starts, it will create a DLQ table with each replicating table

 - The destination table is unavailable.
 - [Loss of quorum]({% link {{ page.version.version }}/architecture/replication-layer.md %}#overview) of the underlying [ranges]({% link {{ page.version.version }}/architecture/reads-and-writes-overview.md %}#range) in the destination table.


oof. sorry i should have caught this earlier. the first 2 bullets are essentially describing the same thing, so you could delete the first one.

katmayb · 2025-05-28T18:13:29Z

@alicia-l2 @jeffswenson can you check we landed where you wanted on the example?

alicia-l2

LGTM just a few comments

alicia-l2 · 2025-05-29T18:15:28Z

src/current/v25.2/set-up-logical-data-replication.md

-When you run LDR in [`immediate` mode](#modes), you cannot replicate a table with [foreign key constraints]({% link {{ page.version.version }}/foreign-key.md %}). In [`validated` mode](#modes), foreign key constraints **must** match. All constraints are enforced at the time of SQL/application write.
+#### Unique secondary indexes
+
+LDR cannot guarantee that the [_dead letter queue_ (DLQ)]({% link {{ page.version.version }}/manage-logical-data-replication.md %}) will remain empty if the destination table has a unique [secondary index]({% link {{ page.version.version }}/schema-design-indexes.md %}). The two clusters in LDR operate independently, so writes to one cluster can conflict with writes to the other.


i wonder if there is a way to rephrase the first sentence? we can't guarantee that the DLQ will remain empty even in situations where the destination table does not have a unique secondary index. maybe we can phrase it from more of a guidance perspective like 'to avoid DLQ entries we recommend no unique secondary indexes on the destination table?'

I rephrased the first sentence here, ok?

src/current/v25.2/set-up-logical-data-replication.md

jeffswenson

LGTM

rmloveland

LGTM!

rmloveland · 2025-06-02T17:40:07Z

src/current/v24.3/set-up-logical-data-replication.md

+INSERT INTO city (100, nyc); -- timestamp 4
+~~~
+
+_Timestamp 5:_ Range containing primary key `1` on the destination cluster is unavailable for a few minutes due to a network partition.


First mention of range in this section could link to https://www.cockroachlabs.com/docs/v25.2/architecture/glossary.html#range

'network partition' could link to https://www.cockroachlabs.com/docs/v25.2/cluster-setup-troubleshooting#network-partition

rmloveland · 2025-06-02T17:43:05Z

src/current/v24.3/set-up-logical-data-replication.md

+
+When the destination table includes unique [secondary indexes]({% link {{ page.version.version }}/schema-design-indexes.md %}), it can cause rows to enter the [_dead letter queue_ (DLQ)]({% link {{ page.version.version }}/manage-logical-data-replication.md %}). The two clusters in LDR operate independently, so writes to one cluster can conflict with writes to the other.
+
+If the application modifies the same row in both clusters, LDR resolves the conflict using _last write wins_ (LWW) conflict resolution. [`UNIQUE` constraints]({% link {{ page.version.version }}/unique.md %}) are validated locally in each cluster, therefore if a replicated write violates a `UNIQUE` constraint on the destination cluster (possibly because a conflicting write was already applied to the row) the replicating row will be applied to the DLQ. 


i think earlier you wrote that the LWW is based on MVCC timestamp - if you said that here as well a good reference link for MVCC is: https://www.cockroachlabs.com/docs/v25.2/architecture/storage-layer.html#mvcc

we also expose a crdb_internal_mvcc_timestamp but it's not really "officially supported" AFAICT since it uses the crdb_internal_* prefix

rmloveland · 2025-06-02T17:45:44Z

src/current/v25.1/set-up-logical-data-replication.md

+INSERT INTO city (100, nyc); -- timestamp 4
+~~~
+
+_Timestamp 5:_ Range containing primary key `1` on the destination cluster is unavailable for a few minutes due to a network partition.


same comments as above re: links to MVCC, network partition, range glossary definition, etc. (non-blocking suggestions only!)

rmloveland · 2025-06-02T17:46:06Z

src/current/v25.2/cockroachdb-feature-availability.md

@@ -87,10 +87,6 @@ The [`VECTOR`]({% link {{ page.version.version }}/vector.md %}) data type stores

 [Organizing CockroachDB {{ site.data.products.cloud }} clusters using folders]({% link cockroachcloud/folders.md %}) is in preview. Folders allow you to organize and manage access to your clusters according to your organization's requirements. For example, you can create top-level folders for each business unit in your organization, and within those folders, organize clusters by geographic location and then by  level of maturity, such as production, staging, and testing.

-### Logical data replication (LDR) for CockroachDB {{ site.data.products.core }}


KILL IT WITH FIRE

rmloveland · 2025-06-02T17:47:08Z

src/current/v25.2/set-up-logical-data-replication.md

+
+When the destination table includes unique [secondary indexes]({% link {{ page.version.version }}/schema-design-indexes.md %}), it can cause rows to enter the [_dead letter queue_ (DLQ)]({% link {{ page.version.version }}/manage-logical-data-replication.md %}). The two clusters in LDR operate independently, so writes to one cluster can conflict with writes to the other.
+
+If the application modifies the same row in both clusters, LDR resolves the conflict using _last write wins_ (LWW) conflict resolution. [`UNIQUE` constraints]({% link {{ page.version.version }}/unique.md %}) are validated locally in each cluster, therefore if a replicated write violates a `UNIQUE` constraint on the destination cluster (possibly because a conflicting write was already applied to the row) the replicating row will be applied to the DLQ. 


same suggestions as above re: sprinkling a few links re: range, MVCC, network partition

katmayb force-pushed the ldr-ga-updates branch from a64a05c to be1c0f1 Compare May 22, 2025 14:57

katmayb marked this pull request as ready for review May 22, 2025 15:01

katmayb force-pushed the ldr-ga-updates branch from be1c0f1 to f4af411 Compare May 22, 2025 15:09

katmayb requested review from alicia-l2 and msbutler May 22, 2025 15:19

alicia-l2 approved these changes May 23, 2025

View reviewed changes

msbutler reviewed May 23, 2025

View reviewed changes

katmayb requested a review from msbutler May 23, 2025 17:55

msbutler approved these changes May 23, 2025

View reviewed changes

katmayb requested a review from rmloveland May 23, 2025 18:57

msbutler approved these changes May 23, 2025

View reviewed changes

katmayb removed the request for review from rmloveland May 27, 2025 15:54

katmayb added 7 commits May 28, 2025 13:08

Update LDR docs for GA & remove validated

f1a9a3c

Add new supported schema changes for 25.2

f1155d9

Feedback 1 DLQ list + foreign keys

4cdccfa

Unique index suggestions incorporated

2cb0c7b

Correct dlq updates

45d776a

Fix

e23d513

Update the unique index example

9c4442c

katmayb force-pushed the ldr-ga-updates branch from 3db4c93 to 9c4442c Compare May 28, 2025 17:08

katmayb requested a review from alicia-l2 May 28, 2025 18:13

alicia-l2 approved these changes May 29, 2025

View reviewed changes

Alicia feedback

b433e3d

jeffswenson approved these changes May 29, 2025

View reviewed changes

katmayb requested a review from rmloveland May 30, 2025 12:39

rmloveland approved these changes Jun 2, 2025

View reviewed changes

		@@ -39,7 +39,6 @@ When the LDR job starts, it will create a DLQ table with each replicating table

		- The destination table is unavailable.
		- [Loss of quorum]({% link {{ page.version.version }}/architecture/replication-layer.md %}#overview) of the underlying [ranges]({% link {{ page.version.version }}/architecture/reads-and-writes-overview.md %}#range) in the destination table.


		When the destination table includes unique [secondary indexes]({% link {{ page.version.version }}/schema-design-indexes.md %}), it can cause rows to enter the [_dead letter queue_ (DLQ)]({% link {{ page.version.version }}/manage-logical-data-replication.md %}). The two clusters in LDR operate independently, so writes to one cluster can conflict with writes to the other.

		If the application modifies the same row in both clusters, LDR resolves the conflict using _last write wins_ (LWW) conflict resolution. [`UNIQUE` constraints]({% link {{ page.version.version }}/unique.md %}) are validated locally in each cluster, therefore if a replicated write violates a `UNIQUE` constraint on the destination cluster (possibly because a conflicting write was already applied to the row) the replicating row will be applied to the DLQ.

		@@ -87,10 +87,6 @@ The [`VECTOR`]({% link {{ page.version.version }}/vector.md %}) data type stores

		[Organizing CockroachDB {{ site.data.products.cloud }} clusters using folders]({% link cockroachcloud/folders.md %}) is in preview. Folders allow you to organize and manage access to your clusters according to your organization's requirements. For example, you can create top-level folders for each business unit in your organization, and within those folders, organize clusters by geographic location and then by level of maturity, such as production, staging, and testing.

		### Logical data replication (LDR) for CockroachDB {{ site.data.products.core }}

Update LDR docs for GA & remove validated #19626

Are you sure you want to change the base?

Update LDR docs for GA & remove validated #19626

Uh oh!

Conversation

katmayb commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Preview

Uh oh!

netlify bot commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for cockroachdb-interactivetutorials-docs canceled.

Uh oh!

github-actions bot commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Files changed:

Uh oh!

netlify bot commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for cockroachdb-api-docs canceled.

Uh oh!

netlify bot commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Netlify Preview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

msbutler left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

msbutler May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

katmayb commented May 28, 2025

Uh oh!

alicia-l2 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jeffswenson left a comment

Choose a reason for hiding this comment

Uh oh!

rmloveland left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

katmayb commented May 21, 2025 •

edited

Loading

netlify bot commented May 21, 2025 •

edited

Loading

github-actions bot commented May 21, 2025 •

edited

Loading

netlify bot commented May 21, 2025 •

edited

Loading

netlify bot commented May 21, 2025 •

edited

Loading

msbutler May 23, 2025 •

edited

Loading