CBG-5153: Implement resync using CBGT for a single node by RIT3shSapata · Pull Request #8057 · couchbase/sync_gateway

RIT3shSapata · 2026-02-05T15:55:44Z

CBG-5153

DRAFT PR

Describe your PR here...

Use bullet points if there's more than one thing changed

Pre-review checklist

Removed debug logging (fmt.Print, log.Print, ...)
Logging sensitive data? Make sure it's tagged (e.g. base.UD(docID), base.MD(dbName))
Updated relevant information in the API specifications (such as endpoint descriptions, schemas, ...) in docs/api

Dependencies (if applicable)

Link upstream PRs
Update Go module dependencies when merged

Integration Tests

GSI=true,xattrs=true https://jenkins.sgwdev.com/job/SyncGatewayIntegration/0000/

torcolvin

A few comments before I look more, I think the test is passing right now because it starts the non distributed resync.

db/background_mgr_resync_dcp.go

db/background_mgr_resync_dcp_test.go

base/dcp_sharded.go

torcolvin

I gave a non exhaustive review for any cases where import was used for code, but I don't think I found all cases. Definitely give a look through the code pathways to see if you can find other locations.

base/dcp_common.go

base/dcp_sharded.go

torcolvin · 2026-02-11T03:52:17Z

base/dcp_sharded.go

 // StartShardedDCPFeed initializes and starts a CBGT Manager targeting the provided bucket.
 // dbName is used to define a unique path name for local file storage of pindex files
-func StartShardedDCPFeed(ctx context.Context, dbName string, configGroup string, uuid string, heartbeater Heartbeater, bucket Bucket, spec BucketSpec, scope string, collections []string, numPartitions uint16, cfg cbgt.Cfg) (*CbgtContext, error) {
+func StartShardedDCPFeed(ctx context.Context, dbName string, configGroup string, uuid string, heartbeater Heartbeater, bucket Bucket, spec BucketSpec, scope string, collections []string, numPartitions uint16, cfg cbgt.Cfg, resyncIndex bool) (*CbgtContext, error) {


This could take DestType instead of a boolean parameter, which would be generally preferred since it would be more reasonable. The name DestType might not be appropriate, but it could be ShardedDCPFeedType instead?

It might be worth figuring out below what is actually needed to do to split on the feed type and whether to pass these arguments into this function or pass a type parameter. Review this comment after looking through the rest of the code and the comments.

base/dcp_sharded.go

db/import_pindex.go

db/background_mgr_resync_dcp.go

db/import_pindex.go

.github/workflows/ci.yml

base/dcp_sharded.go

torcolvin · 2026-02-18T04:03:39Z

base/dcp_sharded.go


 // Given a dbName, generate a unique and length-constrained index name for CBGT to use as part of their DCP name.
-func GenerateIndexName(dbName string) string {
+func GenerateIndexName(dbName string, feedID string) string {


We can pull this into a separate ticket but I think that this value needs to be <200 characters and we need to shorten this.

We can't break code that might work for import but as we make the index name longer, we want to make sure that this code works if there is a long database name.

sync_gateway/rest/config_manager.go

Line 832 in aa83a21

// standardMetadataID returns either the dbName or a base64 encoded SHA256 hash of the dbName, whichever is shorter.

is an example of how we handle this for a metadataID

I think creating a separate ticket is a good plan for this.

Like I mentioned in my comment earlier, I don't think it will ever exceed 34:
#8057 (comment)
we can pair up and discuss it further

torcolvin · 2026-02-18T04:04:45Z

base/dcp_sharded.go

 // the only index defined, and the name is safe.  In that case, continue using legacy index name
 // to avoid restarting the import processing from zero
-func dcpSafeIndexName(ctx context.Context, c *CbgtContext, dbName string) (safeIndexName, previousUUID string) {
+func dcpSafeIndexName(ctx context.Context, c *CbgtContext, dbName string, feedType ShardedDCPFeedType, feedID string) (safeIndexName, previousUUID string) {


I think we can pass feedType to both functions here, and through the callstack.

I did not understand this, could you please elaborate more?

Can you pass just feedType ShardedDCPFeedType? I think that the feedID should actually always be the same for resync, and doesn't need to be unique per run.

That was my mistake in the earlier specification.

db/background_mgr_resync_dcp_test.go

db/background_mgr_resync_dcp.go

torcolvin · 2026-02-18T04:37:50Z

db/background_mgr_resync_dcp.go

+
+		base.StoreDestFactory(loggingCtx, resyncDestKey, resyncDestFunc)
+
+		base.InfofCtx(loggingCtx, base.KeyJavascript, "ResyncID: %s Starting DCP resync for bucket: %q ", resyncLoggingID, base.UD(bucket.GetName()))


Suggested change

base.InfofCtx(loggingCtx, base.KeyJavascript, "ResyncID: %s Starting DCP resync for bucket: %q ", resyncLoggingID, base.UD(bucket.GetName()))

base.InfofCtx(loggingCtx, base.KeyAll, "Resync: Starting DCP resync for bucket: %q ", base.UD(bucket.GetName()))

I believe that the loggingCtx should already contain the CorrelationID and so it doesn't need to be duplicated. You should make sure that is true.

loggingCtx does have correlationID field but it is empty in this scenario. On checking the code, UserLogCtx does not populate the correlationID. So I think this is necessary here?

Agree, this is out of scope for fix in this PR, let me see about fixing this separately. We should still change the log key to be KeyAll and Resync: is redundant.

db/background_mgr_resync_dcp.go

torcolvin · 2026-02-19T03:22:18Z

base/constants_syncdocs.go

+//	format: _sync:{m_$}:cfg[groupID:]   (collections)
+//	format: _sync:cfg:[groupID:]   (default)


Suggested change

// format: _sync:{m_$}:cfg[groupID:] (collections)

// format: _sync:cfg:[groupID:] (default)

// format: _sync:{m_$}:resync_cfg[groupID:] (collections)

// format: _sync:resync_cfg:[groupID:] (default)

skeletal implementation of cbgt working on single node

6b222a1

RIT3shSapata self-assigned this Feb 5, 2026

test commit

6dd1179

RIT3shSapata assigned torcolvin and unassigned RIT3shSapata Feb 5, 2026

torcolvin reviewed Feb 5, 2026

View reviewed changes

fixes based on pr comments

046136b

RIT3shSapata requested a review from torcolvin February 6, 2026 13:47

RIT3shSapata added 2 commits February 10, 2026 20:17

fix unit test for distributed resync

7bbc91d

Squash merge main into CBG-5153

58e51db

torcolvin reviewed Feb 11, 2026

View reviewed changes

fixes based on pr comments

ee7d659

RIT3shSapata requested a review from torcolvin February 17, 2026 10:44

torcolvin mentioned this pull request Feb 18, 2026

Simplify sharded dcp API #8070

Merged

4 tasks

torcolvin reviewed Feb 18, 2026

View reviewed changes

fixes based on pr comments and lint fixes

f8159f2

RIT3shSapata requested a review from torcolvin February 18, 2026 10:01

torcolvin reviewed Feb 19, 2026

View reviewed changes

torcolvin mentioned this pull request Feb 23, 2026

Decouple BucketSpec from DatabaseContext #8075

Merged

4 tasks


		base.StoreDestFactory(loggingCtx, resyncDestKey, resyncDestFunc)

		base.InfofCtx(loggingCtx, base.KeyJavascript, "ResyncID: %s Starting DCP resync for bucket: %q ", resyncLoggingID, base.UD(bucket.GetName()))

	base.InfofCtx(loggingCtx, base.KeyJavascript, "ResyncID: %s Starting DCP resync for bucket: %q ", resyncLoggingID, base.UD(bucket.GetName()))
	base.InfofCtx(loggingCtx, base.KeyAll, "Resync: Starting DCP resync for bucket: %q ", base.UD(bucket.GetName()))

		// format: _sync:{m_$}:cfg[groupID:] (collections)
		// format: _sync:cfg:[groupID:] (default)

Comments

Conversation

RIT3shSapata commented Feb 5, 2026

DRAFT PR

Pre-review checklist

Dependencies (if applicable)

Uh oh!

torcolvin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

torcolvin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants