Skip to content

feat(plugin): add data sync replication from external to internal source#98

Open
whynice724-cell wants to merge 2 commits intoouterbase:mainfrom
whynice724-cell:feat/data-sync-issue72
Open

feat(plugin): add data sync replication from external to internal source#98
whynice724-cell wants to merge 2 commits intoouterbase:mainfrom
whynice724-cell:feat/data-sync-issue72

Conversation

@whynice724-cell
Copy link

@whynice724-cell whynice724-cell commented Feb 26, 2026

Summary

Implements issue #72 by introducing a DataSyncPlugin that replicates data from an external SQL source into internal DO SQLite with incremental checkpoints and cron scheduling.

What is included

  • New plugin: plugins/data-sync
    • POST /data-sync create/update sync task
    • GET /data-sync list tasks
    • DELETE /data-sync/:name delete task
    • POST /data-sync/run/:name run task immediately
  • Incremental pull strategy:
    • First run fetches ordered rows without cursor predicate
    • Next runs use cursorColumn > last_cursor_value
  • Internal table management:
    • Auto-create target table
    • Auto-add missing target columns
    • Create unique index on cursor column for conflict-safe upserts
  • Cron integration:
    • Scheduled sync execution via cron callbacks
    • Removes cron event on task deletion
  • Runtime integration:
    • Registers plugin in worker startup
    • Exposes plugin from dist/plugins.ts
  • Tests:
    • Added service tests for first sync, checkpoint sync, schema expansion/indexing, and config behavior

Claim

/claim #72

Test evidence

  • npx vitest run plugins/data-sync/service.test.ts

Demo video

  • Added to this PR branch: docs/demo/pr98-data-sync-demo.mp4 (commit dff89b7)
  • Demo covers:
    1. Data sync plugin file layout
    2. Plugin test pass output
    3. Exposed endpoints
    4. Incremental sync core functions
    5. Runtime + cron integration

Notes

  • Full test suite still has existing unrelated RLS failures in this repository; plugin-specific tests pass.

@whynice724-cell
Copy link
Author

Demo video added in branch as \ (commit dff89b7).\n\nWhat the demo shows:\n1. New plugin files under \n2. Passing tests:
RUN v2.1.8 /Users/emil/.openclaw/workspace/projects/starbasedb

✓ plugins/data-sync/service.test.ts (5 tests) 2ms

Test Files 1 passed (1)
Tests 5 passed (5)
Start at 01:39:32
Duration 178ms (transform 26ms, setup 0ms, collect 27ms, tests 2ms, environment 0ms, prepare 24ms)\n3. Plugin routes (, )\n4. Core incremental sync logic (, , )\n5. Runtime + cron integration (plugin registration + cron event management)\n\n/claim #72

@whynice724-cell
Copy link
Author

Demo video uploaded in this PR branch:

  • docs/demo/pr98-data-sync-demo.mp4 (commit dff89b7)

Coverage in the demo:

  1. New plugin files under plugins/data-sync
  2. Passing tests: npx vitest run plugins/data-sync/service.test.ts
  3. Plugin routes: GET /data-sync, POST /data-sync, DELETE /data-sync/:name, POST /data-sync/run/:name
  4. Core logic: buildExternalSelect, ensureTargetTable, runDataSyncTask
  5. Runtime + cron integration (DataSyncPlugin registration + cron event cleanup)

/claim #72

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant