Add connector plugin contract for external data sync #509

Context

We would like to support flow into Harper via webhook sync from Salesforce, Zendesk, and Confluence."

Today there is no webhook receiver pattern, no scheduled-sync framework, and no reference connectors. This issue proposes a plugin contract that covers the three modes from the guide: event-driven (webhook), scheduled cadence, and write-through.

Plugin contract

A connector is a Harper component exporting:

interface Connector {
  name: string;                          // 'salesforce', 'zendesk', ...
  version: string;
  configSchema(): JSONSchema;

  install(ctx: ConnectorContext): Promise<void>;
  uninstall(ctx: ConnectorContext): Promise<void>;

  schedule?: CronExpression;                              // null = no scheduled sync
  sync?(ctx, cursor: Cursor): AsyncIterable<Change>;      // scheduled / backfill
  webhook?(ctx, req: Request): AsyncIterable<Change>;     // event-driven
  write?(ctx, change: LocalChange): Promise<RemoteAck>;   // write-through (two-way)
}

type Change = {
  resource: string;          // target Harper resource
  op: 'upsert' | 'delete';
  primaryKey: any;
  record: Record<string, any>;
  remoteVersion?: string;    // for conflict detection
  sourceTimestamp: number;
};

ConnectorContext exposes scope.resources, scope.secrets, ctx.checkpoint(cursor), ctx.retry(fn, policy), ctx.dlq(change, reason).

Shared infrastructure provided by Harper

• hdb_connector_state system table — per-connector cursor, last-success ts, last-error, lag, throughput.
• hdb_connector_dlq dead-letter table — failed changes with replay action.
• hdb_connector_conflicts table — for queue-for-review write-through strategy.
• Webhook router — POST /connectors/:name/webhook dispatches to connector.webhook() with signature verification hook.
• Scheduler — extends core/server/jobs/jobs.js to invoke connector.sync() on cron with cursor passed in.
• Secret resolution — connector config references \${secrets.salesforce.refresh_token}; resolved at install and on rotation.
• Retry & rate-limit helpers — ctx.retry(fn, { policy: 'expo', jitter: true }); shared token-bucket rate limiter per connector instance.
• Observability — every Change emits analytics.connector_change { name, resource, op, latency_ms, success }.

Two-way conflict policy

A write-through connector declares one of:

• last-write-wins (default, timestamp comparison)
• remote-authoritative (remote wins, local field-merge)
• queue-for-review (write to hdb_connector_conflicts, no auto-merge)

CLI

harper connectors list
harper connectors install <name> --config <file>
harper connectors backfill <name> --since <ts>
harper connectors replay <name> --from-dlq [--filter ...]
harper connectors pause/resume <name>

Reference implementations to ship alongside v1

• @harperdb/connector-salesforce — REST + Platform Events CDC
• @harperdb/connector-zendesk — REST + Webhooks
• @harperdb/connector-confluence — REST + cursor scrape (no webhook API)

These are the three named in the PDF's flagship workload. Each can ship as a separate package/issue.

Replication interaction

By default, a connector runs as single-leader with HA failover on Fabric — exactly one node owns the cursor for a given connector instance. Failover via the existing replication leader election. Two-way writes (connector.write()) run on the writing node; cursor updates replicate normally.

Open decisions

1. Schema mapping — ship recommended schema per connector, allow override via config? Suggested: yes, both.
2. Backfill on install — pull historical data or watch-only? Suggested: flag --backfill defaults off; user opts in.
3. Should webhook receivers be a public, generic resource (independent of connectors) for users to roll their own? Suggested: yes — WebhookResource base class, connectors are one consumer.

Acceptance

• Connector interface and ConnectorContext defined in core.
• hdb_connector_state / hdb_connector_dlq / hdb_connector_conflicts system tables.
• Webhook router with signature-verification hook.
• Scheduled sync via existing jobs system.
• CLI commands (list/install/backfill/replay/pause/resume).
• One reference connector (recommend Zendesk — simplest API surface) lands as a separate package.
• Documentation: writing a connector, conflict policy choice guide.

Out of scope (for this issue)

• The reference connectors themselves (separate packages/issues).
• A generic UI for managing connectors (separate from CLI).