Repository navigation
Implement P1 protocol layer: Widget contracts, ETL, Connectors, Data Sync, and OData v4 - #293
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…TL, Connectors, Sync, and OData v4 Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Implements new “P1” protocol-layer schema definitions to support custom UI widgets, enterprise connectors/ETL, bidirectional data sync, and OData v4-compatible querying.
Changes:
- Added widget contract schemas (lifecycle, events, properties, manifest) and updated UI widget protocol exports/docs.
- Added automation protocols for connectors, ETL pipelines, and data sync; updated automation exports and docs navigation.
- Added OData v4 protocol schemas (query/options, response/error, metadata) and exported them via API index; generated JSON Schema + reference docs.
Reviewed changes
Copilot reviewed 91 out of 91 changed files in this pull request and generated 14 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/spec/src/ui/widget.zod.ts | Adds widget lifecycle/event/property/manifest schemas to formalize custom widget contracts. |
| packages/spec/src/automation/sync.zod.ts | Introduces data sync protocol schemas and helper factories for sync configuration. |
| packages/spec/src/automation/index.ts | Exports new automation protocol modules (etl/connector/sync). |
| packages/spec/src/automation/etl.zod.ts | Introduces ETL pipeline protocol schemas and helper factories. |
| packages/spec/src/automation/connector.zod.ts | Introduces connector registry protocol schemas and helper factories. |
| packages/spec/src/api/odata.zod.ts | Introduces OData query/response/error/metadata schemas and helper utilities. |
| packages/spec/src/api/index.ts | Exports the new OData protocol module from the API package. |
| packages/spec/json-schema/ui/WidgetProperty.json | Generated JSON Schema for WidgetProperty. |
| packages/spec/json-schema/ui/WidgetManifest.json | Generated JSON Schema for WidgetManifest. |
| packages/spec/json-schema/ui/WidgetLifecycle.json | Generated JSON Schema for WidgetLifecycle. |
| packages/spec/json-schema/ui/WidgetEvent.json | Generated JSON Schema for WidgetEvent. |
| packages/spec/json-schema/automation/SyncMode.json | Generated JSON Schema for SyncMode. |
| packages/spec/json-schema/automation/SyncExecutionStatus.json | Generated JSON Schema for SyncExecutionStatus. |
| packages/spec/json-schema/automation/SyncExecutionResult.json | Generated JSON Schema for SyncExecutionResult. |
| packages/spec/json-schema/automation/SyncDirection.json | Generated JSON Schema for SyncDirection. |
| packages/spec/json-schema/automation/OperationType.json | Generated JSON Schema for OperationType. |
| packages/spec/json-schema/automation/OperationParameter.json | Generated JSON Schema for OperationParameter. |
| packages/spec/json-schema/automation/OAuth2Config.json | Generated JSON Schema for OAuth2Config. |
| packages/spec/json-schema/automation/FieldMapping.json | Generated JSON Schema for FieldMapping. |
| packages/spec/json-schema/automation/ETLTransformationType.json | Generated JSON Schema for ETLTransformationType. |
| packages/spec/json-schema/automation/ETLTransformation.json | Generated JSON Schema for ETLTransformation. |
| packages/spec/json-schema/automation/ETLSyncMode.json | Generated JSON Schema for ETLSyncMode. |
| packages/spec/json-schema/automation/ETLSource.json | Generated JSON Schema for ETLSource. |
| packages/spec/json-schema/automation/ETLRunStatus.json | Generated JSON Schema for ETLRunStatus. |
| packages/spec/json-schema/automation/ETLPipelineRun.json | Generated JSON Schema for ETLPipelineRun. |
| packages/spec/json-schema/automation/ETLPipeline.json | Generated JSON Schema for ETLPipeline. |
| packages/spec/json-schema/automation/ETLEndpointType.json | Generated JSON Schema for ETLEndpointType. |
| packages/spec/json-schema/automation/ETLDestination.json | Generated JSON Schema for ETLDestination. |
| packages/spec/json-schema/automation/DataSyncConfig.json | Generated JSON Schema for DataSyncConfig. |
| packages/spec/json-schema/automation/DataSourceConfig.json | Generated JSON Schema for DataSourceConfig. |
| packages/spec/json-schema/automation/DataDestinationConfig.json | Generated JSON Schema for DataDestinationConfig. |
| packages/spec/json-schema/automation/ConnectorTrigger.json | Generated JSON Schema for ConnectorTrigger. |
| packages/spec/json-schema/automation/ConnectorOperation.json | Generated JSON Schema for ConnectorOperation. |
| packages/spec/json-schema/automation/ConnectorInstance.json | Generated JSON Schema for ConnectorInstance. |
| packages/spec/json-schema/automation/ConnectorCategory.json | Generated JSON Schema for ConnectorCategory. |
| packages/spec/json-schema/automation/Connector.json | Generated JSON Schema for Connector. |
| packages/spec/json-schema/automation/ConflictResolution.json | Generated JSON Schema for ConflictResolution. |
| packages/spec/json-schema/automation/AuthenticationType.json | Generated JSON Schema for AuthenticationType. |
| packages/spec/json-schema/automation/Authentication.json | Generated JSON Schema for Authentication. |
| packages/spec/json-schema/automation/AuthField.json | Generated JSON Schema for AuthField. |
| packages/spec/json-schema/api/ODataResponse.json | Generated JSON Schema for ODataResponse. |
| packages/spec/json-schema/api/ODataQuery.json | Generated JSON Schema for ODataQuery. |
| packages/spec/json-schema/api/ODataMetadata.json | Generated JSON Schema for ODataMetadata. |
| packages/spec/json-schema/api/ODataFilterOperator.json | Generated JSON Schema for ODataFilterOperator. |
| packages/spec/json-schema/api/ODataFilterFunction.json | Generated JSON Schema for ODataFilterFunction. |
| packages/spec/json-schema/api/ODataError.json | Generated JSON Schema for ODataError. |
| content/docs/references/ui/widget/WidgetProperty.mdx | Reference docs for WidgetProperty. |
| content/docs/references/ui/widget/WidgetManifest.mdx | Reference docs for WidgetManifest. |
| content/docs/references/ui/widget/WidgetLifecycle.mdx | Reference docs for WidgetLifecycle. |
| content/docs/references/ui/widget/WidgetEvent.mdx | Reference docs for WidgetEvent. |
| content/docs/references/automation/sync/meta.json | Adds docs section metadata for Sync. |
| content/docs/references/automation/sync/SyncMode.mdx | Reference docs for SyncMode. |
| content/docs/references/automation/sync/SyncExecutionStatus.mdx | Reference docs for SyncExecutionStatus. |
| content/docs/references/automation/sync/SyncExecutionResult.mdx | Reference docs for SyncExecutionResult. |
| content/docs/references/automation/sync/SyncDirection.mdx | Reference docs for SyncDirection. |
| content/docs/references/automation/sync/DataSyncConfig.mdx | Reference docs for DataSyncConfig. |
| content/docs/references/automation/sync/DataSourceConfig.mdx | Reference docs for DataSourceConfig. |
| content/docs/references/automation/sync/DataDestinationConfig.mdx | Reference docs for DataDestinationConfig. |
| content/docs/references/automation/sync/ConflictResolution.mdx | Reference docs for ConflictResolution. |
| content/docs/references/automation/meta.json | Adds connector/etl/sync pages to automation docs navigation. |
| content/docs/references/automation/mapping/FieldMapping.mdx | Reference docs for FieldMapping. |
| content/docs/references/automation/etl/meta.json | Adds docs section metadata for ETL. |
| content/docs/references/automation/etl/ETLTransformationType.mdx | Reference docs for ETLTransformationType. |
| content/docs/references/automation/etl/ETLTransformation.mdx | Reference docs for ETLTransformation. |
| content/docs/references/automation/etl/ETLSyncMode.mdx | Reference docs for ETLSyncMode. |
| content/docs/references/automation/etl/ETLSource.mdx | Reference docs for ETLSource. |
| content/docs/references/automation/etl/ETLRunStatus.mdx | Reference docs for ETLRunStatus. |
| content/docs/references/automation/etl/ETLPipelineRun.mdx | Reference docs for ETLPipelineRun. |
| content/docs/references/automation/etl/ETLPipeline.mdx | Reference docs for ETLPipeline. |
| content/docs/references/automation/etl/ETLEndpointType.mdx | Reference docs for ETLEndpointType. |
| content/docs/references/automation/etl/ETLDestination.mdx | Reference docs for ETLDestination. |
| content/docs/references/automation/connector/meta.json | Adds docs section metadata for Connector. |
| content/docs/references/automation/connector/OperationType.mdx | Reference docs for OperationType. |
| content/docs/references/automation/connector/OperationParameter.mdx | Reference docs for OperationParameter. |
| content/docs/references/automation/connector/OAuth2Config.mdx | Reference docs for OAuth2Config. |
| content/docs/references/automation/connector/ConnectorTrigger.mdx | Reference docs for ConnectorTrigger. |
| content/docs/references/automation/connector/ConnectorOperation.mdx | Reference docs for ConnectorOperation. |
| content/docs/references/automation/connector/ConnectorInstance.mdx | Reference docs for ConnectorInstance. |
| content/docs/references/automation/connector/ConnectorCategory.mdx | Reference docs for ConnectorCategory. |
| content/docs/references/automation/connector/Connector.mdx | Reference docs for Connector. |
| content/docs/references/automation/connector/AuthenticationType.mdx | Reference docs for AuthenticationType. |
| content/docs/references/automation/connector/Authentication.mdx | Reference docs for Authentication. |
| content/docs/references/automation/connector/AuthField.mdx | Reference docs for AuthField. |
| content/docs/references/api/odata/meta.json | Adds docs section metadata for OData. |
| content/docs/references/api/odata/ODataResponse.mdx | Reference docs for ODataResponse. |
| content/docs/references/api/odata/ODataQuery.mdx | Reference docs for ODataQuery. |
| content/docs/references/api/odata/ODataMetadata.mdx | Reference docs for ODataMetadata. |
| content/docs/references/api/odata/ODataFilterOperator.mdx | Reference docs for ODataFilterOperator. |
| content/docs/references/api/odata/ODataFilterFunction.mdx | Reference docs for ODataFilterFunction. |
| content/docs/references/api/odata/ODataError.mdx | Reference docs for ODataError. |
| content/docs/references/api/meta.json | Adds OData page to API docs navigation. |
| * @example "json" | ||
| * @example "xml" | ||
| */ | ||
| $format: z.enum(['json', 'xml', 'atom']).optional().describe('Response format'), |
There was a problem hiding this comment.
$format includes atom, but Atom/XML payloads were removed in OData v4 (Atom was an OData v3 format). Consider dropping atom (or clarifying/versioning this if you intend v3 compatibility).
| export const ConnectorSchema = z.object({ | ||
| /** | ||
| * Connector identifier | ||
| * Must be globally unique | ||
| */ |
There was a problem hiding this comment.
This new protocol module doesn’t have a corresponding test file (the automation package has extensive vitest coverage for other protocols like flow.test.ts and webhook.test.ts). Adding connector.test.ts to cover basic parse/default behaviors (and invalid cases) would help keep the contract stable.
| export const DataDestinationConfigSchema = z.object({ | ||
| /** | ||
| * Destination object name | ||
| * For ObjectStack objects | ||
| */ | ||
| object: z.string().optional().describe('ObjectStack object name'), | ||
|
|
||
| /** | ||
| * Connector instance ID | ||
| * For external destinations | ||
| */ | ||
| connectorInstanceId: z.string().optional().describe('Connector instance ID'), | ||
|
|
||
| /** | ||
| * Operation to perform | ||
| */ | ||
| operation: z.enum([ | ||
| 'insert', // Create new records only | ||
| 'update', // Update existing records only | ||
| 'upsert', // Insert or update based on key | ||
| 'delete', // Delete records | ||
| 'sync', // Full synchronization | ||
| ]).describe('Sync operation'), | ||
|
|
||
| /** | ||
| * Field mappings | ||
| * Maps source fields to destination fields | ||
| */ | ||
| mapping: z.union([ | ||
| z.record(z.string()), // Simple mapping: { sourceField: 'destField' } | ||
| z.array(FieldMappingSchema), // Advanced mapping with transformations | ||
| ]).optional().describe('Field mappings'), | ||
|
|
||
| /** | ||
| * External resource identifier | ||
| */ | ||
| externalResource: z.string().optional().describe('External resource ID'), |
There was a problem hiding this comment.
DataDestinationConfigSchema requires operation but otherwise allows an empty destination target (no object and no connectorInstanceId/externalResource). This makes it possible to validate configs that can’t execute. Consider enforcing either object OR (connectorInstanceId + externalResource) as required destination identifiers.
| export const DataSyncConfigSchema = z.object({ | ||
| /** | ||
| * Sync configuration name (snake_case) | ||
| */ | ||
| name: z.string() |
There was a problem hiding this comment.
This new sync protocol file introduces multiple schemas and helper factories, but there is no sync.test.ts (automation protocols appear to be covered via vitest in this repo). Adding tests for at least the happy path + invalid configs (e.g., missing source/destination identifiers) would help prevent regressions.
| export const ETLPipelineSchema = z.object({ | ||
| /** | ||
| * Pipeline identifier (snake_case) | ||
| */ | ||
| name: z.string() | ||
| .regex(/^[a-z_][a-z0-9_]*$/) | ||
| .describe('Pipeline identifier (snake_case)'), |
There was a problem hiding this comment.
This new ETL protocol module doesn’t have a corresponding test file (automation protocols like flow/webhook/workflow do). Adding etl.test.ts to validate representative pipelines (and invalid cases) would improve contract stability.
| * const query: ODataQuery = { | ||
| * select: ['name', 'email'], | ||
| * filter: "country eq 'US' and revenue gt 100000", | ||
| * orderby: 'revenue desc', | ||
| * top: 10, |
There was a problem hiding this comment.
The “Programmatic Use” example uses keys like select, filter, orderby, etc., but the schema (and OData URLs) use $select, $filter, $orderby, etc. This example should be updated so it matches the actual ODataQuery shape.
| eq: (field: string, value: string | number | boolean) => | ||
| `${field} eq ${typeof value === 'string' ? `'${value}'` : value}`, | ||
| ne: (field: string, value: string | number | boolean) => | ||
| `${field} ne ${typeof value === 'string' ? `'${value}'` : value}`, | ||
| gt: (field: string, value: number) => `${field} gt ${value}`, | ||
| lt: (field: string, value: number) => `${field} lt ${value}`, | ||
| contains: (field: string, value: string) => `contains(${field}, '${value}')`, |
There was a problem hiding this comment.
The helper filter builders don’t escape string literals. If value contains a single quote, the generated filter becomes invalid (and can change query semantics). OData string escaping requires doubling single quotes (' -> ''); consider applying that in eq/ne/contains before interpolating.
| export const ODataQuerySchema = z.object({ | ||
| /** | ||
| * $select - Select specific fields to return | ||
| * | ||
| * Comma-separated list of field names to include in the response. |
There was a problem hiding this comment.
This file introduces a new public protocol surface (ODataQuerySchema, response/error schemas, and helpers), but there is no odata.test.ts alongside other API protocol tests (e.g., contract.test.ts, endpoint.test.ts). Adding tests for parsing/serialization edge cases (e.g., $select array vs string, $format validation) would improve regression protection.
| export const AuthenticationSchema = z.object({ | ||
| /** | ||
| * Authentication type | ||
| */ | ||
| type: AuthenticationTypeSchema.describe('Authentication type'), | ||
|
|
||
| /** | ||
| * Authentication fields | ||
| * Configuration fields needed for this auth type | ||
| */ | ||
| fields: z.array(AuthFieldSchema).optional().describe('Authentication fields'), | ||
|
|
||
| /** | ||
| * OAuth 2.0 configuration (when type is oauth2) | ||
| */ | ||
| oauth2: OAuth2ConfigSchema.optional().describe('OAuth 2.0 configuration'), |
There was a problem hiding this comment.
AuthenticationSchema allows type: 'oauth2' without providing the oauth2 config (and similarly allows unrelated fields for type: 'none'). Consider modeling authentication as a z.discriminatedUnion('type', ...) so each auth type has the correct required/allowed fields.
| export const DataSourceConfigSchema = z.object({ | ||
| /** | ||
| * Source object name | ||
| * For ObjectStack objects | ||
| */ | ||
| object: z.string().optional().describe('ObjectStack object name'), | ||
|
|
||
| /** | ||
| * Filter conditions | ||
| * Only sync records matching these filters | ||
| */ | ||
| filters: z.any().optional().describe('Filter conditions'), | ||
|
|
||
| /** | ||
| * Fields to include | ||
| * If not specified, all fields are synced | ||
| */ | ||
| fields: z.array(z.string()).optional().describe('Fields to sync'), | ||
|
|
||
| /** | ||
| * External connector instance ID | ||
| * For external data sources | ||
| */ | ||
| connectorInstanceId: z.string().optional().describe('Connector instance ID'), | ||
|
|
||
| /** | ||
| * External resource identifier | ||
| * e.g., Salesforce object name, database table, API endpoint | ||
| */ | ||
| externalResource: z.string().optional().describe('External resource ID'), | ||
| }); |
There was a problem hiding this comment.
DataSourceConfigSchema currently permits an empty object (all fields are optional), which means DataSyncConfigSchema can validate a sync with no actual source configured. Consider enforcing either object (ObjectStack source) OR (connectorInstanceId + externalResource) for external sources.
Implements high-priority protocol definitions from PROTOCOL_OPTIMIZATION_PLAN.md to enable enterprise integration, custom widget development, and data synchronization capabilities.
Widget Contract System (
ui/widget.zod.ts)Complete widget lifecycle and event system for custom field components:
onMount,onUpdate,onUnmount,onValidate,onFocus,onBlur,onErrorETL Pipeline (
automation/etl.zod.ts)Declarative data pipeline with transformations and scheduling:
Connector Registry (
automation/connector.zod.ts)Reusable integration components following Zapier/n8n patterns:
Data Sync (
automation/sync.zod.ts)Bidirectional synchronization with conflict resolution:
OData v4 (
api/odata.zod.ts)Industry-standard query protocol for Microsoft Dynamics/SharePoint integration:
$select,$filter,$orderby,$top,$skip,$expand,$countSchema Conventions
All schemas follow ObjectStack standards:
camelCasefor configuration keys,snake_casefor identifiersOriginal prompt
Start implementation
The user has attached the following file paths as relevant context:
- .github/copilot-instructions.md
[Chronological Review: The conversation began with the user requesting to define objectui, objectql, and objectos using Zod. The user then inquired about the format of plugins and expressed concerns about potential redundancy in definitions. They suggested scanning all spec protocols for better categorization and discussed the merging of kernel and system components. The user continued to explore the organization of data sources and the classification of drivers, leading to discussions about the need for a new stack definition and the potential obsolescence of the definitions file. The conversation evolved into a focus on the ObjectStackConfig format and how to compile ObjectStackDefinition, culminating in a request for a complete work plan and the initiation of implementation.][Intent Mapping: The user explicitly requested to define objects using Zod, inquire about plugin formats, scan protocols for optimization, and evaluate the structure of the stack. They also sought clarification on the differences between files, the necessity of separating configurations, and the potential for CLI enhancements. The user expressed a desire to continue iterating on their work and to assess the overall ecosystem.]
[Technical Inventory: Technologies discussed include Zod for schema validation, JSON for configuration formats, and the structure of the ObjectStackConfig. The user mentioned files such as stack.zod and various package.json files related to examples and configurations.]
[Code Archaeology: The user referenced specific files like /Users/steedos/Documents/GitHub/spec/examples/todo/objectstack.config.ts and /Users/steedos/Documents/GitHub/spec/packages/spec/src/stack.zod.ts, indicating ongoing modifications and the need for clarity in definitions.]
[Progress Assessment: The user has made significant progress in defining the stack and its capabilities, but there are pending tasks related to refining examples, clarifying the purpose of configurations, and implementing a complete work plan.]
[Context Validation: All critical information for continuation is captured, including the need for a clear distinction between configuration and platform capabilities, as well as the user's intent to enhance the CLI for plugin development.]
[Recent Commands Analysis: The last agent command was "Start implementation," indicating the user is ready to begin coding based on the discussions. Prior to this, the user requested a complete work plan, suggesting they are organizing their tasks for effective implementation. The immediate context involved finalizing the structure of the ObjectStackDefinition and ensuring all necessary components are in place before coding.]
1. Conversation Overview: - Primary Objectives: The user requested to define objectui, objectql, and objectos using Zod, inquire about plugin formats, and assess the structure of their ecosystem, stating, "我感觉你定义重复了,你应该扫描所有的spec协议目录." - Session Context: The conversation flowed from defining technical specifications to evaluating the organization of files and configurations, culminating in a request for a work plan and the initiation of implementation. - User Intent Evolution: The user's needs evolved from specific technical definitions to broader organizational concerns and implementation readiness.- Technical Foundation:
- Zod: A schema validation library used for defining object structures.
- JSON: Format for configuration files, particularly in the context of ObjectStackConfig.
- ObjectStackConfig: A proposed structure for defining platform capabilities and configurations.
- Codebase Status:
- /Users/steedos/Documents/GitHub/spec/examples/todo/objectstack.config.ts:
- Purpose: Defines the configuration for the todo example.
- Current State: Undergoing modifications to align with new stack definitions.
- Key Code Segments: Specific configurations related to the todo application.
- /Users/steedos/Documents/GitHub/spec/packages/spec/src/stack.zod.ts:
- Purpose: Defines the Zod schema for the stack.
- Current State: Recent edits to refine the schema based on user feedback.
- Problem Resolution:
- Issues Encountered: Redundancy in definitions and unclear distinctions between configuration and platform capabilities.
- Solutions Implemented: Discussions around merging definitions and clarifying the purpose of configurations.
- Debugging Context: Ongoing refinement of examples and definitions.
- Progress Tracking:
- Completed Tasks: Initial definitions and discussions around the stack structure.
- Partially Complete Work: Finalizing ObjectStackDefinition and examples.
- Validated Outcomes: User has a clearer understanding of the stack's purpose and structure.
- Active Work State:
- Current Focus: The user is preparing to implement the ObjectStackDefinition based on recent discussions.
- Re...
Created from VS Code.
💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.