Skip to content

Feature request: add --migrate-schema flag to start for non-breaking schema changes without resetting indexer state #954

@saurabhburade

Description

@saurabhburade

Problem

Currently, evolving the schema (e.g., adding new fields) risks schema mismatch or requires resetting/reindexing. This makes incremental, backward-compatible schema updates difficult during active indexing.

Goal

Allow additive, non-breaking schema updates while continuing indexing from the existing checkpoint — without resetting indexer state.

Specifically:

  • Add new fields to entities.
  • Ensure new fields are nullable/optional.
  • Continue indexing from the current state.
  • Preserve compatibility with already indexed records.

Proposed Solution

Introduce a schema migration flag for the start command:

Expected Behavior

  • Detect schema differences between the current DB schema and project schema.
  • Automatically apply safe, additive changes (e.g., adding nullable columns).
  • Do not reset indexer state or checkpoint.
  • Fail with a clear error if changes are breaking (e.g., type changes, non-null additions without defaults, column removals).

Acceptance Criteria

  • ✅ Adding nullable fields does not require a reindex.
  • ✅ Indexer continues from the existing checkpoint.
  • ✅ Old records remain valid (new fields default to NULL).
  • ✅ Clear output describing applied migrations.
  • ❌ Breaking changes are rejected with actionable messaging.

Rationale

This enables progressive schema evolution without disrupting historical data or requiring expensive reindexing, while maintaining safety guarantees.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions