Problem
Currently, evolving the schema (e.g., adding new fields) risks schema mismatch or requires resetting/reindexing. This makes incremental, backward-compatible schema updates difficult during active indexing.
Goal
Allow additive, non-breaking schema updates while continuing indexing from the existing checkpoint — without resetting indexer state.
Specifically:
- Add new fields to entities.
- Ensure new fields are nullable/optional.
- Continue indexing from the current state.
- Preserve compatibility with already indexed records.
Proposed Solution
Introduce a schema migration flag for the start command:
Expected Behavior
- Detect schema differences between the current DB schema and project schema.
- Automatically apply safe, additive changes (e.g., adding nullable columns).
- Do not reset indexer state or checkpoint.
- Fail with a clear error if changes are breaking (e.g., type changes, non-null additions without defaults, column removals).
Acceptance Criteria
- ✅ Adding nullable fields does not require a reindex.
- ✅ Indexer continues from the existing checkpoint.
- ✅ Old records remain valid (new fields default to NULL).
- ✅ Clear output describing applied migrations.
- ❌ Breaking changes are rejected with actionable messaging.
Rationale
This enables progressive schema evolution without disrupting historical data or requiring expensive reindexing, while maintaining safety guarantees.