Skip to content

fix(modalprocessors): preserve chunk document IDs during extraction - #360

Open
lllleolin-max wants to merge 1 commit into
HKUDS:mainfrom
lllleolin-max:fix/preserve-multimodal-vector-document-id
Open

fix(modalprocessors): preserve chunk document IDs during extraction#360
lllleolin-max wants to merge 1 commit into
HKUDS:mainfrom
lllleolin-max:fix/preserve-multimodal-vector-document-id

Conversation

@lllleolin-max

Copy link
Copy Markdown

_create_entity_and_chunk initially stores the correct full_doc_id in both chunk stores, then _process_chunk_for_extraction overwrites the vector record with the chunk ID. For example, a chunk belonging to doc-original ends up pointing to chunk-... in the vector store while its text record still points to doc-original.

Preserve full_doc_id from the stored chunk during extraction. Records without that field retain the existing chunk-ID fallback.

Validation:

  • Five regression cases cover the creation/extraction chain with batch mode on/off and an explicit document ID or None, plus extraction from a legacy record missing the field. Baseline: 2 failed, 3 passed; fixed: 5 passed (python -m pytest tests/test_modalprocessor_chunk_document_id.py -q).
  • Full test suite: python -m pytest tests -q400 passed, 1 skipped.
  • Repository Ruff 0.6.4 lint/format checks and git diff --check pass on the changed files.

The regression tests use recording stores and mocked extraction/merge calls; no external database or LLM service is required.

AI assistance was used to investigate, implement, and test this change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant