fix: add standard split metadata when DocumentSplitter uses split_by="function"

[businessarshgoyal]

Related Issues

• No existing issue; found while using DocumentSplitter(split_by="function") together with SentenceWindowRetriever.

Proposed Changes:

DocumentSplitter._split_by_function built its output Documents by hand and only set source_id, unlike every other split mode, which goes through _create_docs_from_splits and sets split_id, split_idx_start and page_number. It also ignored skip_empty_documents, so empty strings returned by a custom splitting function became empty documents.

Consequences:

• chunks produced with split_by="function" cannot be used with components that rely on that metadata. SentenceWindowRetriever raises ValueError: The retrieved documents must have 'split_id' in their metadata.
• no page tracking and no start offsets for custom splitters.

This PR routes the function path through _create_docs_from_splits as well:

• splits are located in the source with content.find(split, cur_start_idx) to compute split_idx_start and the page_number (form feeds before the match).
• because a splitting function may transform the text (e.g. uppercase, strip), a split is not guaranteed to appear verbatim in the source. When it cannot be located, the running offset is used and page numbers are advanced by the form feeds contained in the split.
• empty splits are skipped unless skip_empty_documents=False, matching the other split modes.

How did you test it?

• Updated the existing test_split_by_function assertions and added unit tests for page-number/offset tracking, transformed (non-verbatim) splits, empty-split handling with skip_empty_documents both on and off, and an end-to-end test showing the output now works with SentenceWindowRetriever.
• hatch run test:unit test/components/preprocessors/test_document_splitter.py -> 60 passed
• hatch run test:unit test/components/preprocessors test/components/retrievers -> 559 passed
• hatch run test:types -> Success: no issues found in 468 source files
• hatch run fmt -> All checks passed
• pre-commit hooks run on commit.

Notes for the reviewer

The interesting case is the fallback when a split cannot be found verbatim in the source: offsets then become cumulative lengths of the (transformed) splits rather than true source offsets, which is the best available approximation. The existing test_split_by_function was updated because the function path now emits the same metadata as all other modes; no assertions were weakened.

This change was written with AI assistance (Devin); the behaviour was reproduced locally before the fix and all tests above were run.

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have added unit tests and updated the docstrings.
• I've used one of the conventional commit types for my PR title.
• I have added a release note file.
• I have run pre-commit hooks and fixed any issue.

Written by Devin

[vercel]

@devin-ai-integration[bot] is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[CLAassistant]

All committers have signed the CLA.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
haystack/components/preprocessors
document_splitter.py
Project Total

This report was generated by python-coverage-comment-action

[davidsbatista]

@businessarshgoyal thanks for the contribution.

I've added some missing behaviour, that's when the split_by_function generates an overlapping sliding window, there was a bug:

• Bug: the content.find(split, cur_start_idx) only searches forward, so overlapping splits (sliding-window functions) can't be found; the offsets/page numbers were silently falling back to wrong values, and _split_overlap never got set.
• Fix: on failed forward search, retry from prev_start_idx + 1 (start of previous split, not its end)
• I've added an extra test: test_split_by_function_with_overlapping_sliding_window

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
haystack-docs	Ready	Preview	Aug 31, 2026 1:56pm

[davidsbatista]

looks good, thanks for the contribution!