Skip to content

Conversation

@adamsitnik
Copy link
Member

@adamsitnik adamsitnik commented Oct 29, 2025

Microsoft Reviewers: Open in CodeFlow

@adamsitnik adamsitnik requested a review from tarekgh October 29, 2025 12:46
@adamsitnik adamsitnik requested a review from a team as a code owner October 29, 2025 12:46
Copilot AI review requested due to automatic review settings October 29, 2025 12:46
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements the HeaderChunker class for splitting documents into chunks based on headers while preserving context, along with supporting infrastructure for token-based chunking. The chunker respects token limits and handles various document elements including paragraphs, tables, and images.

  • Adds HeaderChunker and ElementsChunker for document chunking with tokenization support
  • Introduces IngestionChunkerOptions for configurable chunk sizes and overlap
  • Adds validation for IngestionDocumentHeader.Level property (0-10 range)

Reviewed Changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/Libraries/Microsoft.Extensions.DataIngestion/Chunkers/HeaderChunker.cs New chunker implementation that splits documents by headers and delegates element chunking
src/Libraries/Microsoft.Extensions.DataIngestion/Chunkers/ElementsChunker.cs Internal chunker that handles token-based splitting of document elements including tables and paragraphs
src/Libraries/Microsoft.Extensions.DataIngestion/Chunkers/IngestionChunkerOptions.cs Configuration class for chunker behavior with validation logic
src/Libraries/Microsoft.Extensions.DataIngestion/Chunkers/ValueStringBuilder.cs Copied internal type from dotnet/runtime for efficient string building with array pooling
src/Libraries/Microsoft.Extensions.DataIngestion.Abstractions/IngestionDocumentElement.cs Adds validation to IngestionDocumentHeader.Level property
src/Libraries/Microsoft.Extensions.DataIngestion/Microsoft.Extensions.DataIngestion.csproj Adds dependency on Microsoft.ML.Tokenizers package
test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Utils/IAsyncEnumerableExtensions.cs Adds helper method ToListAsync for test assertions
test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/HeaderChunkerTests.cs Comprehensive tests for HeaderChunker including edge cases for tables and token limits
test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/ChunkerOptionsTests.cs Tests for IngestionChunkerOptions validation logic
test/Libraries/Microsoft.Extensions.DataIngestion.Tests/IngestionDocumentTests.cs Adds test for header level validation
test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Microsoft.Extensions.DataIngestion.Tests.csproj Adds test dependency on Microsoft.ML.Tokenizers.Data.Cl100kBase
eng/packages/TestOnly.props Registers package version for tokenizer test data

@adamsitnik adamsitnik requested review from a team as code owners October 29, 2025 14:16
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.

- use raw string literals as indentation is controlled by positioning the closing quotes
- don't allow for headers with level == 0
@adamsitnik
Copy link
Member Author

@stephentoub I believe I have addressed all your feedback. Could you PTAL?

@adamsitnik adamsitnik merged commit d83b99c into dotnet:main Oct 31, 2025
6 checks passed
This was referenced Nov 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants