Chunking Strategies

[arjbingly]

Currently, the package supports a simple recursive chunking. It would be useful to have other chunking strategies. Some that I have in mind include document-specific chunking for well-structured documents like HTML and markdown; and some agentic chunking, where each sentence is compared with other sentences for similarity and added to the chunk only if they meet a a threshold.