forked from opensearch-project/neural-search
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Fix: change max chunk limit exception (opensearch-project#717) (opens…
…earch-project#719) * change max chunk limit exception Signed-off-by: yuye-aws <yuyezhu@amazon.com> * fix integration tests for two chunking algorithm Signed-off-by: yuye-aws <yuyezhu@amazon.com> * update changelog Signed-off-by: yuye-aws <yuyezhu@amazon.com> * add run time parameter string_tobe_chunked_count Signed-off-by: yuye-aws <yuyezhu@amazon.com> * fix unit test for fixed token length and delimiter algorithm Signed-off-by: yuye-aws <yuyezhu@amazon.com> * implement unit test for string to be chunked in fixed token length and delimiter algorithm Signed-off-by: yuye-aws <yuyezhu@amazon.com> * update definition for string to be chunked parameter Signed-off-by: yuye-aws <yuyezhu@amazon.com> * fix text chunking processor ut Signed-off-by: yuye-aws <yuyezhu@amazon.com> * add string to be chunked count in text chunking processor Signed-off-by: yuye-aws <yuyezhu@amazon.com> * add string to be chunked count in text chunking processor Signed-off-by: yuye-aws <yuyezhu@amazon.com> * add more test cases for text chunking processor Signed-off-by: yuye-aws <yuyezhu@amazon.com> * remove chunker util Signed-off-by: yuye-aws <yuyezhu@amazon.com> * change chunk limit check in boht algorithms Signed-off-by: yuye-aws <yuyezhu@amazon.com> * update ut for text chunking processor Signed-off-by: yuye-aws <yuyezhu@amazon.com> * update parameter name to chunk_string_count Signed-off-by: yuye-aws <yuyezhu@amazon.com> * run spot less apply Signed-off-by: yuye-aws <yuyezhu@amazon.com> --------- Signed-off-by: yuye-aws <yuyezhu@amazon.com> (cherry picked from commit 86b70e0) Co-authored-by: Yuye Zhu <yuyezhu@amazon.com>
- Loading branch information
1 parent
9c02141
commit 05b1d68
Showing
9 changed files
with
459 additions
and
157 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
41 changes: 0 additions & 41 deletions
41
src/main/java/org/opensearch/neuralsearch/processor/chunker/ChunkerUtil.java
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.