[Bug]: Delimiter setting not working as intended

[mwd102]

Is there an existing issue for the same bug?

• I have checked the existing issues.

RAGFlow workspace code commit ID

na

RAGFlow image version

v0.15.1-180-g036f37a6 full

Other environment information

docker windows gpu compose file, no changes

Actual behavior

Using mutli-char delimiter is not working as intended. Setting word in backticks does not lead to a chunk on that word. A quick glance at the codebase suggests the way the regex is built may not be handling string delimiters properly (treating them as a collection of individual chars instead)

Expected behavior

text to delimit on a given string match and be visible as chunks

Steps to reproduce

create basic collection
use general parser
update the delimiter section to have `word` as sole value
upload text or md document containing the desired delimiter at intervals
process document in ui and view chunks
note they have not delimited on provided word

Additional information

No response

[mwd102]

Just to add to this by inserting a special char e.g. | before every occurance of my desired word and then using that as a single char match, the system delimits as expected

[Snify89]

Great find. Thank you for reporting

[KevinHuSh]

Could you specify the case?

[mwd102]

@KevinHuSh this happens on any delimiter that is more than one char. Try ingesting a simple text document and chunking on a given word within the doc. It won't work as despite the tooltip ragflow seems to treat a delimiter in backticks e.g. `mydelim` as seperate chars not as a whole token/string.

[DiamondPoirier]

Based on the issues you've provided, after multiple attempts, I found that the system is working fine. Regarding the multi-character segmentation problem you mentioned, I suspect it might be caused by setting the Chunk token number too big, which leads to segmentation issues. I recommend choosing a smaller number like 10 or smaller based on the content.

[mwd102]

The chunking workflow is 100% bugged. I would never recommend creating a vector on less than 128 tokens, that's a surefire way to false positives. I did also notice that on longer documents the chunking seems to 'give up' and just lump the latter half of longer TXT files into one giant chunk or two.

I don't know what else to say, the test case is v simple - try chunking a basic text file on a string delimiter supposedly treated as a single token via backticks and watch it fail. I set no other delimiters and the real clue is if I subbed the word with a single char such as a | it worked fine. I kept min tokens at default throughout.

[fanshuaiy]

This issue has not been resolved yet. When using the General method to parse PDFs, both multiple separators and a single separator do not work. Reproduction method: Control variables to ensure that the same token is used for each parsing method, regardless of whether there is a delimiter. The parsing block of the document will not change, indicating that even if a delimiter is set, it will not work.

[KevinHuSh]

PDFs will not be split just by delimiters.
Uploading TXT files if there're some clear defined delimiters is highly recommanded.