dfget downloads tokenizer.json with same file size as origin, but content is corrupted and MD5 is different

[NelZyhh]

Bug report:

Description

• When downloading tokenizer.json from ModelScope via dfget , the downloaded file looks abnormal.

• Compared with the same file downloaded directly from ModelScope:

  • the file size appears to be the same
  • but the content is corrupted / contains garbled bytes
  • and the MD5 checksum is different

• If the same file is downloaded directly from ModelScope without dfget , the file content is correct.
  Impact

• The corrupted tokenizer.json causes tokenizer/model loading failures.

• This suggests the file content may be altered somewhere in the dfget download path, rather than a simple download failure.

Expected behavior:

• The file downloaded by dfget should be byte-identical to the source file.
• Checksums such as MD5 / SHA256 should match the directly downloaded file.

How to reproduce it:

dfget modelscope://Qwen/Qwen3-14B/tokenizer.json -O /data02/models/tokenizer-dfget.json

Environment:

• Client version: 1.3.5、1.3.10、1.3.11、1.4.0
• OS: linux
• Kernel (e.g. uname -a):
• Others:

[BraveY]

Have you check the content diff piece by piece? If the content of the piece is correct, but the order is misplaced , it could be the the bug reported in #1970

[gaius-qi]

Have you check the content diff piece by piece? If the content of the piece is correct, but the order is misplaced , it could be the the bug reported in #1970

@BraveY This issue addresses the fact that the server may ignore the Range header or return a range different from the requested one (as described by the Content-Range header, see RFC 9110 Section 15.3.7). It is not about the piece download consistency problem.