fix: segmentation fault error in streaming tokens

[bhimrazy]

Before submitting

• Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

What does this PR do?

fix: segmentation fault error while streaming tokens

Fixes #454 and #459.

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[bhimrazy]

Removing the following line does resolve the issue; however, it will prevent memory-mapped files (mmaps) from being properly closed. Ideally, this is not a good solution, as it may lead to resource leaks.

litdata/src/litdata/streaming/item_loader.py

Line 413 in f6660be

self._mmaps[chunk_index]._mmap.close()

[codecov]

Codecov Report

Attention: Patch coverage is 83.33333% with 2 lines in your changes missing coverage. Please review.

Project coverage is 79%. Comparing base (f6660be) to head (f8aceee).
Report is 2 commits behind head on main.

Additional details and impacted files

@@         Coverage Diff         @@
##           main   #485   +/-   ##
===================================
- Coverage    79%    79%   -0%     
===================================
  Files        38     38           
  Lines      5620   5626    +6     
===================================
- Hits       4458   4456    -2     
- Misses     1162   1170    +8     

[tchaton]

Awesome work @bhimrazy @deependujha :)