ref: simplify cache dir creation and remove repeated parts

[bhimrazy]

Before submitting

• Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

What does this PR do?

ref: simplify cache dir creation and remove repeated parts

prep for resolving cace dir issues for hf dataset

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[Copilot]

Pull Request Overview

This PR simplifies the cache directory creation process and removes redundant code for handling cache directories in datasets.

• Consolidated the cache directory creation logic in _try_create_cache_dir.
• Updated documentation and comments to improve clarity, especially around handling the updated_at hash and environment-dependent cache roots.

Comments suppressed due to low confidence (1)

src/litdata/utilities/dataset_utilities.py:211

• [nitpick] The variable name 'input_dir_hash_path' is inconsistent with the parameter name 'input_dir_hash_filepath' used elsewhere. Consider renaming it to 'input_dir_hash_filepath' for clarity and consistency.

input_dir_hash_path = os.path.join(cache_root, dir_url_hash)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79%. Comparing base (b1645b6) to head (39531e2).
Report is 1 commits behind head on main.

Additional details and impacted files

@@         Coverage Diff         @@
##           main   #568   +/-   ##
===================================
  Coverage    79%    79%           
===================================
  Files        40     40           
  Lines      6101   6098    -3     
===================================
+ Hits       4821   4822    +1     
+ Misses     1280   1276    -4     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tchaton]

Nice !