Upgrade clean.py to use Files class instead of accessing `os` directly

[p3nGu1nZz]

Its best to utilize our files.py io module.
we might need a few more methods for is_dir etc..
revamp the clean script

[gauravshinde1816]

Hi @p3nGu1nZz , I want to work on this issue , But I have some questions about installation

• Where can I get the thinking-dataset.toml requirement file, which I think is missing in the repo.
• Can you please add instructions to run the test cases as well , it will be helpful.

Thanks !

[p3nGu1nZz]

@gauravshinde1816 HI o/

• We apologize for the inconvenience. We've recently switched over to using uv and haven't made the necessary updates yet. However, you can install using pip install -e . in the directory where you cloned the repo.
• Our latest working branch is dev-prototyping-alpha.
• Please use this branch 👇

git fetch origin
git checkout 53-upgrade-cleanpy-to-use-files-class-instead-of-accessing-os-directly

• To speed up development, reduce all contractions and terms in the config for NormalizeTextPipe to only one each.
• Another speed-up technique is changing this config:

type: "SubsetPipe"
config:
  rows: [ "all" ]

to

type: "SubsetPipe"
config:
  rows: [ 0, 100 ]

• Apologies, but our tests need updating as per Update all of our PyTests to test our latest version in develop-alpha branch #41. They should be updated to the dev-prototyping-alpha branch. The test system should work okay; there is a script in run_tests_and_generate_report.py that runs all tests and generates the report. I usually use VS Code to run individual tests. We really need help getting our test coverage up to speed with all our new refactors and changes. You should be able to run tests using just pytest, as we are not using anything fancy.
• You can also check the USAGE doc for more explanation on how to run each of the commands. For your test, you would just need to run download and/or process, then run clean, and make sure the directories are removed.

-- Bonus complete #36

Warm Regards
🐧 P3n