[ray] Fix datasets_modules ImportError with Ray Tune
#12749
Conversation
| dynamic_modules_path = os.path.join(datasets.load.init_dynamic_modules(), "__init__.py") | ||
| # load dynamic_modules from path | ||
| spec = importlib.util.spec_from_file_location("datasets_modules", dynamic_modules_path) | ||
| datasets_modules = importlib.util.module_from_spec(spec) | ||
| sys.modules[spec.name] = datasets_modules | ||
| spec.loader.exec_module(datasets_modules) |
There was a problem hiding this comment.
is it possible to use runtime environments here instead? just curious
There was a problem hiding this comment.
Not without editing Ray Tune itself as it would need to be added to an Actor option in trial executor. Also it doesn't appear you can actually import a module from path in a runtime env (only pip and conda), unless I missed that in the docs
Yard1
changed the title
dynamic_modules ImportError with Ray Tunedatasets_modules ImportError with Ray Tune
| dynamic_modules_path = os.path.join(datasets.load.init_dynamic_modules(), "__init__.py") | ||
| # load dynamic_modules from path | ||
| spec = importlib.util.spec_from_file_location("datasets_modules", dynamic_modules_path) | ||
| datasets_modules = importlib.util.module_from_spec(spec) | ||
| sys.modules[spec.name] = datasets_modules |
There was a problem hiding this comment.
another comment: should this be moved upstream to datasets eventually?
There was a problem hiding this comment.
I'm not sure if it belongs there. The actual import needs to somewhere in Tune, and here is the most convenient place
|
Great! @richardliaw , another approval is still required? |
|
@sgugger @LysandreJik could you help take a look :) |
What does this PR do?
This PR fixes an ImportError throws due to
datasets_modulesnot being loaded on Ray Actors when tuning hyperparamters with Ray, fixing the following issues:huggingface/blog#106
huggingface/transformers#11565
https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/34
https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/35
Fixes #11565
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.