-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add grid search notebook #36
Conversation
Also fixed pipeline.py and load 1000 rows instead because questions (in evaluation pipeline) were generated from it NB: to test pipeline_eval.py, must put full path as argument for custom components and use embed_text as a generic component (where fondant 0.8.dev2 is specified in component specs)
not definitive, still component_dir to tackle/adapt
I apparently have too many arguments, should I find a way to have less or should we deactivate this pre-commit check?
- First iteration works well - For second iteration with other parameters, get a "cannot read parquet file" error for evaluation pipeline. to investigate - weaviate_url in component: must put IP address (host.docker or localhost url don't work). to investigate - Cannot run script with Fondant CLI, doesn't find the pipeline. to investigate
- Runs with fondant 0.8.dev3 - Disabled caching everywhere (not needed and actually creates issues in a grid search setting - Provide IP address extractor for docker image to access weaviate locally - Added weaviate_local folder containing a simple weaviate docker compose file. The current one doesn't work because of the contextionary. To investigate. - Fondant CLI on scripts won't work because fondant 0.8.dev3 doesn't support factory functions (Fondant main does)
"too many args" remaining
Also cleaned the notebook
@@ -5,7 +5,7 @@ | |||
|
|||
def create_pipeline( | |||
*, | |||
base_path: str = "s3://sagemaker-fondant-artifacts-robbe/data", | |||
base_path: str = "./data-dir", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Woops
src/utils_param_search.py
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we should abstract this into a separate file. Now the notebook doesn't even create or run any Fondant pipelines. It just calls a magic run_parameter_search
function from a utils file.
src/parameter_search.ipynb
Outdated
"And of course feel free to give us a [**star** ⭐](https://github.com/ml6team/fondant) if you like what we are doing!\n", | ||
"\n", | ||
"<div align=\"center\">\n", | ||
"<img src=\"../art/feedback_is_important.webp\"/>\n", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would still remove this :)
No description provided.