Add grid search notebook #36
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Also fixed pipeline.py and load 1000 rows instead because questions (in evaluation pipeline) were generated from it NB: to test pipeline_eval.py, must put full path as argument for custom components and use embed_text as a generic component (where fondant 0.8.dev2 is specified in component specs)
not definitive, still component_dir to tackle/adapt
I apparently have too many arguments, should I find a way to have less or should we deactivate this pre-commit check?
- First iteration works well - For second iteration with other parameters, get a "cannot read parquet file" error for evaluation pipeline. to investigate - weaviate_url in component: must put IP address (host.docker or localhost url don't work). to investigate - Cannot run script with Fondant CLI, doesn't find the pipeline. to investigate
- Runs with fondant 0.8.dev3 - Disabled caching everywhere (not needed and actually creates issues in a grid search setting - Provide IP address extractor for docker image to access weaviate locally - Added weaviate_local folder containing a simple weaviate docker compose file. The current one doesn't work because of the contextionary. To investigate. - Fondant CLI on scripts won't work because fondant 0.8.dev3 doesn't support factory functions (Fondant main does)
"too many args" remaining
Also cleaned the notebook
RobbeSneyders
approved these changes
Dec 15, 2023
| @@ -5,7 +5,7 @@ | |||
|
|
|||
| def create_pipeline( | |||
| *, | |||
| base_path: str = "s3://sagemaker-fondant-artifacts-robbe/data", | |||
| base_path: str = "./data-dir", | |||
src/utils_param_search.py
Outdated
There was a problem hiding this comment.
I don't think we should abstract this into a separate file. Now the notebook doesn't even create or run any Fondant pipelines. It just calls a magic run_parameter_search function from a utils file.
src/parameter_search.ipynb
Outdated
| "And of course feel free to give us a [**star** ⭐](https://github.com/ml6team/fondant) if you like what we are doing!\n", | ||
| "\n", | ||
| "<div align=\"center\">\n", | ||
| "<img src=\"../art/feedback_is_important.webp\"/>\n", |
There was a problem hiding this comment.
Would still remove this :)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.