Add Single Class Model PR Curves #26
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
Member
Author
|
Unfortunately, during #25 (training single class models) all notebooks were run in my base |
Member
Author
|
This PR is ready for review! |
d33bs
approved these changes
Apr 6, 2023
Member
d33bs
left a comment
d33bs
left a comment
There was a problem hiding this comment.
Nice work! I left a few comments with this review and overall thought it LGTM! Please don't hesitate to let me know if you have any questions or if I may clarify at all.
Co-authored-by: Dave Bunten <ekgto445@gmail.com>
…phenotypic_profiling_model into evaluate-single-class-models
roshankern
added a commit
that referenced
this pull request
Jul 6, 2023
* Refactor Download Module (#18) * refactor module * remove training data file * Update 0.download_data/scripts/nbconverted/download_data.py Co-authored-by: Erik Serrano <31600622+axiomcura@users.noreply.github.com> * eric suggestions --------- Co-authored-by: Erik Serrano <31600622+axiomcura@users.noreply.github.com> * Refactor Split Data Module (#19) * refactor module * greg suggestions * Train module refactor (#20) * refactor format module * use straify function * rerun train module * black formatting * docs, nbconvert * nbconvert * rerun pipeline, rename model * fix typo * Update 2.train_model/README.md Co-authored-by: Gregory Way <gregory.way@gmail.com> * Update 2.train_model/README.md Co-authored-by: Gregory Way <gregory.way@gmail.com> * Update 2.train_model/README.md Co-authored-by: Gregory Way <gregory.way@gmail.com> * notebook run --------- Co-authored-by: Gregory Way <gregory.way@gmail.com> * Refactor evaluate module (#21) * refactor clas pr curves * refactor confusion matrix * refactor F1 scores * refactor model predictions * documentation * dave suggestions * erik suggestions, reconvert * Refactor interpret module (#22) * refactor interpret notebook * docs, reconvert script * greg suggestions * Get Leave One Image Out Probabilities (#23) * add LOIO notebook * LOIO notebook * update notebook * download and split data with cell UUIDs * move LOIO * finish LOIO * black formatting * rerun notebook * rerun notebook, dave suggestions * greg comment * Train single class models (#25) * move multiclass models * rename files, fix sh * single class models notebook * run notebook * binarize labels * train single class models * reconvert notebooks * update readme * rename sh file * remove models * eric readme suggestions * rerun notebook, eric suggestions * Add Single Class Model PR Curves (#26) * get SCM PR curves * shuffled baseline * retrain single class models with correct kernel * rerun pr curves notebook * remove nones * rerun multiclass model * rerun notebook * move file * docs, black formatting * format notebook * Update 3.evaluate_model/README.md Co-authored-by: Dave Bunten <ekgto445@gmail.com> * dave suggestions * reconvert notebook --------- Co-authored-by: Dave Bunten <ekgto445@gmail.com> * Add SCM confusion matrices and F1 scores (#27) * get SCM PR curves * shuffled baseline * retrain single class models with correct kernel * rerun pr curves notebook * remove nones * rerun multiclass model * rerun notebook * move file * create SCM confusion matrix * rerun notebook * add changes from last PR * rerun notebook * add SCM F1, update SCM confusion matrices * documentation * rerun notebook * Update utils/evaluate_utils.py Co-authored-by: Dave Bunten <ekgto445@gmail.com> * Update utils/evaluate_utils.py Co-authored-by: Dave Bunten <ekgto445@gmail.com> * Update 3.evaluate_model/scripts/nbconverted/F1_scores.py Co-authored-by: Dave Bunten <ekgto445@gmail.com> * dave suggestions --------- Co-authored-by: Dave Bunten <ekgto445@gmail.com> * Get SCM Predictions and LOIO Probabilities (#29) * get SCM LOIO probas * reconvert notebook * get model predictions * rerun LOIO * reconvert notebook * save and reconvert notebook * eric suggestions * Add SCM Interpretations (#30) * add scm coefficients * rerun interpret multi-class model * compare model coefficients * nbconvert * readme * make all correlations negative * rerun training * rerun evaluate * rerun interpret * docs * newline * rerun LOIO * Remove unused cp features (#31) * rerun download/split modules * rerun multicalss models * rerun single class model * rerun evaluate module * get LOIO probas * rerun interpret module * rerun download data * Adding CP features to ggplot visualization (#24) * set colors for model types * visualize precision recall with CP and DP+CP * add F1 score barchart visualization * minor tweak of f1 score print * ignore mac files * merge main and rerun viz * change color scheme for increased contrast * add f1 score of the top model, and rerun with updated colors * nrow = 3 in facet * change name of weighted f1 score * update single cell images module (#32) * Refactor validate module (#33) * update validate module * refactor validation * get correlations * convert notebook * update readme * formatting, documentation * reset index * vadd view notebook * docs, black formatting * ccc credit * show all correlations * add notebook * remove preview notebook * convert notebook * add differences heatmaps * preview correlation differences * add docs * black formatting --------- Co-authored-by: Erik Serrano <31600622+axiomcura@users.noreply.github.com> Co-authored-by: Gregory Way <gregory.way@gmail.com> Co-authored-by: Dave Bunten <ekgto445@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In this pull request, precision-recall curves are added for the single-class models trained in #25. A PR curve is made for each label type (eg positive and negative label), for each model (Large single-class model, Metaphase single-class model, etc), for each evaluation dataset (train or test), for each feature type (CP, DP, CP_and_DP).
These PR curve figures have been added to class_PR_curves.ipynb and the PR data have been saved to precision_recall_curves.