Merged
Conversation
Code allows person to go through and annotate, slowly reducing what they can do
It works! Retires clips after 2 people check it if it hits 75% accuracy. Things to consider: 2 users may be too few, perhaps 3. Next: I need to also take away annotations other users have no done for projects running iou metrics
Shows clips the users has never touched and clips that the user has completed themselves
recently annotated clips will be marked 1, older clips will be marked as 2
Added power to change the given threshold, set a max limit for number of users to see the data, and included the number of views + confidence information on data portal (for now, for debuging purposes)
Displays the current confidence threshold and max user limit in the edit project form
Added confidence and retirement information to csv. Also added counted status to csv. (Side note: fixed extra column being added by datetime object)
There was issues using next buttons with a diffrent active other than pending on an IOU project. Fixed this issue so active works as expected
Fixed bugs that appeared during demo today and made confidence metric run during saving
Prevented empty overlap dataframes from breaking apicalls with an if statement. Also changed manual_id from bird to labelCate.value. NOTE: THIS WILL BREAK FOR MUTLICLASS LABELS, NEED TO ADD IN THAT FUNCTIONALITY
Made note of mutliple-select labels
* Replaced IOU with Pairwise Compare IOU with each user instead of grouping all the old users into one group. * Removed old debug and added new note * Reworked pairwise matrix * increased timeoue * Fixed 4 line bug * Added IOU CSV Storage to SQL Added - A row in table to store iou score materix as pandas - ability to download these csvs as a zip folder for more data anaysis -Yeah
Put an general error handler around the clip confidence work just so clip successfully saves
Fixed bug were no Relevant Audio button didn't save the clip
- Changed save all feature so it sends all segmentation data to the backend so it processes all at once as opposed to calling the api call mutliple times - This improves proformance slightly with confidence work as now we don't need to run mutliple iou mertics for each indivual annotation
- Removed unneeded frontend console output - Fixed issue where error message appears if save all does not finish when user moves to a new page - Fixed issue where if a user moves to a new clip, batch saving does not work
Fixed issue where if a project did not have IOU control pipeline enabled, it would not show clips on the data pipeline
Fixed bug where quick start would take you to annotation page even if no data was ready to be annotated!
…into clip-retirement
sprestrelski
approved these changes
Jun 27, 2022
Member
There was a problem hiding this comment.
When testing saving, I was able to save with four different accounts accessing the same clip at once, and it seemed to show up properly according to parameters set for clip retirement.
IOU projects should allow people to redo the clips without showing annotations from other users. Normal projects do not do this
I'm not sure I understand this properly - in both IoU and normal projects, I cannot see spectrogram annotations from other users.
One interesting behavior I found is that if you
- make a normal project
- make annotations
- then, switch the project to have the IOU scores, it does not automatically load the annotations - users need to refresh the page and open each clip again in order for them to show as "annotated"
Collaborator
Author
This is consistent with normal behavior, awesome I think we are good to merge! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Added bug fixes to clip retirement.
The big bug was with saving, triple check with all diffrent kinds of labels to ensure that Pyrenote can save without werid errors
When creating projects, click every button in the admin portal to make sure the settings are good. Be sure to enable next_button on the feature toggle popup and try doing 2 projects: 1 with the experimental quality control pipeline and test that out.
Test on mutliple users to make sure the projects are working as intended. IOU projects should allow people to redo the clips without showing annotations from other users. Normal projects do not do this