[DE-4880] Update annotation related endpoints for multiple ground truth sets #444
Conversation
iz18
force-pushed
the
iz/nucleusTaskBasedAnnotations
branch
from
8818b62 to
421282f
Compare
…m parent classes but not needing the task id field
iz18
force-pushed
the
iz/nucleusTaskBasedAnnotations
branch
from
f4ccf81 to
9c918fa
Compare
|
Removed the black formatting in circle ci because it doesn't behave the same way as local black formatter on the same version. Weird issue and maybe should be looked into? Will put a ticket into backlog. |
| - run: | ||
| name: Black Formatting Check # Only validation, without re-formatting | ||
| command: | | ||
| poetry run black --check . | ||
| # - run: | ||
| # name: Black Formatting Check # Only validation, without re-formatting | ||
| # command: | | ||
| # poetry show black | ||
| # poetry run black --check . |
There was a problem hiding this comment.
see thread: https://scaleapi.slack.com/archives/C05Q7DSPQF9/p1734546043282449, i think we need to up the black versioning, then python dependency version but not sure about how to still support testing for python versions less than 3.8
| ## [0.17.8](https://github.com/scaleapi/nucleus-python-client/releases/tag/v0.17.7) - 2024-11-05 | ||
|
|
There was a problem hiding this comment.
The date this wrong, and the URL is wrong, please update.
| [test_repr(BoxPrediction.from_json(_)) for _ in TEST_BOX_PREDICTIONS] | ||
|
|
||
| [test_repr(LinePrediction.from_json(_)) for _ in TEST_LINE_PREDICTIONS] | ||
|
|
||
| [ | ||
| test_repr(PolygonPrediction.from_json(_)) | ||
| for _ in TEST_POLYGON_PREDICTIONS | ||
| ] | ||
|
|
||
| [ | ||
| test_repr(CategoryPrediction.from_json(_)) | ||
| for _ in TEST_CATEGORY_PREDICTIONS | ||
| ] | ||
|
|
||
| [ | ||
| test_repr(CategoryPrediction.from_json(_)) | ||
| for _ in TEST_DEFAULT_CATEGORY_PREDICTIONS | ||
| ] | ||
|
|
There was a problem hiding this comment.
Why were these removed?
There was a problem hiding this comment.
removed them because predictions are a child class of annotations, so repr includes parent specific fields and fails
With this feature, we will have multiple sets of ground truth if a dataset item is labeled across multiple tasks. To not suddenly break everything on the ML side, we want to have the option to return the legacy results, which is only the set of annotations that correspond to the most recent task. This is also changed to be consistent with the model results shown on Nucleus itself where it will only pull the most recent task. The task id is also now returned with all the annotations.
Moving forward, ML should keep track of which set of ground truth they trained their model with, and they can pull back that set of ground truth by setting
only_most_recent_taskto False. I think a new endpoint where they can request a set of ground truth based on a specifictask_idwould also be helpful in the future.Accompanies this pr