Is your feature request related to a problem? Please describe.
I have trained a model and i would like to know if it performs well and on which datarow it is performing well or not.
Describe the solution you'd like
I would like to be able to filter individual prediction results in labelstudio by their score.
More generally :
- I would like to be able to add one or more metrics to a prediction result (for ex : Score, IoU)
- I would like to be able to add one or more attributes to a prediction result (for ex : Status : "TruePositive")
In this list, there would be two new possibilities :
- pred. result - IoU (from metrics dict) -> filtrable by comparison
- pred. result - Status (from attribute dict) -> filtrable by content
- pred.result - Set (from attribute dict -> filterable by content (exemple train/validation/test)
I would like to be able to filter theses prediction fields directly in the data manager.
Describe alternatives you've considered
I can do this by hand but it's very time consuming. I can use another tool like Voxel51, but because my data are stored in a "cloud like storage", i cannot use their base version.
Impact Analysis
In order to implement this, we'd need to modify the schema of a result, to contain theses field :
We add two new entries :
- metrics : dict[str,int|float]
- attributes: dict[str,str]
They will be stored as json/dict in 2 separates columns in the prediction result database as a list of dictionary aligned on the result column (same number of values).
In order to aware the data manager of the existence of attributes and metrics. We may add a kind of ontology here :
In the ontology, we can add attributes and metrics.
This ontology would be necessary for two reason :
- aware the datamanager of the existence of the fields to allow filtering on them
- filter metrics and attributes during the prediction ingestion in order to only authorize medatadata/attributes that are declared.
It would be stored in a prediction_ontology database (one project == one ontology).
Is your feature request related to a problem? Please describe.
I have trained a model and i would like to know if it performs well and on which datarow it is performing well or not.
Describe the solution you'd like
I would like to be able to filter individual prediction results in labelstudio by their score.
More generally :
In this list, there would be two new possibilities :
I would like to be able to filter theses prediction fields directly in the data manager.
Describe alternatives you've considered
I can do this by hand but it's very time consuming. I can use another tool like Voxel51, but because my data are stored in a "cloud like storage", i cannot use their base version.
Impact Analysis
In order to implement this, we'd need to modify the schema of a result, to contain theses field :
We add two new entries :
They will be stored as json/dict in 2 separates columns in the prediction result database as a list of dictionary aligned on the result column (same number of values).
In order to aware the data manager of the existence of attributes and metrics. We may add a kind of ontology here :
In the ontology, we can add attributes and metrics.
This ontology would be necessary for two reason :
It would be stored in a prediction_ontology database (one project == one ontology).