Skip to content

Default to optimal confidence from model-eval#2206

Draft
leeclemnet wants to merge 1 commit intomainfrom
feat/model-eval-recommended-defaults
Draft

Default to optimal confidence from model-eval#2206
leeclemnet wants to merge 1 commit intomainfrom
feat/model-eval-recommended-defaults

Conversation

@leeclemnet
Copy link
Copy Markdown
Contributor

@leeclemnet leeclemnet commented Apr 7, 2026

What does this PR do?

Model eval calculates F1-optimal confidence thresholds but they aren't currently used for model inference. This PR together with https://github.com/roboflow/roboflow/pull/11053 makes it so. This feature applies only to the inference_models pathway. The legacy inference pathways keep their existing default confidence.

The key changes are

Wire recommendedParameters from model_eval through to inference

  • New RecommendedParameters pydantic model in inference_models/weights_providers/entities.py (confidence, per_class_confidence); parsed from Roboflow API in weights_providers/roboflow.py and threaded through auto_loaders/core.py → initialize_model
  • Auto-loader injects it onto the model instance via hasattr(type(model), "recommended_parameters"), so model classes opt in by declaring a class-level recommended_parameters: Optional[RecommendedParameters] = None

New ConfidenceFilter and post_process_with_confidence_filter() wrapper (inference_models/models/base/)

  • Encodes the 4-tier priority chain: explicit user → per-class optimal → global optimal → hardcoded default
  • Wrapper rewrites the confidence kwarg to the filter's floor before calling post_process (so NMS keeps boxes any class might still want), then refines per-class on the way out
  • Added on ObjectDetectionModel, InstanceSegmentationModel, KeypointsDetectionModel, SemanticSegmentationModel, MultiLabelClassificationModel. Single-label classification deliberately opts out (top-1 always wins)

Inference adapters route through the new wrapper (inference/core/models/inference_models_adapters.py)

  • OD/IS/KP/multi-label classification adapters call post_process_with_confidence_filter instead of post_process
  • Multi-label response builder now reads prediction.class_ids directly instead of re-thresholding the full confidence vector — the model's per-class decision survives to the API response

API request schema (inference/core/entities/requests/inference.py)

  • ObjectDetectionInferenceRequest.confidence default flipped from 0.4 → None so model-eval recommendations can take effect; explicit user values still win. Description updated to document the fallback chain

OLD inference path compatibility (inference/core/models/{object_detection,instance_segmentation,classification}_base.py)

  • Coalesce confidence is None → existing per-class default at the entry of infer() / make_response(), so the USE_INFERENCE_MODELS=false matrix variant still works after the request default flipped to None

Plumbing

  • BackendType moved from auto_loaders/entities.py → weights_providers/entities.py to break a circular import introduced by the new schema
  • CI workflow integration_tests_workflows_x86.yml overlays a local source build of inference_models (pip install --force-reinstall --no-deps ./inference_models) so adapter changes are actually exercised against pinned PyPI requirements

Dependencies:

Type of Change

  • New feature (non-breaking change that adds functionality)

Testing

  • I have tested this change locally
  • I have added/updated tests for this change

Test details:

  • New: test_confidence_filter.py, test_confidence_filter_attribute.py, test_post_process_filter.py, test_recommended_parameters.py, plus expanded test_roboflow.py and test_core.py coverage

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code where necessary, particularly in hard-to-understand areas
  • My changes generate no new warnings or errors
  • I have updated the documentation accordingly (if applicable)

Additional Context

@leeclemnet leeclemnet force-pushed the feat/model-eval-recommended-defaults branch 5 times, most recently from a19cb78 to 5660d57 Compare April 8, 2026 18:37
@leeclemnet leeclemnet changed the title Use recommended optimal confidence from model-eval Default to optimal per-class/global confidence from model-eval Apr 8, 2026
@leeclemnet leeclemnet force-pushed the feat/model-eval-recommended-defaults branch 2 times, most recently from 32fa48d to 473dd08 Compare April 8, 2026 20:07
@leeclemnet leeclemnet changed the title Default to optimal per-class/global confidence from model-eval Default to optimal confidence from model-eval Apr 8, 2026
@leeclemnet leeclemnet force-pushed the feat/model-eval-recommended-defaults branch 4 times, most recently from 8288b2c to 33ea1b0 Compare April 9, 2026 18:42
@leeclemnet leeclemnet force-pushed the feat/model-eval-recommended-defaults branch from 33ea1b0 to 262b5bf Compare April 10, 2026 16:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants