Apply changes from juan5508/sc-3259/Create-Metric-V2-class

[nrichers]

Internal Notes for Reviewers

This PR transfers the changes from @juanmleng's private branch to our public open-source repository.

External Release Notes

This pull request introduces the Unit Metric feature, a new single value measure designed to streamline the risk assessment process in Machine Learning and AI model development. The Unit Metric simplifies complex evidence into actionable insights, allowing for efficient monitoring and comparison of risks across different models or datasets.

Example:

metric_id = "validmind.unit_metrics.sklearn.classification.F1"

inputs = {
    "model": vm_model,
    "dataset": vm_test_ds
}

result = vm.unit_metrics.run_metric(
    metric_id=metric_id, 
    inputs=inputs,
)
result.value

>>> 0.5709401709401709

result.summary

>>> {'f1': 0.5709401709401709}

Changes

• Implemented the UnitMetric class.
• Implemented sklearn classification metrics as unit metrics: F1, Accuracy, Precision, Recall and ROC_AUC
• Added vm.unit_metrics.run_metric to the ValidMind client library to support running metrics.

Testing

• Provided a notebook on defining, executing, and extracting results from Unit Metrics using a customer churn use case. See run_unit_metrics.ipynb

[shortcut-integration]

This pull request has been linked to Shortcut Story #3259: UnitMetric class POC.

[github-actions]

Pull requests must include a description in the release notes section.

[github-actions]

Pull requests must include a description in the release notes section.

[CLAassistant]

All committers have signed the CLA.

[github-actions]

Pull requests must include a description in the release notes section.

[github-actions]

Pull requests must include a description in the release notes section.

[github-actions]

Pull requests must include a description in the release notes section.

[cachafla]

Great job 👏 I left a final comment to remove unused object keys. Other than that it's ready to go 🚀

[nrichers]

@juanmleng looks like Andres approved your pull request. Congratulations on making the very first contribution to our developer framework as open-source software. This will forever be pull request #1 in this repository! 🙌