Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add optional ignore_index in all metrics #2273

Open
fabricecarles opened this issue Dec 18, 2023 · 1 comment
Open

add optional ignore_index in all metrics #2273

fabricecarles opened this issue Dec 18, 2023 · 1 comment
Labels
enhancement New feature or request
Milestone

Comments

@fabricecarles
Copy link

🚀 Feature

Could you please consider incorporating a systematically optional ignore_index argument in all metrics?

For instance, in classification tasks, we currently have the ability to use ignore_index as follows:

torchmetrics.classification.MulticlassJaccardIndex(num_classes=15, ignore_index=255)

However, in the case of regression, there is currently no mechanism to utilize ignore_index when computing metrics such as MSE or MAE.

Motivation

There is a plethora of datasets for regression tasks (but not limited to regression) that contain pixel values representing missing data. It is crucial to have the capability to ignore these values when computing metric statistics.

Pitch

I suggest that all metrics be updated to implement an optional ignore_index argument.

Alternatives

Is it feasible to incorporate the ignore_index functionality in advanced metric settings? For example, as shown in https://lightning.ai/docs/torchmetrics/stable/pages/overview.html#metric-kwargs

Additional context

For instance, in the context of depth maps, ground truth data may contain missing values replaced with an arbitrary ignore_index, and the training loss is already designed to ignore them. This feature would provide consistency and flexibility across various metrics.

@fabricecarles fabricecarles added the enhancement New feature or request label Dec 18, 2023
@fabricecarles fabricecarles changed the title add ignor_index in aall metrics add ignor_index in all metrics Dec 18, 2023
@fabricecarles fabricecarles changed the title add ignor_index in all metrics add ignored_index in all metrics Dec 18, 2023
Copy link

Hi! thanks for your contribution!, great first issue!

@fabricecarles fabricecarles changed the title add ignored_index in all metrics add ignore_index in all metrics Dec 18, 2023
@fabricecarles fabricecarles changed the title add ignore_index in all metrics add optional ignore_index in all metrics Dec 18, 2023
@SkafteNicki SkafteNicki added this to the future milestone Dec 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants
@SkafteNicki @fabricecarles and others