Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metrics and multiple validation/test dataloaders #12765

Open
pietrolesci opened this issue Apr 14, 2022 · 3 comments
Open

Metrics and multiple validation/test dataloaders #12765

pietrolesci opened this issue Apr 14, 2022 · 3 comments
Assignees
Labels
docs Documentation related logging Related to the `LoggerConnector` and `log()`

Comments

@pietrolesci
Copy link
Contributor

pietrolesci commented Apr 14, 2022

As discussed on slack with @justusschock, it would be nice to make explicit the behavior of torchmetrics when used with multiple dataloaders.

From @justusschock:

when using the module-based interface, PL leaves the aggregation to TM since many metrics are in fact non-trivial to aggregate properly. TM however, is designed to also work independently of PL. So it only updates the states when you tell it to and computes results based on its internal states. It does not know about the dataloader concept at all.
What PL does there internally, is that it just caches the metric object to log and for multiple dataloaders it would still cache the same object (since you don’t have different objects per loader). The metric’s internal state however would be global for all of the loaders since it is the same

cc @Borda @rohitgr7 @carmocca @edward-io @ananthsub @kamil-kaczmarek @Raalsky @Blaizzy

@pietrolesci pietrolesci added the needs triage Waiting to be triaged by maintainers label Apr 14, 2022
@justusschock justusschock added docs Documentation related logging Related to the `LoggerConnector` and `log()` and removed needs triage Waiting to be triaged by maintainers labels Apr 14, 2022
@rohitgr7
Copy link
Contributor

What PL does there internally, is that it just caches the metric object to log and for multiple dataloaders it would still cache the same object (since you don’t have different objects per loader). The metric’s internal state however would be global for all of the loaders since it is the same

how are you planning to make it work with multiple dataloaders? since the states are reset on epoch end and epoch end is triggered only once and when all the dataloaders are processed. Can you share more details?

@justusschock
Copy link
Member

@rohitgr7 this isn't about changing the behavior but making it more explicit in the docs :)

@rohitgr7
Copy link
Contributor

got it. btw I remember adding them here: https://torchmetrics.readthedocs.io/en/stable/pages/lightning.html#common-pitfalls.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs Documentation related logging Related to the `LoggerConnector` and `log()`
Projects
None yet
Development

No branches or pull requests

3 participants