Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support all iterator modes for fit/validate/test/predict #16830

Open
1 of 3 tasks
carmocca opened this issue Feb 21, 2023 · 12 comments
Open
1 of 3 tasks

Support all iterator modes for fit/validate/test/predict #16830

carmocca opened this issue Feb 21, 2023 · 12 comments
Labels
data handling Generic data-related topic feature Is an improvement or enhancement help wanted Open to be worked on pl Generic label for PyTorch Lightning package
Milestone

Comments

@carmocca
Copy link
Contributor

carmocca commented Feb 21, 2023

Description & Motivation

trainer.fit only works with CombinedLoader(..., mode="max_size_cycle"|"min_size")

trainer.{validate,test,predict} only works with CombinedLoader(..., mode="sequential")

This constraint is checked in the top-level loops:
https://github.com/Lightning-AI/lightning/blob/0009cde1db1a9ab4e2f1e0a9f69a4affb59d5134/src/lightning/pytorch/loops/fit_loop.py#L351-L354
https://github.com/Lightning-AI/lightning/blob/0009cde1db1a9ab4e2f1e0a9f69a4affb59d5134/src/lightning/pytorch/loops/evaluation_loop.py#L182-L183

Pitch

Have all trainer functions support all modes

TODO:

Alternatives

Not do it

Additional context

This builds on top of #16726

cc @Borda @justusschock @awaelchli

@carmocca carmocca added feature Is an improvement or enhancement data handling Generic data-related topic pl Generic label for PyTorch Lightning package labels Feb 21, 2023
@carmocca carmocca modified the milestones: 2.0, future Feb 21, 2023
@mees
Copy link
Contributor

mees commented Mar 20, 2023

I am migrating my code to PL 2 and it seems that for the val dataloader getting a batch to be of the form {"key_a": batch_dataloader_a, "key_b": batch_dataloader_b} is not implemented in PL 2 yet. Here my old code as a reference:

def val_dataloader(self):
    val_dataloaders = {
        key: DataLoader(
            dataset,
            batch_size=dataset.batch_size,
            shuffle=False,
            num_workers=dataset.num_workers,
            pin_memory=False,
        )
        for key, dataset in self.val_datasets.items()
    }
    combined_val_loaders = CombinedLoader(val_dataloaders, "max_size_cycle")
    return combined_val_loaders

@carmocca
Copy link
Contributor Author

carmocca commented Mar 28, 2023

@mees I added support for that in #17163, if you want to give it a try. The PR only implements it for validation and testing.

@bkmi
Copy link

bkmi commented May 15, 2023

@mees I added support for that in #17163, if you want to give it a try. The PR only implements it for validation and testing.

really helpful! I hope this gets into "stable" soon.... or even the next release!

@FarzanT
Copy link

FarzanT commented May 15, 2023

I really wish there was sequential support in the training loop. Right now, it's not clear how one should handle batches of potentially different sizes in the training_step. We'd have to collate inside the training_step and ensure the given batch size is divided by the number of data loaders to keep gradient accumulation consistent etc. It gets pretty ugly.
@carmocca Thank you for your work on this issue. Not to rush you, but any update on the sequential support in the training loop?
Thanks again!

@carmocca
Copy link
Contributor Author

Unfortunately, I dont have bandwidth to work on this now. If somebody wants to try, I can help getting the PR merged.
You can follow the structure in the EvaluationLoop. The training hooks will need an optional dataloader_idx argument

@surya-narayanan
Copy link

@mees I added support for that in #17163, if you want to give it a try. The PR only implements it for validation and testing.

really helpful! I hope this gets into "stable" soon.... or even the next release!

Me too! Is there any release timeline / nightly version with this supported? I can't use lightning without this and really would like to leverage its other features!

@spfrommer
Copy link

spfrommer commented Aug 17, 2023

Ditto! FYI for others pulling nightly will get the feature: #17163

@chenhaomingbob
Copy link

Thanks! I also need this great feature.

@awaelchli awaelchli modified the milestones: future, 2.2 Oct 6, 2023
@johnathanchiu
Copy link

+1, please release this feature asap!

@awaelchli awaelchli modified the milestones: 2.2, 2.3 Feb 3, 2024
@awaelchli awaelchli modified the milestones: 2.3, future Jun 2, 2024
@lukas-folle-snkeos
Copy link

Is this feature currently worked on?

@carmocca carmocca removed their assignment Jul 30, 2024
@carmocca
Copy link
Contributor Author

As far as I know, nobody is currently working on it, Lukas

@astirn
Copy link

astirn commented Sep 19, 2024

I would also really like this feature. I use CombinedDataloader to encapsulate modality-specific Dataloaders to recycle modalities with fewer data than our largest modality. For this reason, I use CombinedDatloader with "max_size_cycle"/"min_cycle" for train/validation and would like to be able for predict as well. Thanks for considering it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
data handling Generic data-related topic feature Is an improvement or enhancement help wanted Open to be worked on pl Generic label for PyTorch Lightning package
Projects
None yet
Development

No branches or pull requests