Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RichProgressBar in v1.6 is slower than v1.5 #13937

Open
akihironitta opened this issue Jul 29, 2022 · 4 comments
Open

RichProgressBar in v1.6 is slower than v1.5 #13937

akihironitta opened this issue Jul 29, 2022 · 4 comments
Labels
bug Something isn't working help wanted Open to be worked on performance pl Generic label for PyTorch Lightning package priority: 1 Medium priority task progress bar: rich
Milestone

Comments

@akihironitta
Copy link
Contributor

akihironitta commented Jul 29, 2022

🐛 Bug

Similarly to (but independently of) #13179, there's a regression in speed of the rich progress bar between 1.5 and 1.6.

time: 2.059369374997914   # v1.5.10
time: 13.186531708983239  # v1.6.0
time: 11.988152374979109  # v1.6.5
time: 12.897805292042904  # master (aefb9ab43f9a8e6704558a346dbae1a00044bb45)

To Reproduce

from time import monotonic
import torch
from torch.utils.data import DataLoader, Dataset
from pytorch_lightning import LightningModule, Trainer
from pytorch_lightning.callbacks import RichProgressBar

class RandomDataset(Dataset):
    def __init__(self, size, length):
        self.len = length
        self.data = torch.randn(length, size)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.len

class BoringModel(LightningModule):
    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(32, 2)

    def forward(self, x):
        return self.layer(x)

    def training_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("train_loss", loss)
        return {"loss": loss}

    def validation_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("valid_loss", loss)

    def configure_optimizers(self):
        optimizer = torch.optim.SGD(self.layer.parameters(), lr=0.1)
        lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=1)
        return [optimizer], [lr_scheduler]

def main():
    train_data = DataLoader(RandomDataset(32, 64), batch_size=2)
    val_data = DataLoader(RandomDataset(32, 64), batch_size=2)
    model = BoringModel()
    trainer = Trainer(
        max_epochs=100,
        enable_model_summary=False,
        enable_checkpointing=False,
        logger=False,
        benchmark=False,  # True by default in 1.6.{0-3}.
        callbacks=RichProgressBar(),
    )
    t0 = monotonic()
    trainer.fit(model, train_dataloaders=train_data, val_dataloaders=val_data)
    print("time:", monotonic() - t0)

if __name__ == "__main__":
    main()

Expected behavior

There should be no regression unless there's a reasonable explanation.

Environment

  • Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow): n/a
  • PyTorch Lightning Version (e.g., 1.5.0): master (aefb9ab), 1.6.5, 1.6.0, 1.5.10
  • PyTorch Version (e.g., 1.10): 1.12
  • Python version (e.g., 3.9): 3.10
  • OS (e.g., Linux): macOS
  • CUDA/cuDNN version: n/a
  • GPU models and configuration: Apple Silicon M1
  • How you installed PyTorch (conda, pip, source): pip
  • If compiling from source, the output of torch.__config__.show():
  • Running environment of LightningApp (e.g. local, cloud): n/a
  • Any other relevant information: rich==12.5.1

Additional context

Investigating!

https://github.com/Lightning-AI/lightning/pulls?q=is%3Apr+label%3A%22progress+bar%3A+rich%22+milestone%3A%22pl%3A1.6%22+

cc @tchaton @rohitgr7 @kaushikb11 @Borda @akihironitta

@akihironitta akihironitta self-assigned this Jul 29, 2022
@Borda Borda added the priority: 0 High priority task label Jul 30, 2022
@Borda Borda added this to the pl:1.6.x milestone Jul 30, 2022
@Borda Borda added priority: 1 Medium priority task and removed priority: 0 High priority task labels Jul 30, 2022
@Borda Borda modified the milestones: pl:1.6.x, pl:1.7.x Jul 30, 2022
@akihironitta akihironitta added the pl Generic label for PyTorch Lightning package label Jul 31, 2022
@rohitgr7
Copy link
Contributor

rohitgr7 commented Aug 1, 2022

time: 2.059369374997914 # v1.5.10
time: 13.186531708983239 # v1.6.0
time: 11.988152374979109 # v1.6.5
time: 12.897805292042904 # master (aefb9ab)

it's huge... maybe it's the same refresh that is causing the issue

@shenoynikhil

This comment was marked as resolved.

@akihironitta

This comment was marked as resolved.

@kaushikb11
Copy link
Contributor

@akihironitta How does it compare with TQDMProgressBar over recent releases?

@carmocca carmocca modified the milestones: pl:1.7.x, v1.8.x Oct 13, 2022
@Borda Borda modified the milestones: v1.8.x, v1.9 Jan 6, 2023
@Borda Borda modified the milestones: v1.9, v1.9.x Jan 16, 2023
@awaelchli awaelchli added the help wanted Open to be worked on label Dec 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Open to be worked on performance pl Generic label for PyTorch Lightning package priority: 1 Medium priority task progress bar: rich
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants