DDP + static graph can result in garbage data returned by all_gather

[mooninrain]

Bug description

When I use self.all_gather in LightningModule with strategies.DDPStrategy(static_graph=True) for multi-node inference,
the returned values are partially corrupted.

What version are you seeing the problem on?

v2.0

How to reproduce the bug

class ModelWrapper(pl.LightningModule):
    ...
    def on_validation_epoch_end(self) -> None:
        super().on_validation_epoch_end()
    
        test_tensor = torch.tensor([self.global_rank]).to(current_device)
        self.trainer.strategy.barrier()
        test_all = self.all_gather(test_tensor)
        self.trainer.strategy.barrier()
        print(test_all)

And it is called by

    # set up a datamodule
    ...

    trainer = pl.Trainer(
        accelerator="gpu",
        devices=args.devices,
        num_nodes=args.num_nodes,
        strategy=strategies.DDPStrategy(static_graph=True),
    )
    trainer.validate(model, datamodule=datamodule)

Error messages and logs

It should return results as:
torch.tensor([0, 1, 2, ..., world_size-1])

But while most of values are right, a few comes back with corrupted data with very large numbers, like
torch.tensor([0, 1, 2, 3, 913478191043, 5, ..., world_size -1])

Environment

Current environment

#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow): LightningModule
#- PyTorch Lightning Version (e.g., 1.5.0): 2.0.3
#- PyTorch Version (e.g., 2.0): 1.12.1
#- Python version (e.g., 3.9): 3.9
#- OS (e.g., Linux): Linux
#- CUDA/cuDNN version: 11.6
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source): conda

Addtional Information

I do notice the warning that trainer.validate should not be called with DDPStrategy, which makes LightningModule copies some datapoints for the last round of validation. Actually, this is exactly why I use all_gather during validation - to implement a drop-last validation.
It looks like it's caused by failure to block all processes during all_gather. I've tried to investigate why this happens myself, but I can't find any clues.

[adaruna3]

Any updates on this? I'm observing the same problem today. all_gather works fine for 2 GPUs with DDP strategy, but 3 or 4 GPU and some returned values are suddenly corrupted.

[mooninrain]

Any updates on this? I'm observing the same problem today. all_gather works fine for 2 GPUs with DDP strategy, but 3 or 4 GPU and some returned values are suddenly corrupted.

Sadly, nothing from my side. We still face the same problem in PyTorch 2.0 & python 3.10.