Multiple optimizers support in training_epoch_end

[quinor]

🐛 Bug

training_epoch_end only gets the return values from training_step called for the last optimizer each batch.

See:
https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/trainer/training_loop.py#L603
https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/trainer/training_loop.py#L657

To Reproduce

Create a LightningModule with two optimizers. Try to access both training_step return values in the training_epoch_end.

Code sample

(trivial)

Expected behavior

I'd like to have access to both as ie. a list of tuples or a tuple of lists. I'd probably prefer a tuple of lists, ie. I get a separate list of return values for each of the optimizers.

Environment

cuda:
        GPU:
                GeForce GTX 1080 Ti
        available:           True
        version:             10.1
packages:
        numpy:               1.18.2
        pyTorch_debug:       False
        pyTorch_version:     1.4.0
        pytorch-lightning:   0.7.2-dev
        tensorboard:         2.2.0
        tqdm:                4.43.0
system:
        OS:                  Linux
        architecture:
                64bit
                ELF
        processor:           
        python:              3.8.2
        version:             #1 SMP PREEMPT Sun, 22 Mar 2020 16:33:15 +0000

Additional context

[Borda]

@asafmanor pls may you check it?

[awaelchli]

@williamFalcon should we support this?

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!