Description
Description & Motivation
I am currently using PyTorch Lightning for training, and I'm using the CSVLogger callback to record the metric values during training to a CSV file. However, I've noticed that enabling CSVLogger and starting the training process significantly slows down the overall training speed.
I suspect that this slowdown might be caused by the default synchronous writing of metric values to the CSV file by CSVLogger, which could introduce delays due to IO operations. In order to optimize the training speed, I would like to know if there is a way to configure CSVLogger in PyTorch Lightning to perform asynchronous logging.
Therefore, I'm interested in learning how to configure CSVLogger to perform asynchronous logging, which could potentially improve the training speed.
If anyone has any experience or suggestions on how to configure CSVLogger for asynchronous logging in PyTorch Lightning, I would greatly appreciate it. Thank you very much!
Pitch
No response
Alternatives
No response
Additional context
No response
cc @Borda