Translate `mini_epoch` to `istep`

[TillHae]

Describe the task. It can be a feature, documentation, etc.

Translating mini_epoch to istep will enable a more convenient process for the User.

e.g. when using train_continue with mini_epoch instead of istep.

Relates to #1040 #587 #848
CC @clessig @MatKbauer

Hedgedoc URL, if you are keeping notes, plots, logs in hedgedoc.

No response

Area

• datasets, data readers, data preparation and transfer
• model
• science
• infrastructure and engineering
• evaluation, export and visualization
• documentation

[MatKbauer]

@clessig, do you have thoughts on where istep might not yet be used correctly or where mini_epoch needs to be replaced by istep?

[MatKbauer]

@clessig and I had a chat and identified the following topics to proceed with. Some tasks are more involved and Christian will take care of them, as indicated with the name in brackets.

1. Change mini-epoch to istep in chkpt writing. File name should support numbers > 99999 (adjust from five to six digits, that is, make run-id_chkpt55555 -> run-id_chkpt666666) [Till]
2. Write chkpt every 512 isteps, which translates to saving the model after a mini-epoch, when num_samples: 4096 and training on two nodes [Till]
3. In plot_train, use isteps instead of samples on x-axis to prevent sensitivity to number of nodes [Till]
4. Check world_size, that is, world_size_original should be removed entirely [Christian]
5. Disable changing #nodes when train_continue, i.e., if world_size exists already, it must match with the new one [Christian]
6. Regarding Distinguish local dataloader epoch from global epoch #848, let's simply remove at mini_epoch={self.mini_epoch} from the log here [Till]

@TillHae, is this graspable for you and can you proceed from here?

[TillHae]

@clessig and I had a chat and identified the following topics to proceed with. Some tasks are more involved and Christian will take care of them, as indicated with the name in brackets.

1. Change mini-epoch to istep in chkpt writing. File name should support numbers > 99999 (adjust from five to six digits, that is, make `run-id_chkpt55555` -> `run-id_chkpt666666`) [Till]

2. Write chkpt every 512 isteps, which translates to saving the model after a mini-epoch, when `num_samples: 4096` and training on two nodes [Till]

3. In plot_train, use isteps instead of samples on x-axis to prevent sensitivity to number of nodes [Till]

4. Check `world_size`, that is, `world_size_original` should be removed entirely [Christian]

5. Disable changing #nodes when `train_continue`, i.e., if `world_size` exists already, it must match with the new one [Christian]

6. Regarding [Distinguish local dataloader epoch from global epoch #848](https://github.com/ecmwf/WeatherGenerator/issues/848), let's simply remove ` at mini_epoch={self.mini_epoch}` from the log [here](https://github.com/ecmwf/WeatherGenerator/blob/develop/src/weathergen/datasets/multi_stream_data_sampler.py#L289C42-L289C73) [Till]

@TillHae, is this graspable for you and can you proceed from here?

Looks good to me. I won’t have much time over the next two weeks, so progress may be a bit slower, but I should still be able to get it done.
@clessig As for removing world_size_original, I’m almost done with it, so I can pick it up. I just still have an open question regarding the calculation of batch size #1524
I really like to use isteps for plotting as well. I think it is the best solution to prevent the sensitivity to number of nodes.

[clessig]

@clessig and I had a chat and identified the following topics to proceed with. Some tasks are more involved and Christian will take care of them, as indicated with the name in brackets.

1. Change mini-epoch to istep in chkpt writing. File name should support numbers > 99999 (adjust from five to six digits, that is, make `run-id_chkpt55555` -> `run-id_chkpt666666`) [Till]

2. Write chkpt every 512 isteps, which translates to saving the model after a mini-epoch, when `num_samples: 4096` and training on two nodes [Till]

3. In plot_train, use isteps instead of samples on x-axis to prevent sensitivity to number of nodes [Till]

4. Check `world_size`, that is, `world_size_original` should be removed entirely [Christian]

5. Disable changing #nodes when `train_continue`, i.e., if `world_size` exists already, it must match with the new one [Christian]

6. Regarding [Distinguish local dataloader epoch from global epoch #848](https://github.com/ecmwf/WeatherGenerator/issues/848), let's simply remove ` at mini_epoch={self.mini_epoch}` from the log [here](https://github.com/ecmwf/WeatherGenerator/blob/develop/src/weathergen/datasets/multi_stream_data_sampler.py#L289C42-L289C73) [Till]

@TillHae, is this graspable for you and can you proceed from here?

Looks good to me. I won’t have much time over the next two weeks, so progress may be a bit slower, but I should still be able to get it done. @clessig As for removing world_size_original, I’m almost done with it, so I can pick it up. I just still have an open question regarding the calculation of batch size #1524 I really like to use isteps for plotting as well. I think it is the best solution to prevent the sensitivity to number of nodes.

Can you please push what you have. Then I can take over from here.

[TillHae]

@clessig and I had a chat and identified the following topics to proceed with. Some tasks are more involved and Christian will take care of them, as indicated with the name in brackets.

1. Change mini-epoch to istep in chkpt writing. File name should support numbers > 99999 (adjust from five to six digits, that is, make `run-id_chkpt55555` -> `run-id_chkpt666666`) [Till]

2. Write chkpt every 512 isteps, which translates to saving the model after a mini-epoch, when `num_samples: 4096` and training on two nodes [Till]

3. In plot_train, use isteps instead of samples on x-axis to prevent sensitivity to number of nodes [Till]

4. Check `world_size`, that is, `world_size_original` should be removed entirely [Christian]

5. Disable changing #nodes when `train_continue`, i.e., if `world_size` exists already, it must match with the new one [Christian]

6. Regarding [Distinguish local dataloader epoch from global epoch #848](https://github.com/ecmwf/WeatherGenerator/issues/848), let's simply remove ` at mini_epoch={self.mini_epoch}` from the log [here](https://github.com/ecmwf/WeatherGenerator/blob/develop/src/weathergen/datasets/multi_stream_data_sampler.py#L289C42-L289C73) [Till]

@TillHae, is this graspable for you and can you proceed from here?

Looks good to me. I won’t have much time over the next two weeks, so progress may be a bit slower, but I should still be able to get it done. @clessig As for removing world_size_original, I’m almost done with it, so I can pick it up. I just still have an open question regarding the calculation of batch size #1524 I really like to use isteps for plotting as well. I think it is the best solution to prevent the sensitivity to number of nodes.

Can you please push what you have. Then I can take over from here.

Looks like almost all of my changes have already been addressed in the meantime, and only the occurrences with batch_size_per_gpu, where I didn’t know what to do, remain. You can go ahead and start from the current develop branch.