I received some feedback from a valued user which I thought might be useful to post here for the record:
I don't remember having many problems with IteratedModel, so I was able to do all I wanted with little effort. It provides a lot of options off-the-shelf in an intuitive way. I only needed to spend some time thinking about two things:
- How to do learning-rate (LR) scheduling through this feature. A complication is that the
Adam type is immutable, so whenever I wanted to update the LR, I had to get its value, alter it, and create a new Adam instance with it.
- The difference between "control cycles" and
Step(n) and how they relate to training batches and epochs when using NNs. Understanding all these "progress measures" is needed to define how often the LR will be updated. When using decay, if updates are too frequent, the LR vanishes too fast and training halts
I received some feedback from a valued user which I thought might be useful to post here for the record:
I don't remember having many problems with
IteratedModel, so I was able to do all I wanted with little effort. It provides a lot of options off-the-shelf in an intuitive way. I only needed to spend some time thinking about two things:Adamtype is immutable, so whenever I wanted to update the LR, I had to get its value, alter it, and create a new Adam instance with it.Step(n)and how they relate to training batches and epochs when using NNs. Understanding all these "progress measures" is needed to define how often the LR will be updated. When using decay, if updates are too frequent, the LR vanishes too fast and training halts