Support for RS2 at the Selector instead of downsampling #466
Comments
MaxiBoether
added a commit
that referenced
this issue
Jun 4, 2024
This implements the random selection from the [RS2 paper](https://openreview.net/pdf?id=JnRStoIuTe) minus the learning rate scheduling adjustments. Note that it is a bit suboptimal to use the downsampling infrastructure here (#466). We might want to think about making the selector a bit more dynamic, but for now, this will suffice to run experiments. #462 should be merged before this is reviewed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In #465, we implement RS2 sampling, which does not need any information from the forward pass. However, the actual training data in RS2 changes each epoch, which the selector currently does not support. We fitted the implementation to the downsampling case, but actually it might be better to extend the selector component to allow updating the TTS if no information from model forward pass is required, since the definition of downsampling more or less is using information from the forward pass.
The text was updated successfully, but these errors were encountered: