Hello! I really like your paper and I am doing similar research currently. I would like to know if you have experienced issues with evaluating models loaded via HookedTransformer? I found that models loaded via HookedTransformer consumed too much memory so I was getting OOM errors during first iterations of chatting with the model. In addition, I was also interested what is the reason behind using lm-evaluation-harness library for evaluation? Thanks in advance!
Hello! I really like your paper and I am doing similar research currently. I would like to know if you have experienced issues with evaluating models loaded via
HookedTransformer? I found that models loaded viaHookedTransformerconsumed too much memory so I was getting OOM errors during first iterations of chatting with the model. In addition, I was also interested what is the reason behind usinglm-evaluation-harnesslibrary for evaluation? Thanks in advance!