questions about NIAH

[Cooperx521]

Congrats for the insightful paper!

I noticed a few points in the figure in the appendix that I find a bit confusing, and here are two questions:

1. Since the 'Training Long Language Model' step uses a context length of only 224k, why does the model still show high accuracy even when the context length reaches 512k?

2. I observed that when the distractor is set to 5, the distribution of the NIAH results appears unusual. It seems that the context length of 224k performs better than the context length of 64k, which is quite different from what is typically seen in NIAH results for other models.

Looking forward to your insights on these points
Best regards

[jzhang38]

1. The first phenomenon is generally observed across long context training for across various models.
2. I do not have a good answer for the second question.