Long-context Pre-Training of Large Language Models

This folder contains the codes and models for our research papers on LongContext Post-Training.

• 🔥 Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Model, accepted as ICLR 2025 conference paper !

• 🔥 NExtLong: Toward Effective Long-Context Training without Long Documents, ranked 1st among LLMs under 10B on the LongBench v2 leaderboard (2025/01/23) and accepted as ICML 2025 conference paper !

Long-context Instruct-Tuning of Large Language Models

• 🔥 [LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions] Synthesizing large-scale Instruct datasets, further improving the long-context ability of Instruct models!