-
Notifications
You must be signed in to change notification settings - Fork 4
Expand file tree
/
Copy pathindex.html
More file actions
441 lines (387 loc) · 28.5 KB
/
Copy pathindex.html
File metadata and controls
441 lines (387 loc) · 28.5 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<!-- Meta tags for social media banners, these should be filled in appropriatly as they are your "business card" -->
<!-- Replace the content tag with appropriate information -->
<meta name="description" content="M2PO: Multi-modal Pre-training and Prompt Optimization">
<meta property="og:title" content="M2PO"/>
<meta property="og:description" content="Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?"/>
<meta property="og:url" content="https://Infini-AI-Lab.github.io/GRESO/"/>
<!-- Path to banner image, should be in the path listed below. Optimal dimenssions are 1200X630-->
<!-- <meta property="og:image" content="static/images/proj_fig.png" /> -->
<meta property="og:image:width" contecdnt="1200"/>
<meta property="og:image:height" content="630"/>
<meta name="twitter:title" content="M2PO">
<meta name="twitter:description" content="Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?">
<!-- Path to banner image, should be in the path listed below. Optimal dimenssions are 1200X600-->
<!-- <meta name="twitter:image" content="static/images/proj_fig.png"> -->
<meta name="twitter:card" content="summary_large_image">
<!-- Keywords for your paper to be indexed by-->
<meta name="keywords" content="RL for LLMs">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?</title>
<!-- <link rel="icon" type="image/x-icon" href="static/images/icon.jpg"> -->
<link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro"
rel="stylesheet">
<link rel="stylesheet" href="static/css/bulma.min.css">
<link rel="stylesheet" href="static/css/bulma-carousel.min.css">
<link rel="stylesheet" href="static/css/bulma-slider.min.css">
<link rel="stylesheet" href="static/css/fontawesome.all.min.css">
<link rel="stylesheet"
href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
<link rel="stylesheet" href="static/css/index.css">
<script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
<script src="https://documentcloud.adobe.com/view-sdk/main.js"></script>
<script defer src="static/js/fontawesome.all.min.js"></script>
<script src="static/js/bulma-carousel.min.js"></script>
<script src="static/js/bulma-slider.min.js"></script>
<script src="static/js/index.js"></script>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]}});
</script>
<script type="text/javascript"
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
</head>
<body>
<section class="hero" style="background-color: #FFFFFF !important;">
<div class="hero-body">
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column has-text-centered">
<!-- <h3 class="custom-font" style="display: inline;">*</h3> -->
<h1 class="title is-2 publication-title" style="display: inline;">Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?</h1>
<br><br>
<div class="is-size-5 publication-authors">
<!-- Paper authors -->
<span class="author-block">
<a href="http://zhenghaizhong.com/" target="_blank">Haizhong Zheng</a><sup>1</sup>,</span>
<span class="author-block">
<a href="https://jiawei-zhao.netlify.app/" target="_blank"> Jiawei Zhao</a><sup>2</sup>,
</span>
<span class="author-block">
<a href="https://www.andrew.cmu.edu/user/beidic/" target="_blank">Beidi Chen</a><sup>1</sup>
</span>
</div>
<div class="is-size-5 publication-authors">
<span class="affliation"><small>
<sup>1</sup>Carnegie Mellon University
<sup>2</sup>Meta AI
</small></span>
<!-- <span class="eql-cntrb"><small><br><sup>*</sup>Indicates Equal Contribution</small></span> -->
</div>
<div class="column has-text-centered">
<!-- ArXiv abstract Link -->
<span class="link-block">
<a href="https://arxiv.org/abs/2510.01161" target="_blank"
class="external-link button is-normal is-rounded is-light">
<span class="icon">
<i class="ai ai-arxiv"></i>
</span>
<span>arXiv</span>
</a>
</span>
<!-- Github link -->
<span class="link-block">
<a href="https://github.com/Infini-AI-Lab/M2PO/" target="_blank"
class="external-link button is-normal is-rounded is-light">
<span class="icon">
<i class="fab fa-github"></i>
</span>
<span>Code</span>
</a>
</span>
<!-- notion link -->
<span class="link-block">
<a href="https://m2po.notion.site/rl-stale-m2po" target="_blank"
class="external-link button is-normal is-rounded is-light">
<span class="icon">
<!-- <i class="fa-brands fa-notion"></i> -->
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 640"><!--!Font Awesome Free v7.1.0 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free Copyright 2025 Fonticons, Inc.--><path d="M158.9 164.2C173.8 176.3 179.4 175.4 207.5 173.5L471.8 157.6C477.4 157.6 472.7 152 470.9 151.1L426.9 119.4C418.5 112.9 407.3 105.4 385.8 107.3L129.9 125.9C120.6 126.8 118.7 131.5 122.4 135.2L158.8 164.1zM174.8 225.8L174.8 503.9C174.8 518.8 182.3 524.4 199.1 523.5L489.6 506.7C506.4 505.8 508.3 495.5 508.3 483.4L508.3 207.2C508.3 195.1 503.6 188.5 493.3 189.5L189.7 207.1C178.5 208 174.8 213.6 174.8 225.8zM461.5 240.7C463.4 249.1 461.5 257.5 453.1 258.5L439.1 261.3L439.1 466.6C426.9 473.1 415.7 476.9 406.4 476.9C391.4 476.9 387.7 472.2 376.5 458.2L285 314.5L285 453.5L314 460C314 460 314 476.8 290.6 476.8L226.2 480.5C224.3 476.8 226.2 467.4 232.7 465.6L249.5 460.9L249.5 277.1L226.2 275.2C224.3 266.8 229 254.7 242.1 253.7L311.2 249L406.5 394.6L406.5 265.8L382.2 263C380.3 252.7 387.8 245.3 397.1 244.3L461.6 240.5zM108.4 100.7L374.6 81.1C407.3 78.3 415.7 80.2 436.2 95.1L521.2 154.8C535.2 165.1 539.9 167.9 539.9 179.1L539.9 506.7C539.9 527.2 532.4 539.4 506.3 541.2L197.2 559.8C177.6 560.7 168.2 557.9 158 544.9L95.4 463.7C84.2 448.8 79.5 437.6 79.5 424.5L79.5 133.3C79.5 116.5 87 102.5 108.4 100.6z"/></svg>
</span>
<span>Notion Blog</span>
</a>
</span>
<!-- Video Link
<span class="link-block">
<a href="https://youtu.be/vRAaAyjr6Jo" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fab fa-youtube"></i>
</span>
<span>Video</span>
</a>
</span> -->
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<!-- TLDR -->
<section class="section hero is-light">
<div class="container is-max-desktop">
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<div class="content has-text-justified">
<p>
<strong style="font-weight: 900;color: #0f598a">TL;DR:</strong>
Our work shows that stale data can be as informative as on-policy data if exploited properly. We introduce M2PO, second-moment trust proxy optimization. M2PO significantly reduces the fraction of clipped tokens under high staleness with maintaining stable optimization.
Extensive evaluation across six model scales (1.7B–32B) shows that M2PO delivers stable off-policy training even with data stale by <i><u>at least 256 model updates</i></u> and matches on-policy performance.
</p>
</div>
</div>
</div>
</div>
</section>
<!-- Paper intro -->
<section class="section hero is-light" style="background-color: #FFFFFF !important;">
<div class="container is-max-desktop">
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3" style="text-align: center;">
<!-- <img src="static/images/introduction.png" style="height: 43px; display: inline; vertical-align: middle;"/> -->
Introduction
</h2>
<div class="content has-text-justified">
<p>
Reinforcement learning has been central to recent advances in large language model reasoning, but most algorithms rely on on-policy training that demands fresh rollouts at every update, limiting efficiency and scalability. Asynchronous RL systems alleviate this by decoupling rollout generation from training, yet their effectiveness hinges on tolerating large staleness in rollout data, a setting where existing methods either degrade in performance or collapse. We revisit this challenge and uncover a <em>prosperity-before-collapse</em> phenomenon: stale data can be as informative as on-policy data if exploited properly. Building on this insight, we introduce <b>M2PO</b> (Second-Moment Trust Proxy Optimization), which constrains the second moment of importance weights to suppress only extreme outliers while preserving informative updates. Notably, M2PO sharply reduces the fraction of clipped tokens under high staleness (from 1.22% to 0.006% over training), precisely masking high-variance tokens while maintaining stable optimization. Extensive evaluation across six model scales (1.7B–32B) and eight reasoning benchmarks shows that M2PO delivers stable off-policy training even with data stale by <i><u>at least 256 model updates</i></u> and matches on-policy performance.
</p>
<!-- <h3 class="title is-5" > More Efficient <em>Rollout Scaling</em> with GRESO </h3> -->
<div class="figure">
<img src="static/m2po-images/fig1.png" alt="fig1" height="400" />
</div>
<p style="font-size: 0.9em;"><strong>Figure 1</strong> Comparison of on-policy GRPO and off-policy training under a staleness of 256 model updates on Qwen-2.5-32B.
<strong>Left:</strong> Standard GRPO suffers from degradation with stale rollouts, while removing the trust region (GRPO no TR) reveals a clear <em>prosperity-before-collapse</em> phenomenon.
In contrast, M2PO achieves stable training and matches on-policy performance even under high staleness.
<strong>Right:</strong> Token clipping ratio comparison shows that M2PO dramatically reduces clipping events compared to GRPO, while avoiding training collapse.</p>
</p>
<p>
<strong>Why is tolerance to staleness in RL important?</strong> Most RL algorithms for LLMs rely on an on-policy setup: the model must constantly generate fresh(or limited staleness) examples to learn from. This makes training stable and reliable, but also very costly, as each update requires waiting for new rollouts to finish.
To get around this inefficiency, researchers have been experimenting with asynchronous RL systems, like AREAL and SLIME. In these systems, rollouts and model training happen independently, often spread across large computing clusters.
Such approaches improve resource utilization and enable training to scale more efficiently across large and heterogeneous clusters, but their effectiveness fundamentally relies on the ability of RL algorithms to tolerate rollout staleness without sacrificing stability or performance.
</p>
<!-- Case study -->
<div class="content has-text-justified">
<!-- <br> -->
<!-- <h3 class="title is-5" > </h3> -->
<p>
<strong>Study Staleness with Stale-k Training.</strong>
To study the effect of stale data in reinforcement learning for large language models, we introduce Stale-$k$ RL training, where the model is updated using data generated $k$ updates earlier.
In our setup, each training step performs four model updates, following configurations commonly adopted in recent work. Consequently, even stale-0 ($s=0$) training involves data with a staleness of 0–3 updates, while stale-256 ($s=256$) corresponds to staleness of 256–259 updates. During the first $k$ updates, when no stale model is yet available, the policy is trained on rollouts from the original base model.
After this initialization phase, all training data comes from stale models, enabling a controlled study of how staleness influences the dynamics and effectiveness of RL training.
</p>
<div style="flex: 0 0 100%; max-width: 100%; text-align: center;">
<img src="static/m2po-images/stale-comparison.jpg" alt="rollout-scaling" width=300 />
</div>
<div style="display: flex; align-items: top; gap: 10px;">
<div style="flex: 1;">
<p>
As shown in the above figure, we train Qwen2.5-Math-7B with GRPO under varying staleness levels and report test accuracy across eight math reasoning benchmarks. The results reveal a clear trend: as staleness increases, model performance degrades and convergence slows. In particular, low-staleness training achieves higher accuracy, whereas high-staleness training converges more slowly to lower accuracy.
</p>
</div>
</div>
</div>
<!-- Case study end -->
</div>
</div>
</div>
</div>
</section>
<!-- Observation Section -->
<section class="section hero is-light" style="background-color: #f5f5f5 !important;">
<!-- <section class="section hero is-light"> -->
<div class="container is-max-desktop">
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3" style="text-align: center;"> Observation: Prosperity before Collapse</h2>
<!-- Rollout end -->
<!-- Wall-clock time -->
<div class="content has-text-justified">
<!-- <h3 class="title is-5" > Up to 2.0x wall-clock time speed-up in training </h3> -->
<div style="flex: 0 0 100%; max-width: 100%; text-align: center;">
<img src="static/m2po-images/pbc.jpg" alt="rollout-scaling" width=450 />
</div>
<div style="display: flex; align-items: top; gap: 10px;">
<div style="flex: 1;">
<p>
<strong>Prosperity before collapse: training without a trust region.</strong>
To disentangle whether the performance drop stems from stale data generated by highly shifted old policies or from biases introduced by the training algorithm, we remove the trust region entirely to remove bias from the training algorithm.
Surprisingly, we observe a distinct <em>prosperity-before-collapse</em> phenomenon. As shown in Figure 2 and above figure, although training without a trust region eventually collapses, it achieves substantially better performance prior to collapse.
In fact, under stale data ($s$=256), the no-clipping setting outperforms clipped training, sometimes even matching on-policy baselines.
</p>
</div>
</div>
</div>
<!-- Wall-clock time end -->
<!-- Case study -->
<div class="content has-text-justified">
<!-- <br> -->
<!-- <h3 class="title is-5" > Case Study: Selection Dynamics </h3> -->
<div style="flex: 0 0 100%; max-width: 100%; text-align: center;">
<img src="static/m2po-images/pivotal-token.jpg" alt="rollout-scaling" width=450 />
</div>
<div style="display: flex; align-items: top; gap: 10px;">
<div style="flex: 1;">
<p>
<strong> Pivotal token masking by $\epsilon$-clipping when training with stale data.</strong>
As also discussed in recent work, $\epsilon$-clipping may inadvertently mask important tokens, preventing them from contributing useful training signals.
We extend this observation to the training with stalenss setting and show that the problem becomes significantly more severe when training with stale data, since larger staleness induces a greater mismatch between the behavior and target policies. As illustrated in the above figure, the clipping ratio increases sharply under large staleness ($s=256$), while remaining much lower in the on-policy baseline.
</p>
<p>
To better understand this phenomenon, we conduct a quantitative analysis on 90 million training tokens collected during Qwen2.5-Math-7B training with staleness $256$. Specifically, we gather all training tokens generated between 800 and 1200 model updates, ensuring the model is already in a stable training phase but before convergence. Above figure (right) shows a clear trend: as $|r-1|$ increases, the average token entropy also rises. This indicates that $\epsilon$-clipping disproportionately prunes high-entropy tokens, which are typically the most informative for model improvement. Consequently, clipping under stale data leads to degraded performance.
This observation reveals a dilemma: <u><em>while high-entropy tokens are crucial for learning progress, they also introduce instability in the off-policy setting</em></u>, which motivates our key research question:
</p>
<p>
<i><strong> Can a more accurate and adaptive trust region strategy preserve the benefits of stale data while ensuring stable training? </strong></i>
</p>
</div>
</div>
</div>
<!-- Case study end -->
</div>
</div>
</div>
</section>
<section class="section hero is-light" style="background-color: #FFFFFF !important;">
<!-- <section class="section hero is-light" style="background-color: #FFFFFC !important;"> -->
<div class="container is-max-desktop">
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3" style="text-align: center;">
<!-- <img src="static/images/icon.jpg" style="height: 28px; display: inline; vertical-align: -2px;"/> -->
M2PO: Second-Moment Trust Policy Optimization
</h2>
<div class="content has-text-justified">
<p>
The main source of instability in off-policy RL lies in the distributional mismatch between the behavior policy that generates training data and the current policy being optimized. As the divergence between these two distributions grows, importance sampling corrections produce high-variance gradient estimates, leading to noisy and unreliable updates.
Our motivation is therefore to constrain the distributional gap between the behavior policy and the current policy at the batch level, directly coupling the constraint with model updates while preventing over-constraining of token-level variations.
</p>
<p>
To achieve this goal, we propose M2 metric to measure the distributional gap between the behavior policy and the current policy:
</p>
</div>
<div class="figure">
<img src="static/m2po-images/M2.jpg" alt="pipeline" height="400" />
</div>
<div class="content has-text-justified">
<p>
There are two key advantages of using the above M2 metric for trust region constraint:
First, each per-token estimate is always non-negative, so the constraint can be reliably applied even when $r>1$.
Second, while the batch KL only measures the mean shift between policies,
M2 also reflects the variance of importance weights.
This makes it more sensitive to outliers and noisy tokens with extreme ratios $r_i$.
</p>
<p>
To maintain training stability, M2PO applies a masking strategy that selectively excludes tokens until the batch-level M2 of the remaining tokens falls below a predefined threshold $\tau_{M_2}$<sup>1</sup>.
Finally, with the result mask $\boldsymbol{M}$, we update the policy by maximizing the following objective:
</p>
</div>
<br>
<div class="figure">
<img src="static/m2po-images/m2po.jpg" alt="pipeline" height="450" />
</div>
<div class="content has-text-justified">
<p style="font-size: 0.9em;"><sup>1</sup>Importantly, we observe that $\tau_{M_2}$ is not a sensitive hyperparameter.
Across all our experiments, we consistently set $\tau_{M_2} = 0.04$, and this single setting proved effective for stabilizing training in all training scenarios.</p>
</div>
</div>
</div>
</div>
</section>
<!-- Results Section -->
<section class="section hero is-light" style="background-color: #F5F5F5 !important;">
<!-- <section class="section hero is-light"> -->
<div class="container is-max-desktop">
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3" style="text-align: center;"> Stable Off-Policy Training without Accuracy Drop </h2>
<!-- Rollout -->
<div class="content has-text-justified">
<!-- <br> -->
<!-- Up to 2.4$\times$ wall-clock time speed-up in rollout and 2.0$\times$ speed-up in training. -->
<!-- <h3 class="title is-5" > No performance drop with up to 3.35x fewer rollouts </h3> -->
<div class="figure">
<img src="static/m2po-images/table1.jpg" alt="table-rollout-comparison" height="350" />
</div>
<p>
<strong>Prosperity without collapse: Stable off-policy training without performance degradation using M2PO.</strong>
To verify the effectiveness of M2PO, Table 1 presents a comprehensive comparison of math reasoning performance across eight benchmarks using models from four different families and scales, ranging from 1.7B to 32B parameters. We evaluate multiple reinforcement learning methods under both on-policy and off-policy settings, including GRPO, GSPO, and our proposed M2PO.
The results show that while both GRPO and GSPO often suffer significant performance drops under large staleness, M2PO consistently achieves comparable accuracy to the on-policy baseline in all training settings.
Surprisingly, we notice that, in some model settings, M2PO with $s=256$ even achieves a better performance than M2PO with $s=0$.
For instance, on the Qwen3-Base-1.7B model, we observe that M2PO with $s=256$ (36.6%) outperforms GRPO with $s=0$ (33.0%).
</p>
</div>
<!-- Rollout end -->
<!-- Wall-clock time -->
<!-- <div class="content has-text-justified">
<h3 class="title is-5" > Up to 2.0x wall-clock time speed-up in training </h3>
<div style="display: flex; align-items: top; gap: 10px;">
<div style="flex: 1;">
<p>
To better understand the efficiency of our proposed methods, we report the detailed end-to-end training time breakdown for different stages: rollout, actor model update, and other overheads (e.g., reference model and advantage calculation).
Qwen2.5-Math-1.5B is trained on 4×H100 GPUs, while the other two models are trained on 8×H100 GPUs.
The right table compares the training time breakdown between GRESO and Dynamic Sampling for models trained on the DAPO + MATH dataset.
For all three models, GRESO significantly reduces rollout time—achieving up to <b>2.4× speedup</b> in rollout and <b>2.0× speedup</b> in total training time compared to DS.
For instance, on Qwen2.5-Math-7B, GRESO reduces rollout time from 155.9 hours to 65.5 hours, cutting overall training time from 178.0 to 88.3 hours.
</p>
</div>
<div style="flex: 0 0 45%; max-width: 45%;">
<img src="static/m2po-images/pbc.jpg" alt="rollout-scaling" width=450 />
</div>
</div>
</div> -->
<!-- Wall-clock time end -->
<!-- Case study -->
<!-- <div class="content has-text-justified">
<div style="display: flex; align-items: top; gap: 10px;">
<div style="flex: 1;">
<p>
Selection Dynamics of different prompts in GRESO. Each row is a prompt, and each column is an epoch.
We present a case study illustrating how GRESO selects or skips prompts over training epochs. We observe that very easy prompts tend to remain easy throughout training; although frequently skipped, GRESO still occasionally selects them to ensure a minimal level of exploration. For prompts of moderate difficulty, as the model becomes stronger over time, these prompts gradually become easier and are increasingly skipped. In contrast, some hard prompts become solvable~(i.e., effective prompts) in later epochs or even easy prompts. However, certain hard prompts remain unsolved throughout training.
</p>
</div>
<div style="flex: 0 0 30%; max-width: 30%;">
<img src="static/images/case-study.jpg" alt="rollout-scaling" width=350 />
</div>
</div>
</div> -->
<!-- Case study end -->
</div>
</div>
</div>
</section>
<!--BibTex citation -->
<section class="section" id="BibTeX" style="background-color: #FFFFFF !important;">
<div class="container is-max-desktop content">
<h2 class="title">BibTeX</h2>
<pre style="background-color: #FFFFFC !important;"><code >@article{zheng2025m2po,
title={Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?},
author = {Zheng, Haizhong and Zhao, Jiawei and Chen, Beidi},
journal={arXiv preprint arXiv:2510.01161},
year={2025}
}</code></pre>
</div>
</section>
<!--End BibTex citation -->
<!-- <section class="section hero is-light" style="background-color: #f5f5f5 !important;"></section> -->
<footer class="footer" style="background-color: #f5f5f5 !important;">
<div class="container">
<div class="columns is-centered">
<div class="column is-8">
<div class="content">
<p>
This page was built using the <a href="https://github.com/eliahuhorwitz/Academic-project-page-template" target="_blank">Academic Project Page Template</a> which was adopted from the <a href="https://nerfies.github.io" target="_blank">Nerfies</a> project page.
You are free to borrow the of this website, we just ask that you link back to this page in the footer. <br> This website is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/" target="_blank">Creative
Commons Attribution-ShareAlike 4.0 International License</a>. The icons are created by GPT4.
</p>
</div>
</div>
</div>
</div>
</footer>
</body>
</html>