[Feature] LLM Tooling

[vmoens]

Stack from ghstack (oldest at bottom):

• [BugFix] device in args of PPO losses #2969
• [Feature] Losses (GRPO) #2968
• -> [Feature] LLM Tooling #2966
• [Feature] History #2965

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2966

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 7 New Failures, 1 Pending, 5 Unrelated Failures

As of commit 41bcb98 with merge base 1b9d2c1 ():

NEW FAILURES - The following jobs have failed:

• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  Process completed with exit code 1.
• Generate documentation / build-docs (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 86b0f30dbf4a847f5c2fefd75e20b4c6b5ac421e318da323586b0445035f35e4 /exec failed with exit code 2
• Habitat Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 6674d335b0bc8f74286ec8a6b4421c913523532aba3c542a8bd4436e9bf58ce2 /exec failed with exit code 1
• Unit-tests on Linux / tests-cpu (3.10) / linux-job (gh)
  test/test_env.py::TestNonTensorEnv::test_parallel[False-False]
• Unit-tests on Linux / tests-cpu (3.12) / linux-job (gh)
  test/test_env.py::TestNonTensorEnv::test_parallel[False-False]
• Unit-tests on Linux / tests-olddeps (3.9, 11.6) / linux-job (gh)
  RuntimeError: Command docker exec -t 13f8172643222665bb707837efbfe085058d0aaf7a79a722f1b31bced6216902 /exec failed with exit code 2
• Unit-tests on Windows / unittests-cpu (3.10, windows.4xlarge, cpu) / windows-job (gh)
  Process completed with exit code 2.

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Libs Tests on Linux / unittests-gym (3.9, 12.8) / linux-job (gh) (trunk failure)
  test/test_libs.py::TestGym::test_gym_fake_td[True-False-3-HalfCheetah-v2]
• LLM Tests on Linux / unittests (3.9, 12.8) / linux-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• Unit-tests on Linux / tests-gpu (3.11, 12.8) / linux-job (gh) (trunk failure)
  test/test_cost.py::TestPPO::test_ppo_value_clipping[False-clip_value4-KLPENPPOLoss-device0]
• Unit-tests on Linux / tests-optdeps (3.11, 12.8) / linux-job (gh) (trunk failure)
  test/test_cost.py::TestValues::test_gae_multi_done
• Unit-tests on Linux / tests-stable-gpu (3.10, 11.8) / linux-job (gh) (trunk failure)
  test/test_cost.py::TestPPO::test_ppo_value_clipping[False-clip_value4-KLPENPPOLoss-device0]

This comment was automatically generated by Dr. CI and updates every 15 minutes.