dm_control soccer 2v2 self-play with opponent league (EnvPool)

[Denys88]

Summary

Multi-agent self-play training for EnvPool's dm_control locomotion soccer envs (BoxheadSoccer2v2-v1, from Denys88/envpool#1): one shared PPO policy controls the home team across 256 parallel matches while an 8-type opponent league plays the away team — scripted anchors (chaser, keeper, random, zero) plus frozen past checkpoints of the training policy (lagged self-play), auto-refreshed from the run dir.

python runner.py --train --file rl_games/configs/dm_control/boxhead_soccer_2v2_selfplay.yaml
python -m rl_games.envs.dmc_soccer_tools video --camera 3 --out match.mp4
python -m rl_games.envs.dmc_soccer_tools tournament --run-dir runs/boxhead_soccer_2v2_selfplay/nn

What's included

• rl_games/envs/dmc_soccer_selfplay.py — vecenv adapter: players-as-actors batching (egocentric, team-relative obs + within-team one-hot player id), shaped rewards with dense-annealing
• rl_games/envs/dmc_soccer_opponents.py — opponent league + a minimal FrozenPolicy (rebuilds the actor from a checkpoint for cheap CPU inference; handles _orig_mod. torch.compile prefixes)
• rl_games/envs/dmc_soccer_tools.py — match video rendering + checkpoint round-robin tournament (goal-diff/ep cross table)
• config + docs/DMC_SOCCER_SELFPLAY.md — the validated recipe, documented as the seven failure modes each choice prevents (each observed and reproduced during tuning):
  1. concede penalty → ball-avoidance ("cowardice")
  2. dense stream ≥ goal reward + terminate-on-goal → dribble-farming
  3. two-sided ball-progress → punished by opponent attacks (learned helplessness)
  4. per-player chase rewards → everyone crowds the ball (fix: team-level chase)
  5. single-opponent training → late collapse (fix: the league)
  6. shaped-reward proxy plateau (fix: dense anneal → broke a 0.34→0.46 goals/ep plateau)
  7. entropy bonus + fixed_sigma → unbounded σ growth melts the policy into noise (fix: entropy_coef: 0)
• Registry entries follow the existing pattern (_create_envpool-style lazy import in vecenv.py, env_configurations.py entry)

Validation

• 1.64B frames / 50k epochs on laptop CPU (~64k frames/s end-to-end incl. PPO), zero collapses
• Final checkpoint beats all scripted anchors (goal-diff/ep: chaser +0.46, keeper +0.56, random +0.53) and its own mid-training self both home and away
• Smoke-tested in this branch: runner.py --train (3 epochs), tournament, and video render all pass

Dependencies

• envpool with the soccer envs + dmc physics-divergence fix: feat(dmc): multi-agent locomotion soccer envs (BoxHead, Ant, Humanoid) envpool#1 (until released)
• opencv-python only for the video tool (lazy import)

🤖 Generated with Claude Code