Gumbel AlphaZero (ctree) does not learn: OOB child indexing, frozen noise, and algorithm deviations

[marco-costa-ml]

Hi! I tried to train Gumbel AlphaZero on 6x6 Gomoku (gomoku_gumbel_alphazero_bot_mode_config.py with mcts_ctree=True) and got no improvement at all over 160k+ env steps. For comparison, gomoku_muzero_bot_mode_config.py with matched settings (same board, same bot, same net size, 50 sims) reaches roughly 0 reward in about 200k steps. I dug into mcts_gumbel_alphazero.cpp and found several real bugs, plus a couple of places where the implementation deviates from the Gumbel MuZero paper. Posting them here so nobody else burns GPU-days on this path.

Bugs:

1. Out-of-bounds child indexing in _select_interior_child. to_argmax has one entry per child, but it gets indexed with raw action ids (up to action_space_size - 1). Mid-game there are fewer children than action ids, so tree descent picks children based on garbage memory. When the garbage happens to be NaN, it falls into the "action == -1, selecting a random valid action" fallback, which fired 816 times in one of my training logs. In other words, random moves were silently injected into the self-play data. This is easy to confirm: build with -D_GLIBCXX_ASSERTIONS and any mid-game search aborts immediately. The same function also computes probs_softmax and then never uses it.
2. The gumbel noise never changes. It's generated once in the constructor and reused for every move of the entire run. _generate_gumbel re-seeds its RNG on every call, so it always returns the same sequence anyway. It's also hardcoded to 36 entries, which goes out of bounds on boards bigger than 6x6.
3. The search tree leaks every move, about 243 KB per move. Fix submitted in fix(mc): fix search tree memory leaks in AlphaZero ctree variants #492.
4. The Dirichlet noise is a frozen constant. It uses an unseeded std::default_random_engine, which is deterministic (ctree_alphazero has the same problem). Per the Gumbel MuZero paper it arguably shouldn't exist here at all, since the gumbel noise is meant to replace it.
5. The executed action is sampled from the sequential-halving visit counts. Those are nearly flat across the considered candidates, so the move actually played barely reflects what the search found. The maintained gumbel_muzero acts from the improved policy instead.

What I tried: I fixed 1 through 4 (branch fix/ctree-memleak-and-gumbel-noise on my fork) and switched the executed action to sample from the improved policy. That gave the first learning signal I've ever gotten out of this code (reward went from about -0.95 to -0.7 over 50k env steps), but it's still nowhere near MuZero's curve. Happy to share the branches and measurements if there's interest in getting Gumbel AlphaZero working properly.

I may dig into this myself at some point, but for now I figured it was best to just report it.

Best,
Marco