calibration of current and new ais

[sanderland]

make an ai that mimics human play and is not just policy based

[SimonLewis7407]

Hi, I have played against the Calibrated Rank AI in KaTrain version 1.2 six times now, at various strengths (10, 11, 12, 14kyu) and they seem quite good. I used to play against the "P-Pick" settings in version 1.1 but those would always outplay me in the middlegame and then go insane in the endgame. The Calibrated Rank AI is definitely more consistent, less dominant in the middlegame, less crazy in the endgame. More like a human.

The only thing that still feels funny or "artificial" to me now (but I am a weak player) is that during the fuseki the AI tenukis a whole lot. I know players like me have the opposite problem, failing to tenuki enough, but the AI really jumps around.

I am sure there will always be minor details and fine-tuning, but you have achieved a whole lot here. The Calibrated Rank AI represents, as nearly as I can tell, both a strong step forward and a pretty good approximation of what a "real" go game ought to feel like.

[sanderland]

Yes, this is due to the pick-based algorithm inherently being a bit tenuki happy, more so if there are fewer moves to choose from. I've played the 0-4kyu ones a bit and they feel more balanced in that respect. Nevertheless, the AI sees the opening as having a large number of decent moves, as seen by e.g. opening tengen in a teaching game.

[SimonLewis7407]

Hi, could you point me to where in the program the code is that calls the "save an SGF file" operations? I would like to see if I can make the Save command write two SGF files -- one is the normal one, and one would be a specific one with just the details that I want, formatted in a particular way, that I can read into a different program.

[sanderland]

@SimonLewis7407

• game.py has write_sgf which sets up some options / filenames and calls
• sgf in sgf_parser.py, which has the actual parser and generator, and hooks into
• sgf_properties of game_node.py, which modifies the comment to include/exclude info based on settings.

[sanderland]

set it to the latest 20b for a day to see what happens, the stronger bots run away with their rank a bit. sgf in the sgf_20b directory @bale-go

- 15b model
katrain-10k[11k]
katrain-14k[14k]
katrain-18k[17k]
katrain-2d[3d]
katrain-2k[1d]
katrain-6k[6k]

- 20b model one day
katrain-10k[10k]
katrain-14k[11k]
katrain-18k[17k]
katrain-2d[5d]
katrain-2k[2d]
katrain-6k[6k]

[bale-go]

Interesting. It seems as you get closer to the rank of the pure policy network the correctness of the move rank estimation becomes really important.
If you settle for a larger NN model for katrain, I can recalibrate the calibrated rank bot.

[Dontbtme]

With the 15b, bots with ranks weaker than 18k weren't stable at all, if I remember correctly. Maybe now it would be possible to have consistent bots for beginners, using the 20b net?

[bale-go]

I do not think much would change at the weaker rank region.
The 15b model is strong enough to be a perfect player there.

[sanderland]

I think at higher strengths you have a lot more chance to hit the top move / near the top move several times in a row and play out a consistent strategy. The lower ones pick like 8 moves? You're kind of stuck choosing the 'least bad' move then, which the policy was definitely not trained for.