monitor material based fitting

[robertnurnberg]

This is not an issue per se, but just a convenient place to regularly check how our material based fitting works.

Below I report on the fits from ./updateWDL.sh --firstrev b59786e750a59d3d7cff2630cf284553f607ed29 (based on move) and from python scoreWDL.py updateWDL.json --plot save --pgnName update_material.png --momType "material" --momTarget 62 --moveMin 8 --moveMax 120 --materialMin 10 --materialMax 78 --modelFitting optimizeProbability applied to the same json data (based on material).

json data: updateWDL.json.gz

[robertnurnberg]

For completeness, I also attach a plot of output of the command python scoreWDL.py updateWDL.json --plot save --pgnName update_moveshort.png --moveMin 10 --moveMax 78 --modelFitting optimizeProbability, to compare the relative "smoothness" of the a and b curves. (The data range here is [10,78], the same as in the material plot.)

[Disservin]

maybe we want to move this into the wiki?

[robertnurnberg]

Let's wait for vondele to return. :)

My plan was to regularly post those plots here, and see if by some miracle the graphs become more smooth in future...

[vondele]

This is actually remarkably smooth compared to what we had in SF11 times (classical eval).

I don't mind this as an issue, but one could also put this in the readme as an example for this option.

[robertnurnberg]

If we are happy with the model in general, and the smoothness in particular, we may want to adopt the material based WDL model for the dynamic normalisation. It would avoid certain problems of the move based one: such as users being confused that the cp eval depends on the move counter when they use SF for analysis of a single position...

[robertnurnberg]

Below the graphical outputs corresponding to official-stockfish/Stockfish#5002 for ./updateWDL.sh --firstrev 6deb88728fb141e853243c2873ad0cda4dd19320 (based on move) and from python scoreWDL.py updateWDL.json --plot save --pgnName update_material.png --momType "material" --momTarget 62 --moveMin 8 --moveMax 120 --materialMin 10 --materialMax 78 --modelFitting optimizeProbability applied to the same json data (based on material).

[robertnurnberg]

Below the graphical outputs corresponding to official-stockfish/Stockfish#5070 for ./updateWDL.sh --firstrev 6deb88728fb141e853243c2873ad0cda4dd19320 (based on move) and from python scoreWDL.py updateWDL.json --plot save --pgnName update_material.png --momType "material" --momTarget 62 --moveMin 8 --moveMax 120 --materialMin 10 --materialMax 78 --modelFitting optimizeProbability applied to the same json data (based on material).

[robertnurnberg]

Here is the latest fitting based on material count:

I have prepared the branch https://github.com/robertnurnberg/Stockfish/tree/wdl-material-dynamic to potentially merge into SF. Do we think it is time to do this now? @vondele

[vondele]

I think this is ready for integration in SF.

[robertnurnberg]

I think this is ready for integration in SF.

Agreed. Since the anchor now only plays the role of fixing the no-longer-needed constant NormalizeToPawnValue (via the sum of the coefficients of the polynomial p_a), I will choose it to be the average material count of my LTC games at move 32. In this way there will be some continuity for NormalizeToPawnValue from the old move-based WDL model to the new one.

Once I got that, and then fitted the WDL model with this new parameter, I will open a PR to SF.

PS: So the average material at move 32 for my fishtest games is 43.19.

[vondele]

should the issue be closed, since we now use material based WDL?