Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

monitor material based fitting #152

Closed
robertnurnberg opened this issue Dec 18, 2023 · 11 comments
Closed

monitor material based fitting #152

robertnurnberg opened this issue Dec 18, 2023 · 11 comments

Comments

@robertnurnberg
Copy link
Contributor

This is not an issue per se, but just a convenient place to regularly check how our material based fitting works.

Below I report on the fits from ./updateWDL.sh --firstrev b59786e750a59d3d7cff2630cf284553f607ed29 (based on move) and from python scoreWDL.py updateWDL.json --plot save --pgnName update_material.png --momType "material" --momTarget 62 --moveMin 8 --moveMax 120 --materialMin 10 --materialMax 78 --modelFitting optimizeProbability applied to the same json data (based on material).

update_move

update_material

json data: updateWDL.json.gz

@robertnurnberg
Copy link
Contributor Author

For completeness, I also attach a plot of output of the command python scoreWDL.py updateWDL.json --plot save --pgnName update_moveshort.png --moveMin 10 --moveMax 78 --modelFitting optimizeProbability, to compare the relative "smoothness" of the a and b curves. (The data range here is [10,78], the same as in the material plot.)

update_moveshort

@Disservin
Copy link
Member

maybe we want to move this into the wiki?

@robertnurnberg
Copy link
Contributor Author

Let's wait for vondele to return. :)

My plan was to regularly post those plots here, and see if by some miracle the graphs become more smooth in future...

@vondele
Copy link
Member

vondele commented Dec 26, 2023

This is actually remarkably smooth compared to what we had in SF11 times (classical eval).

I don't mind this as an issue, but one could also put this in the readme as an example for this option.

@robertnurnberg
Copy link
Contributor Author

If we are happy with the model in general, and the smoothness in particular, we may want to adopt the material based WDL model for the dynamic normalisation. It would avoid certain problems of the move based one: such as users being confused that the cp eval depends on the move counter when they use SF for analysis of a single position...

@robertnurnberg
Copy link
Contributor Author

robertnurnberg commented Jan 20, 2024

Below the graphical outputs corresponding to official-stockfish/Stockfish#5002 for ./updateWDL.sh --firstrev 6deb88728fb141e853243c2873ad0cda4dd19320 (based on move) and from python scoreWDL.py updateWDL.json --plot save --pgnName update_material.png --momType "material" --momTarget 62 --moveMin 8 --moveMax 120 --materialMin 10 --materialMax 78 --modelFitting optimizeProbability applied to the same json data (based on material).

updateWDL

update_material

@robertnurnberg
Copy link
Contributor Author

Below the graphical outputs corresponding to official-stockfish/Stockfish#5070 for ./updateWDL.sh --firstrev 6deb88728fb141e853243c2873ad0cda4dd19320 (based on move) and from python scoreWDL.py updateWDL.json --plot save --pgnName update_material.png --momType "material" --momTarget 62 --moveMin 8 --moveMax 120 --materialMin 10 --materialMax 78 --modelFitting optimizeProbability applied to the same json data (based on material).

updateWDL

update_material

@robertnurnberg
Copy link
Contributor Author

Here is the latest fitting based on material count:

update_material

I have prepared the branch https://github.com/robertnurnberg/Stockfish/tree/wdl-material-dynamic to potentially merge into SF. Do we think it is time to do this now? @vondele

@vondele
Copy link
Member

vondele commented Mar 17, 2024

I think this is ready for integration in SF.

@robertnurnberg
Copy link
Contributor Author

robertnurnberg commented Mar 17, 2024

I think this is ready for integration in SF.

Agreed. Since the anchor now only plays the role of fixing the no-longer-needed constant NormalizeToPawnValue (via the sum of the coefficients of the polynomial p_a), I will choose it to be the average material count of my LTC games at move 32. In this way there will be some continuity for NormalizeToPawnValue from the old move-based WDL model to the new one.

Once I got that, and then fitted the WDL model with this new parameter, I will open a PR to SF.

PS: So the average material at move 32 for my fishtest games is 43.19.

@vondele
Copy link
Member

vondele commented May 8, 2024

should the issue be closed, since we now use material based WDL?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants