From 16b450eeb2cc1a1b56613420f712406a0ec097af Mon Sep 17 00:00:00 2001 From: PraxTube Date: Tue, 23 May 2023 09:15:05 +0200 Subject: [PATCH 1/7] docs: Add basic setup of milestone three --- docs/milestones/three/README.md | 37 +++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100644 docs/milestones/three/README.md diff --git a/docs/milestones/three/README.md b/docs/milestones/three/README.md new file mode 100644 index 0000000..aae706e --- /dev/null +++ b/docs/milestones/three/README.md @@ -0,0 +1,37 @@ +#
Milestone II - Basic AI
+ +##
Group C - The Plebs
+ +A chess AI written in Python. + +## Summary + + + +## Changes to the Chess Backend + + + +## AI + + + +## Benchmarks + + + +## Issues and Bugs + + + +## Lessons learned + +- benchmarks are extremely important, see numpy -> list +- unit tests are also vital, especially when restructuring (breaking changes) + +## Future Improvements + + + +## Final remarks + From 07ce3f61c6487fe23a8be7d69d885681e1bc3a2a Mon Sep 17 00:00:00 2001 From: PraxTube Date: Tue, 23 May 2023 21:18:26 +0200 Subject: [PATCH 2/7] docs: Add chess backend restructure details --- README.md | 17 ++++---------- docs/milestones/three/README.md | 41 ++++++++++++++++++++++++++++++--- setup.py | 2 +- 3 files changed, 43 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index cd23844..6f1cff5 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,9 @@ # Chess AI -Basic chess ai that uses minimax to search for the best move to make. -Given that it's written in python, the depth is very shallow -(currently max depth is 3). +Basic chess ai that uses alpha-beta to search for the best move +on a given board. +The max depth is 3, any deeper and the AI will take way to long +to compute the best move. ## Getting started @@ -81,13 +82,3 @@ if any of these fail, then you will not be able to merge your PR into master. You should check if your commit passed the tests. Note that your branch will **only** be tested if you have an open **pull request** into master. - -## Milestone II - Basic AI - -The deadline for this is the **18th Mai**. - -- [x] Alpha-beta search -- [x] ~~Sophisticated~~ Better evaluation function -- [x] Unit tests for legal moves -- [x] ~~Unit tests for chess engine~~ -- [x] Benchmarks that test different depths diff --git a/docs/milestones/three/README.md b/docs/milestones/three/README.md index aae706e..555ac58 100644 --- a/docs/milestones/three/README.md +++ b/docs/milestones/three/README.md @@ -10,7 +10,35 @@ A chess AI written in Python. ## Changes to the Chess Backend - +The backend was in dire need of a restructure. +It was created it in a rush to meet the deadline +of the first milestone and was therefore not written +with the most maintainability in mind. +The following things were changed: + +- Use `snake_casing` consistently +- Class `CastlingRights` and replace with a `list` +- Add guard clauses to reduce indentation +- Turn `debug_info` from `dict` to a `class` (in `log.py`) +- Use `list` of `int` for board instead of `str` + +For a more detailed description see the +commit history in #37. + +The way I restructured the backend was actually +very teaching. At first I wanted to simply write +the whole backend from scratch with the new changes. +However I quickly realized that this wouldn't really +work. So instead I opted to apply the changes in layers, +where after each change the whole backend should still +be working. + +This was extremely effective. It not only allowed to +assert that the changes I made didn't introduce new bugs, +but it also made commiting very clean, and also made sure +that I would be able to track down bugs if they did occur. +So in future projects I will use the same way to restructure +bigger files. ## AI @@ -18,7 +46,7 @@ A chess AI written in Python. ## Benchmarks - +- show plots of the different AI stages ## Issues and Bugs @@ -28,10 +56,17 @@ A chess AI written in Python. - benchmarks are extremely important, see numpy -> list - unit tests are also vital, especially when restructuring (breaking changes) +- when restructuring, keep atomic commits ALWAYS, don't restructure multiple things at once +- performing tests under same-ish conditions ## Future Improvements - +We found that we cannot improve the engine much further then +what we have right now. +While it would be possible to restructure the whole backend to use +something like `0x88` or bitboards, that would require an extreme amount +time and effort. Also, it's doubtful that the those changes would +have a huge effect in the end (given that we are using Python). ## Final remarks diff --git a/setup.py b/setup.py index f46b55c..4d5c28e 100644 --- a/setup.py +++ b/setup.py @@ -8,7 +8,7 @@ setup( name="chess-ai", - version="0.1.0", + version="0.3.0", description=("A chess ai implementing minimax to find the best chess moves."), long_description=long_description, long_description_content_type="text/markdown", From 36ca9f6132bc0e0c739dceeae3ab6598d364e6c5 Mon Sep 17 00:00:00 2001 From: PraxTube Date: Tue, 30 May 2023 08:29:51 +0200 Subject: [PATCH 3/7] restructure(docs)!: Rename and remove doc files Remove useless files and renames the folders of the milestones. --- docs/board_positions.md | 18 ------------------ docs/external-resources.md | 5 ----- docs/milestones/{one => 1-dummy-AI}/README.md | 0 .../{one => 1-dummy-AI}/benchmark-table.svg | 0 .../{one => 1-dummy-AI}/class_diagram.svg | 0 docs/milestones/{two => 2-basic-AI}/README.md | 0 .../benchmark-depths-table.svg | 0 .../{two => 2-basic-AI}/benchmark-tables.svg | 0 .../{three => 3-advanced-AI}/README.md | 7 +++++++ 9 files changed, 7 insertions(+), 23 deletions(-) delete mode 100644 docs/board_positions.md delete mode 100644 docs/external-resources.md rename docs/milestones/{one => 1-dummy-AI}/README.md (100%) rename docs/milestones/{one => 1-dummy-AI}/benchmark-table.svg (100%) rename docs/milestones/{one => 1-dummy-AI}/class_diagram.svg (100%) rename docs/milestones/{two => 2-basic-AI}/README.md (100%) rename docs/milestones/{two => 2-basic-AI}/benchmark-depths-table.svg (100%) rename docs/milestones/{two => 2-basic-AI}/benchmark-tables.svg (100%) rename docs/milestones/{three => 3-advanced-AI}/README.md (88%) diff --git a/docs/board_positions.md b/docs/board_positions.md deleted file mode 100644 index 93a07cf..0000000 --- a/docs/board_positions.md +++ /dev/null @@ -1,18 +0,0 @@ -# Stellung I (Gruppe C) - -FEN: r1b1kb1r/pp1p1ppp/1q3n2/2p5/4P3/2N5/PPP2PPP/R1BQKB1R w - - 0 1 - -40 Züge - -Nd5, Nb5, Na4, Ne2, Nb1, Rg1, Ba6, Bb5, Bc4, Bd3, Be2, Ke2, Kd2, Qxd7+, Qd6, Qh5, Qd5, -Qg4, Qd4, Qf3, Qd3, Qe2, Qd2, Bh6, Bg5, Bf4, Be3, Bd2, Rb1, e5, h3, g3, f3, b3, a3, h4, -g4, f4, b4, a4 - -# Stellung II (Gruppe C) - -FEN: rn1qkbnr/ppp1ppp1/4b3/3pN2p/8/2N4P/PPPPPPP1/R1BQKB1R w - - 0 1 - -29 Züge - -Nxf7, Nd7, Ng6, Nc6, Ng4, Nc4, Nf3, Nd3, Nxd5, Nb5, Ne4, Na4, Nb1, Rh2, Rg1, Rb1, h4, g3, -f3, e3, d3, b3, a3, g4, f4, e4, d4, b4, a4 diff --git a/docs/external-resources.md b/docs/external-resources.md deleted file mode 100644 index 64babf7..0000000 --- a/docs/external-resources.md +++ /dev/null @@ -1,5 +0,0 @@ -# External resources - -## Other chess AI repo's - -- Python alpha-beta, using chess package [andoma](https://github.com/healeycodes/andoma) diff --git a/docs/milestones/one/README.md b/docs/milestones/1-dummy-AI/README.md similarity index 100% rename from docs/milestones/one/README.md rename to docs/milestones/1-dummy-AI/README.md diff --git a/docs/milestones/one/benchmark-table.svg b/docs/milestones/1-dummy-AI/benchmark-table.svg similarity index 100% rename from docs/milestones/one/benchmark-table.svg rename to docs/milestones/1-dummy-AI/benchmark-table.svg diff --git a/docs/milestones/one/class_diagram.svg b/docs/milestones/1-dummy-AI/class_diagram.svg similarity index 100% rename from docs/milestones/one/class_diagram.svg rename to docs/milestones/1-dummy-AI/class_diagram.svg diff --git a/docs/milestones/two/README.md b/docs/milestones/2-basic-AI/README.md similarity index 100% rename from docs/milestones/two/README.md rename to docs/milestones/2-basic-AI/README.md diff --git a/docs/milestones/two/benchmark-depths-table.svg b/docs/milestones/2-basic-AI/benchmark-depths-table.svg similarity index 100% rename from docs/milestones/two/benchmark-depths-table.svg rename to docs/milestones/2-basic-AI/benchmark-depths-table.svg diff --git a/docs/milestones/two/benchmark-tables.svg b/docs/milestones/2-basic-AI/benchmark-tables.svg similarity index 100% rename from docs/milestones/two/benchmark-tables.svg rename to docs/milestones/2-basic-AI/benchmark-tables.svg diff --git a/docs/milestones/three/README.md b/docs/milestones/3-advanced-AI/README.md similarity index 88% rename from docs/milestones/three/README.md rename to docs/milestones/3-advanced-AI/README.md index 555ac58..a3392dd 100644 --- a/docs/milestones/three/README.md +++ b/docs/milestones/3-advanced-AI/README.md @@ -42,7 +42,14 @@ bigger files. ## AI +There were quite a lot of changes regarding the AI. +I implemented the following changes: +- Check if given node is terminal node (checkmate/stalemate) + as opposed to only checking `depth == 0` +- Add a proper king of the hill victory condition (and properly test it) + +TODO ## Benchmarks From b054e561609b7b03e906dc88150db399952d44a3 Mon Sep 17 00:00:00 2001 From: PraxTube Date: Wed, 31 May 2023 21:21:35 +0200 Subject: [PATCH 4/7] docs: Add `transposition-tables.md` --- docs/milestones/3-advanced-AI/README.md | 33 +++++- .../3-advanced-AI/transposition-tables.md | 109 ++++++++++++++++++ 2 files changed, 137 insertions(+), 5 deletions(-) create mode 100644 docs/milestones/3-advanced-AI/transposition-tables.md diff --git a/docs/milestones/3-advanced-AI/README.md b/docs/milestones/3-advanced-AI/README.md index a3392dd..64fc83c 100644 --- a/docs/milestones/3-advanced-AI/README.md +++ b/docs/milestones/3-advanced-AI/README.md @@ -1,4 +1,4 @@ -#
Milestone II - Basic AI
+#
Milestone III - Advanced AI
##
Group C - The Plebs
@@ -7,7 +7,6 @@ A chess AI written in Python. ## Summary - ## Changes to the Chess Backend The backend was in dire need of a restructure. @@ -48,6 +47,7 @@ I implemented the following changes: - Check if given node is terminal node (checkmate/stalemate) as opposed to only checking `depth == 0` - Add a proper king of the hill victory condition (and properly test it) +- Try to add [transposition tables but fail miserably](./transposition-tables.md) TODO @@ -65,15 +65,38 @@ TODO - unit tests are also vital, especially when restructuring (breaking changes) - when restructuring, keep atomic commits ALWAYS, don't restructure multiple things at once - performing tests under same-ish conditions +- before commiting for something huge, try to gauge if it will be worth it + (evaluation and C scripts as well as numpy refactors) ## Future Improvements -We found that we cannot improve the engine much further then -what we have right now. +I found that I cannot improve the engine much further then +what I have right now. While it would be possible to restructure the whole backend to use something like `0x88` or bitboards, that would require an extreme amount time and effort. Also, it's doubtful that the those changes would -have a huge effect in the end (given that we are using Python). +have a huge effect in the end, given that the main bottleneck +will always be python. + +Much more interesting would be the refactoring of +the evaluation function. If the evaluation function +would be written entirely in C, then there would be +no overhead when the evaluation function communicates +with the transposition table. This alone would increase +performance by around 10%. Additionally, the performance +boost from switching to pure C would probably give at +least another 10% - 20% boost. + +Apart from that, there is not much that can be really +improved for the performance. That is simply because +the legal move generation is just so insanely slow. +For mid game boards it can take a whole millisecond +and the only way to increase the performance there +would be to completely restructure the backend in C. +That is obviously not going to happen. +So instead I would focus my attention on making the +AI stronger by implementing better and more AI +techniques. ## Final remarks diff --git a/docs/milestones/3-advanced-AI/transposition-tables.md b/docs/milestones/3-advanced-AI/transposition-tables.md new file mode 100644 index 0000000..bf2ec02 --- /dev/null +++ b/docs/milestones/3-advanced-AI/transposition-tables.md @@ -0,0 +1,109 @@ +When I tried to implement the transposition tables +I stumbled across many issues. Mainly having way +too many collisions in the hash table. + +## The Problem + +There are far too many collisions in the hash table to +reasonably implement this into the AI. +Roughly, for 10k boards that were hashed there were +around 20 collisions. If you didn't wipe the table +after each ply, then the number of collisions would +skyrocket after that, to the point were searching +50k boards resulted in 25k collisions, so 50%. +Any type of collision handling would be more then +futile here. + +I was also able to observe that the AI was playing +extremely bad (basically random moves) when the +number of collisions was high, which means that +it's unlikely that there was an issue with the +debugging of the collisions. + +## Theory and Birthday Paradox + +It goes without saying that something must be +going wrong here, though if we consider the +[birthday paradox](https://en.wikipedia.org/wiki/Birthday_problem), +then it doesn't seem so bizarre actually. +Using the rule of thumb of `p ~= n^2 / 2 * m` with +the probability of collisions `p`, +the number of boards hashed `n` and the amount +of entries in the board `m`, we get the following: + +``` +m = 2^24 +p = 1 / 2 +n = sqrt(2^24) = 2^12 +``` + +so if we hash `2^12 = 4096` boards we should expect +that there will be at least one collision with +a chance of 50%. +So collisions with the number of boards we hashed are +actually extremely likely, and given that they grow +exponentially, it's not too surprising to find +these huge numbers of collisions. Though there is +still likely a bug in my implementation. + +## Trying to Solve the Problem + +I tried these things to solve the problem, +however none of them fixed it: + +- Generate zobrist keys through a random seed and + test different seeds +- Write unit tests to see how many collisions appear + on depth 3 alone (roughly 3k) +- Differentiate between index collisions and hash + collisions, were index collisions mean different + hashes map to the same index because of `%`, + and hash collisions simple mean different boards + hash to the same value. Hash collisions are by + far the more common collision type +- Make sure that the castling rights also get hashed +- Use 64 bit zobrist hashkeys instead of 32 bit + +I was very confused when I first saw that the hash +collisions are the much more common type of collision. +It seems like the hash function isn't all that great, +but I don't really know how to make that any better, +or why it would be bad in the first place. + +## Lessons learned + +To begin with, given that I am using python, +transposition tables wouldn't be that great anyways. +We can't get much deeper then depth 4, so the effect +they would have would be pretty small anyways. +Despite that, I still wanted to implement them +because I didn't use hash tables before. +My big take away is to avoid using hash tables +if at all possible. Designing a good hash +function is not at all trivial and managing +collisions, either through channing or open addressing +is also something that has to be carefully choosen. + +I did try to implement them in C so that was a good +learning experience. It's also quite fascinating that +you can call into C from Python in less then +3 microseconds (using the +[cffi package](https://pypi.org/project/cffi/)). +The whole evaluation function could actually be rewritten +in C. That could boost the performance by perhaps 25%. + +I also discovered a bug that is pretty sever. +The current move ordering is actually evaluating +every single board. This is obviously extremely dumb, +especially when you try to compare minimax and alpha-beta +and don't really see much of a difference (this is +the reason why the cutoff effect didn't seem that +effective in [milestone II](../2-basic-AI/README.md) +the amount of cutoffs in depth 3 is actually a perfect +cutoff and it also explains why the total time for +the best move generation is so high). + +Overall I learned a lot, however I did cost me a lot +of time as well, with little to nothing to show for now. +Perhaps I will come back to the transposistion tables +in a later stage, but I doubt it. From 1f083222b7d22f1737e38ceb187a7d6728246afa Mon Sep 17 00:00:00 2001 From: PraxTube Date: Sun, 4 Jun 2023 15:14:18 +0200 Subject: [PATCH 5/7] docs: Add reference to transposition table --- docs/milestones/3-advanced-AI/README.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/docs/milestones/3-advanced-AI/README.md b/docs/milestones/3-advanced-AI/README.md index 64fc83c..9b56f1c 100644 --- a/docs/milestones/3-advanced-AI/README.md +++ b/docs/milestones/3-advanced-AI/README.md @@ -16,8 +16,8 @@ with the most maintainability in mind. The following things were changed: - Use `snake_casing` consistently -- Class `CastlingRights` and replace with a `list` -- Add guard clauses to reduce indentation +- Replace `class CastlingRights` with a `list` +- Add guard clauses to reduce indentation and increase readability - Turn `debug_info` from `dict` to a `class` (in `log.py`) - Use `list` of `int` for board instead of `str` @@ -27,14 +27,14 @@ commit history in #37. The way I restructured the backend was actually very teaching. At first I wanted to simply write the whole backend from scratch with the new changes. -However I quickly realized that this wouldn't really +However, I quickly realized that this wouldn't really work. So instead I opted to apply the changes in layers, where after each change the whole backend should still be working. -This was extremely effective. It not only allowed to +This was extremely effective. It not only allowed me to assert that the changes I made didn't introduce new bugs, -but it also made commiting very clean, and also made sure +but also made commiting very clean, and also made sure that I would be able to track down bugs if they did occur. So in future projects I will use the same way to restructure bigger files. @@ -42,12 +42,15 @@ bigger files. ## AI There were quite a lot of changes regarding the AI. -I implemented the following changes: +I mainly tried to implement transposition tables, +however that went horribly wrong. +I wrote a seperate article in which I documented what went wrong, +[see here](./transposition-tables.md). +Apart from that set back, I implemented the following changes: - Check if given node is terminal node (checkmate/stalemate) as opposed to only checking `depth == 0` - Add a proper king of the hill victory condition (and properly test it) -- Try to add [transposition tables but fail miserably](./transposition-tables.md) TODO @@ -67,6 +70,7 @@ TODO - performing tests under same-ish conditions - before commiting for something huge, try to gauge if it will be worth it (evaluation and C scripts as well as numpy refactors) +- hashing is hard to debug ## Future Improvements From ef747261df62f9ed37abe2d866490ab73e29a79c Mon Sep 17 00:00:00 2001 From: PraxTube Date: Sun, 11 Jun 2023 12:21:42 +0200 Subject: [PATCH 6/7] docs: Add `lessons learned` section details --- docs/milestones/3-advanced-AI/README.md | 39 ++++++++++++++++++++----- 1 file changed, 32 insertions(+), 7 deletions(-) diff --git a/docs/milestones/3-advanced-AI/README.md b/docs/milestones/3-advanced-AI/README.md index 9b56f1c..e057abc 100644 --- a/docs/milestones/3-advanced-AI/README.md +++ b/docs/milestones/3-advanced-AI/README.md @@ -51,6 +51,9 @@ Apart from that set back, I implemented the following changes: - Check if given node is terminal node (checkmate/stalemate) as opposed to only checking `depth == 0` - Add a proper king of the hill victory condition (and properly test it) +- Improve time management to be more dynamic (allocate time depending + on the current stage, early, mid late) +- Refactor move ordering to be much more performant TODO @@ -64,13 +67,35 @@ TODO ## Lessons learned -- benchmarks are extremely important, see numpy -> list -- unit tests are also vital, especially when restructuring (breaking changes) -- when restructuring, keep atomic commits ALWAYS, don't restructure multiple things at once -- performing tests under same-ish conditions -- before commiting for something huge, try to gauge if it will be worth it - (evaluation and C scripts as well as numpy refactors) -- hashing is hard to debug +A collection of lessons that I learned during this milestone + +1. Benchmarks are extremely useful to not only see improvements + over different versions of your code, but also to compare + incremental changes to the code. I realized this when I tried + to refactor the evaluation function to use numpy. When I had + many small `np.ndarray` it was actually slower then the python + `list` implementation. I noticed that this was because we always + have overhead when calling numpy, so I should reduce the number + of times I make a numpy call, i.e. use as big as arrays as possible. + That I did, and the evaluation function went from `200μs` to `50μs`. +1. On the note of benchmarks, I also observed that background tasks + can significantly influence the result of benchmarks. One should + try to run them in same environment (or use a server for that if possible). +1. Sticking to atomic commits when restructuring something big is vital. + It allows to keep a running system which can be checked to make sure + no bugs went into the restructure. +1. When I refactored the backend, the unit tests really came in handy. + After every commit I could just run the tests and see if there + were any issues. +1. It's important to now if something will be worthwile before commiting + to it if it will take a long time to complete. I learned this with + the attempt at implementing transposition tables and when trying to refactor + the evaluation with numpy arrays. If I would have known + that the potential performance increase was about `~10%`, I would have + probably not even tried it in the first place, given how much effort + went into it. The numpy refactor was luckily successful. +1. Debug hash tables is actually not as straightfoward as it initally + seemed like. ## Future Improvements From 1c4051a941eb021af0f59b434edbd549021d49ce Mon Sep 17 00:00:00 2001 From: PraxTube Date: Tue, 13 Jun 2023 12:01:50 +0200 Subject: [PATCH 7/7] Add plots and finish up the 3 Milestone --- docs/milestones/3-advanced-AI/README.md | 131 +++-- docs/milestones/3-advanced-AI/plot-depths.svg | 375 +++++++++++++ docs/milestones/3-advanced-AI/plot.svg | 525 ++++++++++++++++++ .../3-advanced-AI/transposition-tables.md | 2 +- 4 files changed, 992 insertions(+), 41 deletions(-) create mode 100644 docs/milestones/3-advanced-AI/plot-depths.svg create mode 100644 docs/milestones/3-advanced-AI/plot.svg diff --git a/docs/milestones/3-advanced-AI/README.md b/docs/milestones/3-advanced-AI/README.md index e057abc..3ae829f 100644 --- a/docs/milestones/3-advanced-AI/README.md +++ b/docs/milestones/3-advanced-AI/README.md @@ -6,13 +6,26 @@ A chess AI written in Python. ## Summary +A quick overview of what happend in this milestone: + +- Restructured the chess backend + - Clean ups like name conventions + - Using `list` of `int` + - Minor bug fixes +- Improve AI + - Speed up evaluation + - Improve move ordering + - Minor bug fixes +- Restructure the debug log +- Add king of the hill conditions +- Better and more tests and benchmarks ## Changes to the Chess Backend The backend was in dire need of a restructure. -It was created it in a rush to meet the deadline +It was created in a rush to meet the deadline of the first milestone and was therefore not written -with the most maintainability in mind. +with maintainability in mind. The following things were changed: - Use `snake_casing` consistently @@ -22,7 +35,7 @@ The following things were changed: - Use `list` of `int` for board instead of `str` For a more detailed description see the -commit history in #37. +[commit history in #37](https://github.com/PraxTube/chess-ai/pull/37/commits). The way I restructured the backend was actually very teaching. At first I wanted to simply write @@ -46,56 +59,91 @@ I mainly tried to implement transposition tables, however that went horribly wrong. I wrote a seperate article in which I documented what went wrong, [see here](./transposition-tables.md). -Apart from that set back, I implemented the following changes: +Apart from that _minor_ set back, I implemented the following changes: - Check if given node is terminal node (checkmate/stalemate) as opposed to only checking `depth == 0` - Add a proper king of the hill victory condition (and properly test it) - Improve time management to be more dynamic (allocate time depending - on the current stage, early, mid late) + on the current stage: early, mid, late) - Refactor move ordering to be much more performant -TODO +The AI is playing both faster and stronger now. ## Benchmarks -- show plots of the different AI stages - -## Issues and Bugs - - +The results of the backend restructure and the AI refactor were +pretty successful, as can be seen in the plots below. +In figure 1 we can see that the advanced AI is overall much faster then +the previous version. Do note that the dummy version implements minimax +with an evaluation function that simply calculates the material difference. +So the evaluation of the dummy is hard to outperform. + +The tests were run on a PC with the following specs + +- CPU: Intel i5-4590, Threads: 4, Cores: 4, 3.7GHz + +- RAM: 24GB DDR3 + +- OS: Zorin 16.2 (Ubuntu based) + +

+ Plot SVG Image +
+ Figure 1: Benchmarks of the different categories across the AI versions. +

+ +In figure 2 we can see that the advanced AI is much faster then the basic AI. +We can also observe that we search a lot more nodes then in the dummy AI case. +This shouldn't be, because the dummy AI implements minimax (and not alpha-beta). +I fixed a bug during this milestone which is most likely the cause for this. +In this bug the amount of boards to search was decreased, so the dummy AI +is actually much slower. +In other words, the actual speed increase from the minimax to the current AI +is much more significant. This also explains the rather weird behaviour we +saw in the last milestone (where the cutoff effect didn't seem that strong). + +

+ Plot Depths SVG Image +
+ Figure 2: Benchmarks of different AI versions in respect to search depth. +
+ Note that the number of nodes searched is in thousands (kilo). +

## Lessons learned A collection of lessons that I learned during this milestone -1. Benchmarks are extremely useful to not only see improvements +1. Benchmarks are extremely useful, not only to see improvements over different versions of your code, but also to compare incremental changes to the code. I realized this when I tried - to refactor the evaluation function to use numpy. When I had - many small `np.ndarray` it was actually slower then the python - `list` implementation. I noticed that this was because we always - have overhead when calling numpy, so I should reduce the number - of times I make a numpy call, i.e. use as big as arrays as possible. + to refactor the evaluation function to use `numpy`. When I had + many small `np.ndarray` it was actually slower then the pure python + `list` implementation. This is because we always + have overhead when calling numpy, so reducing the amount of times + we call `numpy` draws out the full potential of `numpy`, + i.e. use as big as arrays as possible. That I did, and the evaluation function went from `200μs` to `50μs`. 1. On the note of benchmarks, I also observed that background tasks can significantly influence the result of benchmarks. One should - try to run them in same environment (or use a server for that if possible). + try to run them in the same-ish environment as possible + (or use a server for that if possible). 1. Sticking to atomic commits when restructuring something big is vital. It allows to keep a running system which can be checked to make sure no bugs went into the restructure. -1. When I refactored the backend, the unit tests really came in handy. +1. Unit tests are really handy when restructuring. After every commit I could just run the tests and see if there - were any issues. -1. It's important to now if something will be worthwile before commiting + were any issues (of course one shouldn't overly rely on them). +1. It's important to know if something will be worthwile before commiting to it if it will take a long time to complete. I learned this with - the attempt at implementing transposition tables and when trying to refactor - the evaluation with numpy arrays. If I would have known + the attempt at implementing transposition tables. If I would have known that the potential performance increase was about `~10%`, I would have probably not even tried it in the first place, given how much effort - went into it. The numpy refactor was luckily successful. -1. Debug hash tables is actually not as straightfoward as it initally - seemed like. + went into it. +1. Debugging hash tables is actually not as straightfoward as it initially + seemed. If you can't use a debugger properly then it's + a real mess. Something I want to learn in the future. ## Future Improvements @@ -103,29 +151,32 @@ I found that I cannot improve the engine much further then what I have right now. While it would be possible to restructure the whole backend to use something like `0x88` or bitboards, that would require an extreme amount -time and effort. Also, it's doubtful that the those changes would +time and effort. Also, it's doubtful that those changes would have a huge effect in the end, given that the main bottleneck will always be python. -Much more interesting would be the refactoring of -the evaluation function. If the evaluation function -would be written entirely in C, then there would be -no overhead when the evaluation function communicates -with the transposition table. This alone would increase -performance by around 10%. Additionally, the performance -boost from switching to pure C would probably give at -least another 10% - 20% boost. - -Apart from that, there is not much that can be really -improved for the performance. That is simply because +One possible refactor is to implement the evaluation +in pure C. Though I highly doubt that this would increase +the speed by more then `30%`, and given that I just learned +that commiting to things you aren't even sure will pay out +in the end is a bad thing, I think I will try to avoid it. + +Apart from that, there is not much that can be +improved performance-wise. That is simply because the legal move generation is just so insanely slow. For mid game boards it can take a whole millisecond -and the only way to increase the performance there +and the only way to increase the performance would be to completely restructure the backend in C. That is obviously not going to happen. -So instead I would focus my attention on making the +So instead I will focus my attention on making the AI stronger by implementing better and more AI techniques. ## Final remarks +This milestone was pretty successful overall. +While some things didn't work out, a lot of others did. +I was also able to learn from all of it. +Though I hope the workload will be more balanced +in the next milestone, +[see commit history](https://github.com/PraxTube/chess-ai/commits/master). diff --git a/docs/milestones/3-advanced-AI/plot-depths.svg b/docs/milestones/3-advanced-AI/plot-depths.svg new file mode 100644 index 0000000..9dadc71 --- /dev/null +++ b/docs/milestones/3-advanced-AI/plot-depths.svg @@ -0,0 +1,375 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/milestones/3-advanced-AI/plot.svg b/docs/milestones/3-advanced-AI/plot.svg new file mode 100644 index 0000000..e361610 --- /dev/null +++ b/docs/milestones/3-advanced-AI/plot.svg @@ -0,0 +1,525 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/milestones/3-advanced-AI/transposition-tables.md b/docs/milestones/3-advanced-AI/transposition-tables.md index bf2ec02..78b57e2 100644 --- a/docs/milestones/3-advanced-AI/transposition-tables.md +++ b/docs/milestones/3-advanced-AI/transposition-tables.md @@ -103,7 +103,7 @@ the amount of cutoffs in depth 3 is actually a perfect cutoff and it also explains why the total time for the best move generation is so high). -Overall I learned a lot, however I did cost me a lot +Overall I learned a lot, however it did cost me a lot of time as well, with little to nothing to show for now. Perhaps I will come back to the transposistion tables in a later stage, but I doubt it.