Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue/#724 generalize rating initialization #740

Merged

Conversation

Askaholic
Copy link
Collaborator

@Askaholic Askaholic commented Mar 6, 2021

Implement reading the initializer from the database.

One question is whether the global and ladder_1v1 ratings need to continue to be eagerly created. Right now even completely new players will have a global and ladder rating in their ratings dictionary, however is this really necessary?

Closes #724

@Askaholic
Copy link
Collaborator Author

I'm going to change how I'm handling dependency cycles. Instead of disallowing cycles, I'm going to rewrite the code so it only ever looks back 1 step, instead of trying to look back recursively. That way if e.g. global is initialized based on 2v2, and 2v2 based on global, then essentially whichever one gets played first will be used as an initializer for the other.

Also need to do this in the rating service as well, since unfortunately this code is duplicated.

@Askaholic Askaholic force-pushed the issue/#724-rating-initialization branch from 5b5bd65 to 7157b86 Compare April 6, 2021 06:36
@Askaholic
Copy link
Collaborator Author

I decided that the desired behavior is actually to initialize ratings recursively, and use the default rating when a cycle is detected. This would both allow dependency chains like:

  • 2v2 -> global -> ladder1v1. If neither 2v2 nor global exist, but ladder does exist then both global and 2v2 will be initialized based on ladder. The only tricky thing here is if the deviation for ladder is really small we don't actually want to increase it twice.
    and cycles like:
  • 2v2 -> global -> 2v2. If one of them exists, the other will be based on the existing one, and if neither exist then a default will be used for whichever one is requested first.

Given one of the recent client bugs we saw with trying to display map pools when no rating existed, I think it is actually still a good idea to eagerly initialize ratings. However, this can cause a weird scenario where for example the following happens:

  1. Player who has never played 2v2 logs in, and their 2v2 rating is initialized to their global mean
  2. Player plays a whole bunch of global games changing their mean significantly (maybe they're kinda new and dev is high so mean changes fast)
  3. Player then queues 2v2, but the current 2v2 rating used by the matchmaker is still based on their old global rating when they logged in
  4. Game finishes, their 2v2 rating is re-initialized based on the new global rating and persisted to the database, from now on rating should be as expected.

In this case the first 2v2 game will be balanced based on outdated info. So I think it would be ideal for the PlayerRatings dict to keep track of which ratings were set externally (meaning they are more or less tied to the database in some way) and which ratings were automatically set as part of the initialization process. These automatically set ratings should then be re-computed whenever they're queried to ensure they are up to date. They do need to be set on the object so that they can be iterated over, in which case they don't need to be recomputed.

@Askaholic Askaholic force-pushed the issue/#724-rating-initialization branch from 7157b86 to 6066548 Compare July 4, 2021 04:48
@codecov
Copy link

codecov bot commented Jul 4, 2021

Codecov Report

Merging #740 (7ae75a8) into develop (716717f) will decrease coverage by 0.05%.
The diff coverage is 94.11%.

Impacted Files Coverage Δ
server/__init__.py 91.13% <ø> (ø)
server/db/models.py 100.00% <ø> (ø)
server/matchmaker/matchmaker_queue.py 91.85% <66.66%> (-0.58%) ⬇️
server/rating.py 93.58% <92.06%> (-6.42%) ⬇️
server/lobbyconnection.py 94.24% <100.00%> (+0.01%) ⬆️
server/players.py 100.00% <100.00%> (ø)
server/rating_service/rating_service.py 96.44% <100.00%> (+0.72%) ⬆️

@Askaholic Askaholic force-pushed the issue/#724-rating-initialization branch from 6066548 to 916beab Compare July 4, 2021 05:03
@Askaholic
Copy link
Collaborator Author

I think I've implemented everything that I mentioned above.

  • Recursive initialization with cycle detection
  • Recomputing auto initialized ratings whenever they are queried

I also eliminated the duplication by making the rating service use PlayerRatings objects for computing initialization. It now loads all of a players ratings from the database and throws them into a PlayerRatings object, then it queries the object for the particular rating that its interested in.

@Askaholic Askaholic marked this pull request as ready for review July 4, 2021 05:07
Copy link
Collaborator

@BlackYps BlackYps left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought about how we want the dependency chains to be in practice.
I think ideally we would want global <--> 4v4 <--> 2v2 <--> 1v1. However this requires two references and a way to determine which preference should be preferred, if both exist.
With one reference we can build the chain 1v1 -> 2v2 -> 4v4 -> global -> 2v2. I think this one will work good enough, so we can save the trouble of implementing the double reference version.

server/rating.py Show resolved Hide resolved
@Askaholic Askaholic force-pushed the issue/#724-rating-initialization branch 7 times, most recently from fcf76fe to 6deb161 Compare August 6, 2021 02:49
server/rating.py Outdated Show resolved Hide resolved
@Askaholic Askaholic force-pushed the issue/#724-rating-initialization branch 2 times, most recently from 2b894b0 to 19dadf9 Compare August 10, 2021 04:15
@Askaholic Askaholic force-pushed the issue/#724-rating-initialization branch from 434799c to 35205a4 Compare August 17, 2021 17:59
@Askaholic Askaholic force-pushed the issue/#724-rating-initialization branch from 6fca3f5 to 0cc81e7 Compare August 28, 2021 01:59
@Askaholic Askaholic force-pushed the issue/#724-rating-initialization branch from 0cc81e7 to b779653 Compare August 28, 2021 17:34
…bles

Leaderboards can now have arbitrary initializers and initialization
should happen as expected. Ratings are initialized recursively until either
the current rating has no initializer, or a cycle is detected. Any ratings
that are implicitly created during this process are marked as transient and
will be recomputed whenever they are queried. This ensures we are always using
the latest version of the initializer rating when the first rating for a new
leaderboard is saved. For instance if tmm_2v2 is initialized from global, and
a player new to the matchmaker queues for 2v2, their 2v2 rating will be
initialized based on their global rating, and marked as transient. If the
player then cancels the queue and plays a bunch of global games (changing
their global rating), when they go to queue for 2v2 again, their 2v2 rating
will be reinitialized based on the updated global rating (instead of using the
cached version).

Legacy rating tables were migrated into the new leaderboard_rating table
in database migration v98. Therefore we can delete the code that checks
those legacy tables.
Askaholic and others added 2 commits August 28, 2021 09:41
Deviation will now only be increased once when initializing a long chain of
ratings
@Askaholic Askaholic force-pushed the issue/#724-rating-initialization branch from b779653 to 7ae75a8 Compare August 28, 2021 17:41
@Askaholic Askaholic merged commit eb61418 into FAForever:develop Aug 28, 2021
@Askaholic Askaholic deleted the issue/#724-rating-initialization branch August 28, 2021 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Generalize rating initialization based on another rating
3 participants