Adjust global rating with ranked matchmaker games

[Askaholic]

Now that we're adding a number of different matchmakers, we want to make it so that new players can use the matchmaker to establish a global rating for themselves. To do this, we want to apply rating changes from the matchmaker back to global, but only if the rating changes are positive (to prevent rating manipulation and other issues).

The way it would work is:

Proposal 1

Whenever a game is rated and the rating used is not global and not hidden, then compute the normalized adjustment made by trueskill to the rating i.e. `(after_mean - before_mean) / before_mean` and `(after_dev - before_dev) / before_dev)`. If the new displayed rating is higher than the displayed global rating, compute the adjusted global rating by applying the normalized changes to the global rating, scaled by the global rating i.e. `global_mean += global_mean * normalized_mean_change` and `global_dev += global_dev * normalized_dev_change`. If the displayed adjusted global rating is higher than the displayed old global rating, save the rating adjustments, otherwise leave the global rating unchanged.

Example 1

Suppose a player's global rating and matchmaker ratings are both (1500, 500) and they LOSE a matchmaker game where their new rating returned by trueskill is (1400, 400). Since the new displayed rating (1400 - 3 * 400) = 200 is higher than the displayed global rating (1500 - 3 * 500) = 0, we perform the adjustment as follows:

• Normalized mean change is (1400 - 1500) / 1500 = -0.067
• Normalized deviation change is (400 - 500) / 500 = -0.200.
• Global mean adjustment is 1500 * -0.067 = -100
• Global deviation adjustment is 500 * -0.200 = -100

The adjusted global mean is then computed to be (1500 - 100, 500 - 100) = (1400, 400) which is displayed as 200. Since this is higher than the original displayed global rating 0, the global rating adjustments are saved.

Example 2

Suppose a player's global rating is (1000, 150) and their matchmaker rating is (1200, 100) and they WIN a matchmaker game where their new rating returned by trueskill is (1205, 95). Since the new displayed rating (1205- 3 * 95) = 920 is higher than the displayed global rating (1000 - 3 * 150) = 550, we perform the adjustment as follows:

• Normalized mean change is (1205 - 1200) / 1200 = 0.0042
• Normalized deviation change is (95 - 100) / 100 = -0.0500.
• Global mean adjustment is 1000 * 0.0042 = 4.2
• Global deviation adjustment is 150 * -0.0500 = -7.500

The adjusted global mean is then computed to be (1000 + 4.2, 150 - 7.500) = (1004.2, 142.500) which is displayed as 567.2. Since this is higher than the original displayed global rating 550, the global rating adjustments are saved.

Example 3

Suppose a player's global rating is (1000, 150) and their matchmaker rating is (1200, 100) and they LOSE a matchmaker game where their new rating returned by trueskill is (1195, 105). Since the new displayed rating (1195 - 3 * 105) = 880 is higher than the displayed global rating (1000 - 3 * 150) = 550, we perform the adjustment as follows:

• Normalized mean change is (1195 - 1200) / 1200 = -0.0042
• Normalized deviation change is (105 - 100) / 100 = 0.0500.
• Global mean adjustment is 1000 *- 0.0042 = -4.2
• Global deviation adjustment is 150 * 0.0500 = 7.500

The adjusted global mean is then computed to be (1000 - 4.2, 150 + 7.500) = (995.8, 157.500) which is displayed as 523.3. Since this is lower than the original displayed global rating 550, the global rating adjustments are discarded.

Proposal 2

Whenever a game is rated and the rating used is not global and not hidden, then compute the adjusted global mean by replacing the player's non-global rating with the global mean in the call to the trueskill rating adjustments.

Scenario 1

Suppose a player's global rating and matchmaker ratings are the same, then the new global rating will exactly equal the matchmaker rating, as the call to trueskill is unchanged.

Scenario 2 (1vs1)

• MM Rating: (800, 150) displayed 350
• Global rating: (1000, 150) displayed 550
• MM Opponent Rating: (800, 150) displayed 350

Player WINS:

• New MM rating: (848.918, 143.182) displayed 419
• New Global rating: (1031.924, 144.475) displayed 598

Player LOSES:

• New MM rating: (751.082, 143.182) displayed 322
• New Global rating: (931.189, 142.189) displayed 505

[BlackYps]

Why do you want to use normalized mean changes in Proposal 1? Let's say you have (1500, 500) global and (300, 100) ladder rating. If you win a ladder game and you gain 10 mean and the deviation stays the same you will get 50 more mean in global. Doesn't really make sense to me. It gets even worse when you have a negative mean in one of the ratings, because then you invert the direction of the change.
Proposal 2 would allow you to lose global rating by playing matchmaker.

I would just do the following:
Proposal 3:
Whenever a game is rated and the rating used is not global and not hidden and the new displayed rating is higher than the displayed global rating, then apply the absolute mean and deviation change to global while maintaining a minimum deviation of 150.
If the new displayed global rating is higher than the displayed old global rating and less than 1400, save the rating adjustments, otherwise leave the global rating unchanged.

[Askaholic]

Well in proposal 2 you can also add the same rule for only making the rating change if the displayed rating increases.

I'm a little hesitant to do too much weird arbitrary stuff with the ratings, as that goes further and further outside of what trueskill is actually designed for, and I'm kindof worried that we're just going to break it entirely. That's why I actually like something like proposal 2 where we just continue to use the trueskill library to figure out the rating changes for us.

[BlackYps]

Sounds good. If we properly limit the scenarios when the computation will be actually applied, I think we can do it that way. Don't know if that is an error, but Proposal 2 specifies to only use the global mean. For it to really make sense it should use the whole global rating for the adjustment calculations.

[FtXCommando]

So a problem with proposal 1 (admittedly rare now but this is absolutely going to be something people abuse once the system is figured out) is that normalization really starts messing with things when data is too divergent.

Take a new player that is REALLY bad but REALLY stubborn and keeps playing ladder (or a person that is intentionally trying to manipulate the system).

(1500, 500) or 0 shown rating = global
(5, 200) or -595 shown rating = ladder

he wins his next game

(40,195) or -545 shown rating

With normalization for mean (40-5/5) you get 7.
Apply the multiplier to your global mean, you now have a global mean of 10,500.
With normalization for deviation (195-200/200) you get -.025
Apply the multiplier to your global deviation, you now have a global deviation of 487.5
So now your shown rating is 9037.

[Sheikah45]

The normalization isn't applied in that case because the shown rating is lower than the shown global rating.

[FtXCommando]

No, the normalization is thrown out if the adjustment caused a decrease in the shown global rating in the after period. My example does not: 9037 > 0.

Ah I see, I skipped the final check. My bad.

[FtXCommando]

Ok no I'm confusing myself now, it would be (1,500 + 10,500, 500 - 12.5) or (12,000, 487.5) which is 10,537.5 which doesn't get discarded as it's greater than 0.

[Sheikah45]

There are two checks. The first check is if the after of the ladder is greater than the before of global.

If the new displayed rating is higher than the displayed global rating, compute the adjusted global rating

So in your example since -545 is less than zero nothing is changed in global according to proposal 1.

[BlackYps]

It seems like regardless of the algorithm used to determine the new global rating we want the same constraints for actually saving it: global lower than the matchmaker rating, positive displayed rating change, and only up to a certain rating limit.
With these constraints is anyone against just using trueskill to determine the rating change? (i.e. proposal 2)

Edit: I played around with some trueskill calculations and it seems to work very nicely.