Oct 19, 2018
So this has probably been said a thousand times in more ranty fashion, but I'm legitimately curious if there is any reasoning behind the gain and loss of MMR points and if Blizzard has any intention of improving the system.
At present, there doesn't appear to be much if any logic to how many points you gain or lose in competitive. This is exacerbated by the fact that the matchmaking system occasionally throws you into virtually unwinnable games.
Example from my play this morning:
First game (pretty fair match)
Team 1 - Team rank 3400, 3 pugs and 1 3 premade, personal rating 3505
Team 2 - Team rank 3402, 3 pugs and 1 3 premade
Won - 19 points gained
Second game (!@#$ing what the actual %^-* blizzard)
Team 1 - Team rank 3550, 6 pugs, personal rating 3524
Team 2 - Team rank 3600, 2 x 3 premade
Lost (duh) - 27 points lost
Third game (pretty fair match)
Team 1 - Team rank 3497, 6 pugs, personal rating 3497
Team 2 - Team rank 3505, 6 pugs
Won - 20 points gained
Fourth game (MMR gods show me favor)
Team 1 - Team rank 3557, 3 pugs and 1 3 premade, personal rating 3517
Team 2 - Team rank 3496, 6 pugs
Won - 23 points gained
I get that this doesn't seem like a dramatic swing, but think about this. If the MMR system is working as intended you would expect to gain an average amount of points for a fair match victory, a large amount of points for an underdog victory, and a small amount of points for an expected victory. And we would also expect the opposite for losses; average loss for a fair match, lower loss for an underdog loss (expected loss), and a large loss for a match you lose despite an expected victory.
The reality is that the system does not appear to work like that whatsoever.
As far as I can tell, these are the biggest issues currently facing competitive MMR:
Matchmaking Does Not Weight Premades Correctly
Simply put, matchmaking isn't taking premades into account the way it should. A 3 man premade all in voice chat and cooperating with each other is a MASSIVE advantage in this game. Placing 2 x 3 premades against 6 pugs should never happen, period. If pugs have to play against a team to keep matchmaking times reasonable, at least match in higher ranked pugs so the team with the premade advantage doesn't also have MMR advantage. Additionally, victories against premades should award larger point values and losses against premades should result in fewer points lost.
Matchmaking Tends To Match Premades Together And Pugs Together
This feeds into the first point, but matchmaking appears to heavily favor putting premades with other premades and pugs with other pugs. It is far more common to see a game with 2 or more premades on one team and 6 pugs on the other. This is particularly frustrating because if a game is 6 pugs (team rank 3400) versus 2 x 3 premades (team rank 3400) then it obviously could have matched that game as 3 pugs + 1 3 man (team rank 3400) versus 3 pugs + 1 3 man (team rank 3400). It is absolutely baffling that lopsided premade matches like these happen and shows that the matchmaking system either isn't taking premades into account at all or is assigning far to small a value to premades.
Points For Losses And Victories Seem Random
The points assigned for a loss or victory do not seem to take into account MMR of either team or premades on either team. In other words, MMR doesn't seem to work period. If the point of the MMR system is to assign a score based on what tier you should be playing in then it by definition needs to weight victories and losses based on your opponents. In it's current form it simply doesn't.
Oct 19, 2016
Thanks for the well written post on your matchmaking experiences, I'll try to explain some of what's going on.
The system does in fact try to place equal sized groups on opposite teams whenever possible. Your report for game 2 where both 3 player groups were on the same team definitely seems like something that shouldn't happen based on the rules we've setup, so I'll look into it further.
We do need to do a better job of not placing players into "unwinnable" matches. When the matchmaker creates a match, it determines the % chance for each team to win based on the match it made. The VAST majority of matches are usually near to 50% (especially if you're a player closer to median skill rating and you're not in a group), but I've definitely seen logs of matches where that's really not the case and my eyebrows raise.
The unfortunate truth is that there is not always a "perfect" match for you, especially at very high (and very low!) skill ratings where there's fewer players of similar skill. Then you throw in the desire to match groups vs. groups, with everybody having low latency, and doing ALL of this as fast as possible even though it’s the 3AM offpeak... it can get tough. We've tried different tunings with regards to wait times, and the improvements were unfortunately modest as we increased the time to wait. Still, this is an area we're always looking to improve and tune better.
Fortunately, when we do put you in a match that we know isn't a 50/50, we adjust your SR gain or loss based on your calculated change of winning. So if you did get placed into a match with only a 20% chance to win and then you lose, you shouldn't lose much SR.
For matchmaking groups, there's actually two separate issues that we try to solve. The first issue is "How do we handle groups formed of players with different MMRs?". With season 2 we prevented players of REALLY disparate Skill Rating from grouping, but there's still some variance we need to handle. Over time we've tested different models to try and see what's best and are now using what tested most accurately. (Hint: it's not simply averaging the MMRs)
The other issue is how do we model the synergistic effects of players being together in a group. As you noted, they have access to voice chat. Now here's where things get interesting. This "massive" advantage actually differs based upon the skill rating of the group members. Based upon the data we've seen groups of low to mid SR players don't see that much improvement to their win %. Higher SR players do see more notable improvements, but it's not as huge as you might think. Still, we do take this into account when we predict the win% for each team. Regardless of how the data looks, we do know there's a perception of a large advantage for groups. That's one of the reasons why we explicitly try to match similar sized groups together.
So then why do points for losses and wins seem so random? Well, the amount of MMR (and SR) you go up or down isn't simply a matter of whether you won or lost, and what was your predicted chance of winning. There's a couple of other things at work. One is the matchmaker's confidence in what your MMR should be. Play a lot of games, it gets more certain. Don't play Overwatch for a while, it gets less certain. You go on a large win or loss streak, it gets less certain. The more certain the matchmaker is about your MMR, the less your MMR will change in either direction based on a win or loss.
As a minor factor, we also do evaluate how well you played the heroes you used in a match. The comparison is largely based on historical data of people playing a specific hero (not medals, not pure damage done), and we've done a lot of work to this system based on the community's feedback. In fact, I've seen some people indicate that they don't think we're doing this anymore. We still are. While it's a minor factor compared to wins/losses (The best way to increase your SR is still to play together and win as a team!), doing so does help us determine your skill more accurately and faster.
So take all that into account, the SR gain/loss after any single match can be a bit more "noisy" that it seems it should, but we're asking it to look at a lot of different factors to do the best job it can creating fair matches for you.
As an aside to all of this...
"Fair" matches doesn't always mean that every Ilios match goes 3-2 and 100-99 on the final point, or each team gets the payload to the end in overtime on Dorado, etc. Sometimes when two evenly matches teams play, the result can be one-sided. It just means that at that single moment in time the enemy team played better. It's not always the matchmaker's, your's, or your team's(!!!) fault that you got stomped.
Oct 19, 2016
Posted by Light
Based on what you said, it sounds like the more you play the more certain the system is on where you should be. However, over time I hope to improve. Will the system thinking that I am where I should be have a negative result on me climbing in the future (via lower amount of SR gain from wins)?
Not in the long term. Something to remember is that when uncertainty decreases you also won't lose as much SR when you do lose a match.
Oct 20, 2016
Posted by Sciron
Posted by Nessuno
The effect of win/loss streaks seems to be the biggest culprit in the noise, and adds a lot of unnecessary randomness.
Whether or not your wins are strung together will have a huge impact on your "season high", which determines both your end of season reward and your displayed tier level.
Posted by stephofniThese things. Seems streaks only serve to undo the system's attempts to remain accurate.
the other two inputs into certainty make plenty of sense but it's silly for the game to think because i've won 4 in a row, maybe i'm not the same player i've been for well over 200 games. or because i've lost 4, i might be a much worse player than i was yesterday on that +4 streak. someone's skill changes, it's going to reflect in their winrate. let it imo. trying to hurry that process along is just making a mess out of the median ranks.
We've actually done some tuning since launch so it takes several more wins or losses in a row to trigger full expansion of the uncertainty due to streaks. It's been tricky to tune. We don't want to have your SR seesawing wildly due to natural variance in wins and losses, but but we do want it to be meaningful enough so if someone's MMR is incorrect (a new account, someone trying to smurf, etc.) we get them to their "proper" MMR quickly so they're playing against the players who they should be playing.
Oct 20, 2016
Posted by HimeSama
Here is my question: How can a mathematical equation be "uncertain".
I know you can't let us into the heads of your devs or reveal proprietary code, but I'm just curious. Not attacking or anything. Perhaps understanding how the system works will help people not get so frustrated when the MMR/SR system is taking a big one on their day.
The principle of "uncertainty" is in reference to measuring a player's skill. Skill fluctuates, that's just a given. You have good days and bad days. Your MMR is never supposed to cement at a fixed value. It's supposed to ebb and flow. Uncertainty just measures how much it ebbs and flows. It's not the calculation
that's uncertain, it's the measurement of confidence in your past results shaping the system's current impression of your skill.
Let's say I play a match against you and we're both brand new players. We both start at the seed value of 2500. You beat me. My rating drops to 2450 and yours increases to 2550. Is 2550 an accurate representation of your true potential? Probably not. It's going to require more games -- more data points -- to determine whether that current rating is accurate. If you keep winning and winning and winning, the confidence in whether you are really
2550 is very low. If you keep losing and losing and losing, then you getting to 2550 in the first place may have been a fluke, so the confidence in whether you are really 2550 in that scenario is also very low. If you happen to go 50/50 by playing against other 2550 opponents, then confidence will be a bit higher that that's roughly the range you should be at (depending on how consistent you are), so it makes sense to reduce the volatility a bit and reduce your rating change, since we have sufficient data to conclude that you actually belong there.
Oct 20, 2016
Posted by D4G
As much as I love the post explaining match making something definitely needs improving.
This happened at 11:47am PST today and the 6 man said that they waited 6mins. When the average match takes 15 to 20 mins to complete 6 mins is way too short. I get it that it sucks for a 6 man to wait for a match but a match like this doesn't help anyone and is a waste of both teams times.
Yeah, that's a situation we'd definitely like to improve. The core problem is that there might not be a fair match for that 6 player group in your screenshot, not just after 6 minutes but after 15, 30, or 60. I know that people say that they're willing to wait longer, but there are also practical limits to that patience. We've been erring more on the side of people at least playing the game instead of sitting in queue. Those 6 player stacks sometimes lose, too.
Archive Notes - Last Updated: Feb 7th, 2020
Win/Loss streaks no longer greatly affect most players in their SR adjustments and more tighter matchmaking restrictions are now implemented since the start of Season 4.
Starting with season 4, matches will no longer be formed if the predicted win percentage for a team is less than 40% even if this does not mean a large group is unable to find a fair match.
The post with the image link has been edited to directly show the image in-page.