Valve Employee Comments on Matchmaking

Possibly old news for many of you, but a new Valve red name has shown up on the Dota 2 Dev Forum answering questions about how matches are created.  You should be able to find his entire post history here.  For the most part it confirms things we’ve already suspected, but let’s go over the more noteworthy revelations.

http://dev.dota2.com/showthread.php?t=70322&page=59&p=663906#post663906

>Match ID: 246510650

All 10 players in this game were in the 93rd-94th percentile MMR range. The difference in Elo’s between the highest and lowest player was 50 Elo points. The “noob” with only 13 wins actually had the highest Elo of any player in the match. (This was a smurf.) They did play poorly in this match, but in the previous match (246456658), in which they played against several 4000 Elo players, they had 13 kills and 1 death.

First we have the basic confirmation that matchmaking uses Elo (sidestepping the semantic debate on whether it’s “real” Elo or just similar to Elo).  Also of notice here, the match in question is rated Very High in-game.  This suggests that the player percentile rating necessary to be in Very High is larger than what I’ve found the match percentile to be.  There’s a lot of possible explanations for this, such as a relative activity levels in different percentiles, but no real way to test any of the explanations.  Regardless, it appears that player percentiles as low as 93rd will queue into games that are rated Very High.  Finally, this answer also confirms that Valve does attempt to identify smurfs…

>Match ID: 246567215

The 14-win player did have the highest MMR in the game. However, it probably wasn’t really high enough to push him into smurf territory. So this was not a great match. There was a player who had waited 5 minutes, which is why a match of this relatively low quality was accepted. Thanks for reporting this. It’s very helpful.

…and that this smurf detection is still undergoing tuning.

http://dev.dota2.com/showthread.php?t=98317&p=663920#post663920

Just to clarify, there was never any “win rate” calculation. Ever. It is true that a goal of matchmaking is to make even teams, so that the odds of the Radiant winning any given match is 50%. The matchmaker also will raise your Elo and try to put you in players of equivalent skill, which indirectly tries to get the win rate to 50%. However, it has never looked at your historical win rate and, for example, put you in a game where it knew that you were expected to lose, to end a winning streak, or given you a stomp to end a losing streak.

Just another confirmation that players tending towards 50% win rates is an indirect result of the goals of the matchmaking system (create 1) even matches 2) among players of a similar skill level) and never something that the system strives for directly.

http://dev.dota2.com/showthread.php?t=98284&p=663957#post663957

>246572710

The game looks pretty balanced to me. The Elos are all relatively close. Here are the Elo’s on the two teams:

R D
3172 3047
2918 2958
2810 2804
2788 2720
2409 2508

Now it’s a pretty big spread between 2409 and 3172, and it is a legitimate question to ask why in the world would we put people together with that big of a skill differential. The answer is that we didn’t. The Radiant had a 4-stack which covered that range (the highest and lowest Elos on the Radiant were in the same party). We matched them with two 2-stacks on the Dire. One of the two-stacks had the highest and lowest Elos for the Dire, and the other two stack was in 2 of the middle slots. You were a single who was also in the middle, and there was also a single on the Radiant, also in the middle of the Elo range.

So, the average Elos of all the parties were pretty close. And, player-by-player, each team had somebody on the opposing team of roughly equal skill. Given that 4 stack with the big skill spread, I think it’s hard to come up with a better way to get them into a game.

Many of the Dota2 matches with a wide spread in skill are created to cope with pre-made groups that happen to have a wide skill spread.  The only possible alternatives would be to either intentionally give these teams bad games or to disallow teams of this skill spread from queuing entirely.  If you happen to prefer the latter, you can always enable the solo queue only option.

Also of note, in this game the Radiant team was 4+1 with an average rating of 2819.4.  The Dire team was 2+2+1 with an average rating of 2807.4.  So in this case, the larger premade group did not receive much of a handicap and still got stomped.

http://dev.dota2.com/showthread.php?t=98311&page=3&p=664114#post664114

Just one last comment on this. Elo is a TERRIBLE way to give players a sense of “progress.” Many (most?) people reach a plateau, and their Elo stabilizes. It is simply not mathematically possible for Elo to keep increasing in general for players indefinitely as more and more games are played.

Given this reality, if players used Elo to measure “progress”, we would constantly be reminding them that they are NOT making any. That would be really bad.

There are people out there who won’t like this answer, but many of these complainers also think that plateauing is something that exclusively applies to other people.

http://dev.dota2.com/showthread.php?t=98311&page=2&p=663966#post663966

“Noob” is a relative term. We don’t consider a person with 150 games to be a “noob”. We have some good data that by 75 wins (approx 150 games), Elo is pretty accurate, and so we rely on it almost exclusively at around that point. If you are getting matched with those players, it should be because your Elo is approximately the same. Parties can complicate things considerably. I might be able to provide further insight into why it thought the match would be a good one if you provide a MatchID.

Finally, we have this post that states that the matchmaking system relies on Elo almost exclusively for player evaluation once the account has around 150 games played.  I find this one fun because 150 was the number I used way back in my matchmaking FAQ.  I guess if you write enough you eventually nail something.

But anyway, this summary is biased towards the pieces I find most interesting, and that judgment is often at odds with the populace as a whole, so feel free to read his posts in entirety and reach your own conclusions.

On a sad note, the other post I wanted to run with this week has to be delayed.  My script is choking on the last bit of essential data, and I have to suspect that the load of the Steam sale might be the culprit.  We’ll have to see if things are going a bit more smoothly early next week.

Advertisements

6 Responses to Valve Employee Comments on Matchmaking

  1. note says:

    I want to point out that solo queue, as it exists right now, is probably not a reasonable alternative. The smaller player pool, and possibly the tendency of the MM system to favor shorter wait times over a tighter range, means that matches often have very big skill spreads too.

    As long as the pool of players is smaller and the algorithm doesn’t strongly prefer tighter ranges, the only thing that solo MM addresses is not getting flamed/trolled by stacks.

    • phantasmal says:

      There is speculation that as of a recent patch solo queue isn’t a separate queue, and that every solo player that doesn’t opt-in to solo queue only (which is most of them) can be matched into both group and solo queue games. If this is true, the quality of solo MM should have improved substantially since the change.

  2. Effects says:

    I don’t know if an elo system is what’s best for dota, at least at all levels of the matchmaking system. Since early on a lot more variance exists within the game due to lack of proper information about laning, this causes the team which “luckily” ended up with the right amount of carries/supports/2s to have a higher win rate, not to mention unintentional laning combinations resulting in particularly poor performance for one side and particularly good performance for the other. This means that matches, and thus your elo, will swing rather wildly after you settle into your low elo hole as a beginner, potentially into a hole that is very hard to recover from, due to the inherent variance . Without some amount of knowledgeable players to play against, the beginners and not so good non beginners that populate the lower elo field will never know the benefits of warding, carrying tp scrolls, or maybe even last-hitting. It seems like it is against their own development to be stuck in such matches.

    This may seem innocent enough, but the truth is that not only are these skills relatively unheard of, the few people who may be “misplaced” into these low elos can get stuck into bad equilibria in games that will further delay they’re (supposed) ascent out of their mis-assigned position. This can be seen in a rather competent carry player being unable to leverage his skill due to the prevalence of carries in these low skill games, and the obvious diminishing returns they represent. It can also be seen in the inefficient allocation of time often seen by players in these games spent on acquiring resources – why bother meticulously setting up a split-push or baiting enemy movement to take out a tower when you have 3 knuckleheads on your team roaming together without vision that could get insta-gibbed if they run into the enemy team. Clearly, sticking with the team and getting a low expected return on time invested is the less risky decision as it mostly negates this possibility. This, however, leads to 55 min games where players achieve absurdly low xp and gpm and teaches nothing whatsoever about true late game dota mechanics. A testament to this is how rarely the buyback option is used in this level of game, and how the late game often really comes down to very punishing engagements, due especially to the lack of prevalence of tp scrolls.

    While I’ve been ranting mostly about the agony of knowing how badly your team mates are doing in low elo games, the real point I’m trying to make is that it is possible to have your elo misplaced, even after many games, and that this is due in no small part to the variance of the dota 2 game. An elo system also is slow to react to a player who has gotten much better by learning about dota outside of the game, by watching streams, reading strategy content such as what is found on this website, or simply by being a more intelligent human being after coming back from a year long break. The system is very good at classing players beyond a certain skill threshold, but can keep bad players ignorant and also trap players for a time longer than desired, particularly if that player is constrained on the amount of time he can commit to dota.

    Some possible changes that might remedy these issues are:

    1. A threshold on player skill that would exist at the mean level of elo. Below this, elo would be tracked but would matter a lot less for the purposes of matchmaking, where a less experienced player would have a much greater chance of playing with a (slightly) more experienced player, resulting in greater learning by the lower players and more diverse games at the lower elo levels. In effect, the range of elos on each team would be greater than at higher levels but the teams would still be matched up fairly. One critique of this is that low level players may get beat up on more, and thus get stuck in the same kind of hole albeit with the ability to play with better players once they learn a bit more. This “fix” then doesn’t seem ideal, but it is interesting as a thought.

    2. An added “velocity” to elo gain for players below a certain threshold who display sudden surges in in game statistics for several games. For example, a player who has had >x kda ratio or y difference between his and the next highest players’ gpm (assuming highest) for k out of n games would get a certain amount of boost in elo gain. This is set all in terms of variables as I have no data in front of me to start trying to set realistic parameters for this, but the point still stands that such a system would be very benign and would try to increase the speed with which people correct difference between sample elo and true innate skill. This method does not, however, address the relative ignorance of those in lower elos to superior play.

    I suppose I should wrap up. A few things to take away from the wall of text;
    1) elo is a sample variable that we are interested in learning the true value of. Due to the nature of the game of DotA, players may experience an “initial” shock to their elo that might prove very difficult to rectify for a variety of reasons. We are interested in not having this happen.
    2) Due to elo keeping people who truly are bad playing against fairly bad players effectively forever, they may never learn how to play DotA better just from in-game experience. They may not care but this just seems wrong.
    3) It is possible to implement some changes to elo math that could help to rectify some of these problems, with some unknown consequences.

    ———————————————————————————-

    As an added note, I’d also like to say that I may have fundamentally misunderstood the mechanisms already in elo and that these sorts of systems are already in place. I do not believe that they are, however. I’d also like to say that since elo is not always an exact indicator of your innate skill that matchmaking always does have some random chance in who gets the truly better players. This chance decreases as the number of games played by all players approaches n-large but I digress.

    Also, I really would like to be comforted by a dev posting something about number of wards placed mattering towards elo in a direct way, even if it is marginal.That would make me feel warm and fuzzy inside.

  3. Gaspard Savoureux says:

    If you explain that the VH bracket is wider because of player activity inside different brackets, wouldn’t that mean that players from normal/high play more than people from VH? I would have guessed it the other way around.

    My theory is that the ELO distribution inside the VH bracket is way bigger than inside the other brackets (it was somewhat confirmed by DBR spread analysis). This means that if someone who’s deep into the Very HIgh bracket queues with someone who usually play inside Normal or High, he can easily pull the game into the VH bracket because his ELO is super huge. As always that’s probably not the only factor but i’m pretty sure it does come into play.

    • Gaspard Savoureux says:

      Then there’s also the fact that people from VH are probably more likely to smurf than the others. New accounts by experienced players can get matched inside the VH bracket with as little as 5 games, which probably inflates the numbers in a way that’s very hard to predict.

  4. Dávid says:

    Valve please delete my account as u delete my rank … I was 3800 before February … then it comes 7 wins and something like 40 looses … I made my the best loosing also. I am not the best player in the world but u always stick me together with somethink like 0/8 mid solo player etc. I tried all positions. NOTHING work. PLEASE DELETE MY ACCOUNT. I dislike all of you. You ruin my dota life.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: