Looking at Dota’s Anti-Smurf Detection

Today’s post was inspired by http://redd.it/167wx6, though more for the comments section than the actual post.  I previously mentioned (what I call) Dota’s Smurf Detection back in An Extremely Unofficial Dota Matchmaking FAQ part 2, and these comments support my suspicions and also fills in some of the parts that I was unsure of.

So what do I mean by Smurf Detection?

A smurf is a new account made by an experienced player in order to effectively reset their MMR.  This can be for malicious purposes, like wanting to get matched against newbies for easy stomps; and it can also be done for more benign reasons, such as wanting to practice micro-intensive heroes without tanking the MMR of your main account or for using exclusively with friends who are new to Dota.  It’s impossible to eliminate this completely, but Valve does have an interest in eliminating the most egregious smurfing abuses.  What I call Smurf Detection is Valve’s system of accelerated matchmaking placement.

Accelerated placement isn’t exclusively a Valve thing.  As the reddit post mentions, Elo systems have had its own version of accelerated placement for a really long time.  One of the complaints I received about my matchmaking FAQs was that I underestimated the speed at which League of Legends’ ranked placement matches would move a new player around.  This may be true, but there are some real downsides to the way that LoL handles accelerated placement, and looking at that is a good start for explaining what Valve’s method does and why it’s an interesting approach to the problem.

In LoL’s matchmaking system players start at a 1200 ranking.  They gain points when they win, and lose points when they lose.  The amount varies depending on the expected outcome of the matchup, but the possibilities here are pretty tightly bounded by the matchmaking system, and I’ve seen no indication that the distribution of the types of matches you get (even, slight underdog, slight favorite) varies significantly between accounts.  During placement matches, your gains and losses are amplified.

For players on extreme ends of the distribution, this system works fine.  If you’re a 2000 player you’re likely to stomp most of your placement matches and get a good headstart on getting back to 2000.  The first problem though is that players closer to the middle of the distribution aren’t necessarily going to experience such precise movements.

Think about it this way.  We have a placement match with 10 brand new accounts.  Is the matchmaking for this match going to be fair?  Of course not.  Yeah, it’s perfectly fair in the sense that numerically 1200 x 5 = 1200 x 5, but 1200 here doesn’t mean that we think your skill level is 1200.  It means we have no clue what your skill level is so we’re just going to assume it’s 1200 and see what happens.

So if you’re a 2000 player starting a new account, you’re better than 99% of the players in the game.  You’ll dominate 95% of the lanes that you’re in, and you’ll win the vast majority of your placement matches.  If you’re a 1600 player, things change a bit.  You’re definitely better than most of the people who play the game, but not to the same degree the 2000 player is.  Getting a placement match where you and 4 ~1200 players go up against 2 other ~1600 players and 3 ~1200 players is a real possibility.  Not an extremely likely possibility, but one that will come up and will likely net you a loss in one of your placement matches.  Maybe you get stuck with 3 games in a row with a leaver on your team.   Maybe the system gives first pick on your team to a brand new account that goes mid, only it turns out they’re really a sub 800 player and the otherwise average mid on the enemy team snowballs off of them to a point you cannot counter.

So let’s move from a bunch of potential games to a more generalized statement about placement matches.  They’re extremely volatile because the system knows nothing about the skill levels of the players involved, and that volatility means that a streak of bad luck can heavily influence your final placement.

A side effect of this is that even past placement matches the quality of matches close to the starting ranking is going to be hurt.  If you’re a player playing at 1100, you’re going to run into a decent number of people who are 1600 but got unlucky during placement matches and an even larger number of people who are sub-1000 but didn’t get moved down far enough during placement matches.  This hurts the quality of your games in the long term even after your own personal placement matches are a thing of the past.

It appears that Valve has recognized these problems and their solution is to base Dota’s placement acceleration on player performance instead of purely using wins and losses.  If you don’t trust the source in the comments of the reddit post then consider this.  We have evidence of multiple instances of new accounts moving to the Very High bracket in under 10 games.  In one case it happened in two games and the record of that account was 1-1.  If W/L were the only factor these placements would be impossible.

Now I’m not claiming that Valve’s system of player evaluation is flawless.  If it were then we could ignore W/L entirely.  The reality is that Valve’s player evaluation is likely incredibly flawed but nonetheless adequate.

This isn’t the time and place to go into the difficulties of player evaluation.  Go read Moneyball if you want to experience the tip of that iceberg.  What’s more important to recognize here is that the player evaluation system just needs to be more accurate on average than a version of accelerated placement similar to what LoL uses, and frankly, given the volatility of games in this genre that isn’t a hard hurdle to jump.  Valve recognizes that given a sufficient number of games their standard matchmaking system will outperform their system of individual player evaluation, and this is why the individual player evaluation is “[o]nly in the first games you play, to make up for the people smurfing. Later it has less to no effect.”

The more subtle effect here is the old version of accelerated placement would tend to distribute players in a bell curve around the starting rating, leading to the detrimental effects I mentioned earlier.  With individual evaluation driven placement you can have effectively infinite entry points into the rating system, which helps prevent the formation of densely populated uncertainty zones within the rating system.

 

One interesting wrinkle emerges from this system.  I probably get at least 25% of my google searches from people looking for an answer of how to move up in the brackets.  The standard answer is win more often, but for a lot of players the quickest answer might be “Make a new account.”  If you were learning Dota on your current account or just getting re-acclimated, you might have underperformed during your placement period compared to what you do now.  If you’re winning consistently you’ll climb up, but it could easily be much quicker to make a new account and try to trigger the individual evaluation driven placement.  One of my biggest questions about the system was whether this individual evaluation was only in effect for new accounts, and the comments appear to confirm that this is the case.  Perhaps in the future Valve might consider doing periodic re-evaluations of existing accounts, but only they could know whether such a project is feasible.

One last aside, people are going to complain that this system somehow discriminates against their chosen hero or role.  I’m not going to claim that it’s perfect in this regard.  Personally I don’t even think it needs to be in order for it to be a positive addition to the system.  That being said, I highly expect that the evaluation is relative to the collective performances of the hero you’re using.  In other words, the system likely treats a 4 CS/min game on Crystal Maiden as way more noteworthy than a 6 CS/min game on Nature’s Prophet, so complaints about support discrimination are very likely overblown.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: