DotaBuff Rating: First Thoughts (And maybe last)

Good news is that the official API came back up today.  Bad news is I’m still running into the same bug from however many months ago, but apparently the API is in for a major overhaul in the next couple of weeks, so things look to be back on track (oh god I have to look at code again)

Anyway, DBR came out along with some extra site features.  It’s still too early to tell what the new Dota2 privacy setting will mean for the future.  Some rumors suggest that it’ll be pretty devastating to DotaBuff’s plans,  but for now it doesn’t appear to be interfering with much (though it may or may not be currently bugged).  In any case, DBR would have been calculated from DotaBuff’s backlog of matches so the privacy setting would only prevent future DBR adjustments at worse.  So taking everything as it is at the moment, how is DBR looking?

Update: DotaBuff claims the privacy changes will shut them down

The most important thing is that DotaBuff appears to be keeping personal DBR completely private.  This definitely lessens some of the concerns about DBR being used to flame at hero selection.  Had DBR lookup of other accounts been a DotaBuff Plus feature, it’d only be a matter of time before someone would make a script to look up and report the ratings of everyone in the game.  Whether this would pan out for any number of reasons is unclear, but it’s a headache that Valve would prefer to avoid entirely.  That’s not to say there aren’t downsides to DBR as is.  I still stand by everything I said in The Insignificance of Pub Stats, Part 3, but DBR’s current implementation is at least far more innocuous than it could have been.

I feel the most misleading aspect of DBR is that it appears to be heavily inflated by dead accounts.  A DBR distribution by active accounts would likely present a far clearer picture.  It would also be less flattering for most of us, and maybe this played into the decision.  For whatever it’s worth, I have access to the DBR of one account that solo queues between Normal and High and it’s 96th percentile.  Assuming this is accurate (which is a risky assumption; more data points would be necessary to ensure this particular account isn’t an outlier in some fashion) then combined with my normal/high/very high measurements it suggests two things

  1. There are a lot of dead accounts, and the majority of them are somewhere in normal.  This is almost certainly true.
  2. Of the players that are active, High and Very High players play far more regularly than Normal players.  This is also likely true.

Unfortunately we have no way to tell right now which is the stronger force.  Well, at least I have no way to tell right now.  Maybe DotaBuff knows.

The other silly thing I keep running into is that people are ascribing way too much certainty into the pure numbers of the system without understanding the unit of measurement.  It’s not like we have a block of matchmaking skill sitting somewhere in a vacuum sealed jar.  The skill difference represented by 100 points in any particular matchmaking system is completely arbitrary, unique to that particular matchmaking system unless explicitly designed otherwise, and not even necessarily constant at all points in the distribution (in other words, the extremes might be subjected to warping).

Finally, there’s one other issue I want to address.  One of the responses to my Insignificance articles is that while the individual rating might not always matter, skill plateau’s do matter.  I believe the particular example was that all SC2 pros would make it to the grandmaster ladder.  And I do grant you that something like this is true even in Dota.

The problem is that while every pro player might hit this plateau (or would eventually hit this plateau provided they used matchmaking often enough), not everyone who hits that plateau is going to be capable of being a pro player.  This is especially true in Dota where hero specialization, All Pick, and pre-made stacks are all ways to to game the rating system, which makes the predictive power of any particular plateau pretty unreliable.

The more insidious effect is that there may be people who are below that plateau who are capable of playing at and beyond that plateau provided they ever make it into the right environment for honing their skills, that right environment being organized games.  But if we’ve made a big deal about the plateau then some of the people recruiting for organized games might begin to only recruit people who hit the plateau in matchmaking.  This would stunt the development of the community in a really bad way.

So yeah, maybe DBR allows us to make really broad judgments of a player’s current skill, but we should always emphasize that on a narrow scope these judgments are imprecise and that they are a measurement of current results and not actual potential.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: