The other day I came across Statistical Significance: The Value of Pub Stats, and I’m afraid I cannot let it go unanswered. I don’t believe the author intends anything malicious, but he is breathtakingly wrong on nearly every point he raises. And this isn’t merely some philosophical spat. Bad statistics are poisonous to a game community. They warp players conceptions of the game so that they chase higher stats at the expense of performing uncelebrated tasks. They offer ammunition to our ceaseless desire to prove that everything bad that happens is someone else’s fault. They create an implicit hierarchy along some aggregate number that no one understands but everybody trusts. And finally they make us lazy.
Because I’m responding to an article, I’ll take the issues chronologically according to the article. Hopefully this is convenient for those of you following at home.
A common criticism of tracking detailed statistics is that it provides ammunition for bad mannered. With the DotA community’s infamy for bad mannered players, many charge that lifetime statistics will lead to increased harassment.
[But] I believe DOTABUFF is right that flaming exists in gaming communities regardless of stats, and Dota 2 is no exception. In the end, the availability of stats does not increase flaming, it merely alters the content of flaming. (emphasis theirs)
The marginally more minor objection to this is that the availability of stats does increase flaming. When someone instant locks a carry, they might get flamed. In certain skill brackets it can feel like everyone believes that their teams pick too many carries, and so everyone they get matched with should know better and pick supports so they can pick their favorite carry. It’s a frustrating environment and flaming will happen from time to time.
If you take the same environment and give players the ability to see the lifetime KDA (Kills plus Assists divided by Deaths) of every player on their team with a single click, you will see the pre-game flames skyrocket. A big part of this is that ease of availability is really important. Trying to load up a site where you have to type up every player’s name to get their KDR is not only tedious; it’s also tough to do while the pick timer is counting down. And then when you bring it up in chat you have to admit you perform a manual background check on everyone you play with, which makes you look like a creepy, obsessive moron. This serves as a deterrent, which is good because if you do this you are a creepy, obsessive moron and anything that discourages you from using that to ruin games is a big plus.
The bigger threat is that having stats available does alter the content of flaming. If someone’s just ranting at the champion select screen about so-and-so picking a carry the rest of the team is likely to roll their eyes and know precisely who they should /ignore at the first sign of conflict. But if you suddenly give this person access to the carry player’s MMR or KDA, you’re basically handing them a facade of credibility. People don’t have the time or energy to exhaustively research every statistic waved in front of them, and if you let something like MMR or KDA checking become common you risk giving these practices a legitimacy they do not deserve.
The article goes on to say
Comparatively, educated players will understand that poor stats, such as a low total KDR, are not relevant to evaluating the performance of the support players in a match, and that a person’s career KDR is not useful metric of evaluation when they play a variety of heroes with different expected averages. These players will not be led to flame because they have statistics as ammo; they understand such criticisms are misplaced and faulty.
I mean no offense by this but educated players are so rare that they aren’t statistically significant. If you polled every Dota 2 player the majority of them wouldn’t begin to be able to tell you the multitude of ways KDR is flawed as a stat. The majority might not even be able to tell you what KDR is in the first place. What it comes down to is it we’re all struggling to come up with mental models that help explain an incredibly complicated game, and any time someone creates some aggregate stat that purports to tell you how well you’re doing that stat immediately becomes a tempting shortcut.
But almost inevitably this metastat is a false idol, misleading us in ways that are difficult to notice due to their incredibly subtlety. Even worse, these stats absolve us of the responsibility of having to perform that sometimes tedious task of trying to understand the game for ourselves. We create this superficial metric and let that metric becomes an authority figure that establishes an implicit hierarchy. Then we just accept it because it let’s us become comfortable in being able to pretend to know what we are talking about. The only way to prevent this is to cultivate a healthy suspicion of statistics. If you’re not constantly worried that every number in your spreadsheet might be lying to you, you’re not doing it right.
Valve is 100% in the right to not integrate statistics in the client. Maybe some day when we actually have good, nuanced statistics for evaluating player performance that will change, but until then we should collectively admit that judging player performance is hard and that we’re incapable point to some single number somewhere to determine whether you’re doing a good job or not.
Anyway, this is longer than I expected and I won’t have time to finish it tonight. Stay tuned for the future installments where we explain why KDA isn’t a reliable stat, why statistics can never completely replace watching replays, how we can work to eventually build better statistics, and why stat shaming to try to create a more competitive community is the dumbest thing ever. Also, I’d like to mention that for all my complaints about the article, I feel that the sources quoted in the article are generally much more reserved and cautious about the potential misuses of statistics. Reminds me of this.