Saturday, May 19, 2012

Metacritic: A Blight on the Industry

The recent release of Risen 2: Dark Waters was met with mixed reviews from fans and critics alike. I never expected Risen 2 to do well among mainstream gamers (whose definition of a good RPG consists of Skyrim and Mass Effect), so it's no surprise to me that its metascore is currently sitting at a mediocre 69. Honestly, I would've given it a similar score myself when I first played it. Seemingly broken combat, graphical glitches galore, watered-down gameplay elements, and other issues with design and presentation left me feeling very conflicted about a sequel that should have only improved on the success of the first game.

Less than two weeks after its release, however, Piranha Bytes released a patch which addressed nearly every major complaint I (or anyone else) had with the game. They added roll-dodging and the ability to block monster attacks, which greatly improved the feeling of combat, they adjusted some really distracting graphical issues like low draw distance and growing/shrinking foliage, they fixed a few quest bugs and other glitches, they improved enemy and companion AI a little bit. It's still not a perfect game, but it became instantly more enjoyable with just a single patch.

And yet virtually every single review was published before the patch was even released. Surely these review scores would have been more favorable, maybe averaging in the 75-80 range, had they reviewed it with the patch. To be fair, the game should never have launched without these basic components, and PB / Deep Silver deserve to take the flack for its launch status. At the same time, however, shouldn't they be cut a little slack for addressing all of the major issues in a timely manner? Either way, as far as Metacritic is concerned (and any casual consumer consulting it), Risen 2 is just another mediocre game not worth your time or money. 

So I have a few bones to pick with Metacritic. In the full article, expect a fair bit of ranting about what's wrong with Metacritic and how it's destroying the industry, with other examples besides just Risen 2.

Metacritic is a nice tool for consumers, providing links to a variety of review sources to let you quickly and easily inform yourself about a game, but I'm not fond of attaching such a rigidly objective, universal score to a video game. It's impossible to come up with a perfect quantification of a game's quality because a video game will always leave subjective impressions on the reviewer. Some games are notoriously difficult to score in a review because the subjective experience often transcends the typical conventions of game reviewing.

Take Pathologic for example -- it's not a very good game per se, with many noticeable flaws that detract from the overall experience. You've got the awkward interface, the poor translation from Russian to English, the shoddy combat system, the excruciatingly slow walking speed, the poor graphics and animations. Any reviewer would be obligated to give it a score somewhere in the 5-7/10 range, but the honest truth is that, even in its flawed state, Pathologic is a monument of a game that deserves more praise and recognition than a meager "average" score could ever describe. Yet, seeing a 6/10 on Metacritic often serves to dissuade people's interest.

Metacritic uses a 100-point grading scale, but many review sites instead use letter grades or stars. Most of us grew up in education systems where an average grade, a "C," was about a 75/100. But when Metacritic translates a "C" to their 100-point scale, they deem it a 50/100 (the perfect average of 0 and 100), which falls right at the beginning their "red" category for negative review scores. The stars system tends to work more like a recommendation system than a rating, with 3 stars meaning "average, nothing special, but still a good game." But when Metacritic translates "3 stars" to their 100-point scale, they deem it a 60/100 -- "below average" according to our perception of academic grades. 

By the same token, when reviews give a game an "A" or "5 stars," that translates to a perfect 100 on Metacritic. Most people within the industry believe that perfect scores should be extremely rare, but it's not uncommon for a game to receive an "A" or "5 stars" for being truly exceptional and standing above the crowd -- that's what those scores mean on those scales. Because of Metacritic's screwy conversion rates, however, these "A" and "5 star" reviews wind up inflating a game's metascore, while average "C" or "3 star" reviews artificially deflate the metascore. Reading the text of a given review, you often find commentary contradicting the scores Metacritic translates, which shows that the metascore isn't always very accurate.

As is evident with Risen 2, metascores can also become quickly irrelevant and obsolete with game updates. One of my most-played games of all time, Killing Floor, is three years old and has received constant free updates since its release. It currently holds an "average" metascore of 72, with virtually every review having been written near the time of its launch in 2009. Most of the Metacritic reviews criticize a general lack of content, but less than six months after release, Tripwire added several maps, a bunch of weapons, a new perk, and a new enemy type, and the updates have just kept on coming. Those early reviews no longer reflect the current product, and yet the metascore remains what it is.

Some review sources like X-Play specifically opted for their reviews not to appear on Metacritic, due to conflicts over how their review scores should be translated to Metacritic's 100-point scale. On the other side of the coin, old bastions of "no review scores" like The Escapist, who were concerned for a long time that review scores might compromise the quality of their editorial reviews, have started giving review scores with the rising prevalence of Metacritic, seemingly with the primary intention of increasing site traffic by being linked to on review aggregates. 

Metascores are becoming a more prevalent quantification within the industry, with many people relying on a game's metascore as the sole determinant for whether a game is worth their time and money. If a game doesn't receive generally favorable reviews, many people won't buy it. Metacritic is even affecting video game developers -- after the brilliant success of Fallout: New Vegas, Obsidian failed to receive its bonus from publisher Bethesda, consequently having to lay off employees and cancel a project, all because the metascore was a single point shy of Bethesda's target of 85. Publishers and stockholders pay a lot of attention to metascores. 

This kind of obsession with Metacritic is bad for the industry. With people judging a game's success by its metascore, studios are more likely than ever to make games based on current trends that they know will yield a high metascore. Big studios were always wary of gambling with new IPs, for fear of low sales and not recuperating their investments, but now they may be even less likely to gamble with unconventional, creative IPs if they're concerned of getting a lower metascore, which will detrimentally affect sales anyway. It's a shame, because I usually find better games in the 70-80 range than I do in the 90+ range. 

Let's take a look at the RPG genre; Mass Effect, Fallout 3, Dragon Age: Origins, and Skyrim all scored in the 90+ range on PC. Despite their massive budgets and extensive levels of polish, I found each one of these games deceptively shallow and, in some cases, uninspired. Note how the user rating is lower than the critical rating for each title. By contrast, Risen, Drakensang, The Witcher, Gothic II, and Vampire: The Masquerade - Bloodlines all scored in the 70-80 range, and yet they have generally superior RPG elements than any big budget, AAA RPG. Notice how, with the surprising exception of Gothic II, the user rating for each of these games is higher than the critical rating. 

And it really drives me nuts every time I see a comment on a forum, where someone casually glances over the Metacritic page and writes the entire game off just because the metascore was somewhere in the "mixed reception" category. It's training some consumers to look straight at the final score, disregarding the editorial commentary that reviewers spend a lot of time writing. It's training some publishers that metascores are the sole indication of a game's success. When clearly, there are several fundamental flaws with metascores. Consider me annoyed. 


  1. But 90% of these issues are not metacritic driven, they are simply issues with rating a game with a single score.

    All in all it is nice for gamers to have a single number to turn to to see if a game is worth while even is this a very crude approximation of a games worth.

    But if you know this then it can be useful. And Metacritic used right is fabulous. You can check out a game that you think is cool looking and if you see a low score you can click on the review that gave it the highest score and read the review.

    Often that is what I do, read the review that scored the highest. Because people often have biases, sometimes games are buggy on some peoples computers, and sometimes they release patches and latter reviews are higher because of this, but if the reviewer is any good s/he will only score it high if the game is actually good. And if I have already determined that I love the concept and the genre then the most accurate review for me will be from someone else who agrees with both these sentiments (and he/she will likely score it higher then the others).

    It is all the perspective. When I was a gamer scores were the be all end all and a site without them was less likely to get my attention. But as a reviewer it is the exact opposite. None of us like attacking a number to a game and I have personally seen with my own reviews that converting scores (be then letters, stars, or numbers) does not work out as cleanly as you would expect.

    1. "But 90% of these issues are not metacritic driven, they are simply issues with rating a game with a single score."

      Which is precisely the underlying construct of Metacritic -- collating reviews in order to give games a singular, ultimate score. As far as I know, Metacritic was the first to do this with video game reviews, and considering how much weight it carries within the industry, I think it's safe to say that "issues with rating a game with a single score" are specifically driven by Metacritic.

      It should go without argument that there are inherent flaws with the way Metacritic aggregates reviews, which makes their metascores inaccurate and unreliable. This is problematic in and of itself, and Metacritic should really find a way to address these issues -- when you consider the impact Metacritic has on the industry, you'd think they would want their own system to be as robust as possible.

      To be fair, the problem isn't really with Metacritic, it's with the CEOs and consumers who rely on it as their sole benchmark for success. You'd think people would be reasonable enough to understand that there are more important criteria for a game's success than one, single number. It breaks my heart to see a beloved developer like Obsidian (who actually knows what they're doing when it comes to RPGs) not earn their bonus because of one stupid hundredth-of-a-point. This being after they'd already earned millions of dollars for Bethesda.

      But Metacritic does enable this kind of mentality, since its primary intention is to assign a singular, ultimate judgment on games. And when so many people rely on a flawed system to make their financial decisions, there are only two things you can do: fix the flaws in the system, or get people to stop using it.

      Metacritic can be a great tool for sensible consumers. I've consulted Metacritic numerous times, in order to get a general overview of a game's reception or to see a wide array of reviews all in one location. I take everything with a grain of salt, and it helps me make more informed decisions. Plus it just saves me time finding a general consensus when I'm only vaguely interested in a game.

      But this is where Metacritic is the most dangerous -- if I have only a casual, passing interest in something, and I see a low metascore, I'm not going to want to spend 30-60 minutes of my time scouring reviews, investigating every detail to find out why the consensus is so low. I'll probably just pass on it until I find it dirt cheap in a sale or a bargain bin.

      If I were looking to buy Risen 2 in this manner, I'd get the impression it wasn't worth the cost, because nothing on the Metacritic page would have informed me of the recent patch addressing all of the major problems which resulted in its low score in the first place. The "final consensus" of 47 reviews, the ultimate rating for the game's worth, doesn't reflect the final product that I'd be buying.

      I don't think people are going to stop using Metacritic any time soon, but if we're going to use it, and as long as Metacritic's intention is to assign ultimate, final judgment on games (they rank these things, after all), we may as well improve the system to make it as robust and informative as possible.