Everybody else on the internet seems to be talking about review scores, what with the furor over some outlets giving Killzone 2 a less-than-perfect score. I see no reason for us to be different.
So, this week’s Destructoid Discusses is on the value of review scores. Are they important? What importance do they have in our buying decisions? How do you respond when a game gets a review other than what you might have assigned or expected?
We’re joined this week by tsunamikitsune and we managed to wrangle a pretty good conversation, if I say so myself. Hit the jump and you’ll see what I mean. And make sure to contribute your thoughts in the comments.
Conrad Zimmerman: Let’s try not to beat a dead horse on how the number should not be the most important factor and that people should actually read the text of a review rather than skip to the number at the end. I think it’s safe to say this will never happen.
I’m more interested to know what effect it actually has on you as a consumer to see that a game received a seven vs. an eight? How much does anticipation of a title color your judgment on whether a game is scored fairly?
Jim Sterling: I think the number depends on where it’s come from. An IGN 7 is not the same as a Destructoid 7. This is the reason why our reviews tend to be lower on the Metacritic scale, because we invest a lot more value into each of our numbers. For us, a 7 is good, and well worth a purchase for a significant number of people. I only get that impression from Destructoid reviews and Edge reviews, however. Other places seem to have completely devalued anything below an 8.
Dale North: Even more specifically, it depends on which reviewer reviews that game. The flat number from XXX outlet means nothing to me without knowing who wrote it. But I know exactly what a Colette Bennett “8” means, or what a Jim Sterling “5” means.
As a consumer, review scores have no affect on me as solitary ratings. Together, amassed on a site like Metacritic, they give me a general idea. I love to pull up the site on my phone while I’m out shopping for games. But then again, a overall 50% isn’t going to stop me from buying a game either.
Jim Sterling: I think knowing the personality of the review is helpful with reviews, and one of the benefits we have of being a blog of individuals as opposed to a collective voice. As you say, you know what a review from Colette or me means. People with the same tastes as our writers can use that to better help them decide if a game is worth looking into. Same goes for people you tend to disagree with.
Jordan Devore: I think knowing the personality of the review is helpful with reviews, and one of the benefits we have of being a blog of individuals as opposed to a collective voice. As you say, you know what a review from Colette or me means. People with the same tastes as our writers can use that to better help them decide if a game is worth looking into. Same goes for people you tend to disagree with.
Brad Nicholson: I’m a cautious buyer and I always will be. Review scores don’t mean a lot to me, but that’s because I actually read the reviews. The difference between a seven and an eight can be vast from Web site to Web site. Some sites aggregate scores, others assign them flimsily, others (like us) assign meaning behind the number.
As a consumer, I think a seven means that the game has a glaring flaw that I should consider before purchase. The flaw doesn’t have to be detrimental to the game (and it probably isn’t considering the game got a seven) it’s just there and noticeable. Generally, whenever I assign a seven it’s because the game is too short, the story poor, or a specific mechanic doesn’t work as much as I would like it to. With an eight, I think you’re dealing with a game that has a series of negligible flaws that keep it from reaching a certain level of perfection.
It’s all subjective and I need to take that into account as a reader. I need to familiarize myself with the reviewer’s other work. Review scores don’t help the decision making process — it’s the writing. And that’s something I’ve been pounding into heads since I started writing reviews.
Colette Bennett: I utterly ignore review scores. It isn’t that I don’t care what a reviewer has to say so much as I know that no matter what the general consensus is, it’s likely I will not wholly agree with it. Every once in a while if a game I am interested in gets overwhelming low scores across the board, I’ll take it into account, but it may not prevent me from buying the game. I’ve loved a lot of games that have been critical “flops” (Earth Defense Force FTW!).
That being said, if I do happen to read a review I tend not to react, even if the reviewer is harsh about something I liked a lot. I am able to separate a criticism about something I liked from a criticism directly aimed at me.
Jim Sterling: Welcome to seemingly 3% of the gaming population.
Colette Bennett: Happy to be here, Sir. 🙂
Jonathan Holmes: Yeah, I love reading reviews for the info, but the scores (and the opinion or the reviewers) are usually pretty meaningless to me. I’ve read a lot of 10/10 reviews that actually helped me to know that, even though the game is technically well crafted, that it’s just not for me.
I think the IGN review of Gears (a 10/10 review) actually said something like “The plot is stupid and the gameplay gets really mindless and repetitive, but the kick ass explosions and non-stop brown-ness make this game a must have!”
That was very helpful information.
tsunamikitsune: For the longest time I would use IGN for my reviews, but as I got to playing some of the supposedly crappy games (See: God Hand), I realized that I can’t always skip straight to the number score in order to decide my purchases. I continue to use them to get a rough idea of what a game is like when it comes out (assuming Destructoid hasn’t covered it already). I don’t put so much trust in the scores anymore, especially since I realized that many different people write the reviews for IGN, making it difficult for scores to be compared with one another. I agree with Jim that knowing the personality is important, as I feel much more educated when I read a review by someone who has had experience with the type of game they’re reviewing.
I usually read reviews simply to gather even more information on games I’m already hyped about, so the scores do little to change my opinion in most cases. Honestly, I think scores are pretty worthless aside from giving people something to argue about. Why read a wall of text when you can just get a bolded numeric summary at the end? Maybe reviewers should make the readers work for it by hiding the number somewhere in the bulk of the review. That’ll teach them the value of reading.
I must admit, however, that I get a sick pleasure out of skimming down to the score for a crap Wii game when I see them reviewed on Dtoid. A quick laugh at a horrid game always gets my day off to a good start.
Niero: I’d like to chime in on the business of review scores. Even though it’s obvious that the general public doesn’t give a shit as much as some core gamers might, the publishers definitely do. Some even have payout structures based on how Metacritic scores.
Destructoid has had ad campaigns pulled because of our review scores. Nobody’s ever said to me, hey, pull that number or we’ll pull the ads because we work though ad networks that do that stuff for us. When I see a hard pause on a campaign the day after a review when up and ask what happened, they’ll say the client had a change of heart. And you know what? They have the right. They don’t want to look like fools putting ads on a site that hated their game. It’s a waste of their money. I’m not going to cry on their lap or demand that they appreciate our editorial integrity — we don’t want to be a high maintenance bitch, either. They don’t hire media buyers who’s job requirements call for that.
A respected person in the game industry once said to me “Why do you guys do review scores? You’re just hurting your advertising revenue and not getting any traffic for it.” Another guy who used to work for us would often beat me up about our 10 point system — why publish numbers at all? My answer to both questions was simple: “Because enough of the readers appreciate it.” When Destructoid committed to publishing reviews we committed to becoming part of that ecosystem.
So, this is a burden the editors bear — something we all believe strongly enough to lose money over and put us in less than favorable relationships with some of the publishers we love the most. And that’s the reality of that number; the weight of that number is worth real money and real problems. If you read anyone in the industry who doesn’t publish a review number, this is the real reason. They’re being pussies — and better businessmen than me. And that’s fine. That’s why we’re Destructoid and they’re not.
Jim Sterling: And Niero illustrates, as he so often does, why I am proud to serve as his reviews editor.
Jonathan Holmes: For me, adding the score to a review is by far the worst part of the process of doing a review. I’d love to be able to skip it.
I’d also love for people to be able to read my mind without me having to type a single word, but that’s not possible. If you want to communicate what you think, you need to use the language, and that’s what review scores are; the language of reviews.
tsunamikitsune: I can’t imagine how awful it is for you guys to try and assign a number score to your reviews. It seems like it takes a lot of thought that most sites spend little time on (which is evident when you read a review that hates on every aspect of the game, then see that the score is much higher than the review makes it out to be).
I think most sites decide their review scores by wearing a blindfold and throwing a dart at a dartboard. It may not be as accurate, but it’s probably a lot more fun.
Jonathan Holmes: That’s being optimistic, Kit.
Honestly, I think a lot of sites (and moreso magazines) assign their review scores based on what people want to hear. Gaming magazines and websites feed off of the excitement of their readers. Financially, they couldn’t exist without that excitement. More so, magazines and websites need to make their readers happy in order to stay in business.
Sadly, giving a game 10/10 review scores make people happy. So do 3/10 scores for games that people want to hate.
Jim Sterling: I’ve tried to make it so assigning a score is as painless as possible. All you have to do is line up the tone of your review with the summary of each number. Of course, that’s not an exact and simple science, but I’ve found it’s taken a lot of thought out of the number process. One shouldn’t be wasting time agonizing over slapping an arbitrary numerical figure to a review.
I personally feel scores do have an important place in the overall review process, but if you spend too long deliberating on it, you’re doing something wrong. Your text should tell you what the score is.
Jonathan Ross: Scores are a tough thing. I personally hate giving them, but at the same time I like reading them on other sites. I don’t use them as the sole deciding factor in buying shit, but a lot of times they’re helpful if I’m deciding between two games. At the same time though, I know that a lot of them are arbitrary and that everyone uses a different scale anyway, but that doesn’t seem to stop me from taking them into account. I do overall tend to find the review text overall much more helpful, but I HATE when certain sites just copy/paste the entire review text and score for the PC/360/PS3 versions of the game.
Jonathan Holmes: Oh, it’s not your fault Jim, or the review system’s. My issue with placing a score on a review is totally my own.
At times in my life when I had to put a number or letter grade on someone else’s writing or artwork, I always had a tough time with the process. There is so much subjectivity involved, so many factors to keep in mind.
Sometimes it’s easier than others. I had no problem giving Mega Man 9 a 9/10, but some of those Strong Bad games, which were all about two very subjective things (comedy and puzzles), were really hard to assign a score to.
Tom Fronczak: I think a big piece of the rage behind review scores, even if it’s subconsciously, is how far the industry has come financially. When you walk into a store, you know every game on the shelf spent millions bringing their idea from concept to completion. I can see a lot of people walking in and saying to themselves “Fuck yeah! This company just spent ten million making this game over two years and it’s going to be amazing. I’m going to be blown away.” In some ways we want every game we buy to be a 10/10. So it’s not so much that a 8/10 is bad, it’s the idea that, if the game company spent that much money making it, then why isn’t it polished and perfect? If it’s not, then why do we work so hard at our daily jobs when there’s much less money involved? It sucks that every game on that shelf is $60 bucks, but only maybe only two of them will be a good game — so why do you have to pay the same for all of them? Even though it’s a ridiculous and unrealistic frame of mind, I’m pretty sure a lot of people look at a 6/10 game as 40% of its budget was mishandled. There’s a grain of truth to that, because consumers should always demand greatness no matter what, but pieces of art tend to defy normal logic.
Rev wrote a great article a while ago about the 10/10 game review, and how no game or movie or other piece of art is ever a 10, that it’s a mythic unattainable beast with no flaws. It had a lot of great points in it that I agree with, but no matter how much support those arguments have, I have to admit I still really like numbered reviews. I think Jim and Dale hit it on the head — one person’s 9 is not another reviewer’s 9. You need to trudge through a lot of crappy or okay reviews and reviewers and find a few people with the same mindset as you. Someone who is a similar type of gamer, someone who looks for the same things as you in games, and someone with the same set of expectations and demands from the gaming industry.
That’s why I love our extra review staple. At the end of the day, most gamers just want to know if they should rent the game, buy it, completely avoid it, or if they should expect the game to win awards. It’s a good way to cut out all artistic interpretation and put your foot down on exactly how you feel about the game you just played.
Dyson: I just look at the numbers of the review. If the number is higher or lower than what I would have given the game, then I’ll read the review. Or if I’m totally bored. I know whether or not I’m going to get a game long before So-and-so Jones from Nowhereville post his finely crafted 1100 word opus. Making me read that junk just find out the the person gave it the same score I would have is a waste of my time. There are no less than a billion people out there chomping on the bit trying to show the world what their idea of what a good review for a game is, and they all miss their mark horribly.
Dale North: First off, OMG tusnamikitsune — I was out to lunch earlier, thinking about this topic — thinking about the countless examples of how review scores did not reflect the “score” for me. God Hand came to mind. So high five on that.
And, back to the discussion, as far as Destructoid review scores goes, Jim has made it really easy to drop a number on a review. As a member of Dtoid’s review crew, I used to hate numbering these things. Now it’s no big deal at all. Each number actually means something here. I can go back and look at a number that any of us have given for a game and feel really good that that number means something… to us, at least.
tsunamikitsune: High five indeed, Dale. God Hand is pretty high up there on my list of favorite games, so I’m baffled by IGN’s review. How can any game that lets you kick a guy in the nuts get such a low score?
The Buy it/Rent it/Avoid it scale is amazing. That combined with the knowledge of the reviewer’s taste usually gives me a pretty good idea about how much I might like the game in question.
Now that I think about it, scores do have a bit more value than I originally gave them credit for. When I scroll down to the bottom of a review and notice a high score for some game I’ve never heard of or previously didn’t care about, I usually find my way back to the top of the page, curious about what exactly made this game so great. On these occasions, I actually read the review instead of just glancing at the score. This worked for Lock’s Quest and CarnyVale Showtime, both of which I hadn’t heard of until the Destructoid reviews. They’re both great games that I never would have noticed if not for the high scores they were given.
Speaking of CarnyVale Showtime, playable demos are great after my interest has been piqued by a good review (or even a good score). The trial certainly certainly helped when I was on the fence about Fireplace.
Joseph Leray: Once again, I think Tron and I are on the same page here (for the most part). Not that I do a lot of reviewing around here, but I *hate* assigning a number to something. I just wrapped up a review for a game that, while technically pretty unremarkable, I had a lot of fun playing. The games mechanics are really solid, but it’s executed rather poorly. Most sites have roundly panned this game (and I can see why), but I found certain parts of it incredibly fun. Trying to find a score that accurately reflects, on the one hand, the fun I had with it and, on the other, it’s technical and mechanical short comings was kind of tricky.
Where I disagree with Tron, though, is when it comes to what reviews are good for. Yes, I read reviews for information, but also to see how the game makes people feel. I don’t want to read about what the controls do, or the minutiae of an RPG’s battle system — I want to know if it makes me feel good, if it’s fun. I don’t care if the textures are all bland — I want to know if it adds to or detracts from the experience of the game.
Jonathan Holmes: Oh, I agree with you 100% Joseph, if it weren’t for the fact that I just can’t trust that most reviewers have taste anything like my own.
I’ve probably over used this example at this point, but after Game Informer gave No More Heroes a 6.5 for being too “sexual and violent”, despite the fact that they’d later make GTA IV their game of the year, I pretty much gave up on taking anything away from a reviewer’s opinion of a game.
Chad Concelmo: I give this Dtoid Discussion a 9.3 out of 10! <3
Jonathan Holmes: YOU ARE BIAS MAN
Niero: Fanboy.
Dale North: Replay value is shit.
Published: Feb 11, 2009 04:00 pm