GDC Rants - a (belated) response

May 25, 2009

GDC came and went ages ago now, but I only recently got round to watching the Rants panel that had everyone arguing so vehemently a while back. I've managed to track down videos of a few of them, though there are a couple I haven't found that I'd really like to see. If anyone knows where I can find Chaplin's or Croal's, I'd really appreciate that. For now, I'll do my best to respond to those ones based on what I've read about them. The rest will be more direct. Would be interested to hear others' thoughts on the matters tackled.

***

HEATHER CHAPLIN
"Games aren't adolescent; you're a bunch of fucking adolescents."

Heather compared her audience, a room full of primarily male game developers, to a chiuaua: through the concept of neotony, she suggested developers were stuck in the image of a juvenile version of their adult selves, as chiuauas resemble foetal wolves. I'm amazed at the guts Heather displayed in doing this. That definitely takes some courage. And I agree with her concerns: that games are stuck too rigidly in the realms of adolescent fantasy. But her argument is severely flawed, and attacking developers on a personal level seems misplaced.

She points to rock and roll as having matured after 35 years, to the point where it had given us The Beatles, Bob Dylan and The Clash. But music had been around for centuries, and exists in a much wider cultural bubble than games. She points to film having Citizen Kane around this time. But we totally have our own cornerstone of the medium.

As for her astonishingly generalised comment that "men are afraid to show emotions or intimacy," I just can't see that in any way other than an unnecessarily personal attack that sticks out as gender-misrepresentation from the ugly side of feminism. Utter nonsense. There are better ways than this to go about encouraging more intelligent games.

N'GAI CROAL
"Who's more hardcore: the person who plays Peggle sixteen hours a day, or the person who plays Halo once a week?"

N'Gai expressed his distaste at the terms "hardcore" and "casual", and how they act as an avoidance of being more analytical and descriptive of genres and types of play. It's a very specific topic to rant on, and I don't really have much of a response to it, other than to say he's obviously totally right.

STEPHEN TOTILO
"Our reporting is fine; it's our writing that sucks."

Totilo responded to the numerous accusations that games journalism suffers from poor reporting, and swung it round to the idea that it suffers from poor writing. I agree wholeheartedly. His main point was that we're not being concise and confident enough, unnecessarily hedging our opinions and utilising words that we don't need. "The word 'compelling' doesn't tell us anything," Totilo said; "Nobody knows what 'visceral' means; what's the difference between being 'engrossed' in a game and being 'completely engrossed' in it?"

I'm certainly in favour of chopping out the babble from games journalism, and I'm making a conscious effort to be more concise in my own writing. However, I'd want people to be careful not to excise the important qualities of their own personal style. The joy of long-form journalism is that such styles can seep through and engage the readership, something that more concise writing sometimes struggles to do. It's not impossible, but it's a tricky manouevre.

LEIGH ALEXANDER
"The negativity between PR and press has a trickle-down effect to our audience, which is not a healthy thing for the industry."

A usually beautifully articulate Leigh rambled slightly incoherently (although admitted to being hungover and apologised for her behaviour the previous night) about what she called "the three-way ecosystem of negativity" between development, journalism and audience. She pointed out that PRs refusing to reveal specifics of their work to the press leads to factual inaccuracies and extensive speculation, which in turn leads to audience disappointment and a dislike of journalists.

She's spot-on with it, but I was disappointed by the lack of weight to her rant. Of course we'd like everyone to be more honest and friendly with each other. That's kind of a given, and her talk felt somewhat like a plea for the obvious to be fixed. Which is fair enough, I guess.

CHRIS HECKER
"Your reporting impacts people, both personally and professionally, and with this power comes great responsibility."

I really enjoyed Hecker's talk. It focused more on the reporting side of games journalism, and contrasted starkly with Totilo's speech. His argument was one of journalistic integrity, and a plea for fact-checking in news journalism. Chris pointed to an article written about him on 1Up, which had been extrapolated from a speculative forum post about the nature of his involvement with Spore's development. The headline, however, referred not to a forum post, but to an article, meaning the negative comments against him held far more weight.

In the piece, 1Up stated that they would attempt to contact Hecker for his side of the story. They never did. "I've had the same phone number for 12 years," he said. "I'm not difficult to find."

I remember reading this post on 1Up and thinking, well, that's just not news. It's the sort of nonsense that N4G will post about, not a major and respected blog. Pure, forumite speculation may demand some attention if it's interesting enough, but to imply in the headline that it's a professional article is awful, and I'd hope to not see this sort of stuff repeated.

As a lighthearted goodbye, Hecker continued the trend of IGN-bating started by Totilo, by showing a slide with the following quote:

"'There's a tendency among the press to attribute the creation of a game to a single person,' says Warren Spector, creator of Thief and Deus Ex."

IGN, I hope that was ironic. I very much doubt it was.

ADAM SESSLER
"It's no secret that Metacritic is being used as a means of withholding renumeration from you guys out there, and that's bullshit."

Given the vitriol displayed in the Sessler's Soapbox series, it wasn't surprising that Adam was one of the more venemous and interesting speakers of the rant session. He firstly expressed anger at Metacritic's insistance in "translating" other scoring systems to fit their own /100 mark scheme. Upon receiving a complaint from a publisher about G4TV's "40%" score of their game, one that had actually received 2/5, Sessler had contacted Metacritic. "I explained that's not how we do things," said Adam. "I said there's a reason why we have a one-to-five points system, and it's not like everything's equatable in that. They said: 'No, you're wrong.' It's my scale!"

As a quick aside, Sessler pointed out that he didn't understand the concept of a hundred-point scale. "Someone tell me the difference between 73 and 74," he pleaded. Here's a clue for you, Adam: one of the numbers is higher. There's a common assumption that marking on a percentage basis means you're looking for something quantifiably different between two games that score similar, but different, percentages. That's not the case. It's not a scientific gauge. It's an impression of the overall effect of the experience. And though one writer's 68 might be another's 75, it's sometimes a valuable shorthand for expressing something's ever-so-slight superiority over another product - in the opinion of that writer.

Sessler eventually got onto his main point: Metacritic's use from a publisher's perspective as a gague for delivering or withholding bonuses for developers. Which is something I think we can all agree on.

People have said, "stop using scores, then." I don't think that sticks. Our readership demands them, and ultimately, those are the only people we have to answer to. I'm also a little undecided about the merits of Metacritic. In a way, it's a really valuable tool to have, and potentially rather valuable for growing publications to get their name about, should they be accepted into the listings. I also have no problem with scores being collated on a single database for people to browse at their leisure, nor with Metacritic's decision to claim a game has been "universally acclaimed" or received "mixed or average reviews." That's fine. But I agree that translating scores and assuming what they would be equivalent to on a hundred-point scale is maddening. Worse is their decision to take reviews without scores and simply make shit up arbitrarily. That's not acceptable at all. And it needs to stop. But I don't think the concept of a meta reviews site is bad at all. They just need to lose the arrogance.

Most recent blog posts from Lewis Denby...

One A Day

My Top 50 Games of the Ever

BREVITY OR BUST: Stalin vs. Martians

Most recent blog posts from Lewis Denby...
One A Day My Top 50 Games of the Ever BREVITY OR BUST: Stalin vs. Martians

Feedback

honestgamer - May 25, 2009 (10:08 AM)
Conversion to one scale is one of the only reasons that meta sites work, so it's pretty funny to see people asking that meta sites continue to perform their function of gathering reviews together as those people in the same breath try to tell meta sites to stop score conversion.

I don't have much good to say about Metacritic on the subject, though, primarily because the site chooses not to list HonestGamers content and has a policy of not explaining why so that any issues with HonestGamers content and/or its presentation--if they exist--can't be rectified. That feels like censorship, somehow, and I have a hard time believing that HonestGamers is the only worthwhile site that has been excluded from MetaCritic for reasons unknown.

I don't really have a lot to say on the other points, since mostly I agree with them as summarized above and I'm too lazy/busy to check their original context. I guess I should note also that I really don't give a rat's ass if video games are often just adolescent fantasies turned interactive. I hope that a healthy portion of future games always are because let's face it: those are the ones that tend to be the most fun to play! Games may have potential to be art, to tell cinematic stories and all that, but it's important to remember that by their very definition, they're games first and foremost.

zigfried - May 25, 2009 (10:17 AM)
Who's more hardcore: the person who plays Peggle sixteen hours a day, or the person who plays Halo once a week?"

Possibly the person who plays Halo once a week. It's hard to say without hearing the context around this comment. I can understand frustration with the shifting borders of the "hardcore" and "casual" labels, as well as their frequent use as an insult (which could be towards either the audience or the game), but they're not meaningless labels.

When used properly, those words could be an attempt to quickly summarize breadth of appeal. This doesn't have to be related to the game's challenge. People have their own conception of where they fall on the casual<--->hardcore spectrum; if someone says a game is "for hardcore FPS hounds only", even though it's an overused trope, I know that they're saying the game's not for me, because of the limits to its appeal. I would certainly hope to find supporting evidence throughout the review, but it's a decent way to clue me in on what information I should watch for. And, should the review not support that statement, the writer has kindly told me that I should go read other peoples' reviews instead ;)

The two words distinguish people who derive pleasure from the "art/legacy/what-have-you" of gaming from people who play games to kill time. Someone who plays Peggle 16 hours a day isn't a "hardcore gamer" if they're just really bored and killing time.

Taking use of the label in a different direction... "Casual" action gamers won't understand or appreciate why the new Bionic Commando is a travesty, because their level of "personally caring about games" maxed out at three or four. They will want to know how long/challenging/etc the game is (some casual players will want easy, some will want hard -- casual doesn't mean "likes easy games", after all). Basically, they will want to know how effective it is at not being boring. But people who have invested themselves into gaming -- probably way too much -- are likely to be offended by events in BC. Hardcore gamers will want to know about its role in gaming culture.

So I guess I'm saying the labels aren't defined by what you play or how often you play, so much as how and why you play. The good news is that writers don't have to explicitly understand this for the word to still carry meaning because, on a subconscious level, I think everyone gets it. It's just that some articles do a better job of supporting the label than others.

WATCH FOR MORE COMMENTS LATER

//Zig

zigfried - May 25, 2009 (10:37 AM)
His main point was that we're not being concise and confident enough, unnecessarily hedging our opinions and utilising words that we don't need. "The word 'compelling' doesn't tell us anything," Totilo said; "Nobody knows what 'visceral' means; what's the difference between being 'engrossed' in a game and being 'completely engrossed' in it?"

I'm fine with his general thought regarding the need to improve writing, but his specific examples are misguided (unless there's something in context that I'm missing -- not really sure where to go to read the full text).

Saying that a game is "compelling" does mean something. It's a great "marker" to use in writing. Say the game is compelling -- get the reader's mind primed for the concept! -- then use examples and analysis to show why it's compelling. Writers need concept words like "compelling/immersive/engrossing" to establish structure and maintain focus.

I agree that it's pretty useless to just throw out a statement like "this game is compelling" without any support, although even that clues readers in that at least one person (the writer) was able to get emotionally involved in the game's events. If that's what the writer wanted to convey, then hey, that's fine. But if someone is hinging huge portions of "why this game is good" on an unsupported assessment of compulsion, then that's pretty sloppy.

And "visceral" is totally a cool word that carries a ton of meaning behind it. It's concise, and that's a good thing.

//Zig

Lewis - May 25, 2009 (10:55 AM)
Stephen Totilo's talk can be viewed in full on Youtube if you fancy more context. Follow links to a few others from there as well.

EDIT: By the way, the highlighted text on the Destructoid review that everyone laughs at reads: "...though technically a squid is not an animal."

zigfried - May 25, 2009 (10:58 AM)
Not much to say about the other bits, except that I agree with Honestgamer. Score conversion is a necessity for aggregate sites to work. On the pass/fail scale, 2 out of 5 is a fail, so 40% is a fair value. 50% would also be fair -- and to the jilted developer/publisher, they're essentially the same.

How would Sessler prefer "2 out of 5" be converted? With that response in hand, do other 5 point sites agree with that -- can a consensus be reached? Chances are "no". There's not even a consensus between what 75 and 85 mean, so the score-compiling site is as capable as anyone at determining their way to compile values.

"I explained that's not how we do things," said Adam. "I said there's a reason why we have a one-to-five points system, and it's not like everything's equatable in that. They said: 'No, you're wrong.' It's my scale!"

The conversion is Metacritic's scale. They know best how they've converted the plethora of score systems, so they know best how to convert his scale to fit those other systems. It may be true that a 2 on his scale isn't directly equatable to a "40%" (probably represents a range from 40-ish to 60-ish) but it has to be systematically converted into something. If he has issues with the conversion, then he needs to be contacting the other 5-point sites to make a joint appeal. If he doesn't want it to be converted at all, then that's egotistical.

//Zig

honestgamer - May 25, 2009 (12:24 PM)
Sessler is nothing if not egotistical. I don't say that with malice, either. He's just full of himself and not ashamed of it, something that endears some to him and sends others running for the hills. I'm probably more from the latter category because his style tends toward trollish and I dislike trolls, but I understand and appreciate that he at least has genuine passion.

zigfried - May 25, 2009 (12:41 PM)
I originally wrote "silly and egotistical", but then I changed it because being egotistical isn't necessarily silly. I'm not really familiar with him, but in this case, egotism doesn't endear me to his cause because it gets in the way of practicality.

//Zig

Lewis - May 25, 2009 (03:34 PM)
As Jason says, whatever your opinions of Sessler, he certainly has a distinctive style and passionate delivery that makes him a force to be reckoned with. There's a reason why he's one of the most successful in his trade. Think of him as the Jeremy Clarkson of games journalism, if you will.

I don't think Meta sites need to convert scoring systems at all. We're obsessed with the average. What's wrong with a simple collated list of all reviews that have appeared in the press? Are we unable to scroll down and look at individual marks or - goodness - read the reviews? It's worth noting that the "average" isn't even objectively calculated: it's perhaps not a widely known fact that Metacritic doesn't include all the reviews it cites in calculating the overall score. It prioritises certain publications. It will not reveal which ones, or why it considers those sources more valuable.

What I adore about Sessler - and, to be fair, each and every critic on the panel - is that not only does he love games, he's passionate about his craft as a journalist, and all too happy to speak up about his views on the gaming press. Too many people shy away from this. As Chris Hecker said, it's a job that carries responsibility, and critics always need to be striving to report in the best ways possible.

When he says there's a reason he uses a five-point scale, I think the implication behind that is he chose it for G4TV because he didn't want it to be used on sites like Metacritic. I didn't quote this, but he also said in his rant that Metacritic continued to link to the station's reviews without his permission - which is pretty bad. Of course, you could ask why he doesn't just scrap scores all together, but then as long as Metacritic continues to make them up for those who don't provide scores - which is completely awful and, as Sessler pointed out, "removes our authorship over our work." I think at the core of his anger is that there's this huge site saying "Adam Sessler gave this game 40%" - when actually, he did nothing of the sort.

Of course, his main argument is that publishers should not be using Metacritic - a deeply flawed system, objectively - to decide whether or not their developers did good or not. And I totally, totally agree on that point.

(Random aside: of all the talks, Sessler's went down by far the best with the audience. His opening line of "Fuck Metacritic" resulted in the roof being half-blown-off by rapturous cheers.)

zigfried - May 25, 2009 (04:02 PM)
Regarding making scores up, that is absolutely awful, but this is also the first I've heard of that. I guess I'd like to see examples, because I thought that removing scores was a way to get taken off the site (there was a PC gaming site that did some such thing and was actually sad about no longer being on Metacritic because of it). So perhaps Metacritic has changed policies since then. If so, it's a change for the worse. I agree that making up a score is like putting words in someone's mouth. But I don't see assigning 40% towards a 2 out of 5 as making a score up. I see reading a review and then saying "hm, this sounds like a 55" as making a score up.

Regarding whether developers should be judged by Metacritic scores ---

How should they be judged? They could be judged by sales, but using Metacritic scores actually provides some lenience. "Your game didn't sell well, but the critics liked it, so you must have talent. We'll give you another project and hope for better sales."

Publishers need to judge their developers somehow. They'll either do it by sales, or they'll do it by reviews. Or, more likely, they'll use both methods. If a game sells well, then the audience likes it. If it scores well, then the critics like it. But if it sells badly and scores poorly, then really, who did the game please?

Certainly not the publisher, who put forth money hoping for a viable product.

//Zig

PS- the above is an "in theory" response. Since HG is not on Metacritic, that site is obviously corrupt and unreliable. ;)

PPS- Gamerankings' system seems to work pretty well for an aggregate site. They convert scores to come up with an average, but each individual publication's score is represented as they wrote it. That way, they perform the "one score" function without making any individual writer look bad.

honestgamer - May 25, 2009 (04:43 PM)
As long as a review has a score, readers are going to do the math and convert it anyway, possibly even going so far as to do the math wrong because they're not bright enough to figure out what a letter grade means or because such things are ambiguous.

We can talk all we like about what the scores on a 5-point scale really mean and we can post explanations of our rating system--something that we do on HonestGamers--but the fact remains that no matter how much of that info we provide, the reader is still going to make up his own mind.

In the case of Sessler's rating system, most peole will see a 2/5 as 40%. That's how the fraction converts. A 1/5 is a 20% and so forth. If that's not how he wanted them converted, then he should have chosen a rating system that stated how he wanted the review ratings considered... or he should have gone the ballsy route and not included ratings at all.

Of course he did include them because readers want them. So he has to play by the rules, whether he likes the rules or not, just like the rest of us. I know that he has contempt for ratings and for the fanboy response to ratings from a video I watched of him ranting in regards to the reception to his Killzone 2 review.

However, Metacritic has every right to link to his reviews--which are out there for public consumption. I agree that the GameRankings system of indicating what score the site gave is the best (which is why we use a similar method for our Meta section), though, since the reader needs to be clear on whatever actual rating a site has given the game. Providing that information and then an aggregate rating is perfectly fair.

Lewis - May 25, 2009 (05:12 PM)
Zig: not a game, but first search term I enter into Metacritic, and the top review is stated to have scored 100%. Here is a link to the article. No score anywhere in sight. Utterly disgraceful.

Also, his response to the "how do you judge developers" question was this:

"We don't want you to make a game for us. We want you to make a good game you want to make. Sometimes we'll like it, sometimes we won't, but you wanna know if a game is good or not? There's this really, really effective tool to do that: the market! It being sold! Does it sell or not? And you know what: if it's a good game, and you know it's a good game, but it doesn't sell? Well, go talk to your marketing team. Don't come to us."

Jason: I agree with you on the whole there, and it's something I've heard quite a few times in response to Sessler's rant. If everyone stops awarding scores, Metacritic ceases to exist: it only gets away with it because the non-scoring publications are in the minority. Sessler is - I believe - reviews editor of G4TV. It's his choice what scoring system, if any, he uses - and he should have the balls to do what he wants. The argument in his rant - "I hate awarding scores, but that's just what we have to do" - is flawed.

And relating to your earlier point about Metacritic's opaque criteria for listing, Adam agrees:

"[Metacritic] really does "pick and choose" who goes up on that site. It does not respect the wishes of those who might be having their views recalibrated on that site."

Lewis - May 25, 2009 (05:21 PM)
From the MC FAQ. At least they're honest about how corrupt the whole thing is.

This overall score, or METASCORE, is a weighted average of the individual critic scores. Why a weighted average? When selecting our source publications, we noticed that some critics consistently write better (more detailed, more insightful, more articulate) reviews than others. In addition, some critics and/or publications typically have more prestige and weight in the industry than others. To reflect these factors, we have assigned weights to each publication (and, in the case of film, to individual critics as well), thus making some publications count more in the METASCORE calculations than others.

In addition, for our film and music sections, all of the weighted averages are normalized before generating the METASCORE. To put it another way that should be familiar to anyone who has taken an exam in high school or college, all of our movies, games, and CDs are graded on a curve. Thus the METASCORE may be higher or lower than the true weighted average of the individual reviews, thanks to this normalization calculation. Normalization causes the scores to be spread out over a wider range, instead of being clumped together. Generally, higher scores are pushed higher, and lower scores are pushed lower. Unlike in high school, this is a good thing, since it provides more of a distinction between scores and allows you to better compare scores across movies (or CDs).

Many reviewers include some sort of grade for the movie, album, or game they are reviewing, whether it is on a 5-star scale, a 100-point scale, a letter grade, or other mark. However, plenty of other reviewers choose not to do this. Hey, that's great... they want you to actually read their review rather than just glance at a number. (Personally, we at Metacritic like to read reviews, which is one of the reasons we include a link to every full review on our site.... we want you to read them too!)

However, this does pose a problem for our METASCORE computations, which are based on numbers, not qualitative concepts like art and emotions. (If only all of life were like that!) Thus, our staff must assign a numeric score, from 0-100, to each review that is not already scored by the critic. Naturally, there is some discretion involved here, and there will be times when you disagree with the score we assigned. However, our staffers have read a lot of reviews--and we mean a lot--and thus through experience are able to maintain consistency both from film to film and from reviewer to reviewer. When you read over 200 reviews from Manohla Dargis, you begin to develop a decent idea about when she's indicating a 90 and when she's indicating an 80.

If you are the critic who wrote the review, and disagree with one of our scores, please let us know and we'll change it.

This does happen from time to time, and many of the critics included on this site (such as Ms. Dargis) do indeed check their reviews (as well as those of their colleagues) on metacritic.com.

I'm sure Adam would contest that last bit.

zigfried - May 25, 2009 (05:30 PM)
"We don't want you to make a game for us. We want you to make a good game you want to make. Sometimes we'll like it, sometimes we won't, but you wanna know if a game is good or not? There's this really, really effective tool to do that: the market! It being sold! Does it sell or not? And you know what: if it's a good game, and you know it's a good game, but it doesn't sell? Well, go talk to your marketing team. Don't come to us."

If I were developing games, I'd rather put my faith in the critics than in the marketing team. I see his point, I just cannot get behind it. I would want my publisher to look at scores, and not just at sales.

PS - thanks for the link, I consider myself schooled in the ways of MC now!

//Zig

Felix_Arabia - May 25, 2009 (11:56 PM)
It's the money-ey-ey-ey-ey-ey.

Oh, it's just too much fun sometimes.

Feedback
	honestgamer - May 25, 2009 (10:08 AM) Conversion to one scale is one of the only reasons that meta sites work, so it's pretty funny to see people asking that meta sites continue to perform their function of gathering reviews together as those people in the same breath try to tell meta sites to stop score conversion. I don't have much good to say about Metacritic on the subject, though, primarily because the site chooses not to list HonestGamers content and has a policy of not explaining why so that any issues with HonestGamers content and/or its presentation--if they exist--can't be rectified. That feels like censorship, somehow, and I have a hard time believing that HonestGamers is the only worthwhile site that has been excluded from MetaCritic for reasons unknown. I don't really have a lot to say on the other points, since mostly I agree with them as summarized above and I'm too lazy/busy to check their original context. I guess I should note also that I really don't give a rat's ass if video games are often just adolescent fantasies turned interactive. I hope that a healthy portion of future games always are because let's face it: those are the ones that tend to be the most fun to play! Games may have potential to be art, to tell cinematic stories and all that, but it's important to remember that by their very definition, they're games first and foremost.
	zigfried - May 25, 2009 (10:17 AM) Who's more hardcore: the person who plays Peggle sixteen hours a day, or the person who plays Halo once a week?" Possibly the person who plays Halo once a week. It's hard to say without hearing the context around this comment. I can understand frustration with the shifting borders of the "hardcore" and "casual" labels, as well as their frequent use as an insult (which could be towards either the audience or the game), but they're not meaningless labels. When used properly, those words could be an attempt to quickly summarize breadth of appeal. This doesn't have to be related to the game's challenge. People have their own conception of where they fall on the casual<--->hardcore spectrum; if someone says a game is "for hardcore FPS hounds only", even though it's an overused trope, I know that they're saying the game's not for me, because of the limits to its appeal. I would certainly hope to find supporting evidence throughout the review, but it's a decent way to clue me in on what information I should watch for. And, should the review not support that statement, the writer has kindly told me that I should go read other peoples' reviews instead ;) The two words distinguish people who derive pleasure from the "art/legacy/what-have-you" of gaming from people who play games to kill time. Someone who plays Peggle 16 hours a day isn't a "hardcore gamer" if they're just really bored and killing time. Taking use of the label in a different direction... "Casual" action gamers won't understand or appreciate why the new Bionic Commando is a travesty, because their level of "personally caring about games" maxed out at three or four. They will want to know how long/challenging/etc the game is (some casual players will want easy, some will want hard -- casual doesn't mean "likes easy games", after all). Basically, they will want to know how effective it is at not being boring. But people who have invested themselves into gaming -- probably way too much -- are likely to be offended by events in BC. Hardcore gamers will want to know about its role in gaming culture. So I guess I'm saying the labels aren't defined by what you play or how often you play, so much as how and why you play. The good news is that writers don't have to explicitly understand this for the word to still carry meaning because, on a subconscious level, I think everyone gets it. It's just that some articles do a better job of supporting the label than others. WATCH FOR MORE COMMENTS LATER //Zig
	zigfried - May 25, 2009 (10:37 AM) His main point was that we're not being concise and confident enough, unnecessarily hedging our opinions and utilising words that we don't need. "The word 'compelling' doesn't tell us anything," Totilo said; "Nobody knows what 'visceral' means; what's the difference between being 'engrossed' in a game and being 'completely engrossed' in it?" I'm fine with his general thought regarding the need to improve writing, but his specific examples are misguided (unless there's something in context that I'm missing -- not really sure where to go to read the full text). Saying that a game is "compelling" does mean something. It's a great "marker" to use in writing. Say the game is compelling -- get the reader's mind primed for the concept! -- then use examples and analysis to show why it's compelling. Writers need concept words like "compelling/immersive/engrossing" to establish structure and maintain focus. I agree that it's pretty useless to just throw out a statement like "this game is compelling" without any support, although even that clues readers in that at least one person (the writer) was able to get emotionally involved in the game's events. If that's what the writer wanted to convey, then hey, that's fine. But if someone is hinging huge portions of "why this game is good" on an unsupported assessment of compulsion, then that's pretty sloppy. And "visceral" is totally a cool word that carries a ton of meaning behind it. It's concise, and that's a good thing. //Zig
	Lewis - May 25, 2009 (10:55 AM) Stephen Totilo's talk can be viewed in full on Youtube if you fancy more context. Follow links to a few others from there as well. EDIT: By the way, the highlighted text on the Destructoid review that everyone laughs at reads: "...though technically a squid is not an animal."
	zigfried - May 25, 2009 (10:58 AM) Not much to say about the other bits, except that I agree with Honestgamer. Score conversion is a necessity for aggregate sites to work. On the pass/fail scale, 2 out of 5 is a fail, so 40% is a fair value. 50% would also be fair -- and to the jilted developer/publisher, they're essentially the same. How would Sessler prefer "2 out of 5" be converted? With that response in hand, do other 5 point sites agree with that -- can a consensus be reached? Chances are "no". There's not even a consensus between what 75 and 85 mean, so the score-compiling site is as capable as anyone at determining their way to compile values. "I explained that's not how we do things," said Adam. "I said there's a reason why we have a one-to-five points system, and it's not like everything's equatable in that. They said: 'No, you're wrong.' It's my scale!" The conversion is Metacritic's scale. They know best how they've converted the plethora of score systems, so they know best how to convert his scale to fit those other systems. It may be true that a 2 on his scale isn't directly equatable to a "40%" (probably represents a range from 40-ish to 60-ish) but it has to be systematically converted into something. If he has issues with the conversion, then he needs to be contacting the other 5-point sites to make a joint appeal. If he doesn't want it to be converted at all, then that's egotistical. //Zig
	honestgamer - May 25, 2009 (12:24 PM) Sessler is nothing if not egotistical. I don't say that with malice, either. He's just full of himself and not ashamed of it, something that endears some to him and sends others running for the hills. I'm probably more from the latter category because his style tends toward trollish and I dislike trolls, but I understand and appreciate that he at least has genuine passion.
	zigfried - May 25, 2009 (12:41 PM) I originally wrote "silly and egotistical", but then I changed it because being egotistical isn't necessarily silly. I'm not really familiar with him, but in this case, egotism doesn't endear me to his cause because it gets in the way of practicality. //Zig
	Lewis - May 25, 2009 (03:34 PM) As Jason says, whatever your opinions of Sessler, he certainly has a distinctive style and passionate delivery that makes him a force to be reckoned with. There's a reason why he's one of the most successful in his trade. Think of him as the Jeremy Clarkson of games journalism, if you will. I don't think Meta sites need to convert scoring systems at all. We're obsessed with the average. What's wrong with a simple collated list of all reviews that have appeared in the press? Are we unable to scroll down and look at individual marks or - goodness - read the reviews? It's worth noting that the "average" isn't even objectively calculated: it's perhaps not a widely known fact that Metacritic doesn't include all the reviews it cites in calculating the overall score. It prioritises certain publications. It will not reveal which ones, or why it considers those sources more valuable. What I adore about Sessler - and, to be fair, each and every critic on the panel - is that not only does he love games, he's passionate about his craft as a journalist, and all too happy to speak up about his views on the gaming press. Too many people shy away from this. As Chris Hecker said, it's a job that carries responsibility, and critics always need to be striving to report in the best ways possible. When he says there's a reason he uses a five-point scale, I think the implication behind that is he chose it for G4TV because he didn't want it to be used on sites like Metacritic. I didn't quote this, but he also said in his rant that Metacritic continued to link to the station's reviews without his permission - which is pretty bad. Of course, you could ask why he doesn't just scrap scores all together, but then as long as Metacritic continues to make them up for those who don't provide scores - which is completely awful and, as Sessler pointed out, "removes our authorship over our work." I think at the core of his anger is that there's this huge site saying "Adam Sessler gave this game 40%" - when actually, he did nothing of the sort. Of course, his main argument is that publishers should not be using Metacritic - a deeply flawed system, objectively - to decide whether or not their developers did good or not. And I totally, totally agree on that point. (Random aside: of all the talks, Sessler's went down by far the best with the audience. His opening line of "Fuck Metacritic" resulted in the roof being half-blown-off by rapturous cheers.)
	zigfried - May 25, 2009 (04:02 PM) Regarding making scores up, that is absolutely awful, but this is also the first I've heard of that. I guess I'd like to see examples, because I thought that removing scores was a way to get taken off the site (there was a PC gaming site that did some such thing and was actually sad about no longer being on Metacritic because of it). So perhaps Metacritic has changed policies since then. If so, it's a change for the worse. I agree that making up a score is like putting words in someone's mouth. But I don't see assigning 40% towards a 2 out of 5 as making a score up. I see reading a review and then saying "hm, this sounds like a 55" as making a score up. Regarding whether developers should be judged by Metacritic scores --- How should they be judged? They could be judged by sales, but using Metacritic scores actually provides some lenience. "Your game didn't sell well, but the critics liked it, so you must have talent. We'll give you another project and hope for better sales." Publishers need to judge their developers somehow. They'll either do it by sales, or they'll do it by reviews. Or, more likely, they'll use both methods. If a game sells well, then the audience likes it. If it scores well, then the critics like it. But if it sells badly and scores poorly, then really, who did the game please? Certainly not the publisher, who put forth money hoping for a viable product. //Zig PS- the above is an "in theory" response. Since HG is not on Metacritic, that site is obviously corrupt and unreliable. ;) PPS- Gamerankings' system seems to work pretty well for an aggregate site. They convert scores to come up with an average, but each individual publication's score is represented as they wrote it. That way, they perform the "one score" function without making any individual writer look bad.
	honestgamer - May 25, 2009 (04:43 PM) As long as a review has a score, readers are going to do the math and convert it anyway, possibly even going so far as to do the math wrong because they're not bright enough to figure out what a letter grade means or because such things are ambiguous. We can talk all we like about what the scores on a 5-point scale really mean and we can post explanations of our rating system--something that we do on HonestGamers--but the fact remains that no matter how much of that info we provide, the reader is still going to make up his own mind. In the case of Sessler's rating system, most peole will see a 2/5 as 40%. That's how the fraction converts. A 1/5 is a 20% and so forth. If that's not how he wanted them converted, then he should have chosen a rating system that stated how he wanted the review ratings considered... or he should have gone the ballsy route and not included ratings at all. Of course he did include them because readers want them. So he has to play by the rules, whether he likes the rules or not, just like the rest of us. I know that he has contempt for ratings and for the fanboy response to ratings from a video I watched of him ranting in regards to the reception to his Killzone 2 review. However, Metacritic has every right to link to his reviews--which are out there for public consumption. I agree that the GameRankings system of indicating what score the site gave is the best (which is why we use a similar method for our Meta section), though, since the reader needs to be clear on whatever actual rating a site has given the game. Providing that information and then an aggregate rating is perfectly fair.
	Lewis - May 25, 2009 (05:12 PM) Zig: not a game, but first search term I enter into Metacritic, and the top review is stated to have scored 100%. Here is a link to the article. No score anywhere in sight. Utterly disgraceful. Also, his response to the "how do you judge developers" question was this: "We don't want you to make a game for us. We want you to make a good game you want to make. Sometimes we'll like it, sometimes we won't, but you wanna know if a game is good or not? There's this really, really effective tool to do that: the market! It being sold! Does it sell or not? And you know what: if it's a good game, and you know it's a good game, but it doesn't sell? Well, go talk to your marketing team. Don't come to us." Jason: I agree with you on the whole there, and it's something I've heard quite a few times in response to Sessler's rant. If everyone stops awarding scores, Metacritic ceases to exist: it only gets away with it because the non-scoring publications are in the minority. Sessler is - I believe - reviews editor of G4TV. It's his choice what scoring system, if any, he uses - and he should have the balls to do what he wants. The argument in his rant - "I hate awarding scores, but that's just what we have to do" - is flawed. And relating to your earlier point about Metacritic's opaque criteria for listing, Adam agrees: "[Metacritic] really does "pick and choose" who goes up on that site. It does not respect the wishes of those who might be having their views recalibrated on that site."
	Lewis - May 25, 2009 (05:21 PM) From the MC FAQ. At least they're honest about how corrupt the whole thing is. This overall score, or METASCORE, is a weighted average of the individual critic scores. Why a weighted average? When selecting our source publications, we noticed that some critics consistently write better (more detailed, more insightful, more articulate) reviews than others. In addition, some critics and/or publications typically have more prestige and weight in the industry than others. To reflect these factors, we have assigned weights to each publication (and, in the case of film, to individual critics as well), thus making some publications count more in the METASCORE calculations than others. In addition, for our film and music sections, all of the weighted averages are normalized before generating the METASCORE. To put it another way that should be familiar to anyone who has taken an exam in high school or college, all of our movies, games, and CDs are graded on a curve. Thus the METASCORE may be higher or lower than the true weighted average of the individual reviews, thanks to this normalization calculation. Normalization causes the scores to be spread out over a wider range, instead of being clumped together. Generally, higher scores are pushed higher, and lower scores are pushed lower. Unlike in high school, this is a good thing, since it provides more of a distinction between scores and allows you to better compare scores across movies (or CDs). Many reviewers include some sort of grade for the movie, album, or game they are reviewing, whether it is on a 5-star scale, a 100-point scale, a letter grade, or other mark. However, plenty of other reviewers choose not to do this. Hey, that's great... they want you to actually read their review rather than just glance at a number. (Personally, we at Metacritic like to read reviews, which is one of the reasons we include a link to every full review on our site.... we want you to read them too!) However, this does pose a problem for our METASCORE computations, which are based on numbers, not qualitative concepts like art and emotions. (If only all of life were like that!) Thus, our staff must assign a numeric score, from 0-100, to each review that is not already scored by the critic. Naturally, there is some discretion involved here, and there will be times when you disagree with the score we assigned. However, our staffers have read a lot of reviews--and we mean a lot--and thus through experience are able to maintain consistency both from film to film and from reviewer to reviewer. When you read over 200 reviews from Manohla Dargis, you begin to develop a decent idea about when she's indicating a 90 and when she's indicating an 80. If you are the critic who wrote the review, and disagree with one of our scores, please let us know and we'll change it. This does happen from time to time, and many of the critics included on this site (such as Ms. Dargis) do indeed check their reviews (as well as those of their colleagues) on metacritic.com. I'm sure Adam would contest that last bit.
	zigfried - May 25, 2009 (05:30 PM) "We don't want you to make a game for us. We want you to make a good game you want to make. Sometimes we'll like it, sometimes we won't, but you wanna know if a game is good or not? There's this really, really effective tool to do that: the market! It being sold! Does it sell or not? And you know what: if it's a good game, and you know it's a good game, but it doesn't sell? Well, go talk to your marketing team. Don't come to us." If I were developing games, I'd rather put my faith in the critics than in the marketing team. I see his point, I just cannot get behind it. I would want my publisher to look at scores, and not just at sales. PS - thanks for the link, I consider myself schooled in the ways of MC now! //Zig
	Felix_Arabia - May 25, 2009 (11:56 PM) It's the money-ey-ey-ey-ey-ey. Oh, it's just too much fun sometimes.

© 1998-2024 HonestGamers
None of the material contained within this site may be reproduced in any conceivable fashion without permission from the author(s) of said material. This site is not sponsored or endorsed by Nintendo, Sega, Sony, Microsoft, or any other such party. Opinions expressed on this site do not necessarily represent the opinion of site staff or sponsors.