games

Solving MMO review problems

To the tune of: Nina Nastasia – Our Discussion

As an MMO reviewer, I’ve felt both privileged (at occupying a niche few are equipped to explore) and also terrified (at a reviewer’s complete inability to perceive the whole game). As Quintin Smith found last week, and Ed Zitron found previously, reviewers, no matter how professional, can’t just hop back into an MMO when it’s been updated and hope to review it. Especially not, as those two found, when the audience they think they’re talking to isn’t the audience that actually reads and responds to the article. Here’s a putative structure for reviewing MMOs that deals with the problems caused by trying to employ time-poor professional reviewers.

Cpl Smith, M.I.A.

Problems

  1. Must experience enough of content in proper way to do review.
  2. Different experience types for player types – solo, casual, hardcore, obsessive.
  3. Need for humour, quality writing.
  4. Cost of review process must be kept down.
  5. Content alters substantially over game’s lifetime.

Many professional reviewers provide 3, can attempt 2 but usually fail, don’t keep playing so can’t do 5, and to provide 1 would be to disregard 4.

Solutions

  1. Multiple reviewers
  2. Multiple reviewers
  3. Mediated by co-ordinator
  4. Co-ordinator is paid writer-editor – incentivised to find free reviewers and collate & polish their opinion.
  5. Reconvene with original panel at regular intervals.

It's all about getting a good team together.

This system as a narrative. Un-paid enthusiasts are given early access and review title in return for thought-access. Primary writer becomes interviewer, co-ordinating impressions from many different groups. Individual, subjective experience is not of primary relevance, but collation of views is. Common problems can be identified, and the game rated on these – whilst problems specific to groups acknowledged, represented. Panel reconvenes to alter score when game has altered substantially from previous score. (This also provides you with an evaluation structure for up-and-coming writers, as you can test their analysis, reliability and writing ability  through these panels.)

Here’s a question I don’t know the answer to – is this process applicable to reviews other than MMOs? Should all reviews be done this way?

15 comments to Solving MMO review problems

  • Ashton Raze

    I like the thinking behind this. Sure, it’s more time-consuming than the average review, and sure, it requires multiple people to work together towards some kind of goal… sounds pretty suitable for an MMO then. This is my favourite ‘How To Review MMOs’ suggestion yet, it definitely makes a lot of sense.

  • Interesting way of solving the problem, absolutely. But I believe it boils down to the problem that MMOs don’t really review very well, and a lot of reviewers tend to try to review them in the classical sense anyway. Add in editors that want said reviews to be done as any other, and we’re in dangerous territory.

    Instead of going through all those steps I’d
    - remove the score.
    - remove “solid” opinions from the piece, where no such opinions can be formed.
    - do MMO-reviews in a similar way to what Eurogamer used to do (i.e. two articles, one from early on and one from after a month – of course very problematic, considering problem #4 above)
    - be very honest about what has actually been experienced in the game. The fans of the game will be there to point out any flaws, or opinions taken out of context.

    Readers want a review early. Editors usually want a review early. The game publisher prefer to get early reviews too – at least when a good score is assigned. The only publisher I’ve seen so far that actually wanted journalists to take their sweet time is NCsoft when Aion was released. Funcom had a review embargo on the day before Godslayer come out, practically saying “please, review from beta using our auto-level up system”. That’s no good either, for anyone.

  • Petter – FunCom did WHAT? That’s crazy – the most unenjoyable MMO experience I ever had was a demo tour of a high-level boss in FFXI when I was on PCFormat, as without working for your character, the fights are meaningless.

    Removing solid opinions from early reviews is a definitely an idea – but you’ll be undercut by others in the market who do give scores, as some people just want a buy / don’t buy.

    • That was kind of evident in Quintin’s review, though. He mentioned being given a level 80 character and playing around with that. But I agree, absolutely. I don’t mind them including that, I used the service to move myself past Tortage and giving me a bit more freedom to run around Gateway to Khitai, but as a basis for a review it’s pretty worthless. A way to take a closer look at beta, it’s fine.

      Do note that Funcom didn’t do anything except putting the embargo on that day + having a way to auto-level any character you made in beta (up until a certain date). In the end, it’s up to the publications how they use it. I think Eurogamer chose the wrong path, but they did also review Star Trek Online from beta only.

      And yes, undercutting would be a problem – as I said, readers want those reviews early. I should just count myself lucky that I worked under an editor-in-chief that allowed me to take my time, and is now in the position myself to decide when our review comes out. It’s a tricky situation in general.

  • Ben

    Not really a productive comment, for which I apologise, but which MMO is pictured in the post?

  • Anthony

    Kind of off-topic, but Nina Nastasia is fantastic.

  • TeeJay

    “is this process applicable to reviews other than MMOs? Should all reviews be done this way?”

    As a consumer I already use this process to decide whether to buy a game – ie. I read what gamers post on forums/blogs, alongside any traditional “professional” reviews. The professional reviews are often useful for all the condensed details, being the first into print and maybe for humour etc. but it’s the forum posts that I really pay attention to due to having a whole bunch of different opinions, the back-and-forward discussions.

    This still means a central role for games writers – to provide a focus and forum for discussion, attract a good community that will contribute to the discussion and the editorial and moderation role.

    A more general public might prefer the traditional long-form/essay games-mag style review, the single paragraph newspaper-style condensed ‘buy/don’t-buy’ review or a weekly/monthly ’round-up’/'best-of’ – but maybe these can all be constructed on top of the initial blog-community-crowd-sourcing.

    I’m not really saying anything new – this seems to be how things have evolved now already, from where I am sitting (ie. a consumer, rather than a writer), although it seems to have been more successful on newer independent blogs rather than the older sites or magasines maybe because they are actually designed for this and people have the expectation of interactivity/community/relationships/ownership. People’s relationship with older, corporate media seems more resentful, suspicious and antagonistic, even if it has tried to adapt to the newer style.

    • “maybe these can all be constructed on top of the initial blog-community-crowd-sourcing.”

      Exactly, yep. The writer does the writing and most of the analysis – but he has stringers on the ground who provide the core of the information, filtered through their perceptions and prejudices, which any good writer should be aware of. It’d be a foolish writer who doesn’t already take account of other people’s opinions and knowledge when reviewing an MMO.

  • [...] This post was mentioned on Twitter by Dan Griliopoulos. Dan Griliopoulos said: Are you terrified of reviewing MMOs? Find it just more trouble than it's worth? Dr Grill has a solution: http://ow.ly/1LHSi [...]

  • [...] the review structure of paid journalism be changed? This person thinks so, but I disagree. In the end, the editors are right in paying by the review count. [...]

  • [...] the review structure of paid journalism be changed? This person thinks so, but I disagree. In the end, the editors are right in paying by the review count. [...]

  • [...] same amount than if I played it for 4. Should the review structure of paid journalism be changed? This person thinks so, but I disagree. In the end, the editors are right in paying by the review count. [...]

Argue with me