Author Topic: Reviews are sometimes inaccurate

0 Members and 1 Guest are viewing this topic.

Damacon

  • Member
  • *
  • Posts: 148
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #15 on: October 28, 2014, 11:24:28 PM »
Well it should take shape if it's a good review, so I don't agree there.

I will agree I liked a positive, negative or neutral scoring system better, which we actually do for DLC content.

Now I feel back at step one because that was my whole point is some of the reviews didn't take that shape correctly compared to what the person was actually saying in the review. There is quite a few of them that I feel that way about and the biggest problem with them is that they are not giving the truly good games as much justice as they deserve if some rag tag game is rated only a couple percents under them.
« Last Edit: October 28, 2014, 11:32:45 PM by Damacon »

Starmongoose

  • Forum Moderator
  • Contributing Editor
  • Member
  • *
  • Posts: 6,351
  • I Mean Bismuth
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #16 on: October 28, 2014, 11:44:23 PM »
@Glassjawsh: Compared to the time spent actually writing a review, the amount of time deciding on a number is never going to add up to a lot. Most of the time you have a number in your head for what it's going to be. After all, before I go into writing a review I already know whether or not I liked the game, and to what degree. When I said I don't spend a lot of time thinking about the number, I mean I'm not sitting there going "Is this an 85 or is it more of an 86. Hmm, the dilemma." That is just ridiculousness.

As I said the Overall is probably going to get the most thought put into it, but what really is the amount of time spent. Five minutes tops? I'm hardly going to give it a score that's completely contrary to what I said in my review.

Also I'm not trying to make games sell when I write a review. I might like the game and hope it does well, but I never let a commercial factor seep in. You are right that scores influence how many copies a game sells, but all I care about is making sure that my opinions are represented as defined by myself. Not compared to another reviewer either on RPGfan or elsewhere.  

Well it should take shape if it's a good review, so I don't agree there.

I will agree I liked a positive, negative or neutral scoring system better, which we actually do for DLC content.

Now I feel back at step one because that was my whole point is some of the reviews didn't take that shape correctly compared to what the person was actually saying in the review. There is quite a few of them that I feel that way and the biggest problem with them is that they are not giving the truly good games as much justice as they deserve if some rag tag game is rated only a couple percents under them.

That's already been explained as because you're comparing two different reviewers who have different tastes and differing scoring systems. You need to look at reviews on an individual level. If the review is good and answers your questions to satisfaction, what a completely different game was given is irrelevant.

I'm not trying to be antagonistic here (It's 4am in the UK and I might not be on the right level of tact), I'm trying to give perspective. It's actually really good to write for RPGfan as your given practically total freedom to say how you really feel, and when it goes through editing the proof-readers really try to keep your voice. I think a lot of readers really enjoy RPGfan reviews because they feel really honest, but the flipside of that is your going to have wildly different reviews from person to person.

Let's take for example a review that I wrote and a review that Kyle wrote for the same game, Bravely Default.

Kyle's: http://www.rpgfan.com/reviews/Bravely_Default/index.html (Overall: 75%)

Mine: http://www.rpgfan.com/reviews/Bravely_Default_Flying_Fairy/index.html (Overall: 92%)

If you read both reviews, you will find our opinions don't differ nearly as much as a glance at a scoreboard might appear. We both likes the same things, we both disliked the same things. No major disagreements. It's just a matter of degree. If you actually take the average of the scores Kyle gave the review you come up with 80% which is 5% above what he gave it, and pushes the game into a different "level" of quality than he didn't feel comfortable giving it.

My average works out as more or less what I gave it, though I promise you I didn't calculate that at the time, just a happy coincidence. I merely knew I was going to give this game 90% something and did it.  In short: Our reviews are pretty similar, the scores not so much.
« Last Edit: October 28, 2014, 11:46:13 PM by Starmongoose »


Penguin Boss Patrolling The Forums.

Damacon

  • Member
  • *
  • Posts: 148
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #17 on: October 29, 2014, 12:02:00 AM »
 I agree with you completely really Goose and what you just pointed out is probably what you need most more diversity.

Here is a idea that might work and solve a lot of your problems what if you added comment area to your reviews. That way people who want to can write their own opinions/reviews.Finally you guys can scan through them to decide if they were fair and informational enough to keep up as a different opinion on the game.

Really I wish that I could just become a reviewer but I know all to well how bad my engrish is LOL. I hate myself for it but I am horrible at sentence structure and I am one of those people when I write all kinds of ideas of what I want to say pop into my head at the same time so it makes me run things together even more. <--edit like this lol
« Last Edit: October 29, 2014, 12:11:47 AM by Damacon »

Starmongoose

  • Forum Moderator
  • Contributing Editor
  • Member
  • *
  • Posts: 6,351
  • I Mean Bismuth
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #18 on: October 29, 2014, 12:13:27 AM »
Comment sections are something we have discussed at length multiple times actually among the staff. Ultimately we've agreed each time that it's not a direction we want to go in. There are multiple reasons for that, maybe someone else on staff can take over that issue for me, I feel like I've written enough walls of text tonight. If you do want to discuss a review for a game, there is usually a thread for said game in the forums and you can leave a comment there and you will more than likely get a reply from the reviewer themselves.

I will say it's really unlikely you will sway a reviewer to change their opinion in any meaningful way that would require an edit. Short of full-blown errors that is.

Re wanting to be a reviewer: Personally the hardest part for me is being your own editor. To be able to look at your own work hard and get that big red pen out and sort the good from the bad, that's the real challenge. Then you pass it along to someone with an even bigger red pen. Haha.  If you practice enough you will get better at writing but to be your own editor requires you to change your entire way of thinking. If you think you can do that, then I don't think there is anything stopping you from being a good reviewer.


Penguin Boss Patrolling The Forums.

Damacon

  • Member
  • *
  • Posts: 148
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #19 on: October 29, 2014, 01:51:25 AM »
I didn't mean that you would change your review based on what someone else said but just keep posts in the comments that are legitimate well informed opinions/reviews that someone with a different view of the game might of posted. I am sure there would be a lot of weeding to do though with people just posting THIS GAME SUCKS with no real information but from the looks of it you have a lot of levelheaded readers who would probably post some decent stuff that you could keep under the review or however you would like to do it for some different views of the game.

dyeager

  • Contributing Editor
  • Member
  • *
  • Posts: 3,950
  • Code Monkey Like Fritos
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #20 on: October 29, 2014, 09:46:46 AM »
So, just to farther illustrate that as reviewers/staff members we all have different approaches and don't always agree, let me chime back in here.

To glassjawsh's comment - I'd say the number is BY FAR the thing I spend the most time agonizing over. Obviously in terms of time spent, no, since it takes longer to play the game and write about it, but my approach is typically to write the review with the score in mind, look back at what I wrote, and think about adjusting my scores. Part of the process of reviewing a game vs. simply playing it is you are FORCED, by nature of the process itself, to really examine aspects of the game that you might not otherwise examine or that you might simply shrug off as not important to the experience. But when you are forced to provide a score and you know that people might be making PURCHASING decisions, sometimes with limited resources - well I take that very seriously.

The biggest trouble I have with scores is that a score can only capture how you feel about a game at that specific moment in time, and that score gets captured by all kinds of aggregation engines (like Metacritic) and that becomes "your opinion" on the game. Obviously we appreciate our readers here and believe they are a cut above the average video game player and we try to write detailed reviews respecting that - but a large number of people will only ever interact with RPGFan through sites like Metacritic. And that can be frustrating.

What's also frustrating about a score is that quite often I look back and think, in hindsight, a game had more/less staying power than I thought it did and should have been adjusted up/down accordingly. For example, take my respective scores for Torchlight 2 (91%) and Ni no Kuni (95%), two games I just loved.

Now, I REALLY DID agonize over that 4 percent difference. In hindsight, Ni no Kuni for me was still an Editor's Choice caliber game, but Torchlight 2 is the game I've sunk far more hours into and still play on a regular basis. For me Torchlight 2 probably scores higher if I'm scoring on my personal scale!

But then, factor in that you're trying to predict whether the readership of the site IN GENERAL will like one game or the other more. I suspect that my burning love for Torchlight 2 might be a corner case and Ni no Kuni may have the broader appeal. I just don't know. You start to see right away how this task can cause a person to lose hair.

EDIT: Another thing about Ni no Kuni is I stand behind my comment in my review about this being as close to the Platonic form of the JRPG as we've gotten in many years - even if I am more likely to fire up Torchlight 2.

Now, on the topic of comments - we've debated quite a bit about this as well. I come down on the belief that we've got a really cool community on the site that resides in the forums, and if you want to talk about a game you can do it there and I kind of like it that way. We are, again, a volunteer staff, and even with our powers combined I don't think we'd be able to spend the energy necessary to keep comment sections on review pages constructive and civil - many sites have entire staff for that. HOWEVER, I do agree with Damacon that there are often some really good points that come out of comment sections on reviews at other sites. So I'm willing to be persuaded on this point. But I'd say the biggest reason we don't have comment sections has to do with the resources/time it would require to keep them reasonably under control, whereas we still get great conversation about the games we all like in the forums, facebook, etc anyway.

Hopefully this all explains a little bit behind my thought process anyway - I don't speak for the other staff members. :-)

EDIT: Here are links to the reviews if you'd like to see what I mean or provide critique on my scoring, but hopefully I've done at least a decent job illustrating a few of the troubles with scores.

http://www.rpgfan.com/reviews/Ni_no_Kuni_Wrath_of_the_White_Witch/index2.html
http://www.rpgfan.com/reviews/Torchlight_II/index.html
« Last Edit: October 29, 2014, 09:54:04 AM by dyeager »

Tooker

  • Administrator
  • Member
  • *
  • Posts: 14,046
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #21 on: October 29, 2014, 12:47:35 PM »
I don't speak for the other staff members. :-)

You can speak for me any day, Yeager.  I agree on all points.

There's one other thing I think I can add.  Scoring things is the worst... but that's the form that reviews take across all media.  As has been said, we've discussed eliminating scores a number of times, and we've also discussed changing our ratings scale in a number of ways.  But if we do, then when sites like Metacritic add our review into the rest, they interpret the review and put their own score on it.  That is, if we go with "good," "neutral," and "bad," they might choose to call those "80," "50," and "30."  But that's not what we meant at all!  Also, it means that games like Diablo II would end up with the same score as a game like Mugen Souls, which is worthy of being called "good," but isn't remotely in the same class as Diablo II (or Yeager's examples, which I also love).

We've also discussed eliminating our categories and simply giving a game an overall score.  That bypasses the issue of our overall score getting misrepresented on aggregator sites, but we feel like it creates a different issue.  Doing this takes away our ability to tell readers at a quick glance "this game looks pretty bad, but it has an awesome story, and it's really worth playing."
Those who believe in telekinetics, raise my hand.
—Kurt Vonnegut

Damacon

  • Member
  • *
  • Posts: 148
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #22 on: October 30, 2014, 10:30:42 AM »
 I am sorry that I could not respond faster was very busy yesterday had to last minute make a costume for a party I am getting forced to go to. I just got Tears of Tiara 2 and I much rather play it instead of dealing with people screaming over each other sadface.

 I understand and appreciate you going through the trouble to talk to me about this in such length. I also feel Ni No Kuni was one of the last true JRPGs I have played in a long time and was very happy to see it plus a lot of my other favorites get editors choice. That is probably why I trust you guys so much is you do have the same feelings as I toward RPGs when it matters most.  I will have to say I didn't factor in how you were feeling when the games were released and probably would explain a lot of why some games would seem better if you get RPG starved like I do lol.

 Maybe a better/fun solution would be at the end of each year gather up all the games released that year and do like a mini award ceremony to give some extra credit to a lot of the games everyone feels should stand out a little more. The community could join in on the voting and you could post new threads for each game where everyone could talk about them together I think it would be a lot of fun. There could even be some bad awards to make it more interesting like game most likely to be quit halfway, biggest disappointment, and worst translation.

I'm fairly new to site really, I've been reading reviews for a couple years but just recently delved into the community because I was in search of  answers and fellow RPG nakama. I would like to be of help to you all someway because RPGs are what I love, they have always been my hobby and the majority of my life decisions revolved around them sadly lol.  So if I am bothering you  feel free to tell me and I will back off but I feel pretty strongly that there is just something missing with the site to help forge the way to a better RPG tomorrow!

dyeager

  • Contributing Editor
  • Member
  • *
  • Posts: 3,950
  • Code Monkey Like Fritos
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #23 on: October 30, 2014, 10:36:55 AM »
For my part I'm always up for a chat about things we could maybe do better, so I have no problem whatsoever with your posts or suggestions.

As for extra credit - we do Game of the Year awards annually: http://www.rpgfan.com/features/index_goty.html

We also do a "Reader's Choice" (Ni no Kuni won last year) and we post the results of the voting. One of our favorite parts of the annual awards as well is it gives all the editors a chance to talk about THEIR favorite games - we agree as a group (often with much consternation) on category winners, but then each of us gets our own post to talk about our personal favorites. It's a nice illustration of how the staff has diverse tastes and we certainly don't always agree.

Damacon

  • Member
  • *
  • Posts: 148
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #24 on: October 30, 2014, 11:33:05 AM »
 Ahh I see, now that I think about it I almost do remember seeing a awards thing last year and most of the picks are also the ones that have the editors choice next to them most of the time. But do you also take your community into the votes and discussions or is it just the editors? I also feel that just having only the best of type categories  that you wouldn't really get your point across to everyone of whats wrong with a lot of the games in this day and age. What do you do when there really isn't any contenders and the whole genre is gradually declining?  Isn't that the root of the problem that because people only focus on the best and people try to copy it that so many games have lost a way to make their own style to improve over time?
« Last Edit: October 30, 2014, 11:56:02 AM by Damacon »

Tooker

  • Administrator
  • Member
  • *
  • Posts: 14,046
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #25 on: October 30, 2014, 01:48:11 PM »
I just got Tears of Tiara 2

I'm working on a review of that game! :) Life is getting in the way of making swift progress, but I will have good things to say when I write it.

As Yeager mentioned, we do have readers vote as well, and publish the results along side our staff awards each year.  I can't remember the category off the top of my head, but there was one category one year where we chose not to award any game because none of them were worthy.
Those who believe in telekinetics, raise my hand.
—Kurt Vonnegut

Damacon

  • Member
  • *
  • Posts: 148
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #26 on: October 30, 2014, 02:50:49 PM »
 So far I am really liking tears of tiara 2 the characters are a bit cliche as they fall into roles you see a ton in anime but there is nothing wrong with that as it makes them more likeable and helps you understand their depth a lot easier. The combat system is fairly in depth and keeps adding new fun features as the game progresses which keeps things interesting. If I had one complaint its that the map design's so far are really dull where they just kind of throw enemies everywhere they have a few random objectives sometimes but really the map takes no real effect on the strategy which is most of the time spread out and slaughter everyone even on the hard setting. But this is something common with most tactics games very few of them are really tactical and by the end tactics is just thrown out the window all together.

 So Natural Doctrine was a real surprise though the combat system was not flashy at all, it was a solid tactics system that used terrain and forced you to use special formations to keep yourself from dieing. I think most of us who like tactics games would agree we would like to see a more advance form of a game like Natural Doctrine with flashier skills, bigger maps with more strategy, and a likeable story like tears of tiara 2. So this is kind of my point Natural Doctrine didn't rate to well anywhere so it will fade away and a game like Tears of Tiara 2 will do well plus was probably a hell of a lot easier to create other than the story. Is it really fair I am sure the reviewer on this sight who did the review for Natural Doctrine feels the same way that sometimes that underdog deserves its credit so maybe someone might do it again only better instead of just doing what you know works.
« Last Edit: October 30, 2014, 03:03:37 PM by Damacon »

Dincrest

  • Foot-in-Mouth disease
  • RPGFan Editor
  • Member
  • *
  • Posts: 15,482
  • Overuser of analogies, similes, and metaphors
    • View Profile
    • RPGFan
Re: Reviews are sometimes inaccurate
« Reply #27 on: October 30, 2014, 04:55:43 PM »
Even before I joined the staff, my favorite part of the GOTY features was reading editors' individual picks rather than the sitewide picks.  I can already tell that some of the stuff in my top 5 for 2014 will probably not be in anyone else's list.  And that's part of the fun. 
"I think I'm losing it.  I'm pulling out my hair trying to figure out what couch defines me as a person.  This is freaking madness.  This is hopeless.  This... my perfect little habitat here, and I still don't even know who I am."  - from Dow Jones and the Temple of Doom by Burnt By The Sun

Dice

  • I'm fart and I'm smunny
  • AMG A GIRL
  • Member
  • *
  • Posts: 13,464
  • RPGFan and Dice forever...
    • View Profile
    • My DeviantArt site!  Come, stay, laugh, cry.
Re: Reviews are sometimes inaccurate
« Reply #28 on: October 30, 2014, 06:19:23 PM »
Even before I joined the staff, my favorite part of the GOTY features was reading editors' individual picks rather than the sitewide picks.  I can already tell that some of the stuff in my top 5 for 2014 will probably not be in anyone else's list.  And that's part of the fun. 

I actually love clicking your choices because you do have the (for lack of better words) oddball picks.
http://society6.com/Dice963
http://www.redbubble.com/people/dice9633
http://dice963.tumblr.com/

Support your local Dice (and pitch her ideas)!
RAINBOW PIXEL POWER!!!

Tooker

  • Administrator
  • Member
  • *
  • Posts: 14,046
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #29 on: October 30, 2014, 07:16:18 PM »
We treasure Neal (Dincrest) because he's brave enough to go down the gaming roads many shy away from.  He knows that he can find hidden treasure there, and that he's hardcore enough to survive the garbage he's got to wade through to get to it.

One more Tears to Tiara II note - I bet about 0.1% of people will pick up on this, but the main characters are named for historical figures (and gods and places) from Carthage, a kingdom who fought against Rome.  The guys most famous for Hannibal and the elephants crossing the Alps.  I only know because I listen to a history podcast that happens to coincidentally be talking about Carthage now.  (That is, not because I'm some smarty pants history guy.)
Those who believe in telekinetics, raise my hand.
—Kurt Vonnegut