Author Topic: Reviews are sometimes inaccurate

0 Members and 1 Guest are viewing this topic.

Damacon

  • Member
  • *
  • Posts: 148
    • View Profile
Reviews are sometimes inaccurate
« on: October 27, 2014, 03:32:05 PM »
 I debated really hard if I should make this post and if there is really a way to solve the problem without a extreme amount of work. Because for the most part I am pretty satisfied with your reviews, some of the games you like I don't but its for my own selfish reasons and opinions most of the time.

 But anyway I wanted to use another example but I will use Sword Art Online: Hollow Fragment versus Ys: Memories of Celceta. SOA only has 6% difference from Celceta do you really feel it should be this way. Considering SOA really had nothing going for it other than the graphics and the cute characters who half the time are saying some jibberish not to mention the whole review was a joke on how bad it really is. While the only really complaint the writer of Celceta has is the graphics were a little blurry and its ends abruptly. I personally could not put Celceta down where as SOA I got bored after I maxed all my weapons by the second hollow area/floor 78 and have not played it since. But yet if you didn't have it as percent and went by horrible, very bad,bad,average,above average,good, and very good both these games would be in the good department. I just feel that is very misleading to someone who looks to reviews to buy games as I feel SOA should of got a average at best with the complete lack of one of the most important parts of any game and Celceta would be where it should be as good because it is slightly lacking in one department.

Basically what I am saying is maybe you should separate graphics and sound from the percent rating so the reviews more reflect if the game is actually fun or not. I do think they are important to some extent but if thats all the game has going for them they shouldn't boost a rating in playability.

D-Rider

  • Former God of RPGFan
  • Rainbow Club Member
  • Member
  • *
  • Posts: 3,751
  • Giant Paraplegic Cock
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #1 on: October 28, 2014, 05:23:41 PM »
Hey guys, I didn't agree with your review!  Revamp your whole fucking grading system!

dyeager

  • Contributing Editor
  • Member
  • *
  • Posts: 3,950
  • Code Monkey Like Fritos
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #2 on: October 28, 2014, 05:47:27 PM »
Couple things on the way we do grades, because grading systems are something we debate endlessly among the staff.

1) The reviews represent the opinion of the person who reviewed the game, not necessarily the site on the whole. In fact, when there are big disagreements about a game, sometimes we do more than one review (and every once in a while just because). This is obviously terrible for systems like Metacritic that give a single score on behalf of the site. In the case of the two games you're describing, 2 different reviewers did those. That probably feels like a cop out - but since we are entirely staffed by volunteers, it is simply impossible for us to all review every game - our goal as a site is to at least provide SOME coverage of any RPG that comes out, from the perspective of folks who really like RPGs in general, but if you asked different reviewers to provide scores for certain games you'd get wildly different scores.

2) Overall score is NOT an average of all the other scores. An example is this review (I wrote) of To The Moon: http://www.rpgfan.com/reviews/To_The_Moon/index.html
In this example I gave scores in the 70s to everything (70 on our scale means it was pretty okay) except Story and Sound, but my overall score was 90%. Overall is not an average, it's a more general impression.

There has often been debate about whether we even WANT to continue providing scores for things like reviews, and I think most reviewers would agree that the scoring is one of the LEAST favorite parts of the job. I'd be interested to hear what you guys think of it. But we're just like you - each of us has different opinions and we certainly don't always agree with each other.

Taelus

  • Some music guy
  • Editor Emeritus
  • Member
  • *
  • Posts: 12,367
  • You're gonna have a bad Turnabout.
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #3 on: October 28, 2014, 05:51:11 PM »
What Dave said.
Friends, waffles, work. Or waffles, friends, work. Doesn't matter, but work is third.


Dincrest

  • Foot-in-Mouth disease
  • RPGFan Editor
  • Member
  • *
  • Posts: 15,482
  • Overuser of analogies, similes, and metaphors
    • View Profile
    • RPGFan
Re: Reviews are sometimes inaccurate
« Reply #4 on: October 28, 2014, 05:59:52 PM »
1) Every reviewer grades a little differently, just like teachers in school.  You may submit the same exact essay to two different English teachers and Ms. Jones may give you an A for a litany of valid reasons, whereas Ms. Smith may give you a C+ for a litany of valid reasons.  We're human, we all have different tastes, so look to the reviewers whose tastes align with yours.  

We've had the grading system discussion on staff every year, sometimes multiple times per year.  Sure, the school grading system we use (100-90= A, 89-80= B, etc) may not be perfect, but I like the wiggle room a numerical system gives me (then again, I am an educator by trade.)  Plus, aggregate sites go by numbers more accurately than letters.  

2) The overall score is not necessarily an average of all the category scores.  Sure those play a part, but it's what I call the "reviewer's tilt."  Some reviewers may weigh one category more heavily than the others when coming up with an overall score.  Some may go with gut instinct for overall score.  
« Last Edit: October 28, 2014, 06:01:59 PM by Dincrest »
"I think I'm losing it.  I'm pulling out my hair trying to figure out what couch defines me as a person.  This is freaking madness.  This is hopeless.  This... my perfect little habitat here, and I still don't even know who I am."  - from Dow Jones and the Temple of Doom by Burnt By The Sun

Agent D.

  • Member
  • *
  • Posts: 4,723
  • Agent Bewear.
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #5 on: October 28, 2014, 08:12:25 PM »
What Dave said.

...ok shoot me for this one but who is dave? Both d-rider and dyeager have D as the first letter in their handle....I kinda hope you were referring to d-rider though cuz his answer was awesome.

TASTY!

Tooker

  • Administrator
  • Member
  • *
  • Posts: 14,046
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #6 on: October 28, 2014, 08:17:23 PM »
Sorry, he meant Yeager.
Those who believe in telekinetics, raise my hand.
—Kurt Vonnegut

Agent D.

  • Member
  • *
  • Posts: 4,723
  • Agent Bewear.
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #7 on: October 28, 2014, 08:28:18 PM »
Sorry, he meant Yeager.
Yeah I figured, Taelus usually doesn't fit the "asshole" archetype. Fucking nice guys.

TASTY!

Damacon

  • Member
  • *
  • Posts: 148
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #8 on: October 28, 2014, 10:16:03 PM »
 I understand that the reviews were opinions and I wasn't trying to be rude about it honestly. I did not realize you didn't use the values together to do a overall rating that makes a lot more sense knowing that. But I hope you can at least understand where I was coming from as someone who uses the reviews to kind of judge if I want to take a chance and buy the game. A lot of times you all have helped me out like with Eternal Sonata and a few other games that were pretty good that I probably wouldn't of looked into if I had not checked this site. I never would of said anything if I had known you didn't add up the scores cause the problem I was noticing only took into account that you were taking a rounded sum of the scores together to put the overall score.

I appreciate the replies and now I better understand how the site works I still feel the scores are little misleading but really if you take the time to read the whole review most of them are accurate even if the overall score didn't reflect what the reviewer was saying.

Starmongoose

  • Forum Moderator
  • Contributing Editor
  • Member
  • *
  • Posts: 6,352
  • I Mean Bismuth
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #9 on: October 28, 2014, 10:23:37 PM »
Surely if your looking to form an opinion on the game to sway your buying choice, reading the actual words within the review will be infinitely more helpful than a number?

For all you know the game could have a great score but for all the reasons you hate that the reviewer loves, and you would never know until you read the review.

I mean, let's be honest here. Reviews are given easily digestible scores along with the actual review itself because 1) A lot of people are lazy and don't want to read full reviews 2) Game companies want alluring numbers to put on their advertising.



Penguin Boss Patrolling The Forums.

Damacon

  • Member
  • *
  • Posts: 148
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #10 on: October 28, 2014, 10:32:09 PM »
Yeah I never said I didn't read the whole reviews that is why I felt they were inaccurate to begin with actually because the review was telling me one thing but the scores were telling me something else. If you came to the site wanting to buy a game for your system you would go in look at the scores and then pick the ones with the higher rating to actually read the review I think we could agree on that would happen with the majority of the people who are looking to buy a good game.

Starmongoose

  • Forum Moderator
  • Contributing Editor
  • Member
  • *
  • Posts: 6,352
  • I Mean Bismuth
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #11 on: October 28, 2014, 10:39:17 PM »
I'm saying that personally, for me, the concept of attaching a score to a review and expecting it to be any way accurate to your expectations is inherently flawed. As I said, they could be higher or lower for the complete opposite reasons you yourself would give it.

Asking for such a thing to be "accurate" is quite laughable really. :P

As a reviewer I won't even say I hate attaching these scores, because in truth I don't even think about them that hard. I just spent hours trying to write a witty, interesting, informational review for a game I spent even more hours playing, and people expect me to sum it up in a couple of digits? No, I don't think about them very hard. I just go by rough amounts to how I feel, with perhaps a little more care given to the "Overall" score, because that's the one people are going to look at hardest.


Penguin Boss Patrolling The Forums.

Rucks

  • RPGFan Survey Hero
  • Member
  • *
  • Posts: 1,948
  • Salvum est Vita!
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #12 on: October 28, 2014, 10:59:27 PM »
The goose making you feel like a jerk aside (though I do agree with the bit about people being too lazy to read anything other than the score), I'm glad you're asking questions Damacon.  Way to stir the pot buddy. 

It was very interesting to hear what a lot of the admin around here had to say about why they do what they do.  Doubly so when I take into account that this site has had such a large influence on my spending habits for more than a decade.

I think, if tasked with reviewing a game for a reasonably successful website, I'd wrestle with the number I placed on it A LOT.  Whether you like it or not, review numbers sway the masses, so even though you guys aren't IGN or Kotaku, what you say DOES have an impact on game sales.
« Last Edit: October 28, 2014, 11:06:44 PM by glassjawsh »


Damacon

  • Member
  • *
  • Posts: 148
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #13 on: October 28, 2014, 11:08:16 PM »
 Well then it might actually be easier to just go with the basic of basic system where games are just bad,average, or good. Trust me I appreciate the witty long reviews much more then just some numbers any day. But I also feel as a reviewer it is important to be able to sum up your thoughts and give a fair overall review otherwise everything you just wrote doesn't really take a shape.


Thanks Glassjawsh, I try not be rude to people because I hate when people do it to me that is why I will never verbally attack someone directly. I love talking about games plus hearing peoples opinions because I feel if you don't you will never really understand people and where/why they stand where they do.
« Last Edit: October 28, 2014, 11:14:39 PM by Damacon »

Starmongoose

  • Forum Moderator
  • Contributing Editor
  • Member
  • *
  • Posts: 6,352
  • I Mean Bismuth
    • View Profile
Re: Reviews are sometimes inaccurate
« Reply #14 on: October 28, 2014, 11:12:10 PM »
Well it should take shape if it's a good review, so I don't agree there.

I will agree I liked a positive, negative or neutral scoring system better, which we actually do for DLC content.


Penguin Boss Patrolling The Forums.