Author Topic: Data Dump 5: Josh Reviews the Reviews (with sexy results!)

0 Members and 1 Guest are viewing this topic.

Rucks

  • RPGFan Survey Hero
  • Member
  • *
  • Posts: 1,947
  • Salvum est Vita!
    • View Profile
Data Dump 5: Josh Reviews the Reviews (with sexy results!)
« on: September 25, 2015, 01:50:09 PM »
So I went ahead and did another exhaustive statistical survey of something that most of you don't care about.

Why? Because my life leaves me largely unfulfilled, and deep down I am a highly disturbed individual.

ANYHOW, this time I went through every review of a console or handheld game released in North America that this site has ever done (over 1000) and took down the numerical score assigned to it.  Then I compared the RPGFan score to that of the aggregate scores from websites like Metacritic and GameRankings.

THE PURPOSE of this was to see if I could notice any meaningful trends in the grades the RPGFan staff and the general gaming media at large have given rpgs in the last 2 decades plus.

YES.  I realize it's incredibly difficult to get a substantive number from a process that is both subjective and non-uniform.  Aggregating review scores is undoubtedly a mess of differing opinions and stars and percentages  But my (horrendous and stupid) logic for still going forth with this dictates that since our society very often gleans important information about what millions and millions of people think from research polls that question 1001 people (with a relatively small margin of error) the data I ultimately collected still might be able to puke out something significant.

First off, HERE is a link to what I ultimately ended up with (in convenient spreadsheet form) ----> https://www.dropbox.com/s/lwwuaokj7cstn1e/Josh%20Reviews%20the%20Reviews%20%28with%20sexy%20results%21%29.xlsx?dl=0

I admit that it's fairly unwieldy to look at (especially the master list) but there are quite a few things to take away from what I put together.

1. The RPGFan Bias

This is kind of silly to get into but I think it needs to be explained.  Over the course of nearly 20 years and over 1000 reviews RPGFan has, on average, rated games 6 percentage points higher than the gaming "media" (little m) at large (81.5 to 75.5) the reasons for this are two fold (and somewhat obvious).  The first reason is qualitative.  This site is operated and maintained by "RPGFans" (obv) and that lends itself to a certain natural bias towards the games themselves.  They rate them higher because they like them more than the average person (which is a dumb sentence but makes sense in the greater context). 

The second, and far more important, reason is quantitative.  Basically, because Mickey or GhaleonOne or Sensei Phoenix or whoever, chose to use a ratings system that's actually out of 5 stars (but not really) even though it looks like a percentage system, the numbers were always going to skew high. 



In a nutshell that means that:
100 - 90% = 5 stars
89 - 80% = 4 stars
79 - 70% = 3 stars
69 - 60% = 2 stars
59 - 0% = 1 or (very rarely) 0 stars

See the disproportion at the end?  That's were the 6% positive skew comes in.  So, if you've ever thought the sites' reviewers were too easy on a game, that's because they are, but not really (SIMPLE RIGHT?!)


3. YEAR (aka are the games really getting worse?)

Well, are they? Since I haven't played a 7th or 8th gen rpg that I was totally ga ga over, this is a question that is very near and dear to my heart. 

The short answer? The numbers point to "Yes". 

Below you will find a table of the averages of all reviews collected for each year since 1991.  The Results Might Surprise You (#3 LEFT ME SPEECHLESS!)



fuck it, GRAPH DANCE PARTY!



With the scores being the Y axis and the year a game was released being the X, there is actually a very clear trend of leveling off after the high water mark years of 94-04.

At face value it would seem to be fairly obvious that reviewers are less jazzed about new games than they were 10 or 20 years ago (cue the "things aren't like they used to" bunch). But I also think it might have something to do with the game "media" at large (and on the internet in particular) transforming from an amateur collective of gaming enthusiasts into a much more professional outfit of most likely probably "real" journalists.  Basically, it takes a lot more to impress someone who is doing this as a job than it does to impress someone who is at it because they love it. The real answer is probably somewhere in the middle.


4. Devs (aka Everyone Hates Idea Factory)

I'm tired of typing so I'm just going to leave this here and let you guys (or no one) debate over what it means.

All I'm saying is, Square Enix is pretty much the poster child of the "leveling off" I referred to earlier.  Just look at the difference pre and post merger.  You all knew it, now here's your proof.




I will post a bunch of "Top 5/Bottom 5" lists (as well as the mysterious #2) in the comments later today.


I hope everyone hated this and, as always, you can direct any questions or concerns here http://goo.gl/A49MKV


 




Annubis

  • Member
  • *
  • Posts: 5,466
  • ee yan~☆
    • View Profile
Re: Data Dump 5: Josh Reviews the Reviews (with sexy results!)
« Reply #1 on: September 25, 2015, 01:55:43 PM »
How about total number of games in each of these categories?

Quote
In a nutshell that means that:
100 - 90% = 5 stars
89 - 80% = 4 stars
79 - 70% = 3 stars
69 - 60% = 2 stars
59 - 0% = 1 or (very rarely) 0 stars

Extra points for how many of those 0-59% are reviewed by Neal?

Rucks

  • RPGFan Survey Hero
  • Member
  • *
  • Posts: 1,947
  • Salvum est Vita!
    • View Profile
Re: Data Dump 5: Josh Reviews the Reviews (with sexy results!)
« Reply #2 on: September 25, 2015, 02:12:15 PM »
How about total number of games in each of these categories?

Quote
In a nutshell that means that:
100 - 90% = 5 stars (276 reviews)
89 - 80% = 4 stars  (425 reviews)
79 - 70% = 3 stars (227 reviews)
69 - 60% = 2 stars (81 reviews)
59 - 0% = 1 or (very rarely) 0 stars (38 reviews)

Extra points for how many of those 0-59% are reviewed by Neal? (3 - he hated the Inuyasha game the most)

DONE


Annubis

  • Member
  • *
  • Posts: 5,466
  • ee yan~☆
    • View Profile
Re: Data Dump 5: Josh Reviews the Reviews (with sexy results!)
« Reply #3 on: September 25, 2015, 02:37:45 PM »
Doing the (statistics) lord's work =)

Tooker

  • Administrator
  • Member
  • *
  • Posts: 14,046
    • View Profile
Re: Data Dump 5: Josh Reviews the Reviews (with sexy results!)
« Reply #4 on: September 25, 2015, 02:57:01 PM »
Very interesting.  I keep a spreadsheet of my own reviews, and including the one that'll be posted tomorrow, I've reviewed 108 games, and my average score is 78.3 - right between the site average and the global average.

In terms of "stars," here's what I've got:

1   3
2   12
3   38
4   43
5   12

(Spoiler alert, my pending review is a "one star" review.)

Edit: I should mention that I did a similar exercise to this at the end of 2013, and at that time, Neal had written 135 reviews.  He may actually be over 150 now... He is the site's all-time champ in terms of number of reviews!
« Last Edit: September 25, 2015, 03:08:23 PM by Tooker »
Those who believe in telekinetics, raise my hand.
—Kurt Vonnegut

Rucks

  • RPGFan Survey Hero
  • Member
  • *
  • Posts: 1,947
  • Salvum est Vita!
    • View Profile
Re: Data Dump 5: Josh Reviews the Reviews (with sexy results!)
« Reply #5 on: September 25, 2015, 03:27:24 PM »
Very interesting.  I keep a spreadsheet of my own reviews, and including the one that'll be posted tomorrow, I've reviewed 108 games, and my average score is 78.3 - right between the site average and the global average.

In terms of "stars," here's what I've got:

1   3
2   12
3   38
4   43
5   12

(Spoiler alert, my pending review is a "one star" review.)

Edit: I should mention that I did a similar exercise to this at the end of 2013, and at that time, Neal had written 135 reviews.  He may actually be over 150 now... He is the site's all-time champ in terms of number of reviews!

I thought about having a section on particular reviewers (and I still might include it if there is any interest), but I'm not sure it would really mean anything since each reviewer doesn't necessarily always review the type games they like going in.  Some people just have to do it so it's done.  And then the numbers might be a bit slippery from there.  If that makes any sense...

Not that I'll have any takers,
but if anyone wants my original excel file just let me know and I will email it.


divingfalcons

  • Honorary Blue Rogue
  • Administrator
  • Member
  • *
  • Posts: 1,216
    • View Profile
    • Blog
Re: Data Dump 5: Josh Reviews the Reviews (with sexy results!)
« Reply #6 on: September 25, 2015, 04:52:14 PM »
This is really cool, and I'm going to read over the spreadsheet properly when I'm not on my out of date tablet, but that looks like a hell of a lot of effort, and it's really appreciated. I like these kind of statistics.

think the reason our reviews are a little higher on average is because we've all got different tastes, and we're all a bit more subjective in our reviews, which is why I've always liked the site. It's professional, but also there's a sense it's not just someone doing their job, but doing it because they love it, and wants that love (or hate in some cases) to come through.

I'm not surprised the mid/late 90s come at the top of the aggregate, or that 2014 is quite low. Some of my favourite games all come from the former dates.

I've only ever played one Idea Factory game, and I really enjoyed it, although it was a visual novel. I've seen and heard about their RPGs and they certainly look...divisive.

Tooker, your pending review is another Idea Factory 'gem' right? I like the idea of you keeping a spreadsheet of your reviews, too. When I've got a few more under my belt, I might start doing that.

Rucks

  • RPGFan Survey Hero
  • Member
  • *
  • Posts: 1,947
  • Salvum est Vita!
    • View Profile
Re: Data Dump 5: Josh Reviews the Reviews (with sexy results!)
« Reply #7 on: September 25, 2015, 05:50:01 PM »
ugh...I'm already noticing errors.  Grandia is in the Saturn table even though it was never released in this region on Saturn and Final Fantasy Legends is included in the 2000 table even though it came out in 1991.

To make up for these egregious oversights,  here's (the start of) a yearly list of "best of" titles I threw together:


Arranged by aggregated scores



More to come later






Tooker

  • Administrator
  • Member
  • *
  • Posts: 14,046
    • View Profile
Re: Data Dump 5: Josh Reviews the Reviews (with sexy results!)
« Reply #8 on: September 25, 2015, 06:00:49 PM »
Tooker, your pending review is another Idea Factory 'gem' right? I like the idea of you keeping a spreadsheet of your reviews, too. When I've got a few more under my belt, I might start doing that.

It sure is!  I'll send you a copy of mine if you like, so you can see what I've done.
Those who believe in telekinetics, raise my hand.
—Kurt Vonnegut

divingfalcons

  • Honorary Blue Rogue
  • Administrator
  • Member
  • *
  • Posts: 1,216
    • View Profile
    • Blog
Re: Data Dump 5: Josh Reviews the Reviews (with sexy results!)
« Reply #9 on: September 25, 2015, 06:38:46 PM »
I'm glad to say I've played at least one, two in the later years, of the games listed per year, and agree with both scores. The stuff that I haven't played basically consists of my backlog...

That would be really neat, Tooker. It just seems like a sensible way to keep track on personal standards and other things. Or even just for fun.

Dice

  • I'm fart and I'm smunny
  • AMG A GIRL
  • Member
  • *
  • Posts: 13,464
  • RPGFan and Dice forever...
    • View Profile
    • My DeviantArt site!  Come, stay, laugh, cry.
Re: Data Dump 5: Josh Reviews the Reviews (with sexy results!)
« Reply #10 on: September 29, 2015, 07:41:21 PM »
Oh my god, this is amazing. 

Let me know if you want more Data Dump stuff to do for the Social Media team!!!  Math is my kryptonite, it will kill me to use my brain for maths, but what you've done here is truly impressive work and you should be proud. 
http://society6.com/Dice963
http://www.redbubble.com/people/dice9633
http://dice963.tumblr.com/

Support your local Dice (and pitch her ideas)!
RAINBOW PIXEL POWER!!!

Rucks

  • RPGFan Survey Hero
  • Member
  • *
  • Posts: 1,947
  • Salvum est Vita!
    • View Profile
Re: Data Dump 5: Josh Reviews the Reviews (with sexy results!)
« Reply #11 on: September 29, 2015, 10:17:10 PM »
I will post the rest of the list tomorrow. And I am willing to do whatever for the site (as long as its in my ability to do so.)


Rucks

  • RPGFan Survey Hero
  • Member
  • *
  • Posts: 1,947
  • Salvum est Vita!
    • View Profile
Re: Data Dump 5: Josh Reviews the Reviews (with sexy results!)
« Reply #12 on: September 30, 2015, 07:22:43 PM »
second half of the game of the year list



will post the dev lists tomorrow


Dincrest

  • Foot-in-Mouth disease
  • RPGFan Editor
  • Member
  • *
  • Posts: 15,482
  • Overuser of analogies, similes, and metaphors
    • View Profile
    • RPGFan
Re: Data Dump 5: Josh Reviews the Reviews (with sexy results!)
« Reply #13 on: October 01, 2015, 04:33:34 PM »
How about total number of games in each of these categories?

Quote
In a nutshell that means that:
100 - 90% = 5 stars (276 reviews)
89 - 80% = 4 stars  (425 reviews)
79 - 70% = 3 stars (227 reviews)
69 - 60% = 2 stars (81 reviews)
59 - 0% = 1 or (very rarely) 0 stars (38 reviews)

Extra points for how many of those 0-59% are reviewed by Neal? (3 - he hated the Inuyasha game the most)

DONE

I'm amused that I was singled out here.  My lowest reviewed game, period, is probably Higurashi: When They Cry (I scored that pile in the 30s) but I don't think it counts since there are no critic reviews on metacritic (but the user score is 9.1).  That's also the review I've gotten the most hatemail for.
"I think I'm losing it.  I'm pulling out my hair trying to figure out what couch defines me as a person.  This is freaking madness.  This is hopeless.  This... my perfect little habitat here, and I still don't even know who I am."  - from Dow Jones and the Temple of Doom by Burnt By The Sun

Rucks

  • RPGFan Survey Hero
  • Member
  • *
  • Posts: 1,947
  • Salvum est Vita!
    • View Profile
Re: Data Dump 5: Josh Reviews the Reviews (with sexy results!)
« Reply #14 on: October 01, 2015, 05:44:59 PM »
THE DEV LIST:

THOUGHTS:

RIP Sega -  Not technically dead, but really you're dead.

RIP Tales Studio - also not technically dead

Holy Crap! The RPGFan team likes the Atelier series WAY more than the average Joe.

Nintendo was RPGFan's 2nd highest rated dev and they still technically undervalued them compared to the aggregate (which is insane)































« Last Edit: October 01, 2015, 05:47:40 PM by glassjawsh »