Site Related & More > RPGFan: Site

Data Dump 5: Josh Reviews the Reviews (with sexy results!)

(1/4) > >>

So I went ahead and did another exhaustive statistical survey of something that most of you don't care about.

Why? Because my life leaves me largely unfulfilled, and deep down I am a highly disturbed individual.

ANYHOW, this time I went through every review of a console or handheld game released in North America that this site has ever done (over 1000) and took down the numerical score assigned to it.  Then I compared the RPGFan score to that of the aggregate scores from websites like Metacritic and GameRankings.

THE PURPOSE of this was to see if I could notice any meaningful trends in the grades the RPGFan staff and the general gaming media at large have given rpgs in the last 2 decades plus.

YES.  I realize it's incredibly difficult to get a substantive number from a process that is both subjective and non-uniform.  Aggregating review scores is undoubtedly a mess of differing opinions and stars and percentages  But my (horrendous and stupid) logic for still going forth with this dictates that since our society very often gleans important information about what millions and millions of people think from research polls that question 1001 people (with a relatively small margin of error) the data I ultimately collected still might be able to puke out something significant.

First off, HERE is a link to what I ultimately ended up with (in convenient spreadsheet form) ---->

I admit that it's fairly unwieldy to look at (especially the master list) but there are quite a few things to take away from what I put together.

1. The RPGFan Bias

This is kind of silly to get into but I think it needs to be explained.  Over the course of nearly 20 years and over 1000 reviews RPGFan has, on average, rated games 6 percentage points higher than the gaming "media" (little m) at large (81.5 to 75.5) the reasons for this are two fold (and somewhat obvious).  The first reason is qualitative.  This site is operated and maintained by "RPGFans" (obv) and that lends itself to a certain natural bias towards the games themselves.  They rate them higher because they like them more than the average person (which is a dumb sentence but makes sense in the greater context). 

The second, and far more important, reason is quantitative.  Basically, because Mickey or GhaleonOne or Sensei Phoenix or whoever, chose to use a ratings system that's actually out of 5 stars (but not really) even though it looks like a percentage system, the numbers were always going to skew high. 

In a nutshell that means that:
100 - 90% = 5 stars
89 - 80% = 4 stars
79 - 70% = 3 stars
69 - 60% = 2 stars
59 - 0% = 1 or (very rarely) 0 stars

See the disproportion at the end?  That's were the 6% positive skew comes in.  So, if you've ever thought the sites' reviewers were too easy on a game, that's because they are, but not really (SIMPLE RIGHT?!)

3. YEAR (aka are the games really getting worse?)

Well, are they? Since I haven't played a 7th or 8th gen rpg that I was totally ga ga over, this is a question that is very near and dear to my heart. 

The short answer? The numbers point to "Yes". 

Below you will find a table of the averages of all reviews collected for each year since 1991.  The Results Might Surprise You (#3 LEFT ME SPEECHLESS!)


With the scores being the Y axis and the year a game was released being the X, there is actually a very clear trend of leveling off after the high water mark years of 94-04.

At face value it would seem to be fairly obvious that reviewers are less jazzed about new games than they were 10 or 20 years ago (cue the "things aren't like they used to" bunch). But I also think it might have something to do with the game "media" at large (and on the internet in particular) transforming from an amateur collective of gaming enthusiasts into a much more professional outfit of most likely probably "real" journalists.  Basically, it takes a lot more to impress someone who is doing this as a job than it does to impress someone who is at it because they love it. The real answer is probably somewhere in the middle.

4. Devs (aka Everyone Hates Idea Factory)

I'm tired of typing so I'm just going to leave this here and let you guys (or no one) debate over what it means.

All I'm saying is, Square Enix is pretty much the poster child of the "leveling off" I referred to earlier.  Just look at the difference pre and post merger.  You all knew it, now here's your proof.

I will post a bunch of "Top 5/Bottom 5" lists (as well as the mysterious #2) in the comments later today.

I hope everyone hated this and, as always, you can direct any questions or concerns here


How about total number of games in each of these categories?

--- Quote ---In a nutshell that means that:
100 - 90% = 5 stars
89 - 80% = 4 stars
79 - 70% = 3 stars
69 - 60% = 2 stars
59 - 0% = 1 or (very rarely) 0 stars
--- End quote ---

Extra points for how many of those 0-59% are reviewed by Neal?


--- Quote from: Annubis on September 25, 2015, 01:55:43 PM ---How about total number of games in each of these categories?

--- Quote ---In a nutshell that means that:
100 - 90% = 5 stars (276 reviews)
89 - 80% = 4 stars  (425 reviews)
79 - 70% = 3 stars (227 reviews)
69 - 60% = 2 stars (81 reviews)
59 - 0% = 1 or (very rarely) 0 stars (38 reviews)

--- End quote ---

Extra points for how many of those 0-59% are reviewed by Neal? (3 - he hated the Inuyasha game the most)

--- End quote ---


Doing the (statistics) lord's work =)

Very interesting.  I keep a spreadsheet of my own reviews, and including the one that'll be posted tomorrow, I've reviewed 108 games, and my average score is 78.3 - right between the site average and the global average.

In terms of "stars," here's what I've got:

1   3
2   12
3   38
4   43
5   12

(Spoiler alert, my pending review is a "one star" review.)

Edit: I should mention that I did a similar exercise to this at the end of 2013, and at that time, Neal had written 135 reviews.  He may actually be over 150 now... He is the site's all-time champ in terms of number of reviews!


[0] Message Index

[#] Next page

Go to full version