Establishing a baseline for reviewing things can be difficult.
Culturally, it seems like most Americans operate on a rating system that reflects the way that American schools grade. Start with a perfect score, then subtract points for failure.
No complaints about that book you got on Amazon? Five stars. Nothing wrong with your meal at the local Taco Palace? Five stars.
Noticed the bathroom wasn’t as clean as you’d expect at the Shop-o-Rama? Deduct one, maybe two. Now it’s a three star experience.
This inflates scores up towards “perfect.” Anything that essentially passes the bar without complaints is as good as it can get. There’s no room to differentiate the top end.
I’m strongly of the opinion that average experiences deserve an average rating. If you read a book and it was basically fine, no real complaints, but nothing stands out as exceptional? That should get a 50%. On the rare occasions that I see people doing this, it results in a lot of… Well. Let’s call it “feedback.” So many people are accustomed to a perfect score being baseline from which complaints are subtracted that giving something that’s essentially okay a 50% is likely to result in outrage from people that would agree with you that it’s basically fine. The problem isn’t that they think it’s better than you do, it’s that they think that mediocrity deserves a 90%+ score.
Again, it seems to me this is primarily an American thing.
Maybe it’s because I see absolutely no value in maintaining the status quo where it doesn’t make any sense to me, I’ve never used this style of rating. Usually that means I need to publish a rubric.
For example, here’s my rubric for recipes:
1 star: not good as-is, could be redeemed. Research and try again.
2 star: one person liked this, but not both.
3 star: we both liked this and would like to have it again.
4 star: very good, should make once per month.
5 star: the very best, should make every week.
Three stars is a perfectly fine recipe that we both liked. There’s literally nothing wrong with it.
Okay, but, so… Why?
I’m in the process of setting up RPGGeek as the place to track my project of reviewing all these RPG books. They inherited a 10-star rating system from BoardGameGeek. I think it’s awful, that’s far too many stars, but the communities on that site have done a surprisingly good job of fighting back against rating inflation. The absolutely highest-rated board game has achieved an average rating of 8.60 with almost 50k votes.
But still, I disagree a bit with their published rubric. It’s better than most, but I think still trends towards assuming that something basically okay should have a score of 6.
For RPG items, they publish this:
10 - Outstanding. Rules and concepts well presented and a near perfect fit to your play style. Excellent writing. Excellent Editing. Highly consistent throughout. You would highly recommend this book/product.
9 - Excellent. Same as 10 but lacking in one or more of the elements (writing style, editing, consistency).
8 - Very Good. Rules and concepts are a good fit to your play style with some reservations. Solid writing. Solid Editing. Highly consistent throughout.
7 - Good. Same as 8 but lacking in one of the elements (writing style, editing, consistency).
6 - Above Average. Rules and concepts are a reasonable fit to your play style but there are significant sections which you would change or not use in actual play. Solid writing. Solid Editing. Highly consistent throughout.
5 - Average. Same as 6 but lacking in one of the elements (writing style, editing, consistency).
4 - Below Average. Rules and concepts are not a great fit to your play style and there are significant sections which you would change or not use in actual play. You would cherry pick a few ideas from such a book and discard the rest. Solid writing. Solid Editing. Highly consistent throughout.
3 - Well Below Average. Same as 4 but lacking in one of the elements (writing style, editing, consistency).
2 - Poor. Rules and concepts are not likely to mesh at all with your play style and there are precious few things you can cull from this book to use in your actual play. Solid writing. Solid Editing. Highly consistent throughout.
1 - Horrible. Same as 2 but lacking in one of the elements (writing style, editing, consistency).
It’s more thought than I see most sites giving the problem, but I want to propose an alternative.
5 - Average. You would play this game. There is nothing particularly good nor bad about it. It’s fine.
Okay, what moves us up the scale from a 5? I can think of three areas of analysis: mythos, production, mechanics.
Mythos - How interesting is the game’s world? How evocative is the writing? Am I excited about the stories it wants to tell?
Production - How well-produced is the book? Art, writing, graphic design. How do the materials feel? Is this an object of beauty?
Mechanics - How playable is the game? Does it provide systems that are fun to engage with? How portable are those systems?
So let’s pin a perfect 10 in our scale.
10 - Outstanding. The world is vibrant and interesting, the stories are evocative. The item is finely produced and would be worthy of display. It is an art object in and of itself. The mechanics are interesting, clearly presented, and weave together to support the stories of the mythos.
And with that, I think we can pin a 1.
1 - Horrible. This item has no redeeming qualities. The stories are not interesting, the world lacks color and flavor. The item itself is utilitarian and otherwise poorly produced. You would not want people to see it on your bookshelf. The mechanics offer nothing new or interesting. Best case scenario it’s a retread of something other games have done better.
This scale lends itself really well to a four star system, where a book with none of these things would be a 1, and if it had all three it’d be a perfect four. But I don’t get to set the system on RPGGeek. One option is to map my system on to theirs. My four-star would be their 10-star, and then down by 3 for each intermediary. 10, 7, 4, and jump to 1 in the worst case. This has benefits and weaknesses. The best argument against it is that it loses granularity. What if there’s one really good system, but the rest is trash? What if the cover is truly beautiful, but the pages inside phoned it in?
I can smooth out my system by taking my three axis ratings and giving each a half-step version that differentiates between “they did this one thing really well” and “the whole system knocks it out of the park.”
So then I created a spreadsheet…
It turns out there’s an “easy” solution to this problem. If by easy you mean “Contrariwise is a crazy person who is totally unreasonable.”
Say it with me. (Or don’t.) “Weighted averages.”
Imagine my three categories above: mythos, production, and mechanics. I can grade each of those on a scale from 1-5 where:
5 - It knocked it out of the park in this category, the whole item is a delight throughout.
4 - The item mostly succeeded, with a small number of critiques.
3 - It’s fine.
2 - There’s one or two redeemable qualities or systems, but mostly this ain’t it chief.
1 - Nope. No thanks.
If I rate each category on that scale, 1-5, then I can calculate the average of all of those rankings and arrive at a percentage. 5/5 in all three gets you a ten. 3/5 in all three gets you a five. (Really I’d need 2.5 to do that, but it can be fudged in the formula.) 1/5 in all three… You get the picture.
So I think that’s what I’m going to plan to do. For now. I might revisit this after I’ve tried it a few times. I might be missing things that end up being important. But it’s a starting point, and I’ve explained my reasoning so that if anyone ever wonders why I gave THAT BELOVED CLASSIC a “5” there’s some chance of defending why. I mean. There’s no accounting for taste.