I can only speak about this issue as a user, so make of it what you will. Been using the site for quite a few years, even if this account is pretty new so I remember what it was like back in at least '07 or '06 or so. But anyways, here is the way I see the fundamental issue.
I don't think it has to do with review quality entirely. When a lot of reviews aren't top notch the fundamental problem gets worse, but I would like to formulate the problem as follows:For releases with 15+ reviews, the usability of the review page tends to turn to shit irrespective of the quality of the reviews. This gets worse when the average length of the reviews is pretty long. But that is not the root cause of the usability problem. It also gets worse when the average quality of the reviews isn't top notch. But this is also not the root cause of the problem.
To illustrate this, we could use the the review page for Wintersun's Time I
. It has a ton of reviews. 34 of them. And there are both positive and negative ones, all mixed together. There are quality reviews all mixed together with the shoddier ones. Some are very long while others are shorter. Let's disregard that for now, however, and just look at the volume of all this.
I copied all the text on the review page aside from the headers and put it into an office program (LibreOffice in my case) in order to get some data on this. I knew it would be bad, but it's worse than I thought it would be. The content of that page takes approximately 30 seconds to paste into a document on a slightly above average computer. It fills 56 standard A4 pages. It's larger than my graduation thesis was, and that one contained images and graphs and what not. This is all text and whitespace used for formatting. So what are the numbers?
The Time I review page totals in at around 31000 words
composed of around 180000 characters including whitespace
or around 140000 characters excluding whitespace
This seems like a lot already, but what does it actually mean? Let's find out.
Wikipedia tells me the following:
Words per minute, commonly abbreviated WPM, is a measure of words processed in a minute, often used as a measurement of typing speed or reading speed.
For the purpose of typing measurement, each word is standardized to be five characters or keystrokes long, including spaces and punctuation. For example, the phrase "I run" counts as one word, but "rhinoceros" and "let's talk" both count as two.
The average adult reads prose text at 250 to 300 words per minute. While proofreading materials, people are able to read at 200 wpm on paper, and 180 wpm on a monitor. [Those numbers from Ziefle, 1998, are for studies that used monitors prior to 1992. See Noyes & Garland 2008 for a modern tech. view of equivalence]
If we disregard the part about proofreading on monitors and consider reading reviews on MA to be on par with reading printed prose (it's probably not, but I have no data for standard reading speeds on modern monitors), what does this give us? Let's count for a while. We have a character count of about 140000 characters of we disregard the whitespace. This brings our word count to 28000 words if we use the established "a word is 5 characters" definition. If we say that a person reads 275 such words in a minute (middle ground for the provided values), we can estimate the amount of minutes it would take to read all that, assuming it is true. Actually, let's make counting easier and round it to 280 words.
We get a time-to-read of 100 minutes
for all the reviews on the Wintersun page, or 1 hour and 40 minutes
(seeing as I've made some generous assumptions, I might give it up to another hour extra if I was to guess). If you just read it from beginning to end, that is.
In this time, you could listen to Time I
two and a half times.
This obviously is not 100% proportional to the amount of reviews in itself, but more to the amount of reviews coupled with their length. It's still too much to get through if you value your sanity.
Now that we've established that it's quite an undertaking to get through the reviews of Time I
, where does that leave us? Well, not all of those reviews are going to be worth your time (pun unintended). Some are going to suck. Others don't really suck but don't say much that half of the reviews on the page don't already say. The reviews are sorted in the pretty arbitrary order of submission date. One review has no real relation to the next one except for the fact that they were written at approximately the same time, or to the one that came before it. As a user I can not sort the reviews in any way. This means that as a user, I have no way of knowing if a review is worth my time except by at least skimming through it.
Regarding the ones that suck, a reader could at least hope that they are terrible enough from the start that they would be easy to dismiss just based on the first few sentences. But in my experience only the bottom 5-10% of the barrel of MA reviews are like that. Let's now say I want to read reviews of Time I
. What does this mean to me. We have established that it would take me 1 hour and 40 minutes in somewhat ideal cases to get through it all. How much time can I avoid wasting by filtering out stuff? This is not an easy question to answer systematically, and I realize the numbers are going to seem a bit arbitrary, but see this as some kind of proof-of-concept approximation.
For me, the bottom 10% of the reviews can (hopefully) be dismissed quickly within the first few paragraphs. Let's say I get maybe 1 reviewer in there I don't like and dismiss right of the bat as soon as I see the username. I've managed to dismiss 4-5 reviews really quickly. Where does that leave me with the rest of the 29-30 reviews? I have to get more picky. Let's say I dismiss a few others too, like another 10% of them that make only a slightly better first impression than the worst ones, and maybe 2 of them that I just dismiss because they are too damn long. I've managed to filter out 5-6 more. I've managed to filter out about 10 of the 34 reviews thus far.
And here we have it. These are the ones I can easily pass any kind of judgement on without reading them. The other 24 I would have to read. Let's make the generous assumption that these reviews are of approximately equal length, and that this length is close to the average of reviews for the release. I still have somewhere around 99000 characters that I can't easily judge without reading them. If we use the same reading speed as before (I suspect mine is lower), I still have 70 minutes
in front of me. I could almost listen to the entire album two times.
So what would I do in this case?
I'd pick the first good review I found. Usually this means a recent one, since I have no way to differentiate between them without reading them and they are sorted by submission date. Then Maybe I read a couple more (3 or 4 at most, and I'll probably try to find a negative one just for good measure). Then I wouldn't touch the other 20 or so reviews. Or I'd just fuck off and find somewhere else to read about the album. Or maybe listen to it myself (depends on why I'm reading, I've mostly heard the albums I read about).
In any case, the problem remains. About 25 of those reviews are useless. They aren't making anyone happy, maybe aside from the author. They're nothing but clutter taking up space at this point, and they make finding the worthy ones hell. Usability of the service goes to utter shit. The fact that they're sorted by submission date also means that better reviews get drowned out simply because they are older.
And it gets worse all the time, seeing as more reviews are still coming in. The last Time I
review is from February 2015. Give that one a few more years, and it's going to be utter hell.
But then again, the following still applies:
The idea is that it's unfair to people who came into the game late. Being born 50 years after Gutterscream and Napero shouldn't be held against somebody who wants to review. Yeah, we get redundancies and noobs who should really learn their shit before embarrassing themselves but it's just seen as a level playing field. I hate accepting the trillionth Rust in Peace review as much as anybody but it's a side effect of a fair principle, in my eyes.
How to solve a problem like this? No idea. But also not really my problem since I don't run the site. Maybe it's an interface issue, and there's some clever solution that would make navigation easier. Maybe the idea of being able to upvote good reviews would help, simply so you could sort them based on the number of upvotes and get the crap at the bottom. But I don't know.
The point is, when review pages get too large they get problematic and not very user-friendly. And it drags down the quality of the service from a user perspective.