|
Audio Asylum Thread Printer Get a view of an entire thread on one page |
For Sale Ads |
173.77.11.71
In Reply to: RE: That was Salvatore's point, about the Recommended Components.(nt) posted by DAP on May 08, 2015 at 10:06:40
All Arthur Salvatore revealed in his criticism of Stereophile's "Recommended
Components" is that he doesn't understand statistics. He assumes that the
products chosen for review followed a Gaussian distribution (bell curve)
centered on a mean level of performance.
As I have repeatedly explained in both the magazine and on this forum,
this is not correct: our selection process strongly favors better-sounding
components, much like New York City's rating of restaurant public health
safety is dominated by A-rated establishments. (See the graph in the linked
story below.)
John Atkinson
Editor, Stereophile
Follow Ups:
Below is the actual article concerning this issue posted on my website. Decide yourself whether the "statistics" present a "problem" or not.
As also can be seen, despite Atkinson's deliberately misleading assertion, there is more than one "scenario" in which I do NOT use a "bell curve".
The bottom line is obvious to anyone with an independent and mature mind; in no area of serious reviewing, no matter what prior precautions are taken, is it reasonably and LEGITIMATELY possible to find 60+ CONSECUTIVE "anythings" that are worthy of "recommendation" to your readers.
--------------------------------------------------------------------------
A helpful member of Vinyl Asylum posted a revelatory and devastating message on March 28, 2003 (Message 198755). This is the relevant part:
"(Stereophile) went through 18 straight issues in 2001 and 2002 wherein every component they reviewed (yes, every last one) made (their) recommended components (list). In 1989, 70% earned recommendation. I wrote them a letter on this which was never published."
An Analysis of this Important Revelation
I have checked out the number of reviews in 2001 and 2002. The smallest number for any "18 straight issues" was 60. The highest was 69. Let's give Stereophile a real "break", and go with "only" 60 (the minimum). So, what are the odds of 60 components, in a row, being honestly "recommended"?
That depends on the odds of any single component honestly making the RCL. First, using traditional and scientific "Bell Curves", only around 10% (10/100) of audio components (or any "consumer product") are defined as being "excellent". Accordingly, that would mean that if chosen randomly, the odds of 60 consecutive components being "excellent" would be the number "10" to the power of "60" (that is 10 with 60 zeros!).
That's an impossibly (and damningly) large number, which ends any doubt. However, let's now assume that the sample of reviewed components was NOT "random". In short, we're going to give John Atkinson every possible benefit of the doubt. Here are two more "scenarios":
Scenario 1
Let's give John Atkinson a second real "break". We will assume that Atkinson is a true "genius" at "prejudging" components, so only the models with the best chance to make the RCL were reviewed. Accordingly, we will concede that Atkinson has the capability to eliminate 80% of the initial pool of 100 beforehand. Thus, only 20 (100-80) components are still left in the pool, instead of 100, thus greatly increasing the success rate.
This means that 80 of the initial 90 "non-excellent" audio components are now removed from the pool (that's 88.9% of them). (Actually, in "real-life", I know of no audiophile, no matter how experienced, including myself, with this much audio foresight.) Still, this unprecedented ability will now increase the odds of success all the way up to 10/20 or 50%, instead of (the purely random) 10% (10/100).
So, to summarize, we're going from the initial "success rate" of 10% to 50%, just like "flipping a coin". Now, what are the odds of honestly flipping a coin "heads" or "tails" 60 times in a row:
More than 1,000,000,000,000,000,000 to 1
That's One Quintillion; or a Billion Billion. This is also an astronomical number which is beyond any practical use in the human world.
In short, it's Impossible, even if Atkinson is a genius, to have 60 consecutive components recommended when the process is honest.
Scenario 2
Let's give John Atkinson a third real "break". In fact, we're going to go beyond even being "open minded". Let's now assume that Atkinson is even above a genius, and is actually a (secret) "Superhuman", with powers of foresight far greater than anyone who has ever lived on this earth (like "Clark Kent"). We will increase the 50% success rate even further: Up to 75%. So...
What are the odds now of 60 consecutive components honestly making the RCL with even a "Superhuman" prejudging the components and a success rate of 75%?
More than 31,000,000 to 1
Accordingly, "the bottom line" concerning the Stereophile RCL is simple, obvious and incontestable:
The Stereophile Recommended Component List is a Total Fraud
But the components are graded class A+ through E, so presumably only the A's would be claimed as "excellent"? I've always looked at the list as a loose ranking of (nearly) all the components that Stereophile has tested over the last three years.
Daniel
> I've always looked at the list as a loose ranking of (nearly) all the
> components that Stereophile has tested over the last three years.
In essence, that's what it is, though it does include comments based on
longer-term experience with some products, such as those the reviewer has
purchased or has on formal long-term loan.
To address Mr. Salvatore's point, that there are too many products in the
list, the late J. Gordon Holt once made the same point at a Stereophile
writers' conference. So we went through the then-current list and I asked
Gordon which products we should omit, given his feeling that there were
too many. At the end of that exercise, he had to admit that there weren't
any that deserved to be dropped.
John Atkinson
Editor, Stereophile
I disagree with this methodology. The selling price of the component has serious relevance and must be taken into account.
"Excellent" should mean the top 10% performers within any particular component/price category, such as amplifiers in the $ 1,000 range, or speakers in the $ 5,000 range.
Audiophiles are looking for exceptional value/performance in each range, the upper 10%, and audio magazines are supposed to assist them in this quest, which is why they subscribe in the first place.
However, the one exception would be "Class A", which is "the best". That explicit definition should always limit the number to a handful, at most, even if there were hundreds of qualifying components, so it would be even less than 10% in this one instance.
In general, before something is legitimately "recommended", there should be something "exceptional" about its performance, either for its selling price or in absolute terms.
Finding 60+ consecutive exceptional audio components (or any product) is not feasible with any legitimately critical reviewing method currently known to me. That is why it has only happened in a commercial audio magazine and no where else to my knowledge.
of every component reviewed (BTW, I'm pretty sure I've read at least a couple of JA's measurement sections wherein he wrote that he could NOT recommend the DUT). And, the list does include a "$$$" for components considered to deliver exceptional performance for the price.
""Excellent" should mean the top 10% performers within any particular component/price category"
Meh, most audiophiles are on a budget and will look for the best-sounding product they can afford. The RCL lists the relative (and subjective) rating, regardless of price and gives you the price right after the product name. As I see it, the RCL helps you define your short list of products to audition for yourself. I suspect most readers are plenty smart enough to figure it out for themselves (ie, I think your proposed scheme of ranking within certain price points would be LESS helpful). YMMV.
Everyone has an opinion about RCL and I think his conclusion is the poorest assessment yet. Fraud is a terrible accusation in this instance. Who are the victims?
I like Salvatore's ideas about ranking, and would prefer that approach to Stereophile's, and I understand that manufacturers might prefer Stereophile's. But I think it would be very difficult to argue that Stereophile's doesn't come with full disclosure, no fraud. No fraudster ever took that much time and effort to spell out the selection process. It's not difficult to believe Sam Tellig when he says JA runs a straight ship.
Daniel
...quite a few sub-topics in a stream of conscientiousness."No fraudster ever took that much time and effort to spell out the selection process."
Still entertaining the parameters for fraud?
Edits: 05/12/15
if your point is that my statement "No fraudster ever took that much time and effort to spell out the selection process" is not factually correct, you're probably right, fraudsters do work hard for their money.
My point is that contrary to Salvatore's, it is not tenable to claim the Stereophile RCL is a fraud. I regard my conclusion as being stronger than my arguments.
Best regards,
Daniel
Post a Followup:
FAQ |
Post a Message! |
Forgot Password? |
|
||||||||||||||
|
This post is made possible by the generous support of people like you and our sponsors: