|
Audio Asylum Thread Printer Get a view of an entire thread on one page |
For Sale Ads |
4.237.89.51
In Reply to: Characteristics of consumer-oriented audio publications posted by Avocat on March 20, 2007 at 09:37:07:
>I think that most readers would like to see at least some reports of
>blind testing IN SOME FORM...
I see people occasionally making this statement, but I have to say
that it is not born out by my own experience. Back in the 1990s we
published several large-scale blind tests of loudspeakers, at
considerable expense. While I don't regret the effort -- the tests
gave me significant insight into the listening abilities of the
magazine's review team -- they were unpopular with the magazine's
readership. As an editor, I have no future if I publish material that
people have little desire to read. :-)
John Atkinson
Editor, Stereophile
Follow Ups:
Based on your blind tests, years later in 1994 I auditioned used EPOS ES11 speakers so well liked under blind conditions, as new satellite speakers to replace my Spica TC50's.These small one-cap crossover speakers would otherwise have been completely ignored based on their small size and modest price.
I bought them used for $400 and they were excellent satellite speakers when used with a 70Hz./24dB per octave Marchand crossover and playing near-field at the modeast SPL's I prefer.
The original owner thought the bass was too weak and he was a typical high-end anti-subwoofer audiophile with speakers that desperately needed subwoofers.
I still use the EPOS speakers since no other small satellite speakers I've tried at home have been significantly better so far.
To thank you for this unusual test, I will at every opportunity give you a hard time here to encourage you to get back to work on Stereophile and stop wasting your valuable time here, thereby doing you and Stereophile readers a favor (and you probably thought I was just another argumentative old geezer suffering from Irritable Male Syndrome)
Like you are.Small cheap-looking speakers and other components sound better to some people when they can't see them during the audition. So blind tests work for speakers even though everyone agrees they all sound different. They all look different too.
be seeing a complete reversal of SF policy. No doubt shortly after blind testing will be adopted by pretty much everywhere.This will also lead to a revival of the B&M sector however with the twist that most will combine audio with Wardrobe and other types of blinds.
Great news actually, why didn't you speak up earlier?
I would expect that you would have even less future if you were to publish material that your advertisers do not wish their prospective customers to read. ;-)
__________________________________________________
Boo!
> I would expect that you would have even less future if you were to
> publish material that your advertisers do not wish their prospective
> customers to read. ;-)
More uninformed trolling. No, "Ivan303," I don't select either what I
publish or how I choose how to run my magazine based on what
advertisers might wish readers to read. As my mentor John Crabbe,
editor of Hi-Fi News 1964-1982, used to tell me: if you have readers,
you will have advertisers; but if you pander to advertisers, you will
no longer have readers, which in turn means you will no longer have
advertisers.
John Atkinson
Editor, Stereophile
-------------"I have found that if you love life, life will love you back." -Arthur Rubinstein (1887-1982)
nt
Count me among those who would like to see more double-blind testing, that is of those who make a living claiming to hear what many cannot. It is the editors and reveiwers who should be required to prove they actually possess the listening skills which they purport to have. The skill to make the draw the kind of distinctions which they make in their monthly reviews.It is those with the golden ears who should submit to double blind testing to determine whether they in fact hear meaningful differences, e.g. distinguish between the CD and Super CD layer. If they where able to do this consistantly, I might pay more attention.
Testing the golden ears is the best use of DBT.
...those who make a living claiming to hear what many cannot.Don't worry, in the absence of such testing you are protected! Every major audio publication is signatory to the 1989 International Hirsch-Houck Convention. In a nutshell, this states: If you cannot hear it, you cannot read about it, even when what you cannot hear sounds the same. When reviewers write about what, for you, is consigned to silence, their words actually become invisible. In fact magazine racks contain entire audio publications you can't subscribe to, much less see. This is for your own good and is the true meaning of 'double blind tested reviews'.
By the way, could you please identify a few contemporary reviewers who claim "to hear what many cannot".
"'By the way, could you please identify a few contemporary reviewers who claim "to hear what many cannot".'" HTML tag not allowed
I stand for DBT'g reviewers (and their editors).
many of which, the industry needs to 'hang on for dear life'. it isn't going to happen. a top 3,000 recommended components list is more likely.
Start your own DBT-tested publication, and you will make a fortune. You'll be rich beyond your wildest imagination. There has to be literally hundreds of people that share your views.
That is no defense. You assume the editor and his reviewers could not even tell the difference between a CD and a Super CD. But that's my point, I don't think they could either, nonetheless they routinely describe hearing differences which are much more subtle. Small wonder they run a publication constantly loosing money.
Have you ever read a review that correlated written word wth what you actually experience with your ASL Hurrcanes? Or with your Alons? They've been extensively reviewed. If so, then you have a reviewer whose "golden ears" match your own. If not, you've got a reviewer whom you shouldn't trust. While we're at it, did you do DBTs when you chose your equipment? I'm just curious. Bet you just listened...The point? Proving they can hear differences is not as important as finding the reviewer whose music taste and equipment biaases most closely resembles your own. Then, if they describe what you hear with your equipment, you'd be well-advised to listen to what they say.
nt
(nt)
is a pretty good indication of how well they hear. It's a very simple, inexpensive but effective way to do things. BTW, most reviewers have regular day jobs. They don't make a living as a reviewer.
(nt)
However I would like to see magazines publish repeatable reviews/tests, meaning that more than one independent reviewer gets his independent unedited comments published about the same component ... and those reviews end up being similar, rather than seeming like the two or more listeners were NOT listening to the same component!I'd bet you my entire savings (almost $73 and going up, up, up with my investments in Bulgarian bean futures) that no two reviewers would independently write the same or similar comments about any wire!
If a review or test is not repeatable, it is probably worth zero.
Maybe less than zero if your review.
heh hehBut even if a test IS repeatable (Consumer Reports), it still may not be useful for any specific reader, although it's likely to be at least somewhat useful for most readers.
.
.
.
that proved to be pretty interesting. I'd get some, check them out, and then send a pair to a friend (who lives several hours away) and let him try them. Two completely different systems (one tubes, one SS, one with full-range speakers, the other with monitors), two different tastes in music, two different rooms. You'd be surprised by the similarities in our conclusions.
...Mr. Ass Nut thinks, it happens all the time.Some years ago in my TAS days, Cardas sent me three different sets of his cables at a speaker manufacturer's request to be used with his speakers - Golden Reference (fat burgandy, expensive), Golden Section (fat grey, less expensive) and Neutral Reference (thin tan, inexpensive), IIRC.
I listened to them all and formed my opinion and when I was done sent them all to another reviewer, Neil Gader, who wanted to hear them.
After he listened to them, we shared our impressions and they were pretty much the same - we both preferred the smaller, least expensive ones.
We had the same conclusions. (Note: the wires I am talking about aren't "cheap" to begin with, so don't jump to conclusions.)
I thought your publication did a terrific job doing those speaker tests, particularly given the complicated logistical problems. I was also impressed with the statistical analysis of your results.I know it is not practical to do such tests on a regular basis, but I would like to see your magazine mix things up a bit more and do different things once in a while, just like those group tests.
I can guess why such tests may seem unpopular to your readers. Those most likely to react will be readers who own the products under test. If you test ten speakers, then the owners of nine will be unhappy with the results and only one will be happy. By contrast, the results of regular reviews are a bit more ambiguous.
s
> ...they were unpopular with the magazine's readership.>The pro-DBTers are a very tiny but vocal minority of audiophiles.
And then there are the people who hear about DBTs and think it's a great idea but don't really understand about the rigorous methodology, the statistics involved and the expense to do it right.And then once you're done, you've shown that differences between a tubed amp and a solid state amp can be identified blind.
Yawn.
I think most people just want to be entertained reading about audio equipment.
In Reply to: Re: Characteristics of consumer-oriented audio publications posted by John Atkinson on March 21, 2007 at 04:23:25:
> ...they were unpopular with the magazine's readership.>
The pro-DBTers are a very tiny but vocal minority of audiophiles.Please don't categorize me among the DBTers who favor "quick-switch" ABX machines. - I favor blind testing in general, (in ADDITION to conventional reviews), and I would favor letting the SF staff determine what procedure to follow.
And then there are the people who hear about DBTs and think it's a great idea but don't really understand about the rigorous methodology, the statistics involved and the expense to do it right.
And then once you're done, you've shown that differences between a tubed amp and a solid state amp can be identified blind.
Yawn.Yes, some people find, after extended listening, that they prefer the distortion inherent in "tube sound" despite the fact that such distortion might have caused them to downgrade a tubeed component when listening to it in a blind comparison. Nothing wrong with that, but what's new? - Yawn.
"I think most people just want to be entertained reading about audio equipment."You're certainly welcome to your opinions, of course. Some of us want to listen to good music on the best equipment available within a rational budget. And we are sometime helped toward that goal by reading equipment reviews in SF and elsewhere. (SF has noted that the equipment reviews continue to be the most popular articles.)
...reading The Audio Critic instead of Stereophile.They are proponents of DBTs.
> Yes, some people find, after extended listening, that they prefer the distortion inherent in "tube sound"...>
LOL! As opposed to the distortion inherent in transitor sound?
"LOL! As opposed to the distortion inherent in transitor sound?
That statement sounds like something Aczel would say."-- Yes.
Suggest you read the article on "valve sound" available online at: http://en.wikipedia.org/wiki/Valve_sound. - Note the paragraph that begins: "In particular, it turns out that the "warmth" and "richness" typically associated with "valve sound" is DUE TO SIGNIFICANT LEVELS OF 2ND ORER DISTORION, typically coming from a single ended (and thus by definition class A) SE stage, often the output stage. This being a classic tube amp design."
I'm not saying that SS doesn't have it's own forms of distortion, merely that the preference for tubes may relate to a preference for the "warmth and richness" resulting from such 2nd order distortion.
as tubies would have us believe. That ever so slight rolloff of tubes has also been known to smooth out the high frequencies of metal dome tweeters and the dreaded digital glare of red book. This is often miscredited to such things as linearity...It sounds like Aczel makes objective observations and sticks by them. While never having read his widely trashed publication myself, at least he probably never had said that his opinion was subjective. Does he trash subjectivists as much as they freely trash him?
Aczel versus John Atkinson for a discussion of whatever comes up at Hi Fi '07. I'd almost forgo a live music demo at the show for that one.
Certainly not frequencies."Note the more extended ultrasonic bandwidth of tetrode compared with triode in these graphs: –3dB at 115kHz vs –3dB at 87kHz, respectively. "
> Aczel versus John Atkinson for a discussion of whatever comes up...>It was JA vs Arny Kruger a couple of years ago at a Stereophile Show, IIRC.
Kruger is a noted radical objectivist on rec.audio-opinion, rabid pro-DBTer and co-inventor of the ABX box.
Arny was more bark than bite. It was JA by a unanimous decision.
You can probably find a transcript on the Stereophile site.
Azcel is now just an embittered, angry old man - not much theater in seeing him get out-debated.
In the debate, Arny Krueger wiped the floor with John Atkinson. Atkinson wasn't able to make a single point against his arguments.The link below is to a rather self-serving article by Jason Victor Serinus article which includes a link to the an audio file of the debate. The link is in the editor's note just above the picture.
____________________________________________________
"Opposition brings concord. Out of discord comes the fairest harmony."
------Heraclitus of Ephesis (fl. 504-500 BC), trans. Wheelwright.
as tubies would have us believe. That ever so slight rolloff of tubes has also been known to smooth out the high frequencies of metal dome tweeters and the dreaded digital glare of red book. This is often miscredited to such things as linearity... HTML tag not allowed
"I think most people just want to be entertained reading about audio equipment."mikey, I think there are a lot of readers who also want to see their equipment rated highly on the 500 components list.
(nt)
While I will not take sides on the blind testing issue (way too mych hot air on that) I particularly enjoyed the speaker tests that were published in the early 90s. It gave me a chance to evaluate reviewers as well, and it was interesting to read the comaprison of their comments at the blind testing versus the detailed reviews that followed. i certainly understand that they were a ton of work, but I also think they gave an interesting insight into reviewer listening.
> I think that most readers would like to see at least some reports of
> blind testing IN SOME FORM..."I see people occasionally making this statement, but I have to say
that it is not born out by my own experience. Back in the 1990s we
published several large-scale blind tests of loudspeakers, at
considerable expense. While I don't regret the effort -- the tests
gave me significant insight into the listening abilities of the
magazine's review team -- they were unpopular with the magazine's
readership. As an editor, I have no future if I publish material that
people have little desire to read." :-)John Atkinson
Interesting. - What did you discover about the listening abilites of the magazine's review team? Also, what was the basis for your conclusion that blind testing reports were "unpopular" with your readership? For example, did you conduct some sort of poll, or was your conclusion based on informal discussions, discussions at trade meetings and the like, or based on letters to the editor? More to the point, since the blind testing issue has been the subject of heated debate and ongoing discussion (despite determined efforts to make it go away), to what extent have surveys been made of your readers' opinins on the subject. Were they worded by independent professionals in the polling field, or by your own staff?If a further, more scientific survey were to be considered (unlikely of course, but perhaps possible), I think it would be important to remove some of the usual caracatures at the outset. For example, it should be made clear that the reports of blind listening tests would be supplemental to the current equipment reviews and would not replace them. Also, that the testing format would not comprise the rapidly switched "sound bite" ABX approach, and that instead, the methodologies would be selected by Stereophile's own staff. Questions might be asked about the willingness of readers to accept a slight increase in the (extremely low) $11 subscription rate.
Thanks for your consideration of the above.
Avocat, use the search function looking for text "DBT" authored by SamTubes in Critic's Corner. Happy Reading.
Good ol' SamTubes. An early PIA in the AA. Sadly, though he is gone, his meassage and methods continue unabated....
You know, those drums we beat to help people take long vacations. ;~)I think he has been in seclusion somewhere recruting an army to fight his fight. Based on what I've observed in here lately, he must have an army of five already.
But really, what is with these people?? This hobby is supposed to be fun. Reading an enthusiast magazine is fun too.
but anymore I believe it has more to do with the way some people deal with middle age, having more time on their hands and more bucks in their pocket. Some people deal with it well, others apparently don't.Myself, I think they're probably [insert political party opposite your own]. ;~)
Why discuss DBT here when we've done it so many times before.
> What did you discover about the listening abilites of the magazine's
> review team?
That their abilities to characterize small differences under blind
conditions was no different from their abilities to do so
sighted, once interfering factors such as the order of presentation
of the speakers and the effect of the listening position had been
compensated for. In fact, I continue to listen with my writers when
possible, to see if I hear what they describe, though this is
invariably under sighted conditions. I do.
> what was the basis for your conclusion that blind testing reports
> were "unpopular" with your readership?
The usual data: newsstand sales figures, reader's letters, live
feedback from readers at Stereophile shows, surveys on our website.
> since the blind testing issue has been the subject of heated debate
> and ongoing discussion (despite determined efforts to make it go
> away), to what extent have surveys been made of your readers'
> opinins on the subject.
You can find the most recent poll at http://cgi.stereophile.com/cgi-bin/showvote.cgi?427. Interesting results, I am sure you will agree.
> Were they worded by independent professionals in the polling field,
> or by your own staff?
Our own staff. But shouldn't you be the one answering these
questions? For example, when you stated that you "think that most
readers would like to see at least some reports of blind testing...,"
what research have you done to be able to say that? What polls can
you cite? What market research can you refer me to that shows
that "most" magazine readers would like to see blind tests performed?
So far, all you have done is offer unsupported opinion. Not that I
have any objection to that but it seems curious that you appear to be
doubting my own experience in this matter while not feeling the need
to offer any data of your own.
I should also point out that if I am wrong in my editorial decisions,
the marketplace is quite efficient at conferring the usual
consequence, vide the ongoing failures of editors and magazines who
fail to offer readers what those readers value: the editors of Audio,
High Fidelity, The Audio Critic, etc, all got it wrong, for example.
If I am wrong about this, then ultimately my ability to pay my
mortgage will be compromised :-)
John Atkinson
Editor, Stereophile
I second the compromise suggested below. Sounds like a good approach.Jim
Personally I would prefer a compromise of some sort. Logistically doing blind tests for everything can be a nightmare and would slow down the process. As such it's really not practical. Having said that I would like to see some way of blind testing speakers to see if the responses to the sighted reviews hold true. Maybe a sampling once a year? The reason I suggest this is cost. I would like to hear a reviewer critique speakers based on several common parameters and then suggest what price range they feel that quality of sound could substaniate. Of all the arguements the ABX people make I think the one regarding cost is the most relevant. I think one expects certain capabilities when they know how much something costs. I think it's human nature to want to be able to give a favorable review of someone else's life's work - especially when they are extremely expensive to desin and manufacture. (I experiance the same thing when i buy something or try a tweek. You find yourself wanting to support your own decision). As such I think - especially with speakers- a yearly calibration would be of some use. (Given the decline of high end audio it is virtually impossible for the average person to listen to even a fraction of what is available or recommended. As such - while your recommend people listen on their own and don't buy based on your reviews - it is often practical to do just that - as there is really no other choice. I bought Triangle speakers based on Sam review and the fact that I own tubes. When I did this there was no one within 250 miles that was selling them)
And it continues.... - robert young 14:04:01 03/21/07 (0)
In Reply to: Re: Another Inquisition begins :-( posted by Avocat on March 21, 2007 at 12:33:46:
JA is absolutely right. Why does he have to provide you with back-up when you don't do the same?? This is, indeed, Critics' Corner, but it isn't " Make the Editors Play By Rules That We Don't Have to Follow."
---------------------------------------------------What I asked JA was whether SF has conducted surveys of its readership regarding whether they would be interested in seeing articles with the results of blind testing from time to time. It's a reasonable question. From the information provided, SF hasn't conducted any scientifically based surveys. I don't recall then asking JA to get more "backup".
Regarding your requirement that I get more back up, what kind of "back up" would satisfy you Bob? Do you expect me to pay for an independent survey of audiophiles or of SF subscribers before I'm permitted to post opinions or questions on CC? (The point of your note, of course, is that you don't like what I'm saying, and you think demanding that I provide lots of "backup" before being permitted to express my opinions on this forum is a convenient "gotcha" you can use to put me down. -Isn't that about the size of it Bob? - Once again, what specific kind of "backup" would satisfy you?
My OPINION that lots of audiophiles and SF subscribers would like to see at least some blind testing results from time to time, perhaps once or twice a year, is based on reading many, many discussions of the subject over the years on a number of discussion groups, and even from reading letters to the editors on SF and other publications. IT'S ALSO BASED ON READING COMMENTS AND OPINIONS POSTED ON THIS PARTICULAR DISCUSSION STRING THIS WEEK. - Look around you Bob.
First, please try to respond directly to the post you are answering. It's hard to reply in a timely manner when you've answered someone else.Second, my name is not "Bob," it's Robert. This little factual tidbit is available by looking at my username. The patronizing and confrontational tone you take doesn't help your argument(s).
Third, let's look at "backup," shall we? I'll pick just one item. Here's what you posted initially:
"I think that most readers would like to see at least some reports of blind testing IN SOME FORM (for example, permit the use of more extended listening periods) in addition to the extensive, philosophical reviews of single, individual components."
You questioned whether JA had done any research: he replied with his answer, one you didn't like. You have asked him to give you "scientifically based surveys," but you have not provided any "backup" like that at all to suppport your INITIAL claim that "most readers would like to see at least some reports of blind testing..." Sorry, Avocat, there's no "gotcha" there, and no, I'm not looking for you to pay for an "independant survey" just to satisfy me. However, I do expect that you have SOME support when you make a claim like you did above. If not, then you have to be willing to acccept no back-up from JA or anyone else either. This has nothing to do with someone (supposedly me) trying to suppress your opinion. On the contrary, you didn't present this as an opinion, you made a statement of (purported) fact.
"My OPINION that lots of audiophiles and SF subscribers would like to see at least some blind testing results from time to time, perhaps once or twice a year, is based on reading many, many discussions of the subject over the years on a number of discussion groups, and even from reading letters to the editors on SF and other publications. IT'S ALSO BASED ON READING COMMENTS AND OPINIONS POSTED ON THIS PARTICULAR DISCUSSION STRING THIS WEEK. - Look around you Bob."Well, try presenting it as such right from the start. Opinions are fine. Unsubstantiated statements of fact are fair game for questioning. I shouldn't have to tell you that the rules ought to be the same on both sides of any debate.
I too have read the entire thread, Avocat. No need to "look around," either. Doesn't seem like "most people" are on your side, and even if every post here except mine supported your ideas, it would hardly represent the readership.
Really, you should try not to be so condescending in your replies. I have a difference of opinion with you, nothing else. That is at the very essence of debate. If that's not ok with you, if someone challenging your opinion is not kosher, then I respectfully suggest that you either develop some thicker skin, or try a different venue. Disagreement happpens here with nearly every thread: that is the nature of discourse. Disagreement is not an open invitation to boorish behavior.
...it is might brave and noble of you to keep responding to repetitive, difficult and even hostile queries. I certainly would have bowed out a long time ago, ungracefully too.I am dismayed though that the consumers, industry, et al have not railed for the pursuit of DBT's. If an initial attempt to pursue knowledge is unfruitful at first, then abandonment of the pursuit is not the answer [something about a spider on a wall in Scotland comes to mind].
I do not blame you for giving up..realities of survival and all. It would be patently unfair to expect an individual or group to sacrifice for something that has not gained quorum with the masses yet. As with so many other things in life I'll have to bear with my dissapointment of the slowness of real progress.
Posted by John Atkinson (R) on March 21, 2007 at 10:37:33
In Reply to: Re: Characteristics of consumer-oriented audio publications posted by Avocat on March 21, 2007 at 08:13:30:
- - John, my original note in this section discussed five or six areas in which I suggested possible improvements in SF. - You seem to be interested only in the blind testing issue, for some reason. What about the other suggestions? - -
>
> what was the basis for your conclusion that blind testing reports
> were "unpopular" with your readership?"The usual data: newsstand sales figures, reader's letters, live
feedback from readers at Stereophile shows, surveys on our website."Because your readers have been exposed to SF anti-blind-test propaganda over the years, I doubt that these discussions/letters/surveys are very significant. Sales figures (reflecting newstand purchases by non-subscribers unfamiliar with the issue) don't seem very significant. Regarding the poll, the results do seem to suggest a wide variety of opinions. In any event, I note that only 26% of those responding indicated that they didn't care about the subject. Also, many audiophiles seem to equate blind testing exclusively with ABX dbt methodology, with which is often smeared.
Some seem to think that because blind test results are sometimes ambiguous and don't produce a clear "winner" among several cadidates, the tests are meaningless and worthless. To the contrary, results that show differences of opinion or ambiguity amoung certain components can be quite valuable, in that they tell the reader something about the degree of audible improvement he might expect from a particular component, as well as the degree of audible improvement he may get per dollar spent on particular types of components. In other words, disclosing to the reader that the judement of a listening panel was substantially mixed about a particular component, as compared with others, would suggest that perhaps his budget would be better served by the purchase of another type of component about which the panel did discern significant, audible benefits, agreed to by most of the panel. - INCIDENTALLY, I HAVE NOT SEEN THIS FACTOR MENTIONED IN ANY STEREOPHILE ARTICLE ON THE SUBJECT OR DISCUSSED ON THE WEB BY ANYONE OPPOSING BLIND TESTING.
"You can find the most recent poll at http://cgi.stereophile.com/cgi-bin/showvote.cgi?427. Interesting results, I am sure you will agree."Thanks. The poll results are interesting, if inconclusive.
> Were they worded by independent professionals in the polling field,
> or by your own staff?"Our own staff. But shouldn't you be the one answering these
questions? For example, when you stated that you "think that most
readers would like to see at least some reports of blind testing...,"John, get a grip on yourself. - This happens to be an online discussion group, and, in fact, it's the "Critics Corner" secion of AA in which various opinions, questions, and viewpoints are supposedly welcomed. From the comments about my initial note, there does seem to be quite a lot of interest in these issues. Regarding blind testing, there obvioiusly are differences of opinion. But I'm certainly not the only one with such views.
--------------------------
"So far, all you have done is offer unsupported opinion. Not that I
have any objection to that but it seems curious that you appear to be doubting my own experience in this matter while not feeling the need to offer any data of your own."I asked whether, in view of the obvious and continuing interest in the subject, Sterephile has ever conducted a scientifically based survey or poll of the opinons of its readers on the subject. - Clearly, you haven't.
"I should also point out that if I am wrong in my editorial decisions, the marketplace is quite efficient at conferring the usual consequence, vide the ongoing failures of editors and magazines who
fail to offer readers what those readers value: the editors of Audio,
High Fidelity, The Audio Critic, etc, all got it wrong, for example.
If I am wrong about this, then ultimately my ability to pay my
mortgage will be compromised :-)"John, you have done a good job of keeping up the circulation of the mag over the years. ( Along with the editors of the Enquirer, Star, Sun, Playboy, etc.) Seriously, for a publication related to such a narrow interest group, SF has done very well, to your credit. Your staff is certainly doing a good job of selling advertising space. Incidentally, how does the revenue break down between subscription income, newstand sales, and advertising revenue. (Not asking for $$ figures, just interested in where most of the income comes from.)
As mentioned earlier, my comments are partially based on some general concerns about the audio and music industries as a whole, and not just the issues discussed above. - Per the discussion in today's Wall Street Journal, incidentally.
Jim
> my original note in this section discussed five or six areas
> in which I suggested possible improvements in SF. - You seem
> to be interested only in the blind testing issue, for some
> reason. What about the other suggestions?
I did note them. Some we already do, as Kal Rubinson
pointed out. Some are possibilities, some are impractical.
Any magazine continually evolves in response to market changes
and to reader's changing needs, so maybe you might see some
movement in the directions you describe. Or not.
> Because your readers have been exposed to SF
> anti-blind-test propaganda over the years, I doubt that
> these discussions/letters/surveys are very significant.
There are times when I just need to laugh out loud.
Your arrogamce astonishes me, Jim. So _your_ opinion
carries weight but those of other Stereophile readers
doesn't? I respectfully suggest you think again about
what you have just said, sir.
> Sales figures (reflecting newstand purchases by
> non-subscribers unfamiliar with the issue) don't seem
> very significant.
And again, I continue to be amazed by how much more
those without any publishing experience or hard information
know about my business than I do. :-)
> > shouldn't you be the one answering these questions?
> > For example, when you stated that you "think that most
> > readers would like to see at least some reports of blind
> > testing...,"
>
> John, get a grip on yourself. - This happens to be an
> online discussion group, and, in fact, it's the "Critics
> Corner" secion of AA in which various opinions,
> questions, and viewpoints are supposedly welcomed. From
> the comments about my initial note, there does seem to
> be quite a lot of interest in these issues. Regarding
> blind testing, there obvioiusly are differences of
> opinion. But I'm certainly not the only one with such
> views.
No you are not, but my experience has been that those
vociferously asking for a blind test regime in Stereophile
are a minority of readers and, to a large extent, aren't
even readers at all. I am curious, therefore, why you
feel that "most readers" of Stereophile share your
viewpoint? Why do you think that? Please stop avoiding the
question.
> John, you have done a good job of keeping up the
> circulation of the mag over the years. ( Along with the
> editors of the Enquirer, Star, Sun, Playboy, etc.)
Ah, time to reach for my can of "Troll-B-Gone," I see. :-)
John Atkinson
Editor, Stereophile
John
I have been a Sterophile subscriber for almost 15 years (I also subscribe to other publications and read all of the online pubs)I remember some of you blind tests. I believe some were with a panel that all listened together and then each was assigned a speaker to report on sighted.
Wasn't the panel test skewed by only one person being able to sit in the right spot? (I understand doing other wise would be a hassle)
Were the levels matched precisely system to system?
Personally I would prefer a compromise of some sort. Logistically doing blind tests for everything can be a nightmare and would slow down the process. As such it's really not practical. Having said that I would like to see some way of blind testing speakers to see if the responses to the sighted reviews hold true. Maybe a sampling once a year? The reason I suggest this is cost. I would like to hear a reviewer critique speakers based on several common parameters and then suggest what price range they feel that quality of sound could substaniate. Of all the arguements the ABX people make I think the one regarding cost is the most relevant. I think one expects certain capabilities when they know how much something costs. I think it's human nature to want to be able to give a favorable review of someone else's life's work - especially when they are extremely expensive to desin and manufacture. (I experiance the same thing when i buy something or try a tweek. You find yourself wanting to support your own decision). As such I think - especially with speakers- a yearly calibration would be of some use. (Given the decline of high end audio it is virtually impossible for the average person to listen to even a fraction of what is available or recommended. As such - while your recommend people listen on their own and don't buy based on your reviews - it is often practical to do just that - as there is really no other choice. I bought Triangle speakers based on Sam review and the fact that I own tubes. When I did this there was no one within 250 miles that was selling them)
> Wasn't the panel test skewed by only one person being able to sit in
> the right spot?
Yes. The first test used 4-5 listenrs at a time, which meant that the
tests ran over 2 days. Because it became apparent that the position
of the listener in the room was also a variable, in the subsequent
tests, we had just 2 listeners at a time take part, one sitting behind
the other, so that both were close to being in the sweet spot. This
stretched the testing to 6 days. It would have even better if we had
tested just one listener at a time, but this would have required too
much time just for the blind tests, let alone every other aspect of
the review, to be practical for a monthly publication schedule.
> Were the levels matched precisely system to system?
As closely as is possible with speakers, which have an unflat response
compared with amplifiers or digital front-ends. We matched the levels
based on the speakers' B-weighted sensitivities, which downplays the
effect of differences at the frequency extremes. We did publish the
experimental procedures in full detail, BTW.
One point that is continually glossed over by proponents of blind
testing is that when the audible differences are small in absolute
terms, which is very often the case, designing a blind test that has
sufficient sensitivity to detect such differences is not trivial.
John Atkinson
Editor, Stereophile
"One point that is continually glossed over by proponents of blind
testing is that when the audible differences are small in absolute
terms, which is very often the case, designing a blind test that has
sufficient sensitivity to detect such differences is not trivial."Your point here is self-defeating, "audible differences that are small in absolute terms" will not be any easier to detect in a sighted test, the results will just a lot less reliable as a result of uncontrolled listener bias.
Music making the painting, recording it the photograph
> Your point here is self-defeating, "audible differences that are
> small in absolute terms" will not be any easier to detect in a
> sighted test, the results will just a lot less reliable as a result
> of uncontrolled listener bias.
It depends on the nature of the difference, the conditions of the
blind test, and the conditions of the sighted test. When real
differences exist, an endless series of inadequate blind tests that
produce null results, a la Tom Nousaine, is meaningless.
This was the point of the parable I related at the HE2005 debate,
reprinted at www.stereophile.com/asweseeit/705awsi .
The differences between the amplifiers under test had not been
detected by the formal blind test but had been under normal, sighted,
longer-term listening.
John Atkinson
Editor, Stereophile
> This was the point of the parable I related at the HE2005 debate,
> reprinted at www.stereophile.com/asweseeit/705awsi.
> The differences between the amplifiers under test had not been
> detected by the formal blind test but had been under normal, > sighted, longer-term listening.
Your little parable demonstrates that detectable differences are not the sole criteria for long term satisfaction of any given device, I can think of a very good example that captured the public interest in the UK. Volkswagen was gearing up to re-launch the Skoda in UK, In the consumer clinic, the car performed excellently until the test clinic subjects were made aware of the brand, needless to say it was downhill from there on. No matter how you cut it, once the identity of the object under test is known, expectation bias becomes an unavoidable skewing factor. If it were not the case, getting reliable results will simply be a matter of conducting blind tests immediately after sighted tests (where valid differences were reported) and the results of the blind test will correlate with the results of the sighted tests, however that is opposite of what is widely obtained in these tests. Expectation bias depends on a priori knowledge, a priori knowledge is always present in a sighted test.
Music making the painting, recording it the photograph
> > This was the point of the parable I related at the HE2005 debate,
> > reprinted at www.stereophile.com/asweseeit/705awsi. The differences
> > between the amplifiers under test had not been detected by the
> > formal blind test but had been under normal, sighted, longer-term
> > listening.
>
> Your little parable demonstrates that detectable differences are not
> the sole criteria for long term satisfaction of any given device...
Perhaps you didn't read my essay carefully? Perhaps you didn't
comprehend what I wrote? My point was that all the non-audio factors
were working _in favor_ of the solid-state power amplifier. Yet my
dissatisfaction with its sound turned out to be very real. And that
shortfall in sound quality had _not_ been detected in what on the
face of things had appeared to be a well-organized and well-designed
blind test.
John Atkinson
Editor, Stereophile
> Perhaps you didn't read my essay carefully? Perhaps you didn't
> comprehend what I wrote? My point was that all the non-audio > factors were working _in favor_ of the solid-state power amplifier.Well, you are touching at the heart of why sighted test are inherently weaker than blind tests, the non-audio factors at play are not at the sole discretion of the observer, neither is the observer neccessarily aware of all his/her biases, a priori knowledge brings to the fore all biases not just biases the observer is aware of at the time of the test. A priori knowledge is skewing factor irrespective of the duration of the test.
Music making the painting, recording it the photograph
when the resulting preference is contrary to the bias established by the non-audio factors?
"So, what conclusion can you draw when the resulting preference is contrary to the bias established by the non-audio factors?"Your question was implicitly answered in the original post you responded to. The observer is NOT necessarily aware of all his/her biases and there is no realistic method available to the observer to address this weakness. As a result, claiming that the resulting preference was contrary to the bias established by the non-audio factors is not a possible outcome as it assumes that observer is fully aware of all his biases.
preference for the many reasons was not for the solid state unit?I've made quite a few comparisons where my favored brand did not "win" in the outcome.
"What makes you think that Mr. Atkinson's preference for the many reasons was not for the solid state unit?"???? You are going round in circles asking the same question in various guises, the answer remains the same. At any rate that is not the only problem with Mr. Atkinson's parable, just the one that is impossible to reliably mitigate against in a sighted test.
Music making the painting, recording it the photograph
> that is not the only problem with Mr. Atkinson's parable, just the
> one that is impossible to reliably mitigate against in a sighted
> test.
You continue to miss the point I was making, "audiohobby." There are
2 incontrovertible facts involved in my parable:
1) The solid-state amplifier could not be identified in the formal
blind test.
2) The solid-state amplifier did not satisfy me in long-term
listening, despite all the non-audio factors working in its favor.
And two articles of belief among so-called objectivists:
1) That blind tests, merely by being blind, are an effective means of
detecting audible differences.
2) Sighted listening tests are inherently unreliable, due to the
presence of non-audio factors.
If the facts behind my parable are true, then these two latter
beliefs are mutally incompatible. Either the blind test was
ineffective or the non-audio factors don't actually affect the
listener's reaction to any significant degree. You can't continue to
insist that both are correct.
John Atkinson
Editor, Stereophile
> You continue to miss the point I was making, "audiohobby."I am not missing the point, however you are as I addressed the selfsame issue upteen times in my replies to E-Stat.
> There are 2 incontrovertible facts involved in my parable.
Only the first statement satisfies that criterion.
> 2) The solid-state amplifier did not satisfy me in long-term listening, despite all the non-audio factors working in its favor.
Your comments do not take into account hidden or unconscious bias. It is impossible for the observer to reliably claim that they are fully aware of ALL their biases, as a result it is impossible to claim that all their biases favour a particular choice as you have claimed. A priori knowledge informs both the known and hidden biases, since the observer is not necessarily aware of their hidden (or unconscious) biases, how do they mitigate against it effects in a sighted test? They can't, it is an inherent weakness in sighted tests.
> 2) Sighted listening tests are inherently unreliable, due to the presence of non-audio factors.And you will be correct because of the reasons outlined in my previous comments.
Music making the painting, recording it the photograph
> > There are 2 incontrovertible facts involved in my parable.
>
> Only the first statement satisfies that criterion.
In your opinion. I assure you that my dissatisfaction was real. I find
it amazing that you now claim to be a mind reader in order to be able
to wriggle out from the implications of my experience.
> > 2) The solid-state amplifier did not satisfy me in long-term
> > listening, > despite all the non-audio factors working in its favor.
>
> Your comments do not take into account hidden or unconscious bias...
Nor do they take into account ESP or your claims to be able to
discern my state of mind a quarter century ago. Again: despite my
love for the Quad amplifier, despite the fact that it was very much
less expensive than the amplifier it replaced (meaning I made money
on the exchange), despite it being, small, cool-running, and despite
the hero-worship I felt (and still feel) for its designer, in the long
term, and without reference to other amplifiers, it proved extremely
unsatisfying in the task for which I had purchased it. That sir, is a
fact.
I suppose you will next suggest that I should have performed further
blind tests to "prove" that my dissatisfaction was illusory? Ecept
that I imagine that would have produced the same result as before, in
which case I would still be stuck with a sound that dissatisfied.
Only a fool or someone in the throes of religious zeal would regard
that as rational behavior, I respectfully suggest.
John Atkinson
Editor, Stereophile
you can't possibly understand your own preferences! Actually, I know why you didn't like the Quad in the end. It was the heatsinks on the front. That's it. I'll bet you don't like any amplifiers that have heatsinks on the front. It's that simple. ;)
Nice try, Did you forget that you asked that selfsame question a few posts back or you are just being malicious?
Music making the painting, recording it the photograph
Such involves the use of the question mark. Don't you know that?Malicious? Heavens, no. Just thoroughly amused. :)
How does your rhetorical questionC'mon, John you can't possibly understand your own preferences!
follow from this
"We are incapable of determining our own preferences and opinions? "
No, biases reliably.
Or are you simply being deliberately obtuse?
Music making the painting, recording it the photograph
Taken in the context of E-stat's entire post, it would fall far from being malicious, and instead fall into the category of leg-pulling humor.
At least someone understands my humor and the smiley/winky faces!
Now get back to Propheads where you belong.
> I'll bet you don't like any amplifiers that have heatsinks on the front. It's that simple. ;) <Or that start with the letter 'Q'.
Or maybe John equates it with the quadrophonic sound that was such a disaster.
Or maybe 4 is an unlucky number!
The possibilities are endless. :)
> In your opinion. I assure you that my dissatisfaction was real. I find it amazing that you now claim to be a mind reader in order to be able to wriggle out from the implications of my experience.Those are the facts with respect to biases, you might not like them and try and claim that they do not apply to your particular situation, but then you will not be different many other audiophiles who believe that some basic laws of human behaviour do not apply to them.
> Only a fool or someone in the throes of religious zeal would regard that as rational behavior, I respectfully suggest.
You are defending a position that is totally at odds with generally accepted behavioural science on this well-trodden subject with absolutely nothing except a piece of anecdotal evidence. Now that is irrational behaviour, on the otherhand a fool or someone in the throes of religious zeal may also regard it as rational behavior, I respectfully suggest.
Given the tone of last post, I take it you have nothing else worthwhile to say on the matter.
Music making the painting, recording it the photograph
> > In your opinion. I assure you that my dissatisfaction was real. I
> > find it amazing that you now claim to be a mind reader in order to
> > be able to wriggle out from the implications of my experience.
>
> Those are the facts with respect to biases, you might not like them
> and try and claim that they do not apply to your particular
> situation...
Again, you seem to be addressing a different point. My dissatisfaction
with the sound of the solid-state amplifier was indeed real. Why would
I lie about it, either to you now or to myself back then? All I was
concerned about at the time -- and remember that, as described in the
essay, I was not a reviewer at the end of the 1970s but a hard-line
"objectivist -- was buying an amplifier for use in my own system for
as little cash as I needed pay for the amount of power available. I
did what "objectivists" to this day recommend to audiophiles, which
is to choose a well-designed amplifier on the basis of cost, features,
and power, because in a formal blind test, the amplifier I wanted to
purchase had been shown not to sound any different from more exotic,
more expensive designs. As I said, for you now to claim that you know
more about my state of mind at that time than I did then is both
ridiculous and arrogant.
I have been forthright about the biases and belief I had at that time.
All you are doing is postulating that there must have been other,
hitherto unsuspected biases at work. As I said: mindreading.
> but then you will not be different many other audiophiles who believe
> that some basic laws of human behaviour do not apply to them.
I see you like to patronize those whose points you can't argue with
otherwise?
> > Only a fool or someone in the throes of religious zeal would regard
> > that as rational behavior, I respectfully suggest.
>
> You are defending a position that is totally at odds with generally
> accepted behavioural science on this well-trodden subject with
> absolutely nothing except a piece of anecdotal evidence.
No, I am honestly reporting my _experience_, experience obtained
without pressure from anyone else, and experience that ran counter
to my biases and expectations. Such was the cognitive dissonance
engendered by this situation that I didn't recognize the problem
until I realized that listening to recorded music, my passion, had
increasingly become less and less of my life. The change in amplifier
had been the only significant factor. Occam's Razor suggested that
the amplifier was the root cause of my dissatisfaction.
And if that was the case, then either the blind test had misled me
or my biases and expectations had not outweighed the evidence of
my ears.
And to take your argument to its logical conclusion, you appear to be
suggesting that all human experience, if anecdotal and not supported
by scientific experiments, must be discarded as meaningless. I
hardly think that is the case.
Take my recording activities: I must make literally 100s of judgment
calls during the production of a project, some under extreme time
constraints, and I need to be correct on every one of those decisions.
Should I move this mike back 6"? Should I apply +0.5dB of boost at
100Hz to compensate for a cardioid's early rolloff or +1dB? Should
I record at 96kHz or save hard-drive space and file-handling time by
using 48kHz. The list is endless, yet according to you, without
formal blind testing of each of those situations, none of those
decisions can be justified because each will be affected by my
baggage of biases and preconceived notions.
In that case, why do any of us bother doing anything?
I take it, BTW, that all your own decisions regarding what products
you chose to buy and how you choose to use them have been made on
the basis of formal blind tests? Otherwise, you would add hypocrisy
to your religious belief in the efficacy of the blind test!> > Now that is irrational behaviour, on the otherhand a fool or
> > someone in the throes of religious zeal may also regard it as
> > rational behavior, I respectfully suggest.
>
> Given the tone of last post, I take it you have nothing else
> worthwhile to say on the matter.
I see. So while you appear to feel that it is okay for you to
patronize others, you wish to pick up your ball and leave when
some of that attitude reflects back on you. If you dish it out, you
can hardly complain when others respond to you in kind.
John Atkinson
Editor, Stereophile
JA stated:"And to take your argument to its logical conclusion, you appear to be suggesting that all human experience, if anecdotal and not supportedby scientific experiments, must be discarded as meaningless. I hardly think that is the case."
Of course, no one is saying anything of the kind. On the other hand, you seem to be saying something just as absurd. You seem to be basing your conclusions largely on this single, personal experience. (In which you decided, after extended listening, that you prefer the somewhat warm, mellow characteristics of "tube sound" over those of the SS amp). Of course, I'm sure there are lots and lots of other reasons, but you seem simply to love this one. Again, it's a single example (duhh..) without any follow-up. (As would certainly be expected if, indeed, such personal conclusions are an underlying basis for SF's policies.)In any event, why the "either-or" approach? Why the hard over, never, under any circumstances, "case-closed" policy conclusion? And why attack the underlying principle (the principle of using blind testing IN SOME FORM to minimize personal bias), rather than the methodology? (For example, short listening times are often criticised as being a major problem in blind testing. - Instead of criticising the short listening times, why not consider using somewhat more extended listening times, perhaps repeated on subsequent occasions?)
In view of the obvious interest in blind testing, even as evidenced by comments in this very discussion string, why not permit both testing modalities? If costs are a major factor, such testing could be limited to once or twice a year, perhaps with a slight increase in subscription rates.
Jim
We are incapable of determining our own preferences and opinions?
"We are incapable of determining our own preferences and opinions? "No, biases reliably.
Music making the painting, recording it the photograph
JA said:
> You continue to miss the point I was making, "audiohobby." There are
2 incontrovertible facts involved in my parable:1) The solid-state amplifier could not be identified in the formal
blind test.2) The solid-state amplifier did not satisfy me in long-term
listening, despite all the non-audio factors working in its favor.And two articles of belief among so-called objectivists:
1) That blind tests, merely by being blind, are an effective means of
detecting audible differences.2) Sighted listening tests are inherently unreliable, due to the
presence of non-audio factors.If the facts behind my parable are true, then these two latter
beliefs are mutally incompatible. Either the blind test was
ineffective or the non-audio factors don't actually affect the
listener's reaction to any significant degree. You can't continue to
insist that both are correct.>
that's why I don't give much creedence to the individual product reviews here on the asylum. Most of them are by the owner of the item where one can reasonably expect some bias. I think that TAH doesn' understand that reviewers get all sorts of products where such bias just isn't there. Here are six $30k amps. Which one do you like best?I spent a few days in Seacliff the week before last. Currently, he's got a quad of those hideously expensive Western Electric WE-97A amps running the Nolas. Initial reaction? Naturally, I should like them because:
1. They're expensive. Are they $85k per pair or each?
2. They're 100 watt SETs (actually, they were the first SETs I've heard)
3. HP likes them.
4. They're well built and have tasteful backlighting of the stylized Western Electric brand through the faceplate.
5. HP didn't ask me to rebias them (as I have done for VTLs, ASLs, and Joule Electras
I've heard half a dozen different amps in that system (and seen half a dozen others sitting aside) over the years and these are NOT my favorite. In every case, these came from manufacturers where I had zero experience or prior exposure to them in order to form an opinion other than over the aesthetics. In that respect, I clearly prefer the Edge Signature monoblocks or the big Joule Electra amps with sixteen 6C33 outputs. Ever seen the Edge in person? They are positively gorgeous. There's not a hard angle on the unit. They engraved the Edge logo on the top cover. The heat sinks are shaped in a stylized "S" along with that shape being engraved into the front panel. The cabinet was fashioned from aluminum billet with countersunk machine hex nuts. There is a soft red glow emanating from the front panel. They have bi-wiring friendly posts with handles on them. In operation, they run fairly cool with a laser based bias tracking. They win the fashion show hands down. Especially when compared to...
While they are superb amps, my choice was the ugly duckling of the bunch (with an identical price tag): the VTL Wotans. Cosmetically, they are Roseanne Barr to the Edge's Kristanna Loken. I would characterize them as frumpy looking. Definitely NOT cool. They use exposed sheet metal screws. No lights other than an LED. Biasing forty-eight 6550s is a chore. And yet, they produced the most lifelike artifice of natural music reproduction I had ever heard. Brings tears-to-the-eyes good. It was only after that experience that I sold my beloved Audio Research amp and bought a pair of MB-450s. Five years later, I'm still in love with their ability to reproduce music.
> Cosmetically, they are Roseanne Barr to the Edge's Kristanna Loken <Maybe you're biased in favor of Roseanne Barr! LOL!
My experience with the Edge amplifier is that it's the best solid state unit I've ever heard. It doesn't give up as much to the best tube amps that most SS amps do.
And Valhalla all around - now THAT brings tears to my eyes! I never will be able to afford them. (sigh)
er, I mean the Edge is a fine amplifier.
And Valhalla all around - now THAT brings tears to my eyes!
Just wait - there's another Norse god on the loose!
Q. What chance does the average guy have of owning Nordost Valhalla or Ms. Loken.
I'm so naive I don't understand why any of this makes a difference. Do you not listen to gear before purchase?Because your readers have been exposed to SF anti-blind-test propaganda over the years, I doubt that these discussions/letters/surveys are very significant. Sales figures (reflecting newstand purchases by non-subscribers unfamiliar with the issue) don't seem very significant.
What do you want - a double blind test that compares results of sighted reviews with double blind test reviews both written so no methodological descriptions intrude?
""Our own staff. But shouldn't you be the one answering these
questions? For example, when you stated that you "think that most
readers would like to see at least some reports of blind testing...,"John, get a grip on yourself. - This happens to be an online discussion group, and, in fact, it's the "Critics Corner" secion of AA in which various opinions, questions, and viewpoints are supposedly welcomed. From the comments about my initial note, there does seem to be quite a lot of interest in these issues. Regarding blind testing, there obvioiusly are differences of opinion. But I'm certainly not the only one with such views."
JA is absolutely right. Why does he have to provide you with back-up when you don't do the same?? This is, indeed, Critics' Corner, but it isn't " Make the Editors Play By Rules That We Don't Have to Follow."He gave his answers. My guess is he's not terribly interested in the type of "improvements" you suggest.
nt
This post is made possible by the generous support of people like you and our sponsors: