|
Audio Asylum Thread Printer Get a view of an entire thread on one page |
For Sale Ads |
205.145.64.64
In Reply to: The Inquisition Continues posted by John Atkinson on March 22, 2007 at 08:03:58:
John
I have been a Sterophile subscriber for almost 15 years (I also subscribe to other publications and read all of the online pubs)I remember some of you blind tests. I believe some were with a panel that all listened together and then each was assigned a speaker to report on sighted.
Wasn't the panel test skewed by only one person being able to sit in the right spot? (I understand doing other wise would be a hassle)
Were the levels matched precisely system to system?
Personally I would prefer a compromise of some sort. Logistically doing blind tests for everything can be a nightmare and would slow down the process. As such it's really not practical. Having said that I would like to see some way of blind testing speakers to see if the responses to the sighted reviews hold true. Maybe a sampling once a year? The reason I suggest this is cost. I would like to hear a reviewer critique speakers based on several common parameters and then suggest what price range they feel that quality of sound could substaniate. Of all the arguements the ABX people make I think the one regarding cost is the most relevant. I think one expects certain capabilities when they know how much something costs. I think it's human nature to want to be able to give a favorable review of someone else's life's work - especially when they are extremely expensive to desin and manufacture. (I experiance the same thing when i buy something or try a tweek. You find yourself wanting to support your own decision). As such I think - especially with speakers- a yearly calibration would be of some use. (Given the decline of high end audio it is virtually impossible for the average person to listen to even a fraction of what is available or recommended. As such - while your recommend people listen on their own and don't buy based on your reviews - it is often practical to do just that - as there is really no other choice. I bought Triangle speakers based on Sam review and the fact that I own tubes. When I did this there was no one within 250 miles that was selling them)
Follow Ups:
> Wasn't the panel test skewed by only one person being able to sit in
> the right spot?
Yes. The first test used 4-5 listenrs at a time, which meant that the
tests ran over 2 days. Because it became apparent that the position
of the listener in the room was also a variable, in the subsequent
tests, we had just 2 listeners at a time take part, one sitting behind
the other, so that both were close to being in the sweet spot. This
stretched the testing to 6 days. It would have even better if we had
tested just one listener at a time, but this would have required too
much time just for the blind tests, let alone every other aspect of
the review, to be practical for a monthly publication schedule.
> Were the levels matched precisely system to system?
As closely as is possible with speakers, which have an unflat response
compared with amplifiers or digital front-ends. We matched the levels
based on the speakers' B-weighted sensitivities, which downplays the
effect of differences at the frequency extremes. We did publish the
experimental procedures in full detail, BTW.
One point that is continually glossed over by proponents of blind
testing is that when the audible differences are small in absolute
terms, which is very often the case, designing a blind test that has
sufficient sensitivity to detect such differences is not trivial.
John Atkinson
Editor, Stereophile
"One point that is continually glossed over by proponents of blind
testing is that when the audible differences are small in absolute
terms, which is very often the case, designing a blind test that has
sufficient sensitivity to detect such differences is not trivial."Your point here is self-defeating, "audible differences that are small in absolute terms" will not be any easier to detect in a sighted test, the results will just a lot less reliable as a result of uncontrolled listener bias.
Music making the painting, recording it the photograph
> Your point here is self-defeating, "audible differences that are
> small in absolute terms" will not be any easier to detect in a
> sighted test, the results will just a lot less reliable as a result
> of uncontrolled listener bias.
It depends on the nature of the difference, the conditions of the
blind test, and the conditions of the sighted test. When real
differences exist, an endless series of inadequate blind tests that
produce null results, a la Tom Nousaine, is meaningless.
This was the point of the parable I related at the HE2005 debate,
reprinted at www.stereophile.com/asweseeit/705awsi .
The differences between the amplifiers under test had not been
detected by the formal blind test but had been under normal, sighted,
longer-term listening.
John Atkinson
Editor, Stereophile
> This was the point of the parable I related at the HE2005 debate,
> reprinted at www.stereophile.com/asweseeit/705awsi.
> The differences between the amplifiers under test had not been
> detected by the formal blind test but had been under normal, > sighted, longer-term listening.
Your little parable demonstrates that detectable differences are not the sole criteria for long term satisfaction of any given device, I can think of a very good example that captured the public interest in the UK. Volkswagen was gearing up to re-launch the Skoda in UK, In the consumer clinic, the car performed excellently until the test clinic subjects were made aware of the brand, needless to say it was downhill from there on. No matter how you cut it, once the identity of the object under test is known, expectation bias becomes an unavoidable skewing factor. If it were not the case, getting reliable results will simply be a matter of conducting blind tests immediately after sighted tests (where valid differences were reported) and the results of the blind test will correlate with the results of the sighted tests, however that is opposite of what is widely obtained in these tests. Expectation bias depends on a priori knowledge, a priori knowledge is always present in a sighted test.
Music making the painting, recording it the photograph
> > This was the point of the parable I related at the HE2005 debate,
> > reprinted at www.stereophile.com/asweseeit/705awsi. The differences
> > between the amplifiers under test had not been detected by the
> > formal blind test but had been under normal, sighted, longer-term
> > listening.
>
> Your little parable demonstrates that detectable differences are not
> the sole criteria for long term satisfaction of any given device...
Perhaps you didn't read my essay carefully? Perhaps you didn't
comprehend what I wrote? My point was that all the non-audio factors
were working _in favor_ of the solid-state power amplifier. Yet my
dissatisfaction with its sound turned out to be very real. And that
shortfall in sound quality had _not_ been detected in what on the
face of things had appeared to be a well-organized and well-designed
blind test.
John Atkinson
Editor, Stereophile
> Perhaps you didn't read my essay carefully? Perhaps you didn't
> comprehend what I wrote? My point was that all the non-audio > factors were working _in favor_ of the solid-state power amplifier.Well, you are touching at the heart of why sighted test are inherently weaker than blind tests, the non-audio factors at play are not at the sole discretion of the observer, neither is the observer neccessarily aware of all his/her biases, a priori knowledge brings to the fore all biases not just biases the observer is aware of at the time of the test. A priori knowledge is skewing factor irrespective of the duration of the test.
Music making the painting, recording it the photograph
when the resulting preference is contrary to the bias established by the non-audio factors?
"So, what conclusion can you draw when the resulting preference is contrary to the bias established by the non-audio factors?"Your question was implicitly answered in the original post you responded to. The observer is NOT necessarily aware of all his/her biases and there is no realistic method available to the observer to address this weakness. As a result, claiming that the resulting preference was contrary to the bias established by the non-audio factors is not a possible outcome as it assumes that observer is fully aware of all his biases.
preference for the many reasons was not for the solid state unit?I've made quite a few comparisons where my favored brand did not "win" in the outcome.
"What makes you think that Mr. Atkinson's preference for the many reasons was not for the solid state unit?"???? You are going round in circles asking the same question in various guises, the answer remains the same. At any rate that is not the only problem with Mr. Atkinson's parable, just the one that is impossible to reliably mitigate against in a sighted test.
Music making the painting, recording it the photograph
> that is not the only problem with Mr. Atkinson's parable, just the
> one that is impossible to reliably mitigate against in a sighted
> test.
You continue to miss the point I was making, "audiohobby." There are
2 incontrovertible facts involved in my parable:
1) The solid-state amplifier could not be identified in the formal
blind test.
2) The solid-state amplifier did not satisfy me in long-term
listening, despite all the non-audio factors working in its favor.
And two articles of belief among so-called objectivists:
1) That blind tests, merely by being blind, are an effective means of
detecting audible differences.
2) Sighted listening tests are inherently unreliable, due to the
presence of non-audio factors.
If the facts behind my parable are true, then these two latter
beliefs are mutally incompatible. Either the blind test was
ineffective or the non-audio factors don't actually affect the
listener's reaction to any significant degree. You can't continue to
insist that both are correct.
John Atkinson
Editor, Stereophile
> You continue to miss the point I was making, "audiohobby."I am not missing the point, however you are as I addressed the selfsame issue upteen times in my replies to E-Stat.
> There are 2 incontrovertible facts involved in my parable.
Only the first statement satisfies that criterion.
> 2) The solid-state amplifier did not satisfy me in long-term listening, despite all the non-audio factors working in its favor.
Your comments do not take into account hidden or unconscious bias. It is impossible for the observer to reliably claim that they are fully aware of ALL their biases, as a result it is impossible to claim that all their biases favour a particular choice as you have claimed. A priori knowledge informs both the known and hidden biases, since the observer is not necessarily aware of their hidden (or unconscious) biases, how do they mitigate against it effects in a sighted test? They can't, it is an inherent weakness in sighted tests.
> 2) Sighted listening tests are inherently unreliable, due to the presence of non-audio factors.And you will be correct because of the reasons outlined in my previous comments.
Music making the painting, recording it the photograph
> > There are 2 incontrovertible facts involved in my parable.
>
> Only the first statement satisfies that criterion.
In your opinion. I assure you that my dissatisfaction was real. I find
it amazing that you now claim to be a mind reader in order to be able
to wriggle out from the implications of my experience.
> > 2) The solid-state amplifier did not satisfy me in long-term
> > listening, > despite all the non-audio factors working in its favor.
>
> Your comments do not take into account hidden or unconscious bias...
Nor do they take into account ESP or your claims to be able to
discern my state of mind a quarter century ago. Again: despite my
love for the Quad amplifier, despite the fact that it was very much
less expensive than the amplifier it replaced (meaning I made money
on the exchange), despite it being, small, cool-running, and despite
the hero-worship I felt (and still feel) for its designer, in the long
term, and without reference to other amplifiers, it proved extremely
unsatisfying in the task for which I had purchased it. That sir, is a
fact.
I suppose you will next suggest that I should have performed further
blind tests to "prove" that my dissatisfaction was illusory? Ecept
that I imagine that would have produced the same result as before, in
which case I would still be stuck with a sound that dissatisfied.
Only a fool or someone in the throes of religious zeal would regard
that as rational behavior, I respectfully suggest.
John Atkinson
Editor, Stereophile
you can't possibly understand your own preferences! Actually, I know why you didn't like the Quad in the end. It was the heatsinks on the front. That's it. I'll bet you don't like any amplifiers that have heatsinks on the front. It's that simple. ;)
Nice try, Did you forget that you asked that selfsame question a few posts back or you are just being malicious?
Music making the painting, recording it the photograph
Such involves the use of the question mark. Don't you know that?Malicious? Heavens, no. Just thoroughly amused. :)
How does your rhetorical questionC'mon, John you can't possibly understand your own preferences!
follow from this
"We are incapable of determining our own preferences and opinions? "
No, biases reliably.
Or are you simply being deliberately obtuse?
Music making the painting, recording it the photograph
Taken in the context of E-stat's entire post, it would fall far from being malicious, and instead fall into the category of leg-pulling humor.
At least someone understands my humor and the smiley/winky faces!
Now get back to Propheads where you belong.
> I'll bet you don't like any amplifiers that have heatsinks on the front. It's that simple. ;) <Or that start with the letter 'Q'.
Or maybe John equates it with the quadrophonic sound that was such a disaster.
Or maybe 4 is an unlucky number!
The possibilities are endless. :)
> In your opinion. I assure you that my dissatisfaction was real. I find it amazing that you now claim to be a mind reader in order to be able to wriggle out from the implications of my experience.Those are the facts with respect to biases, you might not like them and try and claim that they do not apply to your particular situation, but then you will not be different many other audiophiles who believe that some basic laws of human behaviour do not apply to them.
> Only a fool or someone in the throes of religious zeal would regard that as rational behavior, I respectfully suggest.
You are defending a position that is totally at odds with generally accepted behavioural science on this well-trodden subject with absolutely nothing except a piece of anecdotal evidence. Now that is irrational behaviour, on the otherhand a fool or someone in the throes of religious zeal may also regard it as rational behavior, I respectfully suggest.
Given the tone of last post, I take it you have nothing else worthwhile to say on the matter.
Music making the painting, recording it the photograph
> > In your opinion. I assure you that my dissatisfaction was real. I
> > find it amazing that you now claim to be a mind reader in order to
> > be able to wriggle out from the implications of my experience.
>
> Those are the facts with respect to biases, you might not like them
> and try and claim that they do not apply to your particular
> situation...
Again, you seem to be addressing a different point. My dissatisfaction
with the sound of the solid-state amplifier was indeed real. Why would
I lie about it, either to you now or to myself back then? All I was
concerned about at the time -- and remember that, as described in the
essay, I was not a reviewer at the end of the 1970s but a hard-line
"objectivist -- was buying an amplifier for use in my own system for
as little cash as I needed pay for the amount of power available. I
did what "objectivists" to this day recommend to audiophiles, which
is to choose a well-designed amplifier on the basis of cost, features,
and power, because in a formal blind test, the amplifier I wanted to
purchase had been shown not to sound any different from more exotic,
more expensive designs. As I said, for you now to claim that you know
more about my state of mind at that time than I did then is both
ridiculous and arrogant.
I have been forthright about the biases and belief I had at that time.
All you are doing is postulating that there must have been other,
hitherto unsuspected biases at work. As I said: mindreading.
> but then you will not be different many other audiophiles who believe
> that some basic laws of human behaviour do not apply to them.
I see you like to patronize those whose points you can't argue with
otherwise?
> > Only a fool or someone in the throes of religious zeal would regard
> > that as rational behavior, I respectfully suggest.
>
> You are defending a position that is totally at odds with generally
> accepted behavioural science on this well-trodden subject with
> absolutely nothing except a piece of anecdotal evidence.
No, I am honestly reporting my _experience_, experience obtained
without pressure from anyone else, and experience that ran counter
to my biases and expectations. Such was the cognitive dissonance
engendered by this situation that I didn't recognize the problem
until I realized that listening to recorded music, my passion, had
increasingly become less and less of my life. The change in amplifier
had been the only significant factor. Occam's Razor suggested that
the amplifier was the root cause of my dissatisfaction.
And if that was the case, then either the blind test had misled me
or my biases and expectations had not outweighed the evidence of
my ears.
And to take your argument to its logical conclusion, you appear to be
suggesting that all human experience, if anecdotal and not supported
by scientific experiments, must be discarded as meaningless. I
hardly think that is the case.
Take my recording activities: I must make literally 100s of judgment
calls during the production of a project, some under extreme time
constraints, and I need to be correct on every one of those decisions.
Should I move this mike back 6"? Should I apply +0.5dB of boost at
100Hz to compensate for a cardioid's early rolloff or +1dB? Should
I record at 96kHz or save hard-drive space and file-handling time by
using 48kHz. The list is endless, yet according to you, without
formal blind testing of each of those situations, none of those
decisions can be justified because each will be affected by my
baggage of biases and preconceived notions.
In that case, why do any of us bother doing anything?
I take it, BTW, that all your own decisions regarding what products
you chose to buy and how you choose to use them have been made on
the basis of formal blind tests? Otherwise, you would add hypocrisy
to your religious belief in the efficacy of the blind test!> > Now that is irrational behaviour, on the otherhand a fool or
> > someone in the throes of religious zeal may also regard it as
> > rational behavior, I respectfully suggest.
>
> Given the tone of last post, I take it you have nothing else
> worthwhile to say on the matter.
I see. So while you appear to feel that it is okay for you to
patronize others, you wish to pick up your ball and leave when
some of that attitude reflects back on you. If you dish it out, you
can hardly complain when others respond to you in kind.
John Atkinson
Editor, Stereophile
JA stated:"And to take your argument to its logical conclusion, you appear to be suggesting that all human experience, if anecdotal and not supportedby scientific experiments, must be discarded as meaningless. I hardly think that is the case."
Of course, no one is saying anything of the kind. On the other hand, you seem to be saying something just as absurd. You seem to be basing your conclusions largely on this single, personal experience. (In which you decided, after extended listening, that you prefer the somewhat warm, mellow characteristics of "tube sound" over those of the SS amp). Of course, I'm sure there are lots and lots of other reasons, but you seem simply to love this one. Again, it's a single example (duhh..) without any follow-up. (As would certainly be expected if, indeed, such personal conclusions are an underlying basis for SF's policies.)In any event, why the "either-or" approach? Why the hard over, never, under any circumstances, "case-closed" policy conclusion? And why attack the underlying principle (the principle of using blind testing IN SOME FORM to minimize personal bias), rather than the methodology? (For example, short listening times are often criticised as being a major problem in blind testing. - Instead of criticising the short listening times, why not consider using somewhat more extended listening times, perhaps repeated on subsequent occasions?)
In view of the obvious interest in blind testing, even as evidenced by comments in this very discussion string, why not permit both testing modalities? If costs are a major factor, such testing could be limited to once or twice a year, perhaps with a slight increase in subscription rates.
Jim
We are incapable of determining our own preferences and opinions?
"We are incapable of determining our own preferences and opinions? "No, biases reliably.
Music making the painting, recording it the photograph
JA said:
> You continue to miss the point I was making, "audiohobby." There are
2 incontrovertible facts involved in my parable:1) The solid-state amplifier could not be identified in the formal
blind test.2) The solid-state amplifier did not satisfy me in long-term
listening, despite all the non-audio factors working in its favor.And two articles of belief among so-called objectivists:
1) That blind tests, merely by being blind, are an effective means of
detecting audible differences.2) Sighted listening tests are inherently unreliable, due to the
presence of non-audio factors.If the facts behind my parable are true, then these two latter
beliefs are mutally incompatible. Either the blind test was
ineffective or the non-audio factors don't actually affect the
listener's reaction to any significant degree. You can't continue to
insist that both are correct.>
that's why I don't give much creedence to the individual product reviews here on the asylum. Most of them are by the owner of the item where one can reasonably expect some bias. I think that TAH doesn' understand that reviewers get all sorts of products where such bias just isn't there. Here are six $30k amps. Which one do you like best?I spent a few days in Seacliff the week before last. Currently, he's got a quad of those hideously expensive Western Electric WE-97A amps running the Nolas. Initial reaction? Naturally, I should like them because:
1. They're expensive. Are they $85k per pair or each?
2. They're 100 watt SETs (actually, they were the first SETs I've heard)
3. HP likes them.
4. They're well built and have tasteful backlighting of the stylized Western Electric brand through the faceplate.
5. HP didn't ask me to rebias them (as I have done for VTLs, ASLs, and Joule Electras
I've heard half a dozen different amps in that system (and seen half a dozen others sitting aside) over the years and these are NOT my favorite. In every case, these came from manufacturers where I had zero experience or prior exposure to them in order to form an opinion other than over the aesthetics. In that respect, I clearly prefer the Edge Signature monoblocks or the big Joule Electra amps with sixteen 6C33 outputs. Ever seen the Edge in person? They are positively gorgeous. There's not a hard angle on the unit. They engraved the Edge logo on the top cover. The heat sinks are shaped in a stylized "S" along with that shape being engraved into the front panel. The cabinet was fashioned from aluminum billet with countersunk machine hex nuts. There is a soft red glow emanating from the front panel. They have bi-wiring friendly posts with handles on them. In operation, they run fairly cool with a laser based bias tracking. They win the fashion show hands down. Especially when compared to...
While they are superb amps, my choice was the ugly duckling of the bunch (with an identical price tag): the VTL Wotans. Cosmetically, they are Roseanne Barr to the Edge's Kristanna Loken. I would characterize them as frumpy looking. Definitely NOT cool. They use exposed sheet metal screws. No lights other than an LED. Biasing forty-eight 6550s is a chore. And yet, they produced the most lifelike artifice of natural music reproduction I had ever heard. Brings tears-to-the-eyes good. It was only after that experience that I sold my beloved Audio Research amp and bought a pair of MB-450s. Five years later, I'm still in love with their ability to reproduce music.
> Cosmetically, they are Roseanne Barr to the Edge's Kristanna Loken <Maybe you're biased in favor of Roseanne Barr! LOL!
My experience with the Edge amplifier is that it's the best solid state unit I've ever heard. It doesn't give up as much to the best tube amps that most SS amps do.
And Valhalla all around - now THAT brings tears to my eyes! I never will be able to afford them. (sigh)
er, I mean the Edge is a fine amplifier.
And Valhalla all around - now THAT brings tears to my eyes!
Just wait - there's another Norse god on the loose!
Q. What chance does the average guy have of owning Nordost Valhalla or Ms. Loken.
This post is made possible by the generous support of people like you and our sponsors: