|
Audio Asylum Thread Printer Get a view of an entire thread on one page |
For Sale Ads |
In Reply to: Some questions on double blind testing. posted by Phil Tower on November 12, 2002 at 03:10:40:
Whatever the future test is going to be it will need to be blind. The ones right now...NOPE! The problem is the tests now are looking for huge differences,,,not unlike comparing two loudspeakers. The psuedoscientists latch on and take that same test to cd players, amplifiers etc. Which don't possess the same degree of difference MEASURABLY and are obviously much harder to differentiate.The number of trials need to go WAY up if the difference is smaller. 6/10 ten times to a 59/100 is statistically significant and FAR more informative than 9/10 or 15/17 etc.
So basically the 10and 17 trials are toilet paper. Then go find the bigger trials...oops there are none.
According to typical DBT ABX tests, Tapes are indistinguishable from CD and LP is indistinguishable from live music. Thus if you support that kind of testing then you may as well buy tapes from here on out...besides they're cheaper.
Validity requires the test to be conducted in the environment for which it was designed. This equipment was designed for long term musical enjoyment not 5 second switching. SOme argue you can just make the test longer. BUT NO YOU CAN't because it goes against short term acoustic memory. but then etc etc etc. and you're in a feed back loop of endless problems.
I saw this in the general forum and part of their conclusion consists of noting problems with DBT AB testing...peer reviewed to boot.
http://jn.physiology.org/cgi/content/full/83/6/3548
We don't know enough about the brain yet to fully understand its role on perception. To make gross generalizations either way is wrongheaded. I'm not saying there are differences in any particular piece of gear...but trying to disprove claims on bad tests like magazines and AES journals etc is simply spurious.
I mean crap scientists are still trying to figure out if Red Meat is bad for you, and if aluminum in underarm deoderant will give you brain tumors. Some recent studies on Red Meat are showing benign to all the harmfullness the health nuts have been espousing over the years. But that's another forum.
Follow Ups:
I hardly know where to start. For instance, you say: ***According to typical DBT ABX tests, Tapes are indistinguishable from CD and LP is indistinguishable from live music. Thus if you support that kind of testing then you may as well buy tapes from here on out...besides they're cheaper.***That statement is totally, abjectly and completely false. Why did you even bother to say it. Over and over and over again, positive results have appeared, even in not very good ABX tests. This is a FACT, one that is well documented in the literature, quotes of which we have all seen already.
And then : ** Validity requires the test to be conducted in the environment for which it was designed. This equipment was designed for long term musical enjoyment not 5 second switching. SOme argue you can just make the test longer. BUT NO YOU CAN't because it goes against short term acoustic memory. but then etc etc etc. and you're in a feed back loop of endless problems.***
Another total, abject misstatement. In a properly executed test, YOU control when you switch.
It is a FACT that it is clear that once YOU LEARN WHAT TO LISTEN FOR, you will learn where to switch, why to switch, and you WILL ANSWER THE TEST TASK QUICKLY. Acoustic memory IS fleeting, but you don't have to listen to 5 second snippets to avoid that. 5 seconds, in any case, is a pure straw man, since acoustic memory so 200 milliseconds or less.
The only issue in switching is that when you WANT to switch, you must be ABLE to do so quickly. There is no requirement that you switch band and forth quickly, that's pure nonsense, and if someone is trying to give you a test where they require it, yes, you should raise your eyebrows a wee bit.
Then, as to your assertion that trials must be longer. What you meant is that there need to be more trials, and you're wrong about that, too.
In short-term tests, where 3/4 repeated twice, in a self-training threshold test, is "success", even not very experienced subjects very quickly learn to hear down to their absolute thresholds as measured in other much more difficult ways. This is another result that's all over the literature in JASA, and it shows that if something is detectable AT ALL, it is nearly always detected in A GOOD TEST.
First, your accusations (I can't call them even pseudo-facts) about what DBT's show are simply false. Then, your conclusions, such as they are, fly directly in the face of massive evidence.
Why did you write what you wrote?
JJ - Philalethist and Annoyer of Bullies
Dear JJJust curious, but would you happen to know as a rough estimate how many tests have been done that meet a PROPER TEST of SAY SS amplifiers?
The only one that seems to get thrown about on other forums is the great 1980 Tandberg amp...which was described to me by a forumer "As inaudible from ALL aother SS amps," which I take to mean all working properly SS amps must in turn sound the same if the Tanberg is the same. Kind of if A=B and B=C then A=C logic.
I personally hope that differences don't exist audibly with most non tube, loudspeaker, turntable gear. Those 300 disc changers are quite accomodating and cheaper than the High end single disc players.
Personally, I hate weasal words being used...like MAY offer, or red meat MAY cause, or there is a POSSIBILITY that...X will cause Y etc.
My psych courses are where my high number of trials came in studying those claiming Extra Sensory Perception. Since other forumers linked believing in audible differences among SS amps and CD players to believing in ESP then I used the ESP high trial requirement. In a proper test where far more variables can be accounted for then the trials could be drastically cut.
Where the frustration sets in is when you see magazines like the Sensible Sound claim that an RCA is indistinguishable from cd players coosting thousands in their home brew DBT. And I say OK, but let me try it...WHAT were the other cd players so that I may listen? But they don't list them. And That is puzzling.
And then why would RCA not use it as a HUGE advertising campaign. RCA $89.00 model X is as good, AUDIBLY, as Linn Sondek CD12(you know the ridiculous 20+k CD player). I mean if I were RCA management I would be all over it.
I think a lot of people would rather put the money into speakers - I know I would...but there is a lot of misinformation I presume is out there. Yet access to AES documents etc is costly and generally hard to find for average Joe consumer? Why?
I'm not saying there are differences in any particular piece of gear...but trying to disprove claims on bad tests like magazines and AES journals etc is simply spurious.I certainly agree. There is a point in your statement that to me is personally important. I believe that a person who raises questions regarding the current state of the art in DBTs should not automatically be labeled as making unsubsubstantiated claims about cable differences and likened to people who believe in alien abductions, creationism and astrology. Likewise, a person who questions the lack of reliability in sighted auditions because of bias shouldn't be labeled as pushing a point of view that "wire is wire is wire."
I think your point regarding number of trials is very important, and seems to me to be often overlooked by those who point uncritically to reported cable and component DBTs which have supposedly produced null results.
Here's a post where I set out the material I've gathered on the issue of number of trials:
http://www.audioasylum.com/members/mgeneral/messages/887.html
Also see:http://www.audioasylum.com/members/mgeneral/messages/397.html
For an interesting discussion on objectivity.
All of which leads me to tentative conclude that home brew blind tests may be pratically worthless and that it is somewhat unlikely that anyone soon will be conducting reliable DBTs on cables.
Is a collection of straw men, and little more.
JJ - Philalethist and Annoyer of Bullies
jj:I'm not sure what article you are referring to. If its the Leventhal JAES article, it may be full of straw men; I just haven't seen any convincing rebuttal to his arguments and I don't know enough to see where he is wrong.
Sorry.He seems to be of the "abx/dbt never found a difference" bunch. Even if you take less that great tests from less than refereed places, they have.
Makes his position tough. He attempted a bit of a whipsaw as well, if I recall.
JJ - Philalethist and Annoyer of Bullies
I have not heard of a Whipsaw?As for the differences of Tape VS CD that was largely because several years ago in BC Canada TDK ran one of those booths where if you could tell the difference between their Chrome II tape, recorded from the cd versus the cd in a blind test you would win some sort of prize. Which of course didn't happen. TDK of course has a conflict of interest, so did the UHF magazine (sort of DBT) that indicated the same kind of thing.
I had asked others on other forums to provide support in AES papers etc, that a recorded tape of cd could be distinguished and came up blanks. What I did get provided were several documents that had nothing to do with Tapes. I was provided with a live performance versus LP from the 30s or something where nobody could distinguish the difference.
Like I replied above...I am getting information fromall the wrong places I presume...but if there is incontributable evidence then RCA, APEX, Yorx etc and all the other "so called" low end companies should be marketing the hell out of the TRUTH and save us from buying non value added jewelry. Besides the "so called" lower end products LOOK WAY BETTER. Pioneer has cool blue lights and some stuff has rosewood side panels and piano blaque lacquer facing on their receivers. Compare that to the butt ugly Brystons with tacky looking handles and the Bryston wins hands down.
I'm perfectly willing to admit that when I listened to the two and thought the Bryston had tightened up the bass response dramatically, that I was under a placaebo effect of some sort. Luckily, I have yet to spend that kind of money on equipment - so I can stop before I get caught in a delusion...or was the Bryston better?
Marketing demonstrations ARE NOT GENERALLY WELL RUN TESTS.I'm not sure what your gripes with the JAES is. Could you be a bit more specific?
As to DBT's, I suggest that you visit JASA or a psychometrics journal.
The AES is generally (although not always) pretty good, but is is (and must remain, to be useful) on the applied end of audio science.
JJ - Philalethist and Annoyer of Bullies
This post is made possible by the generous support of people like you and our sponsors: