Propeller Head Plaza

Technical and scientific discussion of amps, cables and other topics.

Return to Propeller Head Plaza

L.A. Times article concerning blind wine taste tests.

76.191.156.247
Posted on September 7, 2009 at 14:51:00
Al Sekela
Audiophile

Posts: 8279
Location: Northern California
Joined: February 18, 2002
 See the link or:

http://www.latimes.com/business/la-fi-wine4-2009sep04,0,3295012.story

The story reports a study by retired Cal State Humboldt professor Robert Hodgson which found little consistency in which wines win medals in blind competitions.

We are often reminded that blind testing is used routinely to judge wine when the topic of blind audio testing is raised. My experience in Sonoma County is that some of the most god-awful wines tend to win medals and top honors in the local Harvest Fair competition, which carefully follows a blind testing protocol. This article confirms my experience.

I believe the blind part of the protocol is well-intentioned, but overwhelmed by the sheer numbers of entries and the small number of 'expert' judges. Even though the judges spit out the samples, they absorb enough alcohol and wine flavor components that, by the end of the session, only those wines with strong flavor peculiarities manage to remain distinct.

The resolution power of the test method is unmeasured and assumed adequate to the task by those with a vested interest in promoting the method.

Like audio reviewers, I think wine tasters may be happy with their judgments, but I satisfy myself., posted on September 10, 2009 at 15:39:55
Norm
Audiophile

Posts: 14914
Joined: September 6, 2000
 I remember the old Pepsi vs. Coke taste tests. Who cared? I like Pepsi best and Coke Diet best.

I like Pepsi best only when it's sweet and not syrupy., posted on September 16, 2009 at 22:01:29
Don Till
Audiophile

Posts: 675
Location: Tempe
Joined: February 16, 2009
 Seems to depend on where you are when you buy it - like it tastes different in different cities. Coke on the other hand varies much less and IMO it tastes best when it has lots of bite (like Dr. Pepper) and is sweet but not too sweet.

So I would normally pick Coke over Pepsi but given a good Pepsi ( a somewhat rare event here in the Valley of the Sun) I would always chose it over Coke.
RE: I like Pepsi best only when it's sweet and not syrupy., posted on September 17, 2009 at 09:54:30
b.l.zeebub
Audiophile

Posts: 810
Location: 52deg 28'N,1deg56'W
Joined: April 17, 2006
 I've done the Pepsi test a few times, always with the same result: Coke win!

I love Coke, it is my favourite pop. It just so happens that every other Cola I have ever tried tasted foul. If an establishment does not have Coke I drink 7up, Sprite, Fanta, juice or tap water. Basically anything that is not a non-coke cola.

The Pepsi challenge is an excellent comparison to audio DBTs..., posted on September 10, 2009 at 21:16:27
mkuller
Audiophile

Posts: 15929
Location: SF Bay Area
Joined: April 22, 2003
Contributor
  Since: December 28, 2003
 ...tasting sips blind, drinkers preferred New Coke.

But it was too sweet so they didn't want to take it home and drink a whole can. They continued to buy Pepsi and New Coke failed.

The problem was that in the test the way people drank was very different than they way the drank at home.

Just like audio DBTs fail because they have little relevance to the way you listen to music at home.
Kuller, you're such a subjectivist..., posted on September 17, 2009 at 14:23:45
SF tech
Industry Professional

Posts: 5219
Location: San Francisco
Joined: March 24, 2005
Contributor
  Since: February 4, 2008
 You took the Pepsi challenge...

And chose Jif!

SF



Makes about as much sense..., posted on September 17, 2009 at 14:34:07
mkuller
Audiophile

Posts: 15929
Location: SF Bay Area
Joined: April 22, 2003
Contributor
  Since: December 28, 2003
 ...as most of your other posts.

Good grief... Have you no sense of humor? nt, posted on September 17, 2009 at 14:46:52
SF tech
Industry Professional

Posts: 5219
Location: San Francisco
Joined: March 24, 2005
Contributor
  Since: February 4, 2008
 nt



Come on...you know we Pepsi-loving subjectivists take ourselves pretty seriously...(nt), posted on September 17, 2009 at 14:59:11
mkuller
Audiophile

Posts: 15929
Location: SF Bay Area
Joined: April 22, 2003
Contributor
  Since: December 28, 2003
 (nt)

:-) nt, posted on September 17, 2009 at 15:27:05
SF tech
Industry Professional

Posts: 5219
Location: San Francisco
Joined: March 24, 2005
Contributor
  Since: February 4, 2008
 nt



They don't have to., posted on September 12, 2009 at 09:47:22
Analog Scott
Audiophile

Posts: 4001
Joined: January 8, 2002
 I use blind comparisons along with sighted ones in my audition process. They work just fine for me. I don't "change" how I listen other than the fact that I am making comparisons. But that is true blind or sighted.

RE: They don't have to., posted on September 12, 2009 at 12:26:23
Tony Lauck
Audiophile

Posts: 3048
Location: Vermont
Joined: November 12, 2007
Contributor
  Since: February 24, 2009
 "I don't "change" how I listen other than the fact that I am making comparisons. But that is true blind or sighted."

Perhaps you don't change, but for many people cognition alters when one sense is blocked. I think it's because a lot of the cognitive processing the mind performs involves integrating information from multiple senses. When my legally blind wife was taking mobility training she had to navigate through downtown Boston. Her trainer made her wear a blindfold. I was surprised to learn this, as I had naturally assumed she would have just kept her eyes closed. I was told this would have been a bad idea, because the muscular work associated with keeping the eyes shut would have its own impact on cognition.

In addition to cognitive factors there may also be psychological factors associated with blind audio tests, perhaps coming from performance anxiety. The extent of this might depend on the social situation, e.g. a self-conducted test in private would probably not be a problem, but a test at an audio club with hostile factions arguing about DBTs could be quite a different matter. (No way I would ever take such a test in the presence of known skeptics!)

Tony Lauck

"Diversity is the law of nature; no two entities in this universe are uniform." - P.R. Sarkar

RE: They don't have to., posted on September 12, 2009 at 23:28:34
Analog Scott
Audiophile

Posts: 4001
Joined: January 8, 2002
 >>>"I don't "change" how I listen other than the fact that I am making comparisons. But that is true blind or sighted.">>>

>> Perhaps you don't change, but for many people cognition alters when one sense is blocked.>>

I think i actually hear better in the dark but that is besides the point. I see fine during blind comparisons. The only thing that makes them blind is I don't know which is A and which is B.


>> I think it's because a lot of the cognitive processing the mind performs involves integrating information from multiple senses. When my legally blind wife was taking mobility training she had to navigate through downtown Boston. Her trainer made her wear a blindfold. I was surprised to learn this, as I had naturally assumed she would have just kept her eyes closed. I was told this would have been a bad idea, because the muscular work associated with keeping the eyes shut would have its own impact on cognition. >>

Not an issue in my blind comparisons. i don't do them literally blind. I just don't "know" which is which when making the comparisons.



>> In addition to cognitive factors there may also be psychological factors associated with blind audio tests, perhaps coming from performance anxiety. The extent of this might depend on the social situation, e.g. a self-conducted test in private would probably not be a problem, but a test at an audio club with hostile factions arguing about DBTs could be quite a different matter. (No way I would ever take such a test in the presence of known skeptics!)>>

Not a problem in my auditions. I generally do them with the help of one friend. No anxiety involved. The items being compared are on trial. not me. I'm not trying to "prove" anything. i'm just using them to help make my choices.

The biggest problems seems to be..., posted on September 12, 2009 at 12:39:12
mkuller
Audiophile

Posts: 15929
Location: SF Bay Area
Joined: April 22, 2003
Contributor
  Since: December 28, 2003
 ...with "forced choice" type DBTs, like with the ABX comparator.

Right brain (emotional, relaxed listening to music) vs left brain (critical, decision-making) type of activity.

Isn't every audition a forced choice?, posted on September 13, 2009 at 10:12:21
Analog Scott
Audiophile

Posts: 4001
Joined: January 8, 2002
 Buy it or don't buy it?

No one is forcing you...(nt), posted on September 13, 2009 at 10:44:47
mkuller
Audiophile

Posts: 15929
Location: SF Bay Area
Joined: April 22, 2003
Contributor
  Since: December 28, 2003
 (nt)

I think my point is being missed, posted on September 13, 2009 at 11:07:47
Analog Scott
Audiophile

Posts: 4001
Joined: January 8, 2002
 we do audition things as audiophiles. *That* process involves changes in our "normal" listening habbits. So the issue of such changes is not unique to blind comparisons. As I said before. When I am making comparisons I do them both blind and sighted. I find the blind part to be very informative. Just because we have assholes trying to shit on other audiophiles' good times by insisting everyone *else* has to prove their perceptions are "scientifically valid" with a battery of ABX tests that may or may not be any good does not mean all blind testing is corupt and useless. even in audio.

Agreed...(nt), posted on September 13, 2009 at 11:57:44
mkuller
Audiophile

Posts: 15929
Location: SF Bay Area
Joined: April 22, 2003
Contributor
  Since: December 28, 2003
 (nt)

RE: The Pepsi challenge is an excellent comparison to audio DBTs..., posted on September 11, 2009 at 15:12:50
Todd Krieger
Audiophile

Posts: 22046
Location: SW United States
Joined: November 2, 2000
 I don't know if this is urban legend, but I've heard claims that New Coke was formulated expressly to "win" the Pepsi Challenge......


RE: The Pepsi challenge is an excellent comparison to audio DBTs..., posted on September 11, 2009 at 08:29:42
tomservo
Manufacturer

Posts: 2517
Joined: July 4, 2002
 "Just like audio DBTs fail because they have little relevance to the way you listen to music at home."

It is true that you can minimize the usefulness of the approach that way, but that is a decision about test protocol and not the validity of the test itself.
Like when you choose not to look, one never finds anything.

Alternately, the best way to test like this is exactly the same as normal listening, in your own room, at your leisure, over time, changing only one thing in your system.
It is not necessary to follow the strict protocols unless your presenting a paper, all that is needed to to be able to go back and forth at your comfort and pace between two things, playing the music that seems to bring out the difference the strongest, WITHOUT KNOWING which case was which.
Apples and Oranges, posted on September 8, 2009 at 21:33:22
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 Wine tasting is for taste preference, which may not be stable.

Same/different audio DBTs do not determine what one prefers, only whether the results show the difference is detectable.

"Probability is the very guide to life."---Cicero
...are both sold by weight., posted on September 9, 2009 at 16:16:56
Al Sekela
Audiophile

Posts: 8279
Location: Northern California
Joined: February 18, 2002
 To stretch your tired metaphor, the weight is determined by a measuring instrument. In most jurisdictions, the accuracy (including precision and resolution) of the measuring instrument is governed by law.

The point of the L.A. Times article's referenced study is the resolution of the tasting methodology is hopelessly inadequate to the purpose. That the method is blind is good, because it eliminates biases based on prior knowledge of the wineries involved. However, this virtue is insufficient to overcome the weaknesses caused by too many wines and too few tasters.

Whether the purpose of the audio test is to determine preference or detectability, it needs to have the resolution established before the results can be used.

Resolution? That would involve scientific validation..., posted on September 10, 2009 at 13:40:10
mkuller
Audiophile

Posts: 15929
Location: SF Bay Area
Joined: April 22, 2003
Contributor
  Since: December 28, 2003
 ...controls and a lot of messy, time consuming science stuff the DBTers have no clue about.

And I suspect they would find that sighted A/B testing has a higher resolution.
RE: Resolution? That would involve scientific validation..., posted on September 10, 2009 at 14:26:11
Todd Krieger
Audiophile

Posts: 22046
Location: SW United States
Joined: November 2, 2000
 "...controls and a lot of messy, time consuming science stuff the DBTers have no clue about."

What's really wacky is the "DBTers" have no clue about DBT.......


RE: Resolution? That would involve scientific validation..., posted on September 10, 2009 at 17:54:51
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 We know about sighted listening, though.

"Probability is the very guide to life."---Cicero
RE: Resolution? That would involve scientific validation..., posted on September 10, 2009 at 23:13:28
Todd Krieger
Audiophile

Posts: 22046
Location: SW United States
Joined: November 2, 2000
 "Sighted" has nothing to do with DBT........ (It's a buzzword that isn't mentioned outside the audio design community, in the context of double-blind studies.) In a real DBT, the listener would "see" the product, but doesn't realize he might have been given a "placebo" to evaluate. The listener is not listening for differences, but simply reporting his likes and dislikes of his evaluation sample. Just like he would report or be examined for the effects of the product or placebo in a medical double-blind study.


RE: Resolution? That would involve scientific validation..., posted on September 11, 2009 at 05:47:58
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 There is a serious misunderstanding here, DBT is the name of a testing protocol rather than a test per se. You are conflating the test protocol with a the test objectives and as a result jumping to a very bizarre conclusion. How do you use the 'real DBT' you've outlined to test for audible differences? Or closer to home, how does an audiologist go about performing a hearing test? The tests objectives such preference trials, drugs efficacy, test for audible differences should be not be confused with the test methods.

Music making the painting, recording it the photograph
True Audio DBT, Explained................, posted on September 11, 2009 at 09:32:31
Todd Krieger
Audiophile

Posts: 22046
Location: SW United States
Joined: November 2, 2000
 "How do you use the 'real DBT' you've outlined to test for audible differences?"

I've explained how this is done **numerous** times here...... It seems like you're not paying attention to what you read......

But, I'll explain it again.......

An audio company has an upgraded product that needs evaluation. And it intends to do a double-blind study regarding the product's effectiveness on sound. The company builds 100 samples of the upgraded product, and 100 samples of what was the previous product, but looks just like the upgraded product. The second group of items is the "placebo".

The items are all serialized, to keep track of the real items and placebo items. The items are then packaged, to be sent out for evaluation. The packers, shippers, and those taking requests for evaluation are equivalent to the administrators, and have no idea half the items being sent out are "placebos"- This is what makes the evaluation "double blind".

So 200 people are now evaluating the new product, where 100 of them have no idea they're evaluating a "placebo".

The detailed opinions are then collected for all 200 evaluators, after at least a month's time. (The time could be longer or shorter. But long enough to draw detailed conclusions.)

Once these evaluation reports are all collected, the serial numbers on the reports are then sorted for the real product and placebo product.

And finally, the collective evaluations of the 100 real products are compared to those of the 100 placebo products. Whether or not the upgrade was effective is determined by how much of a positive difference in collective opinion for the real product relative to the collective opinion for the placebo. (And in some cases, there might be greater preference for the placebo.)

This has nothing to do with "sighted" or the evaluator's ability to hear differences. This is purely an evaluation of the **product**.

It doesn't take a rocket scientist to explain this.


RE: Weird................., posted on September 11, 2009 at 11:14:16
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 You start out with the big statement "I don't ever recall a true double-blind study conducted in audio" and then conclude your true audio DBT explained post with "This has nothing to do with "sighted" or the evaluator's ability to hear differences. This is purely an evaluation of the **product**.

wow! the text sums the central issue, you've proposed a model that does not even attempt to answer a basic audio requirement "testing for audible differences", it's blind test alright but it's incapable of providing useful answers to the pertinent questions.

Music making the painting, recording it the photograph
For Those Bent on Proof and Accuracy, We Must Be Consistent............, posted on September 11, 2009 at 13:19:30
Todd Krieger
Audiophile

Posts: 22046
Location: SW United States
Joined: November 2, 2000
 You start out with the big statement "I don't ever recall a true double-blind study conducted in audio" and then conclude your true audio DBT explained post with "This has nothing to do with "sighted" or the evaluator's ability to hear differences. This is purely an evaluation of the **product**.

wow!

When you cannot refute something with nuts-and-bolts explanation, your only option is "wow"............

For those who are so bent on proof and accuracy, we must be consistent. We cannot make exceptions, simply because they go against your views.

Double blind testing has a standard protocol, and other testing should not conveniently be labeled "DBT" for the express purpose of associating it with this standard protocol. For it's a disingenuous attempt of utilizing **substitute** methods under the guise of this established protocol.


RE: For Those Bent on Proof and Accuracy, We Must Be Consistent............, posted on September 11, 2009 at 13:36:12
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 When you cannot refute something with nuts-and-bolts explanation, your only option is "wow"............

What else do you expect me to say? You dont recall "any true audio DBT" and then proceed to propose a model that's incapable of answering the most basic audio question "Is it audible?"

"Double blind testing has a standard protocol"

No, Double blind testing is a protocol separate from the test objectives. And so far you have failed to distinguish between test objectives and test protocols.


Music making the painting, recording it the photograph
RE: Resolution? That would involve scientific validation..., posted on September 11, 2009 at 08:03:24
Tony Lauck
Audiophile

Posts: 3048
Location: Vermont
Joined: November 12, 2007
Contributor
  Since: February 24, 2009
 I agree with you, Todd is off base.

The justification for blind tests is that they enable one to construct a simplified causal model, e.g. one makes the assumption that the test subject is influenced by the sound waves he hears, but nothing else. This is based on the plausible assumption that the test subject does not know the random choices made in the test protocol, and that the random choices have certain statistical properties to facilitate mathematical analysis.

One can also represent single blind tests with a causal model and reach valid conclusions if the model is correct. The analysis will be the same, but the model makes an additional assumption: the experimenter is unable to influence the subject. Or one can represent non-blind tests with a model, and also reach valid conclusions if the model is correct. Here the model makes an additional assumption that the subject is not influenced by his other senses. Even in this case, the conclusion will be valid, if the model is correct. (Of course in the non-blind test many people reject this assumption, hence they reject the model and the conclusion.)

The reason for listing these different cases is to show their common elements. In every case interpretation of an experiment requires a model that the interpreter believes is correct. Based on this belief the interpreter can use logic and mathematics to reach a conclusion. In each case, he will be correct (at some statistical probability) if the model conforms to reality. If the model does not conform to reality, he may not reach a correct conclusion.

Most of the disputation concerning DBTs ultimately rests on the factions having different models. (In many cases, the factions don't realize that they even need a model, nor have they articulated one.) Todd's medical examples are inappropriate to audio, because the space of medical models is different from the space of perceptual models, for obvious reasons.

I have no objections to positive DBTs, for reasons that I have stated above. However, when negative results are interpreted positively, they do not fit into a simple model, which would conclude nothing from negative experiments. Since there is no obvious implicit model in these cases, it is incumbent on the experimenter to provide a causal model that explains the reasoning from negative results to a positive conclusion. This is simply missing from most disputed audio DBTs.


Tony Lauck

"Diversity is the law of nature; no two entities in this universe are uniform." - P.R. Sarkar

RE: Thanks Tony -nt, posted on September 11, 2009 at 10:26:02
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 *

Music making the painting, recording it the photograph
RE: Resolution? That would involve scientific validation..., posted on September 11, 2009 at 04:57:01
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 "In a real DBT, the listener would "see" the product, but doesn't realize he might have been given a "placebo" to evaluate. The listener is not listening for differences, but simply reporting his likes and dislikes of his evaluation sample. Just like he would report or be examined for the effects of the product or placebo in a medical double-blind study. "

And how would this affect the results of an audio DBT? How does it make in invalid? I have been waiting for several people to provide a reason why it would for years.



"Probability is the very guide to life."---Cicero
Is that a strawman of your own construction? (nt), posted on September 10, 2009 at 21:18:18
mkuller
Audiophile

Posts: 15929
Location: SF Bay Area
Joined: April 22, 2003
Contributor
  Since: December 28, 2003
 (nt)

ROFL: Sighted auditioning is hardly a straw man!!, posted on September 11, 2009 at 04:49:48
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 Pity it's so unreliable for detecting small differences.

"Probability is the very guide to life."---Cicero
RE: ROFL: Sighted auditioning is hardly a straw man!!, posted on September 16, 2009 at 03:55:39
morricab
Dealer

Posts: 4456
Location: switzerland
Joined: April 1, 2005
 "Pity it's so unreliable for detecting small differences"

You know this?? How? Citations please.

The same argument can and has been made about DBTs as well. One could argue in fact that because things that are readily heard sighted and are not heard blind that the blind testing is unreliable for small differences and that the sighted auditioning clearly superior.

According to J. Gordon Holt..., posted on September 11, 2009 at 10:30:46
mkuller
Audiophile

Posts: 15929
Location: SF Bay Area
Joined: April 22, 2003
Contributor
  Since: December 28, 2003
 ...founder of Stereophile, who first embraced the ABX comparator when it was originally released, he said essentially:

"Sighted A/B testing is much more sensitive to detecting small audible differences."

From everything I've seen, heard and read, audio DBTs seem to remove small audible differences. They are sensitive only to gross loudness, noise and frequency repsonse differences.

I'm sure J. Gordon Holt was a fine man,, posted on September 11, 2009 at 14:06:00
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 a nice guy, and so on. However, he was not an expert in psychometrics, and I don't regard him as an audio authority of any kind. So when if he said that sighted A-B testing is more sensitive than a DBT under the same circumstances, I see no reason to believe him.

"Probability is the very guide to life."---Cicero
Can you show me scientific validation..., posted on September 11, 2009 at 14:14:23
mkuller
Audiophile

Posts: 15929
Location: SF Bay Area
Joined: April 22, 2003
Contributor
  Since: December 28, 2003
 ...for the audio equipment comparison DBTs?

And a scientific experiment showing the greater sensitivity you claim for audio DBTs over sighted A/B testing?

Real science requires proof.

Why should anyone take your or anyone else's word for it...
Is sighted auditioning scientifically calibrated?, posted on September 12, 2009 at 15:45:17
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 "And a scientific experiment showing the greater sensitivity you claim for audio DBTs over sighted A/B testing?"

Sorry, I don't have to defend claims I don't make.

"Probability is the very guide to life."---Cicero
Then how about for the claim you made?, posted on September 13, 2009 at 10:54:57
mkuller
Audiophile

Posts: 15929
Location: SF Bay Area
Joined: April 22, 2003
Contributor
  Since: December 28, 2003
 >So when if he said that sighted A-B testing is more sensitive than a DBT under the same circumstances, I see no reason to believe him.>

Got any scientific evidence he's wrong or just a religious belief?


How can I believe or disbelieve a meaningless claim?, posted on September 14, 2009 at 07:00:12
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 I already asked you how to calibrate a sighted audition.

"Probability is the very guide to life."---Cicero
How do you calibrate an audio DBT?, posted on September 14, 2009 at 09:37:30
mkuller
Audiophile

Posts: 15929
Location: SF Bay Area
Joined: April 22, 2003
Contributor
  Since: December 28, 2003
 ...what are your controls?

If you're claiming it's scientific, it has to have been validated, calibrated and controlled for various types of audible differences.

Are you claiming sighted auditions are scientific?

Been there, done that., posted on September 18, 2009 at 08:40:06
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 You're the one who asked me to compare the sensitivity of those unscientific sighted auditions to the sensitivity of auditions applying some scientific protocols.

It doesn't do me any good to explain how a DBT would be calibrated, as you just ignore it.

"Probability is the very guide to life."---Cicero
So you're calibrating the listeners?, posted on September 18, 2009 at 10:55:36
mkuller
Audiophile

Posts: 15929
Location: SF Bay Area
Joined: April 22, 2003
Contributor
  Since: December 28, 2003
 ...LOL!

That's great since the test is only testing their listening skills anyway.

And how many listeners and test trials are required for a scientifically-accepted confidence level?

The question for audio DBTs is what are the sensitivity thresholds for detection of audible differences between components. This would part of the scientific validation process.

To calibrate sensitivity there should be a control, with known parameters, and then a test component where the specific parameter you're testing for is unknown.

For example, take dynamic range or imaging differences - can DBTs detect differences here or is the test too insensitive? And at what level can it detect, them if at all?

It sounds like loudness and frequency response or noise differences are the only things you're able to test for here.

And at very insensitive levels.

Ummmm . . . it's people who listen. (nt), posted on September 20, 2009 at 13:33:58
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 .

"Probability is the very guide to life."---Cicero
I think you've got it!, posted on September 20, 2009 at 13:46:27
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 Are they being used to determine if interconnects can make an audible difference, for example, or are they being tested for their hearing skills?

There are different objectives described in the two options above. One requires a calibrated "testing device," i.e., the listener group, and the other is a silly parlor trick based on braggadocio vs. insecurity.

Of course I do., posted on September 20, 2009 at 19:19:45
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 You, however, are still confused in some degree.

RY
"Are they being used to determine if interconnects can make an audible difference, for example, or are they being tested for their hearing skills?"

Just changing the subject of the sentence around doesn't change the reality. Let's phrase the questions more properly:

-Can the subjects show they can hear the differences when the interconnects are changed under the conditions of the test?

-What are the limits of the subjects hearing abilities?

If one could show that the interconnects make differences large enough to be audible (established by blind testing, of course), one wouldn't have to bother doing blind tests. But lacking that, anyone who has such confidence they can hear the differences should have no objection to doing blind tests--unless of course, they aren't really that confident but in fact are insecure. Hence, your next paragraph involves a non sequitur--it does not follow.

RY
"There are different objectives described in the two options above. One requires a calibrated "testing device," i.e., the listener group, and the other is a silly parlor trick based on braggadocio vs. insecurity."

Get back to me when you come up with something a lot better than that.

"Probability is the very guide to life."---Cicero
You don't get to re-write my question to fit your own agenda., posted on September 21, 2009 at 10:25:56
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 How arrogant can you possibly be?

""Are they being used to determine if interconnects can make an audible difference, for example, or are they being tested for their hearing skills?" (RY question)

Just changing the subject of the sentence around doesn't change the reality. Let's phrase the questions more properly:

-Can the subjects show they can hear the differences when the interconnects are changed under the conditions of the test?

-What are the limits of the subjects hearing abilities?"

No Pat, you can suggest those questions all you want, but they are not the question(s) I asked.I'll give you a break for maybe not following along with the pronouns in my question, as "they" were a bit confusing, but in the original context it shouldn't have strained you too much. If one is trying to determine if there is an audible difference between interconnects, by using a DBT protocol, one must hae a very large group of listeners as well as repeated tests, no? If one is challenging someone's hearing skills, one needs only one listerner and only a few tests. See the difference?


"If one could show that the interconnects make differences large enough to be audible (established by blind testing, of course), one wouldn't have to bother doing blind tests."

Did you mean to type that sentence? So, one would use a blind test to show differences large enough that they don't really need blind tests...I can't stop laughing. And you have the nerve to tell me to get back to you when I come up with something "better?"


"But lacking that, anyone who has such confidence they can hear the differences should have no objection to doing blind tests--unless of course, they aren't really that confident but in fact are insecure. Hence, your next paragraph involves a non sequitur--it does not follow."

This shows you either didn't read the my post (but probably only the name of the author), or you have wilfully misconstrued its meaning. The only non sequuitor here is you not following along with the meaning of the post.


"RY
"There are different objectives described in the two options above. One requires a calibrated "testing device," i.e., the listener group, and the other is a silly parlor trick based on braggadocio vs. insecurity."

Get back to me when you come up with something a lot better than that."

You don't get it still?? "Braggadocio" refers to the attitude of those who claim to hear something that is unlikely. "Insecurity" refers to the state of mind of those who find it necessary to use a flawed protocol, all the while relying on the subject not knowing that one can't prove the null hypothesis, just to "disprove" someone's hearing claim. Either one is trying to use a DBT in as scientifically valid a form as possible, despite the tests inherent flaws, or one is participating in a farce. Where do you fall, Pat?
You are just totally confused., posted on September 21, 2009 at 12:01:28
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 The only thing I need to comment on is your failure to understand the concept of a threshold.

PD
"If one could show that the interconnects make differences large enough to be audible (established by blind testing, of course), one wouldn't have to bother doing blind tests."

RY
"Did you mean to type that sentence? So, one would use a blind test to show differences large enough that they don't really need blind tests...I can't stop laughing. And you have the nerve to tell me to get back to you when I come up with something "better?" "

Well, Stereophile did such a test between a tube amp and a SS amp driving a big B & W speaker, when the FR measurements clearly showed that the difference was above established thresholds thresholds established by other blind tests. So one wouldn't have needed to do that rather poorly run test to show the differences were audible.

One potentially do the same thing with interconnects, but except for phono cables, a difference above threshold is much less likely.

One can establish audible differences between components in two ways:

1) Do measurements which show differences above already established thresholds of audibility.

2) Do controlled blind tests.



"Probability is the very guide to life."---Cicero
Not a bit, Pat, and I still don't accept you hijacking the question asked., posted on September 21, 2009 at 14:36:03
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 RIIIGHHT. Another smoke screen. Now I don't understand thresholds. You just keep fantasizing about my lack of knowledge, if it makes you feel better about yourself. I won't waste the time to continue to call out your arrogance and pig-headed refusal to see your own faulty logic, but I will comment on this duesey:

"1) Do measurements which show differences above already established thresholds of audibility.

2) Do controlled blind tests."

I'm all for #1. In fact, it is my mantra like "do DBTs" is for you, AJ, and that Wile E. Coyote of the internet, RBNG. But I require that one determine the thresholds without resorting to DBTs. Do the threshold deterination blind, of course, but don't neglect to solve all the other problems with the protocol as well. It is absurd to propose that the only alternative to currently "accepted" DBT protocols would be sighted testing.

As for #2, please do us all a huge favor and explain how you are proposing that be done. Please exlain about controls, thresholds, types I and II errors, correlation, the Placebo effect, forced-choice problems in testing, etc., etc. I think I know already what you will post as a reply. Come on and surprise me. Convince me to drink the HydrogenAudio Kool-aid...
RE: Worthwhile discussion, posted on September 22, 2009 at 02:36:35
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 You've already dicussed some of these issues with mls-stl in a separate thread, why are you bringing them up again?

"As for #2, please do us all a huge favor and explain how you are proposing that be done. Please exlain about controls, thresholds, types I and II errors, correlation, the Placebo effect, forced-choice problems in testing, etc., etc. "

You question ask for answers to a gamut of issues associated with controlled tests in general. While at it should Pat D also give a concise response to the meaning of life?

"I think I know already what you will post as a reply. Come on and surprise me. Convince me to drink the HydrogenAudio Kool-aid..."

Statements such as this suggest that you are not really interested in any worthwhile discussion. Question #2 is too open-ended and as such unanswerable.

Music making the painting, recording it the photograph
If only it were a worthwhile discussion..., posted on September 22, 2009 at 05:39:24
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 "You've already dicussed some of these issues with mls-stl in a separate thread, why are you bringing them up again?" (TAH)

Actually, it is in this same thread, different branch. And I "bring it up again" for two reasons: first, the discussion with mls-stl was about forced-choice; and second, the "discussion" with PatD was abuot calibration. I was responding to a comment by PatD, who posted: "Ummmm . . . it's people who listen." when asked if he was planning to calibrate listeners.


"As for #2, please do us all a huge favor and explain how you are proposing that be done. Please exlain about controls, thresholds, types I and II errors, correlation, the Placebo effect, forced-choice problems in testing, etc., etc. " (RY)

You question ask for answers to a gamut of issues associated with controlled tests in general. While at it should Pat D also give a concise response to the meaning of life?" (TAH)

Yes, I'm asking for answers to a gamut of issues associated with controlled tests in general. Well, isn't an audio DBT supposed to be a controlled test? Why can't you or PatD provide any answers for anything other than the "sighted vs blind" problem? Just because a test satisfies one problem doesn't make it valid. How about addressing one of those problems?. That would certainly be a start to having a "worthwhile discussion."

Oh, and the comment about the meaning of life is absurd. Leave the strawmen behind.


""I think I know already what you will post as a reply. Come on and surprise me. Convince me to drink the HydrogenAudio Kool-aid..." (RY)

Statements such as this suggest that you are not really interested in any worthwhile discussion."

Do you ever read PatD's posts? I'd love to have a real discussion of the issues, but having my questions rewritten by Patd is NOT participating in debate. PatD plays semantic and rhetorical games in lieu of actual participation, and I'm simply acknowledging that no matter how many times I ask him a question, he avoids answering it in any way useful or participatory. I'm all for a worthwhile debate. It is PatD and yourself who won't engage (your own refusal is embedded within this thread).



"Question #2 is too open-ended and as such unanswerable."

To which "Question #2" are you referring? Here's mine: "...are they (participants) being tested for their hearing skills?"

Here's PatD's "Question Number 2:"-What are the limits of the subjects hearing abilities?"

My question was not only somewhat rhetorical, but also is reasonable in context: that audio DBTs are often used to "show up" someone who claims to have heard a difference and says they can back up the claim. The objective is very different from trying to determine if there really is an audible difference between cables, say. One question is about listeners, the other is about equipment. The question is hardly "open-ended," and for any given DBT ought to be answerable.

Now if you are referring to PatD's question #2, you'll have to take that up with him, as it is not the question that I posed. However, I find his version to be quite important. If it is indeed "unanswerable," than the audio DBT world has even bigger problems...

Equipment isn't audible in itself, only in relation to listeners., posted on September 22, 2009 at 08:41:48
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 It's the listeners who do the hearing. You can't fob that off on the equipment.

"Probability is the very guide to life."---Cicero
Try not to be such a twit, Pat., posted on September 22, 2009 at 09:27:15
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 "Equipment isn't audible in itself, only in relation to listeners."

Actually, only the medium through the equpment is audible, and then in relation to listeners.

However, its about the objectives of the DBT. Either the purpose of the DBT is to determine if there are audible differences between components, or, the purpose is to DBT is to test someone's hearing, primarily to "prove" to them that they fantasize when they claim they can hear differences, existing or not.



Do you realize that you continue to play childish semantic games, and that your technique for personal entertainment can be read by all?

You are still confused., posted on September 22, 2009 at 10:39:40
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 RY
"However, its about the objectives of the DBT. Either the purpose of the DBT is to determine if there are audible differences between components, or, the purpose is to DBT is to test someone's hearing, primarily to "prove" to them that they fantasize when they claim they can hear differences, existing or not."

It isn't fantasy, it's just the way perception works. We overdetect differences. I presume this is an evolutionary advantage in that it would be an advantage for survival.

Get it straight: what is tested is whether the listeners can show they can detect the differences under the conditions of the test.



"Probability is the very guide to life."---Cicero
No, Pat. But you can keep telling yourself that., posted on September 22, 2009 at 16:40:01
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 Maybe you can think about the questions some have about the DBT protocol on your own, outside of the Forum. Maybe someday you'll have the confidence to answer questions in a public forum without obfuscation. Remember that its ok to have been wrong about something. Its even better to admit it, and even better than that to learn from the experience.

RE: Surely the test design follows the test objectives., posted on September 23, 2009 at 05:14:47
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 Surely if a certain population claim that certain differences are audible, but only demonstrate such ability under sighted tests, the goal of a control test is to determine whether such claimed audible differences are still detectable under blind conditions, listener calibration is a red herring under such circumstances as the goal of the DBT to test the claim rather than "someone's hearing".

Music making the painting, recording it the photograph
partially agree...., posted on September 23, 2009 at 06:05:04
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 "Surely if a certain population claim that certain differences are audible, , the goal of a control test is to determine whether such claimed audible differences are still detectable under blind conditions, listener calibration is a red herring under such circumstances as the goal of the DBT to test the claim rather than "someone's hearing"."

Your example implies that the "certain population" fails to demonstrate the claim(s) under tests other than sighted ("...but only demonstrate such ability under sighted tests"). You also imply that this test is a DBT. So far, so good. I'm all for eliminating potential biases, and sight is surely one of them. But calling something a DBT doesn't answer how the protocol addresses a myriad of other problems, many of which may have an effect on the listeners' ability to discern differences. (I have listed a few of these in previous e-mails, but there has yet to be a description of how the protocol addresses them. Honestly, I'd love to read how it could.)

Listener calibration is potentially not as important in the single-listener hearing test scenario. It becomes extremely important, though, if one tries to extrapolate the results of the hearing test to the broader population, or even more egregiously apply the results to a geralization about the DUT ("Joe couldn't identify which amp was which in a DBT, therefore, all amps sound the same"). The occurrence of this logical travesty is denied by most, and yet it seems to appear with some frequency in this forum (less so since RBNG took leave). "...the goal of the DBT (is) to test the claim rather than someone's hearing." Unles you are talking about a ery large sample group of listeners as opposed to a single claimant, and no calibration has been performed, and no thresholds of audibility determined against the calibrated listener, then the DBT is indeed a "hearing test," more geared towards disabusing someone of their braggadocio than towards a scientifically-valid experiment.
RE: calibration provide answers to a different question, posted on September 23, 2009 at 07:32:20
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 Could you collate from a previous emails the myriad of problems that affect listener's ability to discern differences to forestall any tangential comments from myself.

To your main point, a very large sample of listeners is not required to verify a claim, what's required is a representative sample and the sample size can be quite small, as small as 30 (I think?), a larger sample size would give a smaller sampling error however the results of the smaller sample size is still valid albeit with a larger sampling error. This is a fundamental principle in Statistics that is somewhat counter-intuitive but true nonetheless.

Even so, calibration is a non-issue if the population claim they can discern audible differences. The results of a calibration test provide answers to a different question. Asking whether athelete A can complete a 100m race under 10secs is different question from why athelete A did or did not complete the 100m dash under 10secs.

Music making the painting, recording it the photograph
RE: calibration provide answers to a different question, posted on September 23, 2009 at 08:36:08
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 "Could you collate from a previous emails the myriad of problems that affect listener's ability to discern differences to forestall any tangential comments from myself."

Actually, I'm a bit busy, so can't be a search engine right now. We can explore those questions at another time.


"To your main point, a very large sample of listeners is not required to verify a claim, what's required is a representative sample and the sample size can be quite small, as small as 30 (I think?), a larger sample size would give a smaller sampling error however the results of the smaller sample size is still valid albeit with a larger sampling error."

We're talking audio here, right? 30 people is a very large sample. For the conditions to be the same, the DBT must be done at the same time, in the same room, with the same source material. All things must be equal, so that the variables don't compound. And 30 people in a sweet-spot? Not too likely! on- vs. off-axis makes a difference when one is talking about "subtle differences."


"The results of a calibration test provide answers to a different question. Asking whether athelete A can complete a 100m race under 10secs is different question from why athelete A did or did not complete the 100m dash under 10secs."

Yes, but I'm not asking why. I'm asking that the sensitivity of the participant be calibrated againt a known threshold of audibility. That really shouldn't be too hard. I've done several (I have loat hearing in my right ear due to a failed otosclerosis surgery, and thus have spent much tome listening to test tones in the both :( ) Clearly one wouldn't use a rod marked in feet to measure inches. Nor would one use a person with low-level hearing loss to opine on the audibility of low organ notes. The yard-stick has an advantage that the poor guy doesn't have: it maintains its acuracy. The listener can't do that.

RE: calibration provide answers to a different question, posted on September 23, 2009 at 15:31:57
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 "Actually, I'm a bit busy, so can't be a search engine right now. We can explore those questions at another time."

Fair enough, another time.


"For the conditions to be the same, the DBT must be done at the same time, in the same room, with the same source material. All things must be equal, so that the variables don't compound. And 30 people in a sweet-spot? Not too likely!"

Well you've implicitly stated that the conditions you've put forward are unrealistic and cannot be replicated in actual tests. Similar test conditions can be established without resorting to such restrictions. Identical source material is fairly easy to replicate. But insisting that all tests take place at the same time, in the same room is a red herring. The only condition that need be satisfied is that listening environment be non-intrusive. After all, it's unlikely that the claim under investigation is that differences are only audible at a particular time in a particular room.

"Yes, but I'm not asking why. I'm asking that the sensitivity of the participant be calibrated against a known threshold of audibility. That really shouldn't be too hard... Clearly one wouldn't use a rod marked in feet to measure inches. Nor would one use a person with low-level hearing loss to opine on the audibility of low organ notes. The yard-stick has an advantage that the poor guy doesn't have: it maintains its accuracy. The listener can't do that. "

The whole idea of taking a representative sample is to ensure that the results are not skewed by data of the type you've mentioned. It seems to me that the request for listener calibration stems from a lack of the understanding of the fundamental statistical principles that underpin studies of this nature.

Music making the painting, recording it the photograph
sorry, been away..., posted on September 28, 2009 at 14:46:19
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 ""For the conditions to be the same, the DBT must be done at the same time, in the same room, with the same source material. All things must be equal, so that the variables don't compound. And 30 people in a sweet-spot? Not too likely!" (RY)

Well you've implicitly stated that the conditions you've put forward are unrealistic and cannot be replicated in actual tests" (TAH)

Yes. That is one of the reasons I think a scientifically valid audio DBT cannot exist.


" Identical source material is fairly easy to replicate. But insisting that all tests take place at the same time, in the same room is a red herring."

Not at all. Audio systems sound very different over the course of the day. i think you would be hard-pressed to find anyone (besides yourself, of course!) who would not support this as fact. For the large sample to be valid, the SAME equipment must be used, and it must be operating in the same fashion. Thus the test must be done at the same time.


"The only condition that need be satisfied is that listening environment be non-intrusive."

Umm, I don't think you mean that. One cannot increase the number of variables and expect the test to be valid.


"After all, it's unlikely that the claim under investigation is that differences are only audible at a particular time in a particular room."

Talk about a red herring! The test is never about time or space. It is about a difference that is either audible or not. If the time difference between trials is an issue that affects the test, or the room changes and thus skews the results, then the test is clearly invalid. If the test shows a difference, then the particular time and particular space issue is irrelevant.


"The whole idea of taking a representative sample is to ensure that the results are not skewed by data of the type you've mentioned. It seems to me that the request for listener calibration stems from a lack of the understanding of the fundamental statistical principles that underpin studies of this nature."

I'll ignore the egregious ad hominem, and reply that it seems that you are not aware of the number of variables involved beyond those which affect the listeners. Again I'll ask you, as I asked PatD, what are your tests for? Are you testing someone's hearing claim, or are you testing for audibility of a difference between equipment? In the first case, the sample size is irrelevant: it is a test of one person's claim, and thus the sample size need only be one. In the case of the latter, the equipment better be operating exactly the same way. If not, unless the test takes place at the same time at the same space, the sample size is meaningless.

RE: sorry, took a while to respond, posted on September 30, 2009 at 02:37:04
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 Could you clarify a few comments

"For the large sample to be valid, the SAME equipment must be used" (RY)

As in same model, same serial number? Could you expand on what you mean by same equipment.

"I'll ignore the egregious ad hominem" (RY)

It isn't and your response unhelpfully sidesteps the issue rather than address it i.e. the relevant statistics. As it stands you seemingly hold a self-contradictory position but I will await your clarification before commenting your post in further detail.


Music making the painting, recording it the photograph
RE: sorry, took a while to respond, posted on September 30, 2009 at 12:06:31
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 "Could you clarify a few comments
"For the large sample to be valid, the SAME equipment must be used" (RY)

As in same model, same serial number? Could you expand on what you mean by same equipment."

Yes: same EXACT equipment. One cannot assume before any tests are performed that the same model, same brand is sufficient to insure that a constant doesn't become a variable.


""I'll ignore the egregious ad hominem" (RY)

It isn't and your response unhelpfully sidesteps the issue rather than address it i.e. the relevant statistics."

An ad hominem argument, also known as argumentum ad hominem (Latin: "argument to the person" or "argument against the person") is an argument which links the validity of a premise to a characteristic or belief of a person advocating the premise. You (TAH) proposed that my argument stems from a 'lack of understanding of the fundamental principles..." when it most certainly does not. Your argument was about me and my education (or lack thereof) rather than the substance of the position.Perhaps you might consider challenging my position based on a reasoned argument againt that position, rather than relying on an erroneous assumption about my education and all that implies. Furthermore, my response most certainly doesn't sidestep anything: it simply asks what I believe to be fundamental to answer before addressing any further audio DBT questions: what is the purpose of the test?


Ad Hominem:

RE: sorry, took a while to respond, posted on October 11, 2009 at 13:10:11
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 I have mulled over this post for a while now and there is no getting away from it, your comments do not stand up to scrutiny. To claim that an audio DBT is only valid if performed on the SAME equipment is faulty. The test designer performs validation tests to establish that the suitability of the equipment, as every piece of equipment mustpass this validation test it's a tad illogical to insist only a single set of equipment is valid for the test.

Insisting that experiments take place at the same time and in the place is even worse supposedly to reduce the number of variables, You forget that we are interested in designing a valid experiment not a perfect one. Insisting that an experiment for testing audible differences is only valid if all tests are performed at same time is to imply that the environment is one of the key determining factors.

To answer your question, We are testing for audible differences, audible across a variety of reasonably non-intrusive listening environments i.e. a valid experiment should lend itself to unambiguous repeatability.




Music making the painting, recording it the photograph
sorry, but you are quite incorrect., posted on October 11, 2009 at 14:52:43
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 The equipment has to be the same, or there has been a variable introduced. The time, place, and even listening position have to be the same, or variables have been introduced.

"Insisting that an experiment for testing audible differences is only valid if all tests are performed at same time is to imply that the environment is one of the key determining factors."

Hey there, you got it!! It is!! It is extremely important. Time changes everything, even the impact of the environment on a test....

This is why audio DBTs to test for audible differences between components are impossible.

Either you are debating something that has nothing at all to do with the actual world of testing audio equipment and listener ability, or you have no knowledge about the workings of the equipment.

"To answer your question, We are testing for audible differences..."

So who is this "we?" They don't have much of a clue, no matter who they are.

RE: sorry, but you are quite incorrect., posted on October 12, 2009 at 09:24:54
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 "Hey there, you got it!! It is!! It is extremely important. Time changes everything, even the impact of the environment on a test...."

How does time change everything? How does the environment change everything? Seems to me, we are back to the "Audio is different" routine. You are effectively saying the it is impossible to replicate a controlled environment for audio hence why one must insist on using the same environment and equipment within a single time window, that's a gross overstatement with very little basis in reality IMO. Or do you have something to back that statement up beyond your personal opinion?





Music making the painting, recording it the photograph
RE: sorry, but you are quite incorrect., posted on October 13, 2009 at 03:20:38
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 "How does time change everything?"

Listeners change over time. I hear differently at various times of the day. This has been tested by my surgeon, Dr. Darius Kohan in NYC. I suffer from otosclerosis, and from a surgery that was unsuccessful. I have had multiple tests performed on my hearing since the surgery in September of '01. Diet, temperature, humidity, and various non-quantifiable aspects of brain function all can have an affect on a person's hearing. (I am nearly deaf in my right ear, so in a sense, my interest in the "audibility" debates is more metaphysical than physical. I don't have a personal horse in the race.)

"How does the environment change everything?"

I'm a bit surprised at your questioning of this. The listening environment is another "component" in the chain. If the room is different, the sound is going to be different. Anything that may affect the performance of a system must be removed from "variable" status, unless that variable is itself the DUT. I'm not sure it is necessary to provide proof that this is the case.

"Seems to me, we are back to the "Audio is different" routine."

I'm assuming that you mean "different from medical testing." If my assumption is correct, then yes, audio IS different. Please show me the controls, the placebos, the protocol correlations between the two very different worlds.

"You are effectively saying the it is impossible to replicate a controlled environment for audio hence why one must insist on using the same environment and equipment within a single time window, that's a gross overstatement with very little basis in reality IMO. "

You have that one a bit sideways, I'm afraid. For a test to be valid, the environment (including the listener, and including the variable of time) for audio must be a constant. As this is impossible, a valid test is therefore impossible.

"Or do you have something to back that statement up beyond your personal opinion?"

I'm curious about which piece of the logical construct you would have me prove? Or can we narrow down the potentials by answering a few questions:

Does the hearing abilities of listeners change over short durations (over the course of a day)" Yes or no.

Does equipment operate differently over the same time durations? (Those of us who have lived in New York City appreciate the difference that time makes given mains issues during the summer season.) Yes or no.

If one is testing for audibility of differences in interconnects, is it ok to change amplifiers in mid-test? Yes or no.

If one is again testing for audibility of differences in interconnects, is it ok to change the acoustics of the test space by switching rooms? Yes or no.
Well, if you can't ask a good question . . ., posted on September 21, 2009 at 19:08:17
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 As for the extent of controls, it all depends on what you want to do. What is the purpose of doing a blind test?

But of course, if you don't want to do blind tests, for heaven's sake don't worry about them.

"Probability is the very guide to life."---Cicero
If you can't answer ANY question...., posted on September 21, 2009 at 19:30:34
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 "good" or otherwise, then you really haven't got anything to offer, have you?

Unlike your buddy Bassnut, who would go out of his way to repeat his position endlessly, howling logical fallacies be damned, you don't ever take a position at all. I don't think you can answer critical questions about DBTs, but I know that even if you could, you most likely wouldn't. You're like your namesake from the '90s Saturday Night Live. No one knows what you are, or what you believe in. Always an excuse for why you won't answer a question, or enter a debate: your opponent isn't smart enough, the questions need rephrasing, "it depends..." Just rhetorical devices, with a paucity of substance.

When questions contain contrary to fact assumptions or are illogical, they need restating. (nt), posted on September 22, 2009 at 10:41:33
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 .

"Probability is the very guide to life."---Cicero
Bullshit., posted on September 22, 2009 at 11:23:09
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 Poor excuse for another attempted manipulation on your part.

There was nothing "contrsary to fact" or "illogical" in the question. You ust won't admit you misunderstood the point being made (since repeated 3 times).

Without validation and calibration of the test's sensitivity..., posted on September 20, 2009 at 19:34:31
mkuller
Audiophile

Posts: 15929
Location: SF Bay Area
Joined: April 22, 2003
Contributor
  Since: December 28, 2003
 ....how do you know the forced choice DBT doesn't interfere with the listener's ability to detect small differences?

It certainly appears to.

Good luck!, posted on September 14, 2009 at 06:13:47
E-Stat
Audiophile

Posts: 9838
Location: Central boonies
Joined: May 12, 2000
Contributor
  Since: April 5, 2002
 More ambiguous teflon to follow, if anything at all.

rw

Nope, posted on September 12, 2009 at 18:55:36
kerr
Audiophile

Posts: 3695
Location: Central Indiana
Joined: November 10, 2003
 But then again, there are no pseudo-scientists running around making such a claim, as there are for DBT's.

>Sorry, I don't have to defend claims I don't make.<

Why back off now? You've been claiming this for years!

A Forum, not a Tribunal, posted on September 13, 2009 at 13:27:27
Tony Lauck
Audiophile

Posts: 3048
Location: Vermont
Joined: November 12, 2007
Contributor
  Since: February 24, 2009
 "Why back off now? You've been claiming this for years!

You don't get it. :-) Pat D doesn't make claims. He asks people to defend claims. More generally, he asks questions, but seldom answers questions. Perhaps Pat was Grand Inquisitor in a previous incarnation.

Pat D is not unique. There are others around here or on vacation who play a similar game. When I hear people accusing others of making "claims" and demanding that they defend them by providing "evidence" a red flag goes up. I put the person into my "process" category, not my "outcome" category. Ironic that most of the people who think that social process is an effective way to ascertain truth also believe that they have a hold on objective truth and others are obviously delusional, or at least "biased".

We are audio hobbyists and Prophead is a forum, not a tribunal.


Tony Lauck

"Diversity is the law of nature; no two entities in this universe are uniform." - P.R. Sarkar

RE: A Forum, not a Tribunal, posted on September 14, 2009 at 04:45:59
kerr
Audiophile

Posts: 3695
Location: Central Indiana
Joined: November 10, 2003
 >You don't get it. :-) <

I guess not! :)

I would swear that for years Pat D was saying DBT's for audio are more sensitive than sighted A/B comparisons. He's now stating that is a figment of my imagination, presumably meaning that DBT's are NOT more sensitive than sighted A/B comparisons.

Glad I finally got that straight!

You have to realize his arguements are circular and rely on semantics, rather than substance...(nt), posted on September 13, 2009 at 11:59:05
mkuller
Audiophile

Posts: 15929
Location: SF Bay Area
Joined: April 22, 2003
Contributor
  Since: December 28, 2003
 (nt)

RE: Nope, posted on September 13, 2009 at 10:02:19
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 Please keep in mind that Mr. Kuller has attributed to me a specific claim, a rather silly one.

mkuller
"And a scientific experiment showing the greater sensitivity you claim for audio DBTs over sighted A/B testing?"

That is a figment of someone's imagination.

"Probability is the very guide to life."---Cicero
Hmmm..., posted on September 14, 2009 at 04:42:57
kerr
Audiophile

Posts: 3695
Location: Central Indiana
Joined: November 10, 2003
 mkuller
"And a scientific experiment showing the greater sensitivity you claim for audio DBTs over sighted A/B testing?"

Pat D "That is a figment of someone's imagination."

I agree it's a figment of someone's imagination, but I would have sworn it was your figment and your imagination. My mistake, sorry.

Mkuller made a meaningless claim., posted on September 14, 2009 at 07:02:38
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 I wouldn't bother to believe or disbelieve a meaningless claim.

"Probability is the very guide to life."---Cicero
What makes it meaningless?, posted on September 14, 2009 at 10:27:17
kerr
Audiophile

Posts: 3695
Location: Central Indiana
Joined: November 10, 2003
 The claim itself or that he attributed it to you? Or something else?

As I said, he only argues semantics, not substance..., posted on September 14, 2009 at 11:37:22
mkuller
Audiophile

Posts: 15929
Location: SF Bay Area
Joined: April 22, 2003
Contributor
  Since: December 28, 2003
 ...what he IS claiming is that audio DBTs, with their probably 90% null test percentage, are more *reliable* than sighted testing.

That's his word and the one the pro-DBT debaters use.

I'd say he's wrong about that, too.

DBTs tend to have mostly false negatives - where sighted listening can have false positives.

But since he claims the only way you can know for sure about an audible differences is with a positive DBT, it's a circular argument.

Even though he can't tell us about the validation, calibration and sensitivity of DBTs.

RE: As I said, he only argues semantics, not substance..., posted on September 15, 2009 at 04:59:53
kerr
Audiophile

Posts: 3695
Location: Central Indiana
Joined: November 10, 2003
 I guess I'm missing how a DBT could be more reliable but not more sensitive. Is it that semantics thing you mentioned, or am I just not getting it?
RE: As I said, he only argues semantics, not substance..., posted on September 15, 2009 at 11:44:41
Tony Lauck
Audiophile

Posts: 3048
Location: Vermont
Joined: November 12, 2007
Contributor
  Since: February 24, 2009
 "I guess I'm missing how a DBT could be more reliable but not more sensitive. Is it that semantics thing you mentioned, or am I just not getting it?"

Sensitivity and reliability are not the same thing. Sensitivity is a measure of ability to detect small differences. Reliability is a measure of ability to produce correct answers. So for example, one could have a very reliable test that was 100% accurate with big differences, but which had no sensitivity at all for small differences. Alternatively, one could have a very sensitive test that generally gave correct answers even with small differences, but which was very flaky and unreliable on occasion. Consider a golden-eared listener who had a spastic hand. He might identify all of the sounds perfectly, but would randomly hit the wrong button 10% of the time. This subject would be sensitive, but not reliable.

There are trade offs between sensitivity and reliability that depend on sample size and other factors. There is considerable theory associated with designing equipment that makes correct decisions on signals in the presence of noise. Presumably similar considerations occur in the "wetware" between our ears. :-)

Tony Lauck

"Diversity is the law of nature; no two entities in this universe are uniform." - P.R. Sarkar

RE: As I said, he only argues semantics, not substance..., posted on September 17, 2009 at 04:05:54
kerr
Audiophile

Posts: 3695
Location: Central Indiana
Joined: November 10, 2003
 >So for example, one could have a very reliable test that was 100% accurate with big differences, but which had no sensitivity at all for small differences.<

Such is the problem with DBT for audio, I'm told.

RE: As I said, he only argues semantics, not substance..., posted on September 16, 2009 at 04:13:09
morricab
Dealer

Posts: 4456
Location: switzerland
Joined: April 1, 2005
 "Sensitivity and reliability are not the same thing. Sensitivity is a measure of ability to detect small differences. Reliability is a measure of ability to produce correct answers."

Hi Tony,
I would like to point out that reliability does not have anything to do with ability to produce correct answers. It can produce the wrong answer consistently and still be reliable. Your definition of sensitivity is correct. Humans tend to be sensitive and even accurate but not reliably.

A good example from the scientific world is regarding peak integration in mass spectrometry. I saw a presentation where they compared the accuracy and precision (ie. reliability or reproducibility) of 3 different peak integration algorithms to a human manual integration. The winner in terms of accuracy (ie. getting closest to the true value) was the human but all three algorithms had better precision. In other words, they had more difficulty getting the right answer but could get the SAME answer more reliably.

In the world of analytical chemistry we are trying to have accurate tests that are also very precise...and sensitive!


"Alternatively, one could have a very sensitive test that generally gave correct answers even with small differences, but which was very flaky and unreliable on occasion. "

The only solution for such a test, assuming that the method cannot be improved, is to make lots of trials to get a good idea of the statistical variance and then to establish confidence limits. It will never be as clean as an accurate and precise method but it may still be useable.
It makes validation difficult...
RE: As I said, he only argues semantics, not substance..., posted on September 16, 2009 at 05:21:59
Tony Lauck
Audiophile

Posts: 3048
Location: Vermont
Joined: November 12, 2007
Contributor
  Since: February 24, 2009
 Perhaps I should have thrown "accuracy" (or some other word) into the fray. I would use the words "repeatability" or "consistency" for what you call "reliability". But not to argue definitions. My main point was that there are at least two terms involved, and you have pointed out that there are at least three. (Perhaps another inmate will come up with a fourth orthogonal category.) In other words, in the context of the thread there is substance here, not just semantics.

In some circles "guesswork" is frowned upon, and those who produce "guesses" that aren't correct nearly 100% of the time (or even consistent) are frowned upon. (The worst offender in this regard is presently on vacation from the Asylum.) However, it is well known that "guesses" can do better than chance in many cases, but at the cost of noise in the output. (Come to think of it, that's what dither does in an analog to digital converter.)

Tony Lauck

"Diversity is the law of nature; no two entities in this universe are uniform." - P.R. Sarkar

One of mkuller's repeated refrains . . ., posted on September 15, 2009 at 08:01:12
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 something like this: DBTs underdetect, sighted auditions overdetect.

You can understand this if you reflect that in a sighted audition, identifications are 100% (barring careless errors), whether there are any audible differences or not. So any pieces of equipment with audible differences will be identified as different, even without listening, before listening. It may be that the person listening doesn't perceive any differences, but he/she still knows which one is playing. Is that more sensitive? Mkuller thinks it is.

In DBT methodology, sighted auditioning may be used for training.

Controlled DBTs seldom overdetect but may underdetect small differences, unless they are large scale.

"Probability is the very guide to life."---Cicero
RE: What makes it meaningless?, posted on September 14, 2009 at 10:34:06
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 The claim itself is meaningful. That he attributed it to me is false.

"Probability is the very guide to life."---Cicero
And..., posted on September 11, 2009 at 13:58:01
E-Stat
Audiophile

Posts: 9838
Location: Central boonies
Joined: May 12, 2000
Contributor
  Since: April 5, 2002
 they proved that using lossy 128k MP3 codecs is good enough for music. :)

rw

RE: And..., posted on September 11, 2009 at 14:08:05
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 Who are "they?"



"Probability is the very guide to life."---Cicero
That would be The Imperial Codec Wizards, posted on September 11, 2009 at 14:37:07
E-Stat
Audiophile

Posts: 9838
Location: Central boonies
Joined: May 12, 2000
Contributor
  Since: April 5, 2002
 who deemed that such was the standard for commercial sales. Ever heard of iTunes? Amazon.com? Napster? Ring any bells?

rw

That's an "ABX" Test............., posted on September 9, 2009 at 01:07:52
Todd Krieger
Audiophile

Posts: 22046
Location: SW United States
Joined: November 2, 2000
 As I responded elsewhere, I don't ever recall a true double-blind study conducted in audio.


RE: That's an "ABX" Test............., posted on September 9, 2009 at 04:00:55
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 "I don't ever recall a true double-blind study conducted in audio."

uh....an ABX test is a type of double-blind test.

Music making the painting, recording it the photograph
Link To My Original Post on "DBTs" in Audio............, posted on September 9, 2009 at 09:46:51
Todd Krieger
Audiophile

Posts: 22046
Location: SW United States
Joined: November 2, 2000
 What's called a "DBT" in the audio world is totally removed from the standard protocol of double-blind testing outside the audio world. In fact, in real DBT evaluations, there's no such thing as "sighted".........

The reason why "DBT" results are inconsistent in audio testing is the simple fact the "DBT evaluations" in audio have nothing whatsoever to do with real double-blind studies conducted in other industries........

A true double-blind study in audio would be to hand a large number of prospective consumers a product to be evaluated. (These consumers need not be "audiophiles". And don't even need to have "listening skills".) The greater the quantity of evaluators, the better.

Half of the evaluators are given the real product, and half of them a substitute product, a placebo, which is basically the real product (visually identical) minus its unique design parameters. None of the evaluators realize that some of the samples being evaluated are *not* the real product. (This "blind" vs. "sighted" B.S. doesn't even exist outside the audio world. Just Google "sighted" and "double blind" and filter out "audio" and "listening".)

The products are evaluated over a period of time, results are submitted by the evaluators, and then the results of the real product are compared to those of the placebo product.

If the results exhibit a collective improvement using the real product demonstrably greater than that using the placebo, then the design might have a tangible impact on the advancement of sound reproduction.

If you were to Google "double blind study", you'll see numerous examples of this.


RE: Link To My Original Post on "DBTs" in Audio............, posted on September 9, 2009 at 11:21:33
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 "What's called a "DBT" in the audio world is totally removed from the standard protocol of double-blind testing outside the audio world"

You are quite wrong here, you are comparing apples to oranges, in this case comparing a 'detection test' to a 'placebo test' and coming up with the bizarre conclusion that a double-blind 'placebo test' is an example of DBT but a double-blind 'detection test' isn't.

Music making the painting, recording it the photograph
RE: Link To My Original Post on "DBTs" in Audio............, posted on September 9, 2009 at 11:40:32
Todd Krieger
Audiophile

Posts: 22046
Location: SW United States
Joined: November 2, 2000
 An ABX test is a "blind" test, but it doesn't follow the protocol of a true DBT. An ABX test doesn't even require a "test conductor", who would administer a DBT without himself knowing if it's the genuine or placebo product. This is where the "double" in "double blind" comes from. Where neither the administrator nor evaluator knows which product is being evaluated.

In the past, I've stated whether difference would "pass a DBT"...... Which is erroneous....... I should have been saying "pass an ABX".


RE: Link To My Original Post on "DBTs" in Audio............, posted on September 9, 2009 at 12:10:55
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 " An ABX test doesn't even require a "test conductor", who would administer a DBT without himself knowing if it's the genuine or placebo product. This is where the "double" in "double blind" comes from"

There seems to be an error in this phrase, could you clarify the wording?

PS: To move the discussion forward I have included a link to the wikipedia ABX article, note that the text "ABX tests can easily be performed as double-blind trials, eliminating any possible unconscious influence from the researcher or supervising technician." contradicts what I think you are attempting to put across in this post.

Music making the painting, recording it the photograph
RE: Link To My Original Post on "DBTs" in Audio............, posted on September 9, 2009 at 13:19:10
Todd Krieger
Audiophile

Posts: 22046
Location: SW United States
Joined: November 2, 2000
 "There seems to be an error in this phrase, could you clarify the wording?"

Double-blind testing implies neither the test conductor nor the evaluator (two people -> "double") knows whether the real product or the placebo product is being administered.

An ABX test does not even *need* a test conductor. It's a "blind" evaluation on the part of the evaluator. Aside from the double-blind protocol, there are a lot of types of "blind" testing. And ABX is merely one of these other forms of "blind" testing.

An ABX test does not even imply one of the two items ("A" and "B") is a "placebo"...... It's just a blind test to determine whether two items are tangibly different.


RE: Link To My Original Post on "DBTs" in Audio............, posted on September 9, 2009 at 14:14:10
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 "ABX tests can easily be performed as double-blind trials, eliminating any possible unconscious influence from the researcher or supervising technician."(Wikipedia ABX Test)

Per my revised post. An ABX test can be structured as double-blind test. The Meyer/Moran SACD/CD Comparison test is a good example of a double-blind ABX test, or do you disagree? If yes, why?

Music making the painting, recording it the photograph
RE: Link To My Original Post on "DBTs" in Audio............, posted on September 9, 2009 at 17:55:36
Todd Krieger
Audiophile

Posts: 22046
Location: SW United States
Joined: November 2, 2000
 The Wiki page states:

ABX tests can easily be performed as double-blind trials, eliminating any possible unconscious influence from the researcher or supervising technician.

It just says it can. But it never states **how**....... ABX does **not** involve comparing the effect of the actual product amongst a group of individuals against a placebo amongst a different group of individuals, which is how the double blind protocol is administered. So I question the statement's validity, without supporting examples.

If an audio company were to evaluate an upgraded product using the true DBT protocol, it would not use an ABX test (which is "blind", but not the standardized "double blind" protocol), but would compare the new design versus the existing (placebo) one, by comparing the collective satisfaction from a group of people using the new product versus the collective satisfaction from a different group using the existing (placebo) product, in which the two versions of the product are otherwise identical. This would be the only method to conduct a true double-blind study in audio.

In the double blind protocol, unlike an ABX test, nobody actually administers or evaluates *both* "A" and "B"...... Some will listen to only "A", others to only "B", in which one is the real product, the other is a placebo. The positive result is if there is a striking difference in consensus satisfaction between the product group and placebo group.

I don't know of a documented double-blind study that has actually taken place in the audio industry. Whatever that has been called "DBT" in audio is totally removed from the established and standardized methods for DBT in other industries.


RE: Link To My Original Post on "DBTs" in Audio............, posted on September 11, 2009 at 06:02:26
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 "It just says it can. But it never states **how**......."

let's revisit the text

"ABX tests can easily be performed as double-blind trials, eliminating any possible unconscious influence from the researcher or supervising technician"

The word easily would suggest that it's trivial to perform double-blind ABX tests and various hardware and software based tools have accomplished exactly that. Moran/Meyer SACD/CD comparison is an example of a hardware-based double-blind ABX test, while the link below is a software based example.

Per my other post, your comments suggest that you are confusing the test method with the test objective and jumping to a very bizarre conclusion.

Music making the painting, recording it the photograph
You're Confusing "Blind" with "Double Blind".........., posted on September 11, 2009 at 13:06:38
Todd Krieger
Audiophile

Posts: 22046
Location: SW United States
Joined: November 2, 2000
 All the tests that the so called "DBT" crowd is advocating are "blind" tests, but not the tried and true "double blind" protocol established and standardized in other industries. I really don't get why some want to pound "double blind" into something that really isn't.


RE: I am not, you are .........., posted on September 11, 2009 at 13:48:47
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 The primary condition for a valid "double blind test" is that subjective bias from the test participants as well as the test conductors is eliminated. I have given you with two audio examples. Could you explain how experimenters introduce subjective bias into either example?

Music making the painting, recording it the photograph
You two seem to be using different definitions of "double-blind.:, posted on September 11, 2009 at 14:29:54
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 Here's one: "Double-blind: Term used to described a study in which both the investigator or the participant are blind to (unaware of) the nature of the treatment the participant is receiving. Double-blind trials are thought to produce objective results, since the expectations of the researcher and the participant about the experimental treatment such as a drug do not affect the outcome. Also called double-masked." This is from Medicine.net.

Note the key phrase, "both the investigator or the participant are blind to (unaware of) the nature of the treatment the participant is receiving." This is somewhat different than trying to determine a difference between two amplifiers, say. It is clear that in medical DBTs the "blind" refers to awareness (of whether the treatment is a placebo or not), not necessarily to the opposite of "sighted."

RE: You two seem to be using different definitions of "double-blind.:, posted on September 11, 2009 at 15:05:14
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 In both examples given there are no test adminstrators (unlike the medical example you cited) in attendance as the test is adminstered by a software/hardware device with samples randomised and participants' answers collated by the said device. The design setup precludes contact between the researcher and the participants, therefore the researcher is unable to affect the participants' choices.

Music making the painting, recording it the photograph
What does that have to do with the medical definition of "double-blind??", posted on September 11, 2009 at 16:50:47
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 "In both examples given there are no test adminstrators (unlike the medical example you cited) in attendance as the test is adminstered by a software/hardware device with samples randomised and participants' answers collated by the said device."

What are the two examples I gave?? What medical example did I cite? I gave only a definition. If you wish to contradict or challenge that definition, please do so clearly. Are you talking about some examples of ABX? Just for giggles, let's revisit a post by EBradMeyer, one of the objectie's favorites:

"Micha Schattner and I discovered that the ABX high-current module, used for power amp switching and equipped with high-current speaker relays (but not for line-level tests) made a soft mechanical clang that was different depending on whether A or B was being chosen. We tested the audibility of this with the relay module a few feet away from us and no signal in the system, in a very quiet room, and we could hear the difference. I don't remember the number of trials but it was 8 or 10 apiece, all correct, so there was no doubt we were hearing it. This effect is irrelevant to ABX testing, for at least three reasons. First, you can muffle that sound easily even with no music playing -- a couple pillows over the box and you can't hear it any more. Second, with music playing you don't even have to muffle it (though I always do); it's so soft that any music masks it. And third, neither this box nor the reed relays used for line-level switching affect the audio signal at all. -- EBM "

Reason number 2 is so laughable it has to call into question the quality of thought that goes into a EBM ABX test: "Second, with music playing you don't even have to muffle it (though I always do); it's so soft that any music masks it." So it is possible to mask a sound that was clearly audible in tests WITH THE VERY MUSIC ONE USES IN THE TEST. Sounds a lot like EBM admitting that sounds, which could be differences between DUTs, are maskable by the very process that is supposed to reveal them. And reason 3? Just as ludicrous. There are all sorts of clues that are not in the audio signal that can affect the process. It's the equivalent of a test administrator winking every time the answer is "a." Now that the giggles are over, let's get back to your post:

"The design setup precludes contact between the researcher and the participants, therefore the researcher is unable to affect the participants' choices."

Yes, that's very nice. But one has many factors to address before any particular protocol can be deemed valid. You are still asking a participant to be an evaluator, and that's a very naughty thing, because it adds all sorts of bias potential that double-blind and ABX can't eliminate. The medical DBT works because the participant either gets better or not, and the relationship between those taking the actual medication and those taking (and reporting effects from ) a placebo explains whether the improvement is statistically significant. I'm afraid I don't see that kind of control in an audio DBT.

But I'd love it if you would show me.
RE: What does that have to do with the medical definition of "double-blind??", posted on September 12, 2009 at 16:34:57
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 "Yes, that's very nice. But one has many factors to address before any particular protocol can be deemed valid. You are still asking a participant to be an evaluator "

By the way the same applies for obvious reasons in a number of medical DBTs focused on visual or auditory perception but I digress....



Music making the painting, recording it the photograph
But you don't regress at all! On the contrary, you're at the heart of the matter!, posted on September 13, 2009 at 06:24:57
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 "By the way the same applies for obvious reasons in a number of medical DBTs focused on visual or auditory perception but I digress...."

Those types of tests are the ones that have the greatest number of challengers, for the same reasons. It has been very difficult to achieve consensus within the ranks for the protocol when evaluation of an effect is relegated to the participants. It is not the "bindness" of the test that is the problem per se. There are other parts of the test that also need to be addressed for the results to be acceptable.

In the case of the evaluation of results, one has passed into a "forced choice" scenario, i.e., the participant has to decide if there is an effect at all (choice between an effect and no effect).

I find no problem on a cenceptual level with the need for "double-blindnes" in an audio DBT. The problem for me lies in the lack of scientific rigor equally applied to the other potential problems inherent in the testing protocol. Of course, it is entirely possible to construct and apply a less-than-rigorous protocol, and still have statistically meaningful and valid results. Large differences in sound are hard to mask. It is when the differences are acknowledged to be subtle (if indeed they are audible at all) when the protocol must reach the degree of rigor to assure that the results are not tainted by a myriad of other influences besides "blindness...."

It is unfortunate that on this forum (and other sites as well) that it is the results of the DBT that seems to drive the degree of testing protocol analysis. The protocol must be challenged regardless. Expectation bias of the evaluators should have no influence on whether the protocol passes muster. I'd be interested in seeing a protocol that would, for example, eliminate the potential influence of the forced choice. I'm not sure that can be achieved as long as the participant has to decide anything...perhaps if we could tap into the auditory nerve and measure the response before the brain has a chance to mess it up, we might eliminate the bias effects. Then all we'd need would be a large enough sample to have statistical significance. Of course, that deals with general audibility, but not at all with the specific "prove it" demands on individuals...;)

RE: My original assessment was correct....., posted on September 12, 2009 at 10:43:46
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 It's pointless discussing DBTs with you, the essence of your post was captured brilliantly in an old post by another AA member.

"They will pick and choose among "facts". A science tidbit that supports their position will be adopted immediately and uncritically. A neighboring tidbit that doesn't support their belief will be challenged unmercifully, sometimes to the point of silliness."

I wouldn't bother responding to any of the 'issues' outlined in your post as I said earlier, you are free to believe Todd's statement it's your lookout.



Music making the painting, recording it the photograph
Another cop-out., posted on September 12, 2009 at 11:27:50
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 "It's pointless discussing DBTs with you, the essence of your post was captured brilliantly in an old post by another AA member."

Why is it pointless? Because I have a point of view that doesn't coincide with yours? That's about as weak a position as one can take. I also found that post you linked to be quite interesting, as I think it applies very accurately to you.


"I wouldn't bother responding to any of the 'issues' outlined in your post as I said earlier, you are free to believe Todd's statement it's your lookout."

Then why DID you respond at all? You have been explicitly invited by me to provide an argument to change my mind. I have not been holding my breath waitng, but I would honestly like to hear your rebuttal(s). For all your "objective/scientist" posturing, that you rely on the unquestioning acceptance of the scientifically questionable without engaging in any debate whatsoever is just pathetic. Your posts have less to do with any kind of knowledge being distributed, or decent debate being engaged in, than in marking your philosophical territory like a dog pissing on the neighborhod trees.

And just to be clear (again), since you appear to tailor your posts based on the poster's name and their perceived "side" rather than the post's contents: I don't entirely agree with Todd either. When you mature a little bit, you'll find that thoughts and ideas don't fall so neatly into black-or-white categories. Once you feel more confident in your knowledge base, you may be able to respond to thoughts with which you disagree, rather than closing your eyes, cupping your hands over your ears, and stomping your little feet.

RE: I can see you are raring to go, posted on September 12, 2009 at 15:35:16
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 but the goading won't work here. I do not have the inclination nor time to waste engaging in a discussion with you on the issue. I am not particularly bothered whether you change your mind or not. So thanks for the offer but no thanks.

By the way, I gave you the benefit of the doubt but your subsequent response affirmed my original decision so I am backing out of this one. If you feel Todd's position has some merit thats fine by me.




Music making the painting, recording it the photograph
As Kerr posted below..., posted on September 13, 2009 at 05:04:22
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 your crash and burn is noted.

You seem to miss the fact that the forum is not a medium for private discourse, but one where a very large number of people can read and participate. It is not simply me that you would be "re-educating" if you tried.

And again, I'll try to make this as clear as I can, as I have numerous times with RBNG, PatD, and others: I do not propose that because I find DBTs to be problematic that I support a sighted alternative. It is that because there are other problems, some indeed caused by the DBT protocol for eliminating sighted bias, that I think there need to be vast modifications to any testing procedure before it truly becomes scientifically valid, and thus meaningful.

Too bad you won't play along. Your refusal to engage is a reflection of your arrogance, not your knowledge.

RE: You two seem to be using different definitions of "double-blind.:, posted on September 11, 2009 at 14:55:44
Todd Krieger
Audiophile

Posts: 22046
Location: SW United States
Joined: November 2, 2000
 A simple "blind" comparison isn't even unbiased, because in a lot of cases, the individual might have a "bias" toward the first or second item (even if he's told the administrator doesn't know which is which). Such "bias" may be subconscious expectations, based on what the individual noticed in the first item. Not to mention "memory" of the first item affecting how he perceives the second. (This is an issue with wine tasting.) The double-blind protocol even removes this variable from the evaluation, since each individual is evaluating just one item. (If correctly done, the possibility of a placebo shouldn't even cross his mind, unless he's familiar with how such studies are conducted.)

When I used to take the "Pepsi challenge", I initially tasted both, wanted to choose "Coke" every time, and ended up "guessing 50-50" thinking the second "tasted more like Coke or Pepsi" than the first, and other expectations that "tricked" my senses.

But when I started tasting the first one, "tastes like Pepsi" or "tastes like Coke", and *ignoring* what I tasted with the second drink, I started choosing Coke every time. (The key is developing a long term memory of what the two drinks tasted like, from prior experience.)

An ABX type "blind" test, unlike the DBT protocol, only detects differences. Where the DBT protocol actually refines a possible improved or regressed state, due to the placebo being a reference.


Yep. Forced choice = potential for bias effect., posted on September 11, 2009 at 16:56:28
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 No medical DBT protocol asks a participant to decide which pill is "better," or "different" for that matter.


A double-blind-ish audio comparison is a very fancy parlor trick that cranky audio cynics like RBNG use to put non-existent "Golden Ears" in their place. That's a whole different world from scientifically valid testing for effect.

RE: Yep. Forced choice = potential for bias effect., posted on September 14, 2009 at 08:55:08
mls-stl
Audiophile

Posts: 1621
Location: St. Louis
Joined: April 11, 2006
 >> No medical DBT protocol asks a participant to decide which pill is "better," or "different" for that matter.

That's not correct. While some pills are evaluated by external test results (blood, CT, MRI, etc.), medical test participants are quite frequently asked to report how they feel either mentally or physically and whether it represents a change from before they took the new pill. What they, and the immediate test administrator don't know, is whether the pill they are taking is the new drug, a placebo, or perhaps a continuation of what they were taking before.

The problem, when it comes to human testing in a field like audio, is that there are inherent limitations for a variety of reasons. It is difficult to control for all variables, there are endless combinations of equipment, the cost and time for truly extensive tests is often not worth the effort. Those with the budget and facilities for elaborate tests, such as universities, often don't consider consumer audio worthy of their time.

However, one can still discern a pattern that has emerged over the years. That is the "big" differences talked about under sighted conditions often become much smaller when the listener lacks prior knowledge of make, model or appearance information about the product. The differences may even disappear completely.

There is nothing surprising about that type of result. Audio is just as subject to human vagaries as any other aspect of human perception. The problem is the subjective responses of one person often cannot reliably predict that others will respond in the same manner.

It is interesting that some elaborate excuses have arisen when blind tests don't duplicate the results of sighted ones. The "stress" of having to make an unsighted choice is a popular defense of sighted test right now. But that is an interesting concept since the influence of peer pressure for social conformity is completely ignored on the other end of the scale in spite of it being a well established factor in how people behave in all manner of circumstances.

It also hasn't been well explained how, if one has to make a decision about which of two pieces of equipment is better, how it is less of a "forced choice" if done sighted instead of blind.

And I find interesting, posted on September 14, 2009 at 15:14:23
E-Stat
Audiophile

Posts: 9838
Location: Central boonies
Joined: May 12, 2000
Contributor
  Since: April 5, 2002
 It is interesting that some elaborate excuses have arisen when blind tests don't duplicate the results of sighted ones.

those excuses that arise when null results aren't acknowledged as a lack of any conclusion. I especially like it when the "conclusions" of null results are extrapolated to DNUTs. Devices not under test - or that which didn't even exist at the time of the null result. :)

rw

You're right..., posted on September 14, 2009 at 09:21:42
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 I should have been clearer in my wording, as well as in the type of medical DBT I had in mind. Asking the patient how they feel is indeed part of the protocol, but it not a critical factor in determining the efficacy of a test for a cancer drug, for example. Correlation of patient impression with results is important. There is valuable information to be learned from a patient's own feelings, but the reduction of a tumor, say, is not something determined by asking a patient how they feel. There is an objective means for determining that fact.

"There is nothing surprising about that type of result. Audio is just as subject to human vagaries as any other aspect of human perception. The problem is the subjective responses of one person often cannot reliably predict that others will respond in the same manner." Absolutely agree. That is one of the more significant issues when the purpose of the DBT is left vague (is it proof of an individual's claims, or a general teast for audibility, or...?)

"It is interesting that some elaborate excuses have arisen when blind tests don't duplicate the results of sighted ones. The "stress" of having to make an unsighted choice is a popular defense of sighted test right now. It also hasn't been well explained how, if one has to make a decision about which of two pieces of equipment is better, how it is less of a "forced choice" if done sighted instead of blind."

I'm not sure why that would be. I don't challenge audio DBTs because I favor sighted tests: on the contrary, I agree with the criticisms of sighted listening and the inherent biases they contain. But because an audio DBT can eliminate sighted bias doesn't mean it deals effectiely with any other problem that might exist in the testing. And again, I agree with you that i can't see how a "sighted" test makes the problems of forced choice go away. A test is still a test, sighted or not....
RE: You're right..., posted on September 14, 2009 at 11:22:25
Tony Lauck
Audiophile

Posts: 3048
Location: Vermont
Joined: November 12, 2007
Contributor
  Since: February 24, 2009
 A non-blind test isn't likely to be an identification test. It's likely to be a preference test.

There is going to be less pressure in a preference test than in an identification test, because everybody knows that we all have our preferences and that there is nothing wrong with having a particular preference. On the other hand, nearly everyone would like to believe that their senses are working well and that they can hear as well, if not better than average. Hence an identification test is potentially more stressful, because the answers are really "right" and "wrong". The most stressful kind of identification test is a blind test that follows a successful sighted test, because it adds the further fear that our minds themselves may not work in an unbiased fashion, a much scarier possibility for most people.

Proper experimental design can mitigate these factors, but this requires a deep knowledge of psychology, acoustics, technology and statistics.

Tony Lauck

"Diversity is the law of nature; no two entities in this universe are uniform." - P.R. Sarkar

RE: You're right..., posted on September 14, 2009 at 14:58:17
mls-stl
Audiophile

Posts: 1621
Location: St. Louis
Joined: April 11, 2006
 >> The most stressful kind of identification test is a blind test that
>> follows a successful sighted test, because it adds the further fear that
>> our minds themselves may not work in an unbiased fashion, a much scarier
>> possibility for most people.

That is simply a slightly different manifestation of the peer pressure issue. In sighted comparisons the fear is that a listener will look foolish or inferior in front of his peers if he states he cannot hear what others say they are hearing. One can see that fear has a reasonable basis given the catcalling that takes place regularly in this forum when divergent opinions are posted.

The fear in your blind comparison example comes from the concern that the listener won't be able to live up to their sighted boasts, and once again look foolish in front of their peers.

Fear of looking foolish in front of one's peers is the same factor in both cases. Yet the subjective influence of sighted testing is often dismissed on this forum while the scariness of blind testing is justification to void results.

They may be slightly different flavors, but they are both still ice cream.

>> Proper experimental design can mitigate these factors, but this requires
>> a deep knowledge of psychology, acoustics, technology and statistics.

Granted some tests are better designed than others, but it is still interesting to watch how each group overlooks the deficiencies of tests where results favor them and discount the results the do not match their expectations.

RE: You're right..., posted on September 14, 2009 at 16:49:54
Tony Lauck
Audiophile

Posts: 3048
Location: Vermont
Joined: November 12, 2007
Contributor
  Since: February 24, 2009
 I didn't make myself clear. I was talking about a deeper issue, a more profound mystery than how one behaves in a social situation. Consider one of these identification tests done in the privacy of one's home, and my point should be clear: no peer group, no peer pressure, but often the same puzzling results.

There is a real mystery here, and it's not so much about how the ears work, it's about how the mind makes inferences. This is a matter of epistemology as much as psychology.

Tony Lauck

"Diversity is the law of nature; no two entities in this universe are uniform." - P.R. Sarkar

RE: You're right..., posted on September 14, 2009 at 17:10:05
mls-stl
Audiophile

Posts: 1621
Location: St. Louis
Joined: April 11, 2006
 >> There is a real mystery here, and it's not so much about how the ears
>> work, it's about how the mind makes inferences. This is a matter of
>> epistemology as much as psychology.

Yes, the brain operates in a very complex manner and we will always have more to learn. I just finished the book "The Craving Brain" by Ronald Ruden and it goes into extensive detail about the various levels at which our brain operates from the mesolimbic dopamine system on up the chain. Combine that with Cordelia Fine's "A Mind of Its Own - How Your Brain Distorts and Deceives" and you've got a very interesting insight into just how difficult it is to separate the sound waves coming from our speakers from what our brain does with them. (Comment: neither book is about audio.)

The discussion of peer pressure in the prior post was meant as only one example of subjective factors. It was not a statement or indication that this one issue is the root cause of all subjectivity.

I certainly have a healthy respect for just how complex this is and even the brightest scientist isn't going to be able account for all the variables in play. However, the inability to achieve perfect knowledge from a test is hardly reason to avoid doing it.

My stereo has great days, so-so ones and poor ones. I long ago came to the conclusion that I am a far bigger variable than it is. ;-)
RE: You're right..., posted on September 14, 2009 at 15:36:44
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 "Granted some tests are better designed than others, but it is still interesting to watch how each group overlooks the deficiencies of tests where results favor them and discount the results the do not match their expectations."

What if one isn't in one of the groups? What if there are more than a few who think that both types of test are tragically flawed? What if by stating that one finds a type of test flawed, a fairly large sampling of that test's supporters automatically lump one into the opposite camp?

The difficulties that arise from these "groups" is that both are too arrogant to be self-critical.

RE: You're right..., posted on September 14, 2009 at 17:12:07
mls-stl
Audiophile

Posts: 1621
Location: St. Louis
Joined: April 11, 2006
 >> What if one isn't in one of the groups?

Then you're probably not too far from where I sit. ;-)

Blind vs Sighted, posted on September 14, 2009 at 10:25:36
kerr
Audiophile

Posts: 3695
Location: Central Indiana
Joined: November 10, 2003
 >don't challenge audio DBTs because I favor sighted tests<

I do favor sighted tests but only because they are simpler, and also because there are as many problems with DBT as there are with sighted tests. The problems are just different.

I have done some single blind tests but I perform them in a non-stressful manner. I have an entire evening of listening to decide which component is playing. There's no stress anyway; I either hear a difference or I don't, and I honestly couldn't care less which occurs. But I don't do a lot of these tests, either, because the opportunity to pull it off doesn't exist as easily as the opportunity to A/B components sighted. I've learned to respect both blind and sighted comparisons, especially since my blind experiments tend to confirm what I hear (or don't hear) sighted.

RE: Blind vs Sighted, posted on September 14, 2009 at 12:07:18
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 I have no problem at all with what you are doing with the various types of tests you employ. What has irked me for quite a while with the "debate" on DBTs is that first, the audio DBT is held up by a certain crowd as scientifically validated, whch is highly dubious; second, that the DBT itself is not questioned as to other protocol problems, but nearly always presented as legitimate in opposition to the (implied illegitimate) sighted test; and third, that the point of the test seems to be a moving target (are we testing for general audibility, or that interconnects can be identified to statistical significance, or are we just trying to figure out which amp we prefer?). It is beyond unlikely that an individual checking his own preferences can pul off a scientifically valid test, assuming for a moment that a scientificaly valid audio DBT is even possible....;)

If I *could* have..., posted on September 15, 2009 at 06:15:09
kerr
Audiophile

Posts: 3695
Location: Central Indiana
Joined: November 10, 2003
 ...written your post, I would have! Totally agree on all points.

Or as the link TAH provided terms it..., posted on September 11, 2009 at 16:42:48
E-Stat
Audiophile

Posts: 9838
Location: Central boonies
Joined: May 12, 2000
Contributor
  Since: April 5, 2002
 An ABX type "blind" test, unlike the DBT protocol, only detects differences.

"The test is an "enforce blind A/B comparison"".

Exactly. Not DBT.

rw

For some unknown reason..., posted on September 11, 2009 at 07:54:43
E-Stat
Audiophile

Posts: 9838
Location: Central boonies
Joined: May 12, 2000
Contributor
  Since: April 5, 2002
 I'll enter this particular theatre of the absurd. :) When you actually read your link, here's what you find which is not what you assert:

The test is an "enforce blind A/B comparison"

Read the further description of the objectives which are so totally different from a *real* DBT trial. BTW, what are your results to taking this test? How far are you able to discern the added distortion?

rw

uh....., posted on September 9, 2009 at 04:47:34
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 the operative word was "true."

RE: rather....., posted on September 9, 2009 at 05:32:02
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 the operative word is "recall" because a claim that no "true double-blind study (is) conducted in audio." is idiotic.

Music making the painting, recording it the photograph
Time to back up that statement., posted on September 9, 2009 at 06:16:05
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 I look forward to you providing verifiable information that contradicts Todd's statement. Just one link to one audio DBT that can pass muster. That shouldn't be too difficult, should it?

RE: Not interested in having a pointless discussion, posted on September 9, 2009 at 08:10:56
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 "I look forward to you providing verifiable information that contradicts Todd's statement. "

What a laugh! Not interested in having a pointless discussion, if you consider Todd's statement as having merit that's your lookout.

Music making the painting, recording it the photograph
Just exactly what I thought you'd say..., posted on September 9, 2009 at 08:20:53
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 though it is truly sad. I am more than happy to have you convince me I'm wrong, and as I'm not asking you to use your highly-refined rhetorical skills, but instead am asking you simply to provide factual support for a position, this should not have created such a crisis for you. You see, it is a good thing to be convinced to change one's mind: it means one has learned something new, as opposed to merely honing one's previous argument. I'll continue to give you the chance, but as long as you continue to avoid entering the debate, I'll contiune to find your position on audio DBTs ridiculous and anti-scientific.

"highly-refined rhetorical skills" - that's one of the meanest put-downs I've read here, in a long time., posted on September 21, 2009 at 11:00:21
carcass93
Audiophile

Posts: 2634
Location: NJ
Joined: September 20, 2006
 Here's another pearl for your enjoyment. I seriously thought about making that phrase my permanent signature line.

When you are unable to answer questions and all else fails..., posted on September 9, 2009 at 10:34:03
E-Stat
Audiophile

Posts: 9838
Location: Central boonies
Joined: May 12, 2000
Contributor
  Since: April 5, 2002
 I'm not asking you to use your highly-refined rhetorical skills...

quote a Professor of Rhetoric!

rw

Thanks for the wander down memory lane..., posted on September 9, 2009 at 11:48:03
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 It is bad enough when one is arrogant and right, but arrogant and wrong? I thought that was the exclusive domain of Soundmind, and that 1919191 dude....;)

I have enjoyed a number of your responses, posted on September 9, 2009 at 11:58:28
E-Stat
Audiophile

Posts: 9838
Location: Central boonies
Joined: May 12, 2000
Contributor
  Since: April 5, 2002
 in this thread. In particular, your lament to Tony about the state of "debates" here. And, of course one of your responses to Pat D regarding his er conclusions. :)

rw

RE: I have enjoyed a number of your responses, posted on September 9, 2009 at 15:05:26
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 There's a limit to the enjoyment to be derived from arguing with a brick. I'm glad that we all have different limits to the enjoyment of futility, though, as I'm getting a kick out of reading Todd's responses to TAH....
RE: Thanks for the laugh, posted on September 9, 2009 at 09:02:37
theaudiohobby
Audiophile

Posts: 4438
Joined: January 16, 2003
 "You see, it is a good thing to be convinced to change one's mind: it means one has learned something new, as opposed to merely honing one's previous argument..I'll contiune to find your position on audio DBTs ridiculous and anti-scientific. "

LOL! And round and round it goes, thanks but no thanks.

Music making the painting, recording it the photograph
Another embarrassing cop-out., posted on September 9, 2009 at 10:34:10
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 I'm not even asking for anything difficult. And you can't provide it.

Thanks for clarifying your credibility.

Your crash and burn is noted. Anyone else care to pilot DBT Airlines? (nt), posted on September 9, 2009 at 09:10:08
kerr
Audiophile

Posts: 3695
Location: Central Indiana
Joined: November 10, 2003
 nt

Aren't wine tasting events preference tests?, posted on September 8, 2009 at 12:30:26
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 The audio DBTs discussed here are mostly same/difference tests.

"Probability is the very guide to life."---Cicero
It's a "Blind Taste Test".........., posted on September 9, 2009 at 01:12:21
Todd Krieger
Audiophile

Posts: 22046
Location: SW United States
Joined: November 2, 2000
 And it is indeed preference.............

"The audio DBTs discussed here are mostly same/difference tests."

There is no real DBTs in audio...... A double-blind study would require a "placebo" product that is visually identical to the test product...... And a bunch of people getting either the real product or the placebo, but not both...... So it isn't even "hearing differences" or "preference", but the collective satisfaction of the real product versus the collective satisfaction of the placebo.


so if you've taken care of the biases..., posted on September 8, 2009 at 15:04:51
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 wouldn't one have to be able to discern a difference to form a preference?

A question of terminology?, posted on September 9, 2009 at 10:27:29
Tony Lauck
Audiophile

Posts: 3048
Location: Vermont
Joined: November 12, 2007
Contributor
  Since: February 24, 2009
 "wouldn't one have to be able to discern a difference to form a preference?"

No. Here is a case where I seem to agree with Pat D and not you! (Perhaps the first time?) Or perhaps it is merely a disagreement over terminology. It comes down to what one means by "preference".

If one takes an subjective viewpoint that a preference is something conceptualized and then verbalized, one requires conscious knowledge and one needs to be able to discern a difference before expressing a preference. If one takes an objective viewpoint that preference is expressed by specific actions, it is possible to take an take an action without a difference being consciously discerned. There is a difference between saying "I prefer A to B" on a marketing survey, and laying out one's hard cold cash and buying A instead of B. (I am sure the marketing "scientists" understand this distinction quite well.)

There are many cases in practical life (not just confined to audio) where people make decisions unconsciously. Indeed, most decisions are probably made this way and they are not necessarily "irrational". Human actions are done in real-time under uncertainty and the costs of making a good decision can easily exceed the benefits of making the best decision.

Tony Lauck

"Diversity is the law of nature; no two entities in this universe are uniform." - P.R. Sarkar

Yes, a question of terminology..., posted on September 9, 2009 at 10:45:30
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 "If one takes an subjective viewpoint that a preference is something conceptualized and then verbalized, one requires conscious knowledge and one needs to be able to discern a difference before expressing a preference."

Yep, that's the one.

Thanks for reminding me that one ought always to define terms clearly before proceeding with any kind of debate....


"There are many cases in practical life (not just confined to audio) where people make decisions unconsciously. Indeed, most decisions are probably made this way and they are not necessarily "irrational". Human actions are done in real-time under uncertainty and the costs of making a good decision can easily exceed the benefits of making the best decision."

I agree, but I think (at this point in the formulation of a thesis anyway...) that these "unconscious" or subconscious decisions are bias-related, which is why I qualified the statement by suggesting that bias effects had been eliminated...


"Here is a case where I seem to agree with Pat D and not you! (Perhaps the first time?)"

That's OK, I won't get too distraught...;) Part of what used to be quite valuable and fun about this Forum was that one could actually disagree, have a debate, and learn something. It's OK to have someone else's good argument thrash one's own, as long as one learns something in the process. However, it is more likely now that the a priori battlements are already manned with combatants, and the debate is replaced with a bloodbath of disagreement...

See theaudiohobby's reluctance to engage in debate.
RE: Yes, a question of terminology..., posted on September 9, 2009 at 12:15:53
Tony Lauck
Audiophile

Posts: 3048
Location: Vermont
Joined: November 12, 2007
Contributor
  Since: February 24, 2009
 "these "unconscious" or subconscious decisions are bias-related, which is why I qualified the statement by suggesting that bias effects had been eliminated..."

No problem here. But if one uses the term "bias" one needs to realize that one is making a value judgment that certain factors influencing cognition are inappropriate. It's not just a matter of saying that every sense other than hearing conveys inappropriate information, as is sometimes argued.



Tony Lauck

"Diversity is the law of nature; no two entities in this universe are uniform." - P.R. Sarkar

I know "bias" has value connotations..., posted on September 9, 2009 at 15:16:28
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 but does it really, in regard to testing? A bias can be something "for" or "against," and can certainly be subconsciious. But I do agree with your last sentence. To me, one of the important factors that allows me to perceive the world is my subconscious bias. Listening to music with all other senses turned off (the elimination of a large percentage of potential bias) is not actually listening. The disengagement of senses eliminates one of the more profound aspects of "hearing..." (that the senses are very strongly linked).
RE: so if you've taken care of the biases..., posted on September 8, 2009 at 21:16:42
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 Well, no. As Sean Olive has shown, people's preferences are likely to be different under blind conditions than under sighted conditions, when other factors than the sound influence judgment. Under sighted conditions, one can form a preference even though the choices sound the same.

But you miss the point. There is no guarantee that a preference determined in one test will be the same as that found in another test. Simply being able to discern the difference under blind conditions does not guarantee a constant preference. The subject's mood may change between tests, for example, or other variables may have changed (i.e., the speakers may be different, and one might prefer one amplifier with Speaker A and another amplifier with speaker B--or of course, the DBT may not show that the person can hear the difference).

So, pointing to variable results in a preference test of wine does not at all mean that same/difference audio DBTs are not good. The thread is based on an apples and oranges comparison.


"Probability is the very guide to life."---Cicero
Did you not understand the subject line?, posted on September 9, 2009 at 04:22:49
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 "Well, no. As Sean Olive has shown, people's preferences are likely to be different under blind conditions than under sighted condition, when other factors than the sound influence judgment. Under sighted conditions, one can form a preference even though the choices sound the same."

Well, you missed it completely, Pat. Did you understand the meaning of my post? Clearly not. Let me try again, with some added emphasis through capitalization, so you can get it: "So IF you've taken care of the biases, wouldn't one have to be able to discern a difference to form a preference? " (note the order: first, "difference," then "preference." A preference is not necessary for there to be a difference, but there must be a difference for there to be a preference. The statement was not a universal affirmative: the syllogism should have been clear.)

I've already addressed your "blind conditions than under sighted conditions" problem by the first part of the sentence. See it there? The part that states, "so if you've taken care of the biases...?" Nothing was said about "sighted conditions." You've tried to hide a red herring in my pot of lobsters.


"But you miss the point. "

No, not at all, but you clearly did.


"There is no guarantee that a preference determined in one test will be the same as that found in another test. Simply being able to discern the difference under blind conditions does not guarantee a constant preference."

Yes, but as my post to which you responded made clear, IF one has a preference, AND the biases that a DBT supposedly eliminates have been addressed, THEN the formation of a preference is the result of identifying a difference (or certainly THINKING one has). So what could be an alternative explanation, if one can form a preference without bias effects in play, and without identifying a difference? Could it be that the human brain is so unreliable from moment to moment that it cannot distinguish difference or sameness reliably? Wow. If that is the case, it would make a DBT kind of useless, wouldn't it? This possibility is even supported by RBNG's oft-repeated mantra that audiophiles identify a difference 75% of the time when the a component is compared with itself.


"The subject's mood may change between tests, for example, or other variables may have changed (i.e., the speakers may be different, and one might prefer one amplifier with Speaker A and another amplifier with speaker B--or of course, the DBT may not show that the person can hear the difference)."

Very good, Pat. You just described a few of the reasons why an audio DBT is as worthless as a parlor trick.


"So, pointing to variable results in a preference test of wine does not at all mean that same/difference audio DBTs are not good. The thread is based on an apples and oranges comparison"

You provided NOTHING that supports your conclusion. Sighted biases are irrelevant, because the tests are...well, BLIND. And the subject's own inconsistencies? Those occur all the time, in every circumstance. A more appropriate criticism of "apples to oranges" applies to comparing medical DBTs to audio DBTs. In audio, there is no control, no placebo, and the test subject is asked to act as test equipment and interpreter of stimuli simultaneously. Hardly acceptable scientific protocol.







"

Yes, and I didn't disagree with it., posted on September 9, 2009 at 10:21:32
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 ""So IF you've taken care of the biases, wouldn't one have to be able to discern a difference to form a preference?"

I don't disagree with that if it's a blind test with sufficient trials. What makes you think I do?

But during a blind test, one may easily perceive the sound to be better in any given trial. Thus, one may perceive that the present trial is better than in the previous trial whether the choice is actually different or not. That's the reason for doing a number of trials and figuring the probabilities.

"Probability is the very guide to life."---Cicero
RE: Yes, and I didn't disagree with it., posted on September 9, 2009 at 10:27:53
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 "I don't disagree with that if it's a blind test with sufficient trials. What makes you think I do?"

How about your first line: "Well, no." followed by this gem: "But you miss the point," ending with, "The thread is based on an apples and oranges comparison."

Those quotes from yuor reply might lead one to believe that you disagreed. I'm pleased to learn that your typical obfuscation tactics extend into the realm of agreements. ;)
If you would keep track of the actual question you asked . . ., posted on September 10, 2009 at 07:48:57
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 not one you may have thought you asked, you could have understood my answer.

"Probability is the very guide to life."---Cicero
Pathetic, Pat. As I repeated it to you twice..., posted on September 10, 2009 at 07:55:53
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 it can't hurt to do it again: "So IF you've taken care of the biases, wouldn't one have to be able to discern a difference to form a preference?"

That was the first question, which you not-so-skillfully tried to counter. Then you followed with this beaut: "I don't disagree with that if it's a blind test with sufficient trials. What makes you think I do?" I answered that one too. Your response? That I can't keep track of my questions??!!??

Please try to pay attention, and stop wasting eeryone's time.

It just wasn't quite the simple answer you wanted., posted on September 10, 2009 at 11:05:43
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 I didn't try to counter it, I did. As a general proposition, the answer is "no" but I noted it was not entirely incorrect.

"Probability is the very guide to life."---Cicero
You just keep telling yourself that, Pat., posted on September 10, 2009 at 11:59:33
robert young
Audiophile

Posts: 2226
Location: new york
Joined: October 19, 2003
Contributor
  Since: July 13, 2004
 It's remarkable to what extent some people will go to keep the fantasy alive.

Maybe you should try wine same/difference tests and let us know how it goes..., posted on September 8, 2009 at 14:42:38
mkuller
Audiophile

Posts: 15929
Location: SF Bay Area
Joined: April 22, 2003
Contributor
  Since: December 28, 2003
 ...confusion would be my guess.

Blind testing does have many drawbacks...

Yes indeed! Blind testing does have drawbacks for subjective audio reviewers!, posted on September 8, 2009 at 20:56:46
Pat D
Audiophile

Posts: 7558
Joined: June 20, 2000
 They likely would not be able to identify the sound of a lot of equipment under blind conditions.

Their reviews of many pieces of electronics, interconnects, speaker cables and many tweaks would be shown to be unreliable.



"Probability is the very guide to life."---Cicero
LOL! It's pseudoscience and has drawbacks for everybody...(nt), posted on September 9, 2009 at 09:41:30
mkuller
Audiophile

Posts: 15929
Location: SF Bay Area
Joined: April 22, 2003
Contributor
  Since: December 28, 2003
 (nt)

RE: L.A. Times article concerning blind wine taste tests., posted on September 8, 2009 at 10:48:51
Todd Krieger
Audiophile

Posts: 22046
Location: SW United States
Joined: November 2, 2000
 "My experience in Sonoma County is that some of the most god-awful wines tend to win medals and top honors in the local Harvest Fair competition, which carefully follows a blind testing protocol."

So you're suggesting that those who rely on blind testing in audio end up choosing awful audio equipment?? .......

Jeez, no wonder why these people are so resentful..........


At a local Supermarket, posted on September 8, 2009 at 04:25:30
bjh
Audiophile

Posts: 11876
Location: Toronto, Canada
Joined: November 22, 2003
 the wine store was doing a testing test - same wine in three different package options (one was bottled).

When I approaced I witness the lady pouring the third sample so we decided I would do the test with just two.

I was able to distinguish and the one I identified as best was the bottle.

Then she suggested I do the full test. Again the one I identified as best was the bottle and of the other two the favored was in the more expensive packaging option.

The 3 sample test was more difficult. I found I had to repeatedly sample the choices. While I don't remember the exact details I do recall that sampling in certain order helped me notice 'nasties' (metalic after-taste, etc.) identifying the sample without (or diminished) as "better".

I like wine (reds) but I'm hardly a connoisseur.
Everything matters, don't forget to tweak your placebos!
At a regional wine shop,, posted on September 8, 2009 at 15:20:21
Al Sekela
Audiophile

Posts: 8279
Location: Northern California
Joined: February 18, 2002
 I tasted some local Zinfandel wines, including one I had found very good in a previous sampling. This was not a blind test.

All the wines tasted bad that day, and the one I recalled as being good tasted like it had been cooked.

I mentioned this to the proprietor and found the assistant had left the wine bottles on the window sill in the sun to 'warm them to room temperature.'

I'm sure your experience resulted in a clear preference, but it could have been the result of how the samples had been treated as well as the style of packaging.

On a related note, avoid any wine in clear bottles that are exposed to fluorescent light. Some stores have chilled wine in display cases illuminated by fluorescent lamps. The UV in the light kills the flavors quickly.

Big surprise..., posted on September 7, 2009 at 22:21:56
Ugly
Audiophile

Posts: 1420
Location: Des Moines, WA
Joined: August 22, 2006
 judges have tastes that may vary over time and can be different than others tastes. The real surprise for me is that anybody would be surprised by this, much less doubt it.

RE: L.A. Times article concerning blind wine taste tests., posted on September 7, 2009 at 17:44:05
caspian@peak.org
Audiophile

Posts: 386
Location: Oregon
Joined: January 12, 2008
 A teetotaling defender of ABX double-blind testing might construe this as proof that all wines (or at least all Pinots, all Rieslings, all whatevers) taste the same!

A meterhead like myself might argue that it shows up the shortcomings of DBT, which is fraught with unreliable human variables, and suggest that the only proper way to test wines is by chemical analysis showing the exact proportions of various sugars, tannins, enzymes, terpenes, bioflavonids, etc. A chemist who was also a wine enthusiast, and had a great deal of experience both analyzing and tasting a wide variety of wines (no more than one a day), could probably draw reasonably accurate inferences about the flavor of a given wine just by looking at its chemical profile.

RE: L.A. Times article concerning blind wine taste tests., posted on September 7, 2009 at 18:40:19
manor999
Audiophile

Posts: 87
Joined: September 24, 2001
 this has been done and there is no direct correlation between the % of the phenolic compounds,and there are hundreds,and the tastes.for one thing the permutations are too vast.plus the fact that all the various chemicals in wine have not even been identified.
if this could be done ,it already would have.stick to audio,it is much less intricate.
jim buck
RE: L.A. Times article concerning blind wine taste tests., posted on September 9, 2009 at 13:05:56
caspian@peak.org
Audiophile

Posts: 386
Location: Oregon
Joined: January 12, 2008
 "the permutations are too vast.plus the fact that all the various chemicals in wine have not even been identified."

Funny, that's pretty much what a lot of subjectivists say about sound quality. That we don't yet know how to measure everything we hear, and can't necessarily correlate all the measurements we CAN make to what we DO hear.

RE: L.A. Times article concerning blind wine taste tests., posted on September 7, 2009 at 22:07:41
Ugly
Audiophile

Posts: 1420
Location: Des Moines, WA
Joined: August 22, 2006
 That and a tasters taste will probably vary through time.

Wonder if Jerry is related to Julian..., posted on September 7, 2009 at 15:25:51
mkuller
Audiophile

Posts: 15929
Location: SF Bay Area
Joined: April 22, 2003
Contributor
  Since: December 28, 2003
 ...while blind tasting and listening may eliminate certain biases, it introduces many more problems.

Page processed in 0.612 seconds.

[ Top Top of Page ] [ Contact Us ] [ Support/Wish List ] [ Copyright © 2009, Audio Asylum, all rights reserved ] [ Go Green ] [ Take5 ]
[ General ] [ Speakers ] [ Tubes ] [ Vinyl ] [ Digital ] [ Hi-Rez ] [ Video Asylum ] [ Cables ] [ Tweaks/DIY ] [ Music ] [ Films ]