|
Critic's Corner Discuss a review. Provide constructive feedback. Talk to the industry. |
For Sale Ads |
Use this form to submit comments directly to the Asylum moderators for this forum. We're particularly interested in truly outstanding posts that might be added to our FAQs.You may also use this form to provide feedback or to call attention to messages that may be in violation of our content rules.
Original Message
Blind testing is a hypothesis test and has the same mathematical rules
Posted by Timbo in Oz on November 8, 2016 at 19:02:17:
If you set the confidence limit too low / don't have enough trials /subjects aka 'n', your probability of type 2 error (beta) can exceed 50%. Which is into bull-shit territory.
The double blindness in ABX DB testing is NOT comparable to the DB nature of medical hypothesis testing. It is ignorant to suggest that it is. And ignorant to suggest that blindness is the scientific aspect of testing. Proper and open use of statistics is where 'proofs' lie.
Far too many ABX DB tests that get published don't publish their alpha OR beta figures. It ought to be essential.
I wouldn't bother publishing or promoting any ABX DBT results where beta was significantly larger than alpha. None of us should tolerate it from the hard line objectivists.
For me, any blind home-audio testing should involve a single seated listener in the sweet spot, in a treated room. Not a room-full of people.
Thus, getting enough n is a problem, and yet it is the only way to get alpha and beta down while having a reasonable confidence limit. 10% / 90% is a LOT more appropriate most of the time that 5%/95 was used.
Bear in mind that unwillingness to post the two salient numbers - of alpha and beta - is rife in published hypothesis test science, of all kinds.
Where the science is an estimate of the value of a variable, the question of what sort of estimator they chose to use is very rarely discussed. This is not surprising because everyone seems to think that an unbiassed estimator is the best in all cases.
But, it rarely is appropriate. It is only appropriate if errors above and below the mean are of equal concern to the people likely to be affected by the use of the estimated value.
Science? Schmience!
Having an abiding respect for real, solid science it worries me that the prevailing 'scient-ism' about so much of 'science' might not be justifiable, because of these, to me, manifest flaws. You know, those "fuck I LOVE Science" threads on social media. ? yes?
Value judgements just do have to be made, when we are using statistical techniques, and using the usual settings isn't 'objective' it is simply ignorant, perhaps even arrogant.