Critic's Corner

Discuss a review. Provide constructive feedback. Talk to the industry.

Return to Critic's Corner


Message Sort: Post Order or Asylum Reverse Threaded

Blind Testing

173.52.245.236

Posted on October 28, 2016 at 04:13:08
John Atkinson
Reviewer

Posts: 4045
Location: New York
Joined: November 24, 2003
In an archived thread, Doug Schneider wrote:

>As I wrote to John [Atkinson] on a thread that's presumably way, way
>down...I also noted that in his pseudo-blind test he said he identified
>MQA 5 out of 8 times.

Just to clarify, I scored 4 our of 7 correct in this blind test (see link
below). But as I later explained, 2 out of the 3 I got wrong were the same
track, Steely Dan's "Babylon Sisters," where I misidentified it as what I
thought was the better-sounding MQA version. Specifically, I preferred the
quality of the bass on what turned out to the original hi-rez file.

If you eliminate those 2 failures, I got 4 out of 5 correct in blind
comparisons on the other tracks, which, while still not statistically
significant, is relevant information, I feel.

John Atkinson
Technical Editor, Stereophile

 

Hide full thread outline!
    ...
RE: Blind Testing, posted on October 28, 2016 at 04:33:32
John Atkinson
Reviewer

Posts: 4045
Location: New York
Joined: November 24, 2003
Just to address another statement made in the MQA thread,: Doug Schneider
wrote that:
>There was a lot of misinformation about MQA files being the same size as
>CD-resolution files. They're not. CD resolution is 16-bit/44.1kHz, while
>MQA is 24/44.1 or 24/48. At minimum, they're 50% larger.

I compared the file sizes with one of my recordings in the article linked below.
From that article:

"The 24/88.2 WAV master file for my recording of the Portland State
Chamber Choir's 2014 performance of "Amazing Grace" is 169.5MB. The
MQA-encoded FLAC version is just 51.5MB: 30% of the original size. For
reference, the 24/88.2 Apple Lossless version is 90.7MB, or 53.3% of the
original, and the version on the CD, limited to 16-bit/44.1kHz, is 55.7MB,
or 33%. MQA, therefore, gives the greatest reduction in file size and
thus necessary streaming bandwidth..."

The MQA FLAC file is 51.8MB, the CD is 55.7MB. It is fair to point out
that the CD file is uncompressed and that a lossless compression codec
like FLAC would reduce the file size to around 30MB. But I don't believe
it was that comparison that Doug was referring to in the quoted text. If
it was, then it wasn't clearly expressed.


John Atkinson
Technical Editor, Stereophile

 

My Mistake..., posted on October 28, 2016 at 05:11:31
Doug Schneider
Reviewer

Posts: 881
Location: North America
Joined: April 16, 2005
Hi John,

Oops, my mistake on those numbers. I'd known it was 7, actually, because at the time read it, I was hoping you were going to go for 8.

For the record, I applaud your effort for doing this. In fact, I suspect you did that because you know that's what me done -- compare.

Just so people know, Bob Stuart contacted me by phone just before my original MQA article was set to run. At that time, I made the offer to send him material produced by one of Canada's most prolific and most respected recording and mastering engineers -- material that goes all the way back to the '80s with those early digital recorders. He had the material set. Bob had also mentioned to me that he was making a trip to Canada and I even offered to bring him to his studio, which is in Toronto. Bob didn't appear interested on the phone, and a follow-up email to wasn't replied to. So it never happened.

These lack of proper comparisons are not only suspect, but also noticed by many -- you can read the responses below and, when I talk to many digital designers, they say the exact same thing. Like politicians' claims that have no substance or backing, manufacturers that do the same thing have stand the very strong chance of being doubted.

So good for at least doing that. Your results certainly don't convince me, but that's at least what has to be done.

Doug Schneider
SoundStage!

 

I think I'm the source of the "misinformation" :), posted on October 28, 2016 at 07:09:09
E-Stat
Audiophile

Posts: 37470
Joined: May 12, 2000
Contributor
  Since:
April 5, 2002
where I once summarized that the claim for MQA was that it could pack all the goodness of 24/96 into a 16/44 wrapper. Indeed, it remains 24 bits and the amount of space that implies.

I sit corrected.

 

RE: My Mistake..., posted on October 28, 2016 at 07:16:01
Even though it has been pointed out to you numerous times that "proper comparisons" have been done, you continue with your story. Interesting.

I'm curious Doug - exactly what are suggesting when you say that the comparisons that have been done were not "proper"?

Are you saying that MQA is purposefully misleading people?

That recording engineers who have compared their own music pre- and post-MQA have somehow been mislead?

That every equipment manufacturer and record label who has or is in the process of implementing MQA are doing so based on ignorance of the outcome?

I ask these questions because it appears as if your real argument is simply the fact that *you* have not had an opportunity to compare MQA to non MQA music. Therefore, every person who has is somehow suspect. Do I have that right, Doug?

 

RE: My Mistake..., posted on October 28, 2016 at 07:20:37
Isaak J. Garvey
Industry Professional

Posts: 1207
Location: Hollywod, CA
Joined: January 7, 2016
Perhaps you should read the Roon forum. There, representatives of both Roon and Signalyst (HQ Player) clearly state their opinions that MQWA/Meridian's first concern is commerce, regardless of inconvenience to consumers. It is all there in black and white. Lots of interesting discussion there.

 

RE: My Mistake..., posted on October 28, 2016 at 07:22:58
I typically do not read forums. I find it to be a waste of time and rarely does it have any relevance to what I do.

I decided to comment here to try to staunch the flow of BS and lies about me, mainly perpetrated by you, Isaak.

 

RE: My Mistake..., posted on October 28, 2016 at 07:28:05
Isaak J. Garvey
Industry Professional

Posts: 1207
Location: Hollywod, CA
Joined: January 7, 2016
The Roon forum is a wealth of information. With direct participation of the principals of Roon and other important hardware and software developers. I have found easy and quick solutions to little issues I have had with Roon.

Are you delusional? Are you having paranoid fantasies about people "spreading lies and BS" about you? Perhaps you need to seek help?

 

Which is the reason MQA only works for files and not CD-type discs, posted on October 28, 2016 at 07:28:19
John Marks
Manufacturer

Posts: 7775
Location: Peoples' Democratic Republic of R.I.
Joined: April 23, 2000
It would have been nice were MQA backward-compatible with CD transports, but...

That is not the case.

That said, I have wondered whether one could...

1) Take an early digital recording such as Glenn Gould's early-1980s Goldberg Variations remake

2) Process it with the front end of the MQA system, that is, correct the file's digital timing errors

3) Sample Rate Convert the corrected file and press a CD from that?

Obviously, the question is "Why?"

The answer is, "Perhaps there are a few recordings where there is pent-up demand for the best possible physical medium version that is not an LP or an SACD."

Demand, but not among audiophiles or computer geeks.

On the other hand, I could pick out a Desert Island recordings list that did not have one recording that was originally digital: Court and Spark, Sleeping Gypsy, Aja, Avalon, Famous Blue Raincoat, etc.

ATB,

John

 

RE: My Mistake..., posted on October 28, 2016 at 07:30:35
Nice try, Isaak. Do you really think your childish attempts at diversion work?

Your silly misleading "pimping" post being the most recent example of your BS.

 

RE: My Mistake..., posted on October 28, 2016 at 07:31:59
Isaak J. Garvey
Industry Professional

Posts: 1207
Location: Hollywod, CA
Joined: January 7, 2016
Very informative post Doug.

I asked john to post samples of his pre and post MQA files for download and he declined. Not the full tracks mind you, but 30 second samples, as Michael Fremer does when he tests ADCs, carts, and phono stages.

 

RE: Which is the reason MQA only works for files and not CD-type discs, posted on October 28, 2016 at 07:45:40
John Atkinson
Reviewer

Posts: 4045
Location: New York
Joined: November 24, 2003
>I have wondered whether one could...
>1) Take an early digital recording such as Glenn Gould's early-1980s
>Goldberg Variations remake
>2) Process it with the front end of the MQA system, that is, correct the
>file's digital timing errors
>3) Sample Rate Convert the corrected file and press a CD from that?

I discussed this possibility with Bob Stuart at the recent RMAF. Whether
any additional resolution could be packed into the LSBs of the 16-bit CD
container would depend on the level of the recording's analog noisefloor.

John Atkinson
Editor, Stereophile


John Atkinson
Technical Editor, Stereophile

 

I typically do not read forums, posted on October 28, 2016 at 07:52:10
fmak
Audiophile

Posts: 13158
Location: Kent
Joined: June 1, 2002
Why should anyone read his?

For a reviewer to say this is to admit blatantly that he/she does not do sufficient research before writing pieces/posts giving unqualified support to a new product combination which others cannot readily access or judge for themselves.

As I pointed out, CA takes a much more balanced view, as do posters in other forums.

 

RE: I typically do not read forums, posted on October 28, 2016 at 07:56:01
"For a reviewer to say this is to admit blatantly that he/she does not do sufficient research before writing pieces/posts giving unqualified support to a new product combination which others cannot readily access or judge for themselves."

You think reading posts like yours is research? I write about audio, not psychology.

 

Comparisons/Marketing/Etc., posted on October 28, 2016 at 08:13:48
Doug Schneider
Reviewer

Posts: 881
Location: North America
Joined: April 16, 2005
Hi Michael,

I don't think I've ever read an article on your site, but you've taken quite a defensive position on this. At SoundStage!, I believe if our writers are going to put their opinions out there, they better have thick-enough skin.

As far as your words like "misleading" go, when a manufacturer publishes specifications or claims, do you automatically believe them? When a loudspeaker manufacturer says a bookshelf speaker has bass below 30Hz and a sensitivity of 90dB, do you buy that? Companies try to sell products -- let's leave it at that. But when a company claims "lossless" (which, in the world of computer files, has a pretty strict meaning), but takes a file that's, say, 24/192 and makes it 24/44.1 but gives no way to verify if you can take it back again, doesn't that raise your eyebrows? Maybe not...

Insofar as comparisons go, I believe JA is knowledgeable enough to know that, when claims like the ones MQA are making, comparisons are warranted. And what he did I'd call a start -- and what should have been done LONG ago, before guys like me (and many others) got on the Internet and said, "Hey, where are the comparisons to back up all these claims?" Obviously, guys like us don't just buy in...

As for the kinds of comparisons I'd like to see:

-- Varying file resolutions from 16/44.1 up to whatever.

-- Varying file vintages, meaning the years they were made -- key, I think.

-- Comparisons of varying resolutions of the same file to whatever MQA creates.

-- Ideally, I'd like to see the conversion to ensure any other remastering isn't being done beyond the process itself. Engineers know that a dB difference here or there can change sound drastically.

I could go on, but my cohort in Toronto was ready to supply recordings that go back to the '80s, digitally recorded at the bit depths and frequencies popular back then, right up to modern high-resolution recordings done at 192-384kHz. The whole nine yards, and see what differences there are, and, ultimately, which is better. These were recordings he would've done himself, so, like in JA's case, a good knowledge of what it's supposed to sound like, not like what you think it sounds like.

Is that really asking too much?

Doug

 

RE: Which is the reason MQA only works for files and not CD-type discs, posted on October 28, 2016 at 08:19:53
Doug Schneider
Reviewer

Posts: 881
Location: North America
Joined: April 16, 2005
Hi John,

I have to wonder: Why do you want it compatible with CD transports? I'm having trouble understanding the rationale for that.

Doug

 

I write about audio, not psychology, posted on October 28, 2016 at 08:20:42
fmak
Audiophile

Posts: 13158
Location: Kent
Joined: June 1, 2002
Now we are really off the rail.

Egocentricity embodied by this rush of posts of little meaning says quite a lot.

 

Well said, posted on October 28, 2016 at 08:21:36
fmak
Audiophile

Posts: 13158
Location: Kent
Joined: June 1, 2002
as do the posts in CA.

 

RE: Which is the reason MQA only works for files and not CD-type discs, posted on October 28, 2016 at 08:23:56
Doug Schneider
Reviewer

Posts: 881
Location: North America
Joined: April 16, 2005
Hi John,

Please don't take this as an attack or anything like that. I see this forum as an open discussion.

In terms of what is mentioned about the noise floor, I think this is where MQA's definition of "lossless" and the computer world's term for it differs. And I'd argue that if someone is going to use the terms "lossless" and "compression," they should be using the computer world's.

In computers, lossless compression is, well, a ZIP file or similar. Take the file, compress it, then uncompress it and bring it all back again.

When I hear things like it depends on the noise floor, I think they're using it as "analog recording lossless." So, for example, if 15 bits are used, then 9 bits (out of 24) are just noise, then if we only retain the 15 bits, say, we're "lossless" because the remaining 9 didn't matter anyway.

Do you agree with how I'm thinking they mean that MQA is lossless?

Doug

 

RE: Which is the reason MQA only works for files and not CD-type discs, posted on October 28, 2016 at 08:28:10
John Atkinson
Reviewer

Posts: 4045
Location: New York
Joined: November 24, 2003
>Do you agree with how I'm thinking they mean that MQA is lossless?

No.

John Atkinson
Technical Editor, Stereophile

 

RE: Which is the reason MQA only works for files and not CD-type discs, posted on October 28, 2016 at 08:32:17
Doug Schneider
Reviewer

Posts: 881
Location: North America
Joined: April 16, 2005
So do you think they mean lossless in the computer-world sense? You can take the original, say, 24/192 or higher file, MQA compress it, then uncompress that same file and get it all back again? Like millions do with ZIP files daily? (Or, in the music-file world, FLAC or ALAC or similar?)

Doug

 

Why would you eliminate those two?, posted on October 28, 2016 at 08:43:12
mkuller
Audiophile

Posts: 38130
Location: SF Bay Area
Joined: April 22, 2003
...they seem as relevant as the others.

 

RE: Comparisons/Marketing/Etc., posted on October 28, 2016 at 08:46:06
Well Doug, when I make a claim, especially one that reflects on colleagues, I make it a point to know what I'm talking about.

You haven't read my MQA review, yet you have an opinion on my stance re. MQA. That's some shoddy opinion-making.

You continue to make claims about a lack of comparisons that have been done. You don't know about them because you have not bothered to do the research.

My skin is just thick enough to still know when someone is misinformed. The fact that I point this out typically unearths other people's thin skin.

 

I've found, posted on October 28, 2016 at 09:16:36
that I can occasionally get useful tips from a forum, like how to repair a leaky faucet or get Roon to behave as it should. But the notion that reading forums constitutes "research" is preposterous, unless I suppose its public-opinion research.

 

RE: Comparisons/Marketing/Etc., posted on October 28, 2016 at 09:19:42
Doug Schneider
Reviewer

Posts: 881
Location: North America
Joined: April 16, 2005
Hey Michael,

Well, it's obvious that you don't like when people do respond to your questions -- you choose to ignore that.

So let's just call it a day on this thread... again.

Doug Schneider
SoundStage!

 

RE: My Mistake..., posted on October 28, 2016 at 09:57:21
Mercman
Audiophile

Posts: 6580
Location: So. CA
Joined: October 20, 2002
Isaak, you have been baiting Michael with imagined stuff recently and can't seem to stop. Who is in need of help?

 

RE: Why would you eliminate those two?, posted on October 28, 2016 at 09:58:01
John Atkinson
Reviewer

Posts: 4045
Location: New York
Joined: November 24, 2003
I didn't eliminate them for the result that was published in the magazine.
But I am conjecturing that perhaps I should have because I was keying
my blind identification off of an audible difference but incorrectly
ascribing that difference to what I believed to be the better-sounding
version of the file.

This confusion of "difference" with "preference" often rears its head with
blind testing. See my discussion at the linked article below.


John Atkinson
Technical Editor, Stereophile

 

RE: My Mistake..., posted on October 28, 2016 at 10:00:14
Isaak J. Garvey
Industry Professional

Posts: 1207
Location: Hollywod, CA
Joined: January 7, 2016
Baiting? Give me a break..is this pre school?

Did you, or did you NOT claim your system was "hacked"? You clearly did.

 

RE: My Mistake..., posted on October 28, 2016 at 10:01:08
Mercman
Audiophile

Posts: 6580
Location: So. CA
Joined: October 20, 2002
My system has never been hacked whatever that means. I'll ask Lucy.

 

RE: My Mistake..., posted on October 28, 2016 at 10:02:51
Isaak J. Garvey
Industry Professional

Posts: 1207
Location: Hollywod, CA
Joined: January 7, 2016
You should be ashamed. You know what you claimed. I never took you for a lap dog.

 

RE: My Mistake..., posted on October 28, 2016 at 10:04:05
Mercman
Audiophile

Posts: 6580
Location: So. CA
Joined: October 20, 2002
Isaak, what are you talking about?

 

RE: Comparisons/Marketing/Etc., posted on October 28, 2016 at 11:11:54
ahendler
Audiophile

Posts: 5151
Location: San Antonio, Texas
Joined: January 24, 2003
I would like to report on something our club did about 4 months ago. It is the River City Audio Society in San Antonio Texas. I purchased a Meridian Explorer2 with MQA decoding. I then went to The 2L Audio testbench and downloaded 12 tracks both in 16/44 and MQA. Both are derived from the same DXD master.. At our meeting we had about 20 people and set up a system with B&W speakers and Audio Research tube integrated amps. The files were played from JRiver 21 on a Macbook Pro. I simply would play the 16/44 file followed by the MQA file and then asked for opinions. After all the tracks were played the results were as follow. 5 out of 20 felt they heard differences but they were small . 15 people said they heard no difference in any of the comparisons. Of the 5 who heard something the term natural came up the most. I do want to say that the 2L files are so good it is hard to hear differences among any of the different different formats they offer. I am still waiting for streaming so I have a lot more music available to listen to.
Alan

 

Great Story -- But Small Differences Again..., posted on October 28, 2016 at 11:56:43
Doug Schneider
Reviewer

Posts: 881
Location: North America
Joined: April 16, 2005
Hi,

This is a great story you told. That's also a pretty respectable sample size.

What's interesting to me is that 3/4 found no difference, 1/4 found a tiny difference. What's missing is if the 1/4 found the MQA better, or if they found the other better. Can you elaborate?

What's especially intriguing about this is that a portion of the press has thrown around some pretty big phrases, such as "game changer" and "beyond high resolution." But here we have JA's differences being not reliable and 3/4 of the people here hearing no difference, with 1/4 hearing a tiny difference. That doesn't sound like a "game changer" to me. I also can't help but wonder if that's why MQA doesn't like comparisons. In Munich, I asked for a comparison -- and didn't receive.

Doug Schneider
SoundStage!

 

RE: Great Story -- But Small Differences Again..., posted on October 28, 2016 at 12:02:43
Isaak J. Garvey
Industry Professional

Posts: 1207
Location: Hollywod, CA
Joined: January 7, 2016
Doug:

This has been my concern as well. The only group that consistently seems to be utterly knocked out by MQA are a relatively small group of audio writers..maybe a dozen?

I have read about similar group tests like Alans, as many as 10 others written about on regional audiophile society pages and other forums like Audiocircle, and NONE, not one, had the same reactions as the press. Yes, some heard a difference, some preferred it, some did not. But the huge difference in reaction between ordinary listeners and reviewers is just too large to ignore.

Not once did I encounter the word "stunning", or a phrase like a "new era in digital".

 

I wonder how much Robert Harley was paid?, posted on October 28, 2016 at 12:11:03
hawkmoon
Audiophile

Posts: 903
Location: cleveland
Joined: July 11, 2003
Nt

 

RE: Great Story -- But Small Differences Again..., posted on October 28, 2016 at 12:12:45
Doug Schneider
Reviewer

Posts: 881
Location: North America
Joined: April 16, 2005
I'm not sure it's a dozen!

As far as I can tell, there are two aspects to MQA:

1) Compression -- this is useful for folks with bandwidth issues. I'm not one of them (heck, my generous cellphone plan now follows me around the world), but for those who are, knock your socks off and use it.

2) Sound Improvement -- this is all that interests me. This supposed "time-smearing" fix.

Out of those two things we have gotten from -- I'd argue, less than a dozen -- absolute raves that this encoding scheme is the Next Big Thing. But the company won't do comparisons, and when they are done, they're small to nonexistent. It makes me wonder, too.

Doug

 

RE: Great Story -- But Small Differences Again..., posted on October 28, 2016 at 12:16:59
Isaak J. Garvey
Industry Professional

Posts: 1207
Location: Hollywod, CA
Joined: January 7, 2016
Well..Robert Harley, John Darko, Lavorgna, Jim Austin, John Atkinson, and little tid bits from show reports from a few others. So yeh, maybe you are right. I think Chris Connaker has beeen mostly neutral leaning slightly positive when he was sent a Meridian DAC and files.

Darko is interesting because he used his usual technique, feigning tremendous skepticism, and then having a "conversion."

 

The biggest problem with MQA is .., posted on October 28, 2016 at 12:42:48
Posts: 2763
Location: Orange Co., Ca
Joined: September 19, 2001
it is too clever for most to understand!

13DoW

 

But isn't that an issue with any blind listening test? (nt), posted on October 28, 2016 at 13:40:46
mkuller
Audiophile

Posts: 38130
Location: SF Bay Area
Joined: April 22, 2003
(nt)

 

RE: Comparisons/Marketing/Etc., posted on October 28, 2016 at 13:44:09
Hey Doug,

It's obvious that you cannot accept the fact that you do not know what you are talking about and when this is pointed out, you continue talking without knowing what you are talking about.

This seems like a good place to call it a day.

 

RE: Comparisons/Marketing/Etc., posted on October 28, 2016 at 14:19:39
Doug Schneider
Reviewer

Posts: 881
Location: North America
Joined: April 16, 2005
Sure Michael, carry on...

Doug

 

RE: Great Story -- But Small Differences Again..., posted on October 28, 2016 at 14:19:55
ahendler
Audiophile

Posts: 5151
Location: San Antonio, Texas
Joined: January 24, 2003
I stressed that I wanted participants to tell if they could hear a difference. 5 said yes. In a followup I asked if they could describe what the difference was they were hearing? Two clearly said the MQA felt more natural sounding. More musical (Whatever that means). That has also been my reaction to MQA. The other 3 could not put words to there experience of a difference.
I came away from this meeting that at least with this material and hardware differences in sound quality are very small. I still insist that the jury is out on MQA until a lot more familiar material is available to the consumer
Alan

 

What!!! No Crying?? No Fits of Ecstasy?? Nothing like that video on the MQA site? No birth of new world? nt, posted on October 28, 2016 at 14:26:57
bjh
Audiophile

Posts: 18614
Location: Ontario
Joined: November 22, 2003
.


 

RE: Great Story -- But Small Differences Again..., posted on October 28, 2016 at 14:32:05
Doug Schneider
Reviewer

Posts: 881
Location: North America
Joined: April 16, 2005
Hi Alan,

Thanks for this. I agree -- a lot more of this stuff needs to be done. And you're right about 2L recordings -- they're very, very good to begin with.

Doug Schneider
SoundStage!

 

RE: My Mistake..., posted on October 28, 2016 at 15:05:25
Peter Breuninger
Reviewer

Posts: 601
Joined: August 28, 2002
Thumbs up Doug.

 

RE: Comparisons/Marketing/Etc., posted on October 28, 2016 at 15:08:23
Peter Breuninger
Reviewer

Posts: 601
Joined: August 28, 2002
It's time for our videos of this... follow the money.

 

RE: Blind Testing, posted on October 28, 2016 at 17:59:09
fstein
Audiophile

Posts: 2989
Location: fstein
Joined: May 18, 2006
Because I am on my wife's tablet, I admit to not slogging thru all this, but have a moest suggestion.
Many years ago a British audio magazine t3ested the hearing of a large group. Those with superior audiograms were indeed "golden ears".
Perhaps we could do this again followed by testing with files of varying resolution?

 

We have, posted on October 29, 2016 at 03:58:47
fmak
Audiophile

Posts: 13158
Location: Kent
Joined: June 1, 2002
another reviewer who do not value the opinion of users and readers and therefore do not care to collate and listen to them.

By research, I refer to the collation of views, especially those of other repected reviewers that are reflected through forum postings.

This attitude is simply an egocentric appraoch to audio reviewing in particular and journalism in general.

 

Let me get this straight:, posted on October 29, 2016 at 13:39:58
b.l.zeebub
Audiophile

Posts: 9361
Location: 52deg 28'N,1deg56'W
Joined: April 17, 2006
You got some wrong, published the actual result (credit to you) but are now trying to justify to yourself that you should have cheated when publishing to make yourself look better because the result you got is statistically about as good as pure guessing.

Not exactly the point of blind tests now, is it?

 

it is too clever for most to understand, posted on November 5, 2016 at 10:29:31
fmak
Audiophile

Posts: 13158
Location: Kent
Joined: June 1, 2002
assertions within the model cannot be understood until they have been subjected to audience testing with a large sample size.

 

Page processed in 0.046 seconds.