The Golden Ear Challenge

[Raphie]

Wie durft? 

http://www.trustmeimascientist.com/2013/10/07/update-on-the-golden-ear-challenge-who-will-conquer-audio-mount-everest/

By Justin Colletti | Published: October 7, 2013
tmias_thumbLast month, we issued "The Golden Ear Challenge": Our promise to write a glowing, feature-length article about whoever becomes the first person on record to show he or she can reliably hear an improvement offered by any super-high-resolution file format under properly controlled conditions. So far, this prize remains unclaimed.

Although I have my doubts that anyone will ace this challenge, I hold out hope that it is possible. If, in the spring of 1952, you asked the average person whether someone would ever reach the top of Mt. Everest or run a mile in under 4:00, he might have said "Nah." But within 2 years, both of those records were broken.

It could very well be that 16/44.1 PCM or 320kbps MP3 represent the very pinnacle of audio quality as far as the human ear is concerned. (At least that's what the overwhelming weight of science seems to suggest so far.)

But hey, it's worth trying, right? Especially since so many marketers claim there's a real difference. And if that difference is real, it should be acknowledged.

Claims to the Contrary

Since issuing this challenge, we've heard from a few people whose responses have gone a little something like this: "What are you talking about?? The difference is obvious. I hear it all the time. What are you, deaf?"

Of course this is al lot like a runner in April, 1954 saying: "What are you kidding? I run a sub 4 minute mile all the time. It's just that no one has ever been there to record it." It's also just like a daredevil in April, 1953 saying: "Psssshhh, Mt Everest? I've been to the top. And I don't need your silly 'photographic evidence' to prove it."

So what gives? Why are a handful of people so certain they can hear differences that have never yet been documented under properly controlled conditions?

The usual answer to this is either: A) placebo effect, B) expectation bias C) a poor understanding of probabilities, D) a deep unfamiliarity with the body of evidence on the subject, E) wishful thinking or F) all of the above.

(For an overview of some of these crucial pro audio concepts, I strongly recommend Ethan Winer's classic AES panel "The Audio Myths Workshop." If yu've never seen it, it might just change the way you think about sound.)

Still, I can't hold it against anyone who is convinced he has heard real differences based on sighted tests and marketing materials, because honestly, no one is born knowing this stuff. I know I wasn't. And the truth is you don't need to know the science to make great-sounding recordings. (Although it can't hurt!)

Unsuccessful Attempts so Far

My favorite thing about the Golden Ears Challenge is that it has opened the eyes and ears of a few very reasonable people.

Thanks to this challenge, I've had several friends and colleagues participate in properly-controlled blind tests for the very first time. By and large, they've been astonished by the results.

The response from these participants has usually been something along the lines of: "Well, I thought I could hear a difference – Until you stopped telling me which was which!"

I remember feeling exactly the same way when I was first exposed to blind testing. It's a huge perspective changer. Once you've felt for yourself just how strong the power of our unconscious biases are, it's hard to ever look at the world the same way again.

Through blind testing, you come to realize firsthand that there is no such thing as an "unbiased" person. There are only people who are aware of their own biases, and those who are not.

For some of my colleagues, this also turned out to be their first real experience thinking about probabilities and how they apply to listening tests.

When they think about it for a minute, listeners quickly come to realize that it's not enough to get the right answer once in a blind test. Just like flipping a coin and correctly calling "heads" is not proof of ESP, getting the answer right in a single blind AB comparison is no proof of an audible difference.

Some of us are surprised to realize that even getting the answer right three times isn't enough! If you were to just randomly guess, you'd get the right answer three times in a row on 1 out of every 8 tries, purely by chance. You can see how cherry picking results could be a real problem! We wouldn't conclude that 1 in 8 people are genuine psychics for correctly guessing a coin flip three times in a row. And there's no reason to apply a different standard to audio.

What People Can Hear

So far, all the results we've collected during the Golden Ear Challenge have confirmed what scientists already know about human hearing. We just haven't found anyone – including a few very capable GRAMMY award-winning audio engineers – who has achieved a result that flies in the face of the current body of evidence.

To date, no one who has taken this challenge has gotten a statistically significant result when comparing a final playback of a 16/44.1 file or 320 kbps mp3 to any higher resolution format. This includes 24/192, 24/96, 24/88.1 and SACD. In each and every case, the results have been no better than a coin flip.

For me, the most interesting result came from one participant's test which suggested that he might be able to tell a difference between a 24 bit and 16 bit at the low levels commonly used for tracking. If he can hear that difference, it would be no surprise. But it does give us an extra insights into why, against all evidence, some engineers might swear by higher-resolution formats for playback.

During the tracking process, engineers tend not to use the full resolution of their digital systems so that they can avoid clipping. At this stage, it may be possible to hear a difference between the noise floor of 16 bit and 24 bit audio. This has been known for a long time. But we also know that this difference disappears as soon as the file is mastered or normalized in order to use the full resolution of either format.

For instance: If you set a peak level of -18dbfs during a recording session, and compared the resulting file at 16 and 24 bits without bringing up the level, you might hear a difference. That would be like comparing an effective resolution of 13 bits to 21 bits. This is a difference we'd expect many (but not all) trained listeners to hear. But in a case like this, you're just not using all the bits available in the file! Once you bring your peak level up to anywhere near 0dbfs for final playback, that difference in noise floor would disappear completely.

(If those last two paragraphs make no sense to you, I recommend reading our recent article on what bit depth really means for audio quality.)

Placebo effect aside, it could be that since some engineers really do hear a difference in the noise floor between 16 and 24 bits while they're recording. (Which is to be expected.) And this might lead them to incorrectly assume that higher resolutions must be better for final playback as well. That's a possibility that's worth considering.

Potential Victors?

So far, out of all the responses to the Golden Ears Challenge, we've heard from just one confident challenger who A) was already familiar with the concepts of placebo effect and expectation bias, B) already understood the importance of blind testing and statistical significance, and C) already trains his ears with blind ABX listening tests. If anyone can win this thing, it's bound to be somebody like this!

We made contact via social media, and although I begged him to take the challenge, he did not pursue. His claim was that he's able to reliably differentiate between 320kbps mp3 and CD quality sound 68% of the time. If we could even get that kind of result with any statistical significance, I'd consider this challenge won. But so far, we have no confirmed evidence. Just a claim on Facebook.

We've also heard from another impressive candidate who's currently studying for a PhD in ear training. He is confident that he can demonstrate a clear positive at 192kbps, and seems optimistic that he might be able to succeed at 320kbps. Reliably differentiating a 192kbps file is an impressive feat in its own right (studies suggest that the majority of musicians and engineers can't even hear that difference) but we already know that it can be done. If we could get any solid evidence of this happening at 320kbps, it would be news worth reporting.

So there you have it: Until we find a winner, you can consider this an ongoing challenge. To date, we've heard a lot of talk, and seen zero evidence. So if anyone comes to you with big claims about their ability to hear any improvement offered by extreme audio resolutions, send them our way and tell them to "Take The Challenge"!

Until then, the jury says: "Come on in; The music is good and the audio's fine."

[martin778]

Klinkt als behoorlijke onzin? ABX tests zijn dat sowieso 

MP3 320 vs Flac is duidelijk hoorbaar en ook meetbaar.

[Raphie]

Doe je mee? gaat er namelijk niet om of er verschillen zijn maar of je ze waarneemt

Als je het artikel had gelezen had je dat er wel uitgehaald 

[snikkeltje]

Klinkt als behoorlijke onzin? ABX tests zijn dat sowieso 

MP3 320 vs Flac is duidelijk hoorbaar en ook meetbaar.

Het is zeker meetbaar, maar mijn oortjes horen het verschil niet meer tussen de 320Kbs Rip of de originele CD.

[Nahpets]

maar mijn oortjes horen het verschil niet meer tussen de 320Kbs Rip of de originele CD.

Been there, done that. And lost. 

[AbZ]

Been there, done that. And lost. 

Dat zijn toch juist bevrijdende conclusies in het audiogebeuren 

A    B     X     Kost het onderscheiden (te)veel moeite, dan is er een probleem minder 

[Raphie]

genoeg nuchtere reacties hier, het gaat meer om de zweefteven met "gouden oortjes" die stekkertjes draaien en die zoals het artikel aangeeft het er "zonder meer direct uitpikt" hebben altijd wel een grote mond, maar als het er op aankomt is het stil..... 

[daanvanduif]

Moet eerlijk toegeven dat ik deze test thuis heb geprobeerd, kon zelf het verschil er niet uithalen....

[Raphie]

Moet eerlijk toegeven dat ik deze test thuis heb geprobeerd, kon zelf het verschil er niet uithalen....

niemand kan dat zonder visuele houvast, blind of dubbel blind

[morca]

Ik zie zo gauw niet welk nr gebruikt is.
Maar met een rustig nr vd de dire straits was het niet te doen.
met een nr van bublle met ze bigband kon je duidelijk horen welke de MP3 was.
Dit is al eens eerder aangehaald door leden.

Dus met het juiste nr,ja durf ik het wel te doen.
Ik pikte hem er zo uit,en dat was ook blind.

[Raphie]

Ik zie zo gauw niet welk nr gebruikt is.
Maar met een rustig nr vd de dire straits was het niet te doen.
met een nr van bublle met ze bigband kon je duidelijk horen welke de MP3 was.
Dit is al eens eerder aangehaald door leden.

Dus met het juiste nr,ja durf ik het wel te doen.
Ik pikte hem er zo uit,en dat was ook blind.

nou geef je op dan? Je mag zelf je tracks kiezen

[Nahpets]

Best wel stil hier. Ik had eigenlijk wel meer reacties verwacht.

[Raphie]

ja, het is eng dcihtbij voor de gouden oortjes, slechts 1 stapje op Facebook en je hebt wereldwijde faam (one way or the other )
Zo zie je maar, je kan de grote broek aantrekken in je eigen topic maar als het erop aan komt geven ze niet thuis prutsers

[kippekop]

Ik kan niet eens het verschil horen tussen 256 AAC en de originele cd... en dat op een echt mooie cd-speler (SONY ES van 12 kg)

[morca]

nou geef je op dan? Je mag zelf je tracks kiezen

Ik had geen zin alles te lezen,maar je kan hier zelf kiezen?.
Nog maar eens lezen dan.

Ik geef nooit op     

EDIT nou zie ik het,de abx software van foobar,die heb ik al getest zoals ik al zei 

[Deleted member]

genoeg nuchtere reacties hier, het gaat meer om de zweefteven met "gouden oortjes" die stekkertjes draaien en die zoals het artikel aangeeft het er "zonder meer direct uitpikt" hebben altijd wel een grote mond, maar als het er op aankomt is het stil..... 

ja, het is eng dcihtbij voor de gouden oortjes, slechts 1 stapje op Facebook en je hebt wereldwijde faam (one way or the other )
Zo zie je maar, je kan de grote broek aantrekken in je eigen topic maar als het erop aan komt geven ze niet thuis prutsers

Je gaat weer lekker...  Anyway, ik denk dat de 'prutsende zweefteven' helemaal geen interesse hebben in ABX tests, en terecht. Zo veel mag toch wel duidelijk zijn na tig jaar deze zelfde lame discussies.

[Raphie]

Dat is het punt van het artikel niet, natuurijk kietel ik een beetje maar het punt is dat hetgeen ze beweren nooit gebeurt als er anderen bij zijn
het gebeurt altijd als niemand kijkt, er zijn nooit getuigen