Extreme Tech's Audio Codec Quality Shootout

[Jason Dunn]

<div class='os_post_top_link'><a href='http://www.extremetech.com/article2/0,1558,1561566,00.asp' target='_blank'>http://www.extremetech.com/article2...,1561566,00.asp</a><br /><br /></div>"Millions of people convert their CD collections to a compressed digital audio format, often without considering the format or settings they use. With several different popular formats and lots of quality settings, many users just pick an application they like and use whatever the default settings are. ExtremeTech readers -- generally a bit more hands-on and do-it-yourself -- are more likely to seek out other applications or players and encode their audio exactly how they want. <br /><br />Certainly a top consideration should be compatibility. We all have different audio devices, and it doesn't matter how good your music sounds if it won't play on your hardware. We'll leave that decision up to you; you know what software and devices you have and what they're capable of playing. Encoding time is another possible consideration, but honestly, modern computers encode audio so quickly that the difference in speed between one format and another is practically a non-issue."<br /><br />This is one of the best, most hard-core looks at audio codec quality that I've seen. If you've been wondering about what audio format to use for your portable music player (whether it be an iPod, Pocket PC, or Smartphone), this article is a good substitution for doing all the testing yourself. I won't ruin the article for you, but let's just say that I wasn't surprised by the final result - I made the switch myself about a year ago, and have been very pleased with the results. It's also greatly simplified my ripping/encoding process, saving me time, which is a good thing.<br /><br />Ultimately, however, it's important to realize that the best codec for me might not be the best one for you - it depends on factors such as speakers, headphones, music type, and the most important one: your ears. Check out the article - it's a great read!

[SubFuze]

This is a very biased review, so much so that I wonder if MS paid them to do the test. The article gets blasted in the comments and over at Hydrogen Audio - http://www.hydrogenaudio.org/index.php?showtopic=20443.

Some major flaws of the test:
*Only 6 listeners
*They try to make the listening group out to be both a group of experts (which they're not- they admit one of the listeners is "even a young child") and a "real-world listening test" (which it's not- with a sample that small, their results are irrelevant)
*Of those 6 listeners, some of them gave 5 (the highest rating in their scoring system) to the 64k samples
*They are very unspecific as to their settings (huh?! 98% quality? Is that 313.6 ABR? --alt-preset insane with 2% of the audio spectrum cut off? what is it?)
*Their "98% quality" VBR scores were lower than their 128k scores (they speculated that it was "because they are actually more accurate than the 128k versions")

Sorry for the rant, but it really irks me when people try to pass off advertising as unbiased journalism. I do agree that WMA has good compatibility, but it is not as leaps and bounds ahead of MP3 as they claim. In *unbiased* listening tests, AAC has proven itself to be a much more transparent codec than WMA, especially at higher bitrates (which they conveniently left off because "iTunes doesn't support VBR AAC"). They also freely admit that OGG sounds the best of all codecs at 64k (in the low bitrates, where it was optimized to sound best). If device support was a non-issue, I'd do all my high bit-rate encoding in AAC and all my low bit-rate encoding in OGG. However, since I am concerned with compatibility, I just do everything in MP3 because it is even more ubiquitous than WMA (it plays in my car CD MP3 player, my portable MP3 CD players, my Pocket PC, my computer, and my DVD player).

[Jason Dunn]

This is a very biased review...

Heh. Interesting - I found it to be the exact opposite. I think if anything can be learned from this article, looking at the scores, the reality is that at 128 kbps and higher, it's difficult for the average person with average audio speakers/headphones to tell the difference between what each codec offers. If you have an iPod, you're going to use AAC. If you don't, WMA offers better compatibility with devices while still maintaining great audio quality. MP3 still has the best compatibility though, and if you're encoding at 160 kbps, I doubt most people could tell the difference - so I think they shorted MP3 a bit in the article. All sorts of car stereo decks play MP3, but finding one that supports VBR WMA 9 is a bit of a challenge (at least in these parts).

Saying that it's a MS-sponsored piece is pretty insulting BTW. :?

[BitBandit]

This article seems to have really stirred the passions of audio enthusiasts and I'm not sure that I understand why. The author clearly stated at the beginning that the tests were based on a very small sample size (and here I do agree that six people really don't create any kind of significance, statistically speaking) and that it was specifically done without the usual controls and constraints of a scientific inquiry. They were really just seeking some real-world opinions and that's exactly what they published. It's simply another way of comparing audio codec performance and seems like a worthy look to me, yet there is so much fervor over it's lack of merit as a comparison.

According to the author's instructions, it's meant to be more of an editorial piece than a comparative study, but people seem to refuse to take it for what it's intended to be &lt;insert heavy sigh here>. Oh well.

[klinux]

Ultimately, however, it's important to realize that the best codec for me might not be the best one for you - it depends on factors such as speakers, headphones, music type, and the most important one: your ears.

Well said and cannot agree you with your more.

If you cannot tell the difference between a 64k MP3 and 256k WMA, great, you save space! If you have to have play vinyl with a vaccum tube amp and nothing else, it's great that you appreciate and enojoy your hobby!

As soon as people realize that they should be their own best judge on what codec to use, the less forum battles need to be devoted to audio codec and be then used to talk about the never-ending topics of PPC vs Palm, Apple vs Windows vs Linux, religion, politics, etc! :)

[SubFuze]

Heh. Interesting - I found it to be the exact opposite. I think if anything can be learned from this article, looking at the scores, the reality is that at 128 kbps and higher, it's difficult for the average person with average audio speakers/headphones to tell the difference between what each codec offers. If you have an iPod, you're going to use AAC. If you don't, WMA offers better compatibility with devices while still maintaining great audio quality. MP3 still has the best compatibility though, and if you're encoding at 160 kbps, I doubt most people could tell the difference - so I think they shorted MP3 a bit in the article. All sorts of car stereo decks play MP3, but finding one that supports VBR WMA 9 is a bit of a challenge (at least in these parts).

Saying that it's a MS-sponsored piece is pretty insulting BTW. :?

I guess one of the reasons I found it to be so biased is that it contradicts almost every other listening test that I've ever seen. I agree that for the most part, above 128k, most people can't tell much of a difference, however I can and so I'm much more picky about my codecs than most. Because of my pickiness, I've been reading about encoding methods and codecs for years. Currently, I encode all of my CDs with LAME MP3 and use --alt-preset extreme which gives me about a 224k average bit rate (which is a good balance between quality and size for me since most of my portable devices are CD based). At that high of a rate, the difference between codecs is so small that it's not worth losing compatibility.

I in no way meant my speculation about their motives for writing the article to be a slam on either MS or your site. As far as enthusiast communities go, PPCT is one of the best at realizing where to draw the line between enthusiasm and propaganda. If there are problems with MS, they're freely discussed and never censored. The ads are always clearly marked as such and I've never felt like anyone here was paid-off to be a supporter. That's why this is one of very few sites that I visit daily :D .

Honestly, I'd be happy if WMA was the best codec because of the widespread support, but from almost every other listening test I've seen, it just hasn't been. WMA has come a long way, but it's still not the top codec in my opinion. As such, I'd love to see wider support for more codecs in more devices (I'd love to see that regardless, having more choices is (almost) always a good thing). That's one of the reasons I love my Pocket PC- it doesn't just play WMA and MP3...

[Spiral]

I thought the review was pretty decent. I'm not surprised Ogg did well in the low end, that's generally recognized. WMA usually sounds quite a bit better than mp3 (esp at low-end, where mp3 just fails imo). And i haven't listened to aac much. at lower bitrates i prefer to use wma because it's convenient.

On computer i rip to 192 kbps mp3, with 160 sometimes i can hear a few artifacts, 192 is nearly unnoticeable from original (for me).

[jeffmd]

my opinion on wmv9 has changed some what since I technicaly see its actually pretty hot at high bitrates. Im now wondering if players like my iriver 250's wma codec properly decodes wma9. I know it CAN decode wma9, but is it chinsing on quality in a sort of compatability mode? or does it fully decode it like media player 9 would?

As for the real world test, 6 subjectes was so poor they should have saved the time. I have NO idea why they didn't just burn a crap load of these cd's off, and offer them for free with a checklist. 6 is just..sad.

[allenalb]

i don't really consider myself that much of an audiophile, but i can DEFINITELY hear the difference between 160kbps mp3 and anything lower through even cheap headphones... in fact i would go so far as to say that i would call anyone out if they say they CAN'T hear the difference on headphones.

what i REALLY don't get is why anyone willing to plunk down a couple hundred bills would then decide to shortchange their music, when it's easy to upload music to any mp3 player made... especially if it's a hdd based model which already holds more music than anyone could possibly listen to on a month long vacation...

just my opinion though... i could certainly be wrong...

[gpspassion]

Same here, anything encoded below 160kbps sounds terrible to my ears (except maybe AAC), so that someone could give a "5" to something encoded at 64kbps is cause for concern.

I remember when WMA first came out in 1999 (1998?) they grabbed people in supermarkets and made them listen to mono samples and they said it was better than mp3...

[Jason Dunn]

but i can DEFINITELY hear the difference between 160kbps mp3 and anything lower through even cheap headphones... in fact i would go so far as to say that i would call anyone out if they say they CAN'T hear the difference on headphones.

I think you'd be very, very surprised at what most people can and can't hear. Hell, I have trouble hearing artifacts with most music on a 128 kbps WMA - I would never call them obvious at any rate. You have to know what to listen for in order to hear them. I encode 160 kbps WMAs and rarely, if ever, hear any type of audio artifact. But with poor quality headphones and external noise, it doesn't surprise me at all that many people think 64 kbps sounds just fine - and with some music, it does!

Again, your ears are different than mine, and everyone elses - it's a bit insulting that you'd accuse people of lying rather than accepting that they just can't hear the difference.

[WindWalker]

I have to think that this argument is another example of how we, as enthusiasts and experts, sometimes forget that "ordinary people," that is, the folks who don't spend hours tweaking their computers/PPCs/etc. and won't bother to go out and get LAME and other very specific tools, react to differing, subjective levels of technology. Yes, it would be better if everyone knew as much as we do (well, maybe not.....then, how special would we be? :lol: ), but that's not going to happen. Most people CAN'T hear the difference between some levels of sound encoding....and frankly, they don't care.

Thoughts?

[ctmagnus]

I find that, since I lost 90%+ of the hearing in one ear (other ear's still fine), I can pick out the difference between a low-bitrate file and a high-bitrate file if I listen carefully.

My advice to everyone is to not take any of your senses for granted; go out and buy the highest-quality headphones/monitor/TV set/stereo/whatever you can afford and make use of your senses. Don't take your senses for granted, there's an entire world out there to experience!