Notes by ff123
Test started 10/28/01
Test closed 1/12/02
Listener
comments for fossiles.wav
Listener
comments for rawhide.wav
Listener
comments for wayitis.wav
Test Objective: Determine how audio codecs which average about 128 kbit/s rank in subjective quality. The intended application would be general-purpose file-based, audio-only storage. Compatibility with real-time streaming, portable devices, or video is not considered in choosing the codecs or settings to be compared.
Description of test: 6 codecs are tested at settings which average 128 kbit/s. I have prepared 3 short WAV samples using each of the 6 codecs for people to listen to and rate against the original. The codecs are:
| Format Type | Encoder Name and Version | Settings (128 kbit/s) | Decoder Name and Version | Settings |
| MP3 | Lame 3.89beta | --abr 134 -h --nspsytune --athtype 2 --ns-bass -8 | in_mp3.dll (version 2.75i): default mp3 decoder within Winamp 2.76 | none |
| MP3 | Xing within AudioCatalyst 2.1 | 128 kbit/s, high frequency mode disabled, simple stereo disabled | in_mp3.dll (version 2.75i): default mp3 decoder within Winamp 2.76 | none |
| AAC | Liquifier Pro 5.0.0 Beta 2, Build 24 | streaming 128, equalization disabled, dynamics disabled, dual mono encoding disabled, audio bandwidth overridden by the program, set at 17995 Hz . | in_lqt.dll (v. 1.055) | none |
| MPC | *mppenc.exe version 1.7.9c | -radio -ltq_gain 10 -tmn 12 -nmt 4.8 | mppdec.exe 1.7.8c | none |
| WMA8 | Windows Media Player 7.1 (version 7.01.00.3055); wmadmoe.dll version 8.0.0.0371 | 128 kbit/s | Windows Media Player 7.1 (version 7.01.00.3055); output captured by Total Recorder | none |
| Ogg Vorbis | **Oggdrop pre-RC3 for Windows 32; from CVS (10/26/01) | 128 kbit/s | in_vorbis.dll: Nullsoft Vorbis Decoder v1.15b (RC2) | none |
* Uses settings which lower the bitrate below the
nominal preset (-radio).
** This is not an official release version. Ogg Vorbis RC3 had
not yet been released as of the start of the test. It is released
now with some improvements since the date of this compile.
1. Choose download source: I have made compressed files available from my website. If you can, choose something other than just the plain zipped archives to save some of my bandwidth. For each column choose one losslessly compressed archive file to download. You'll also need to download the appropriate decompressor. You may test one, two or all three of the samples below, as time and inclination allows. The most difficult file to hear differences in is "fossiles" and the easiest file is "wayitis."
| lossless compression type | "fossiles" | "rawhide" | "wayitis" |
| LPAC'd and zipped | fossiles_lpac.zip (7.4 MB) |
rawhide_lpac.zip (9.6 MB) |
wayitis_lpac.zip (10.7 MB) |
| FLAC'd and zipped | fossiles_flac.zip (8.8 MB) |
rawhide_flac.zip (10.8 MB) |
wayitis_flac.zip (12.7 MB) |
| SHN'd and zipped | fossiles_shn.zip (9.2 MB) |
rawhide_shn.zip (11.5 MB) |
wayitis_shn.zip (13.2 MB) |
| Just plain zipped (if you can, save my bandwidth by
using the other archives) Note: I initially forgot to upload the originals for the plain zips, so I added them afterwards. Sorry to those who couldn't find the originals. |
fossiles.zip (13.2 MB) fossiles_orig.zip (2.2 MB) |
rawhide.zip (16.2 MB) rawhide_orig.zip (2.7 MB) |
wayitis.zip (20.1 MB) wayitis_orig.zip (3.3 MB) |
Description of musical selections:
1. fossiles: 14 second excerpt of "Fossiles" from Camille Saint-Saëns's Carnival of the Animals (Philips 416 841-2). Piano, violin, and xylophone are the prominent instruments in this selection.
2. rawhide: 18 second excerpt of Frankie Laine singing the theme to the TV series, "Rawhide" (Telarc CD-80141). Guitar and male vocals.
3. wayitis: 21 second opening excerpt (keyboards) from "The Way It Is" by Bruce Hornsby & The Range.
2. Randomize the order of the files: Before you listen to any of the files, randomize the order as follows. Go to my web-based randomizer utility at http://ff123.net/random/random.html. Enter the numeric identifier from each of the six codec files and randomize the order. This is the order you should listen to the files in. A convenient way to make sure you do this is to add a letter in front of the numeric ID and have your file manager alphabetize everything for you. For example:
| before randomization | randomized order | added letter in front of numeric ID |
| fossiles_200.wav | fossiles_299.wav | fossiles_a299.wav |
| fossiles_234.wav | fossiles_854.wav | fossiles_b854.wav |
| fossiles_299.wav | fossiles_321.wav | fossiles_c321.wav |
| fossiles_321.wav | fossiles_234.wav | fossiles_d234.wav |
| fossiles_413.wav | fossiles_200.wav | fossiles_e200.wav |
| fossiles_854.wav | fossiles_413.wav | fossiles_f413.wav |
3. Turn off any equalization or sound-enhancement processing. If you play the WAV files through Winamp, for example, make sure that you are listening to the unprocessed files. In Winamp, to turn off the equalization, make sure it's turned off from within the equalization window itself. The button on the main panel is just to open up the equalization window -- it doesn't turn the equalization on or off! And just setting the slider to zero without turning the equalization off is not good enough either.
4. Rate each codec against the original and comment on the defects. Listen to each of the codecs and rate each one in comparison with the original. Use a 1.0 to 5.0 scale, as described below:
5.0 = imperceptible (not perceptible)
4.0 = perceptible but not annoying
3.0 = slightly annoying
2.0 = annoying
1.0 = very annoying
Specify your rating to 1 decimal place, for example 3.3, 2.7, etc. The lowest score you can give is 1.0, and the highest score you can give is 5.0, which means that you can't tell it apart from the original.For each file that you rate less than 5.0, describe what is wrong with it.
If you are unsure if you hear a difference from the original, you are encouraged to perform ABX tests. Various ABX utilities are listed below:
Windows: PC-ABX -- http://www.pcabx.com/
Perl script written by xiphmont (of Ogg Vorbis fame)
*NIX: abx.c from the Lame Source
linux: Linabx -- http://www.beryllium.net/~remco/linabx/
Mac ABX -- http://www.pcabx.com/
You are assumed to actually be hearing a difference from the original (with 95% confidence) if you score at least 12 correct out of 16 trials. You should report a score of 5.0 if you score less than 12 correct out of 16.
5. Message me (ff123) from the r3mix or hydrogenaudio forums with your results, or email me at miyaguch at eskimo dot com. Please do not publicly discuss results until the tests are completed.
I will make the ratings/comments available for participants to read from a private page while the test is ongoing.
Your comments and ratings will be anonymous unless you request otherwise.