Testing Guidelines for the 3D DAW-SUM CD-Final Version

(Modified sample length as of 2/25/03)
(Added Track IDs 3/5/03)
(Clarified sample length 3/10/03)

The purpose of this exercise is to compare multiple DAW platforms and see if they can achieve identical mixes if each is supplied with identical input material and each is calibrated exactly the same. This test has been designed with the help of many recording and mastering engineers. It has many calibration stages and hopefully is designed to overcome or compensate for any inherent differences between DAW platforms. There will no winners or losers and no list will be published. This is an educational exercise to see if there are indeed sonic differences between supposedly "identical" mixes on different DAW mixers. It should be noted that several hardware digital mixers will be included in this test just to see if there is a difference in their summing as opposed to a DAW mixer.

For now, these rules apply only to those people who have been chosen to participate in the testing. They have been notified via email. After the initial round of testing is complete, others will have the opportunity to get the same source soundfiles on a CD-ROM so they can compare their results with the ones we have recorded. (If you think you have a platform that may not be represented, you may contact me at 3daudio@comcast.net.)

INSTRUCTIONS:

Each participant will download 24 mono .wav files from the server. They are labeled "01_L.wav" through "24_R.wav".
All soundfiles are 24-bit/ 48 kHz files. Odd numbered tracks, (1, 3, 5, 7, etc.) should be panned entirely to the left channel. Even numbered tracks (2, 4, 6, 8, etc.) should be panned entirely to the right channel.

Each track begins with a 1 kHz tone at -12.00 dB, confirmed using Spectrafoo. The tone lasts for 18 seconds, to allow for confirmation of levels. NOTE: Each track of the 24 begins with the same identical tone. Do NOT play all 24 tracks at the same time. Doing so will seriously overload your system.

You may notice that the tones on alternate tracks are polarity reversed. That is intentional. The odd numbered tracks have the tone starting with a positive swing. The even numbered tracks have the tone starting with a negative swing. This will offer visual confirmation that the tracks are correctly positioned.

I should note that the tones are intentionally 16-bits only. This was a suggestion from Bob Katz that will allow us to see whether each mixer actually passes a 24-bit clean signal. If you have a bitscope or Spectrafoo's Code Meter, please use it to confirm that the tone you are returning is "16-bit clean," meaning there is no activity in bits 17-24.

After the initial alignment tone, there is a timing pop positioned identically across all 24 tracks. When the tracks are imported, all these sync pops should line up across all 24 tracks. They are accurately timed to single sample resolution.

After that, the musical program starts within 2 seconds.

All faders should be set to "0 dB" which should be unity gain. Using a single pair of faders, feeding a master fader also set to "0 dB", the output from the master fader should read "-12 dBFS." Any other readings mean that the alignment is incorrect. It is best to use external metering (Spectrafoo, Mytek Digital metering, or some other standalone) to confirm these output settings.

After you have confirmed that each fader outputs a level of -12 to the master fader, please eliminate all the tones and sync pop except forthe first two tracks. This will make sure you don't blow your speakers if you happen to play from zero and you will be able to print the tones back to the stereo mix files. The last test is to play all 24 tracks at the same time and observe the output level.

Tracks 23 and 24 are the resulting mix of Tracks 1-22 except for one thing--the polarity is inverted. This means that if your levels are correctly calibrated, the resulting output from the stereo bus should be almost nonexistent. In my tests, I was able to achieve a -138 dB reading with all 24 tracks open. Granted, mine is the system where the mixes were generated. If you see any output when all 24 faders are played at the same time. In my testing, moving one of the faders by .1 dB raised the output level of the difference mix up to ~ -50 dB, a level jump of 80 dB! I realize that not everyone has metering that will enable them to see the 24 bit or signals in the -130 range, but everyone should be able to tell if there is signal at -60 dBFS.

If there is a signal greater than -60 dBFS present when all 24 tracks are played simultaneously, then you have not accurately adjusted your faders or there is something wrong with your setup. Ideally, the signal should be around -130 dBFS. Either recalibrate your levels or try to null the mix by ear. I will say again. If the fader settings are correct and all 24 tracks are played simultaneously, the difference signal should be inaudible or extremely low level.

After the calibration and difference tests have been successfully performed, you can print your mixes. I need to receive three mix files from each participant, labeled in the following fashion.

The first mixfile will be :20 long and will contain only the test tone from two tracks at unity gain. This will be used to confirm whether the mixer is passing a clean 24-bit signal. This mix file will be labeled with ONLY the number of your assigned platform and no suffix, for example: 01.wav.

The second mixfile, Mix A, will be a stereo interleaved 24/48K .wav file that needs to be precisely 1:18 in length. It should begin at the leading edge of the sync pop and It will include a) the sync pop sourced from only two tracks, and b) the resulting mix when all faders are set to their properly calibrated 0 setting. Do NOT include tracks 23 and 24 in this mix, since they are only used for calibration. If properly done, this mix file will be 21.6 Megabytes in size.

This mix will be the "unity gain" mix and should represent a summing only example of each platform. This mix will be labeled with the number of your platform (sent to you via email) and the capital letter A followed by the wave suffix, .wav. For example: 01A.wav.

VERY IMPORTANT NOTE

Mix A should not include any level adjustments at all and more importantly, CANNOT contain any dither or noise shaping at all. For those who use PT, you MUST use the non-dithered mixer for this test to be valid. If not, there will be activity in the 2 LSBs that could invalidate this test.


The soundfiles that are submitted to me will not be modified in any way. I will only import a copy of them to confirm their viability. Other than a verification of a copy, they will not be processed or modified at all.

The third mixfile, Mix B, will also be a stereo interleaved 24/48K .wav file that needs to be precisely 1:18 in length. This mix will include level adjustments that are performed with the faders of the platform itself. No plugins are allowed. The individual faders should be lowered by exactly 3 dB and the master fader should be raised by a corresponding 3 dB. Some have noted that the fader coefficients between individual and master faders are not the same and level differences could result. In my research, with the help of others, these differences would not exceed .1 dB and I feel are therefore allowable since the fader resolution of most platforms does not allow increments smaller than .1 dB.

When you lower the individual faders, you need to use the test tone again and verify that each track is feeding a -15 dB signal to the master fader. Then using only one set of tones (2 tracks only), make sure that the adjusted output of the master fader (with the 3 dB boost) reads -12 dB. Once this have been confirmed, then you are ready to print the second mix, or mix B. It will need to be exactly the same length as the first mix (1:18) and will include the same elements: a) the sync pop sourced from only two tracks, and b) the resulting mix when all faders are modified to the -3 and +3 settings.

NOTE: Since this mix will generate words that are longer than 24-bits, then dither needs to be applied. In the case of a DAW with built-in dither, like PT's dithered mixer, then use that. This mix is intended to compare only the effect of the fader level changes, not the dithers. But since there is no universal dither, I have decided to use whatever is implemented in each platform. Do NOT use any third party dithering for this test.

This third mixfile will be labeled similarly to the second, with a B instead of an A. For example: 01B.wav.

I have received a few requests from people who have developed techniques for maximizing the accuracy of their preferred platform. So, in response to their requests to show off their techniques and see if the differences are indeed audible and improved, I will offer the possibility of including a third mix (a fourth file). This mix must follow the same
procedure and gain changes as Mix B, except that the dithering or gain changing techniques may be non-standard. Mix C is a gain changed mix done exactly as Mix B, except third party plugins may be allowed to do the gain changes or dithering. It can only be 1:18 in length and will not necessarily be included in the compilation project. If you think that you know a better dithering plugin or like to use a plugin for level changes instead of the standard faders or have some other technique for maximizing the resolution of your DAW, you may include it here. NOTE: These mixes must follow the same guidelines are the previous mixes. If any level differences are detected, the mixes will never be heard by anyone other than me. Do NOT feel obliged to offer a third mix.

The main focus of this exercise is comparing the "A" mixes. Those are the only ones where everything will truly be equal. The "B" mixes will be interesting but of less value, since there will be fader changes and also dithering introduced.

The purpose of this exercise is to compare apples and apples as much as possible. We are trying to ascertain whether on each platform 2 + 2 = 4. The object is not to "beat" the competition but purely to set up a level playing field upon which we can compare the multitude of platforms.

There is much speculation about what we will find when they are all lined up side by side. But since I have never heard of anyone doing a test like this in the history of DAWs, then it is all purely speculation. No one knows, including me, what we will find. There may be mixers that won't pass 16-bits cleanly from input to output. There may be mixers that will not do precise level adjustments regardless of what their faders say. We've already discovered some things that could have sonic significance just during the week of research for this project. I'm sure we'll discover much more as we audition all of the sound files.

One more thing. Keep copious notes of all aspects of your system and platform. These will be required to accompany your submissions.

Here is the identification of what instruments are on what tracks.

01_Bass_L.wav
02_Bass_R.wav
03_KickSn_L.wav
04_KickSn_R.wav
05_Drums_L.wav
06_Drums_R.wav
07_Piano_L.wav
08_Piano_R.wav
09_B3_L.wav
10_B3_R.wav
11_ElectricGtr_L.wav
12_ElectricGtr_R.wav
13_ElecPrc_L.wav
14_ElecPrc_R.wav
15_Perc_L.wav
16_Perc_R.wav
17_Horns_L.wav
18_Horns_R.wav
19_Choir_L.wav
20_Choir_R.wav
21_Reverb_L.wav
22_Reverb_R.wav
23_InvertedMix_L.wav (polarity inverted)
24_InvertedMix_R.wav (polarity inverted)

[ February 24, 2003: Message edited by: 3D Audio Inc. ]

[ February 26, 2003: Message edited by: 3D Audio Inc. ]

[ March 05, 2003: Message edited by: 3D Audio Inc. ]

[ March 10, 2003: Message edited by: 3D Audio Inc. ]

[ March 14, 2003: Message edited by: 3D Audio Inc. ]