Test 24bit/96kHz vs. CD resolution audio

Discussion in 'High resolution audio' started by tigre, Sep 10, 2003.

Page 1 of 2
  1. tigre

    tigre Moderator Staff Member

    Joined:
    Feb 18, 2003
    Messages:
    789
    Likes Received:
    0
    Trophy Points:
    26
    Here's the test, finally:



    Test CD resolution (16bit/44.1kHz) vs. high resolution (e.g. 24/96)

    Test samples
    2 short samples are here: http://64.41.69.21/product/reference/keys.wav and http://64.41.69.21/product/reference/triangle-2.wav
    Anyone who can get 24/96 material please post short samples (~20 seconds, reasonable volume = peak at -0.1 - -10dB) here: http://www.hydrogenaudio.org/index.php?showtopic=13054&

    Hardwardware needed: (*)
    Soundcard capable of 24/96 playback.

    Software needed: (*)
    foobar2000 http://foobar2000.org
    ABCHR http://ff123.net/abchr/abchr.html (*)
    ______________________
    (*)
    for those who have
    DVD-A authoring software and a DVD-A player OR
    DVD-V authoring software that can handle 24/96 audio and a DVD player capable of DVD-V-with-24/96-audio playback,
    instead of ABCHR use KikeG's fileABX (http://www.hydrogenaudio.org/index.php?showtopic=12387&) to create randomized files and use your authoring software to create DVD-A/V

    ========================


    How to test

    0. Install the software needed etc.

    1. Take some audio sample(s) recorded at 24/96 = "A" sample(s)

    2. Convert "A" to "A_downsampled": 16/44.1 using fb2k's diskwriter, dither (strong noiseshaping), slow resampling

    3. Create "B" samples by converting "A_downsampled" to 24/96 using fb2k's diskwriter, slow resampling, dither.

    4. Try to hear differences and prove that you hear them by ABXing using ABCHR (**)
    ________________
    (**)
    If you want to use DVD-A/V + hardware player:

    4.1. Use KikeG's fileABX to create randomized files from "A" and "B"
    4.2. Author a DVD-A/V using the randomized files
    4.3. Try to ABX = play back with your hardware player and tell which files are "A" and which "B"; compare to fileABX's log *AFTERWARDS*

    =======================


    Details/suggestions for "How to test":

    Given the directory you want to use for the test is d:\test

    0. Install the software needed etc.
    - create subdirectories: \test\A_samples; \test\temp; \test\B_samples
    - put ABXHR (or fileABX) to d:\test
    - install foobar2000 using installer

    1. Take some audio sample(s) recorded at 24/96 = "A" sample(s)
    - put the samples you want to use for testing to A_samples directory

    2. Convert "A" to "A_downsampled" ...
    - run foobar2000
    - enqueue (e.g. drag'n'drop) all samples from \A_samples to foobar2000's playlist
    - highlight all files
    - Rightclick -> convert -> Settings ...:
    -- Output directory: d:\test\temp
    -- Output filename formatting: %_filename%
    -- Output format: WAV (PCM 16bit dithered)
    -- [x] Use DSP
    -- [ ] Don't reset DSP between files
    -- [ ] Use replaygain
    -- Press "go to DSP settings" (DSP Manager) : Move *ONLY* Resampler (SSRC) to "Active DSPs" window
    -- Go to DSP Manager -> Resampler:
    --- Target sample rate: 44100 Hz
    --- [x] Slow mode
    --- close preferences
    - Rightclick -> convert -> Run conversion
    Finished. Now 44/16 versions of the source file(s) should be in d:\test\temp

    3. Create "B" samples by converting "A_downsampled" to 24/96 ...
    - clear foobar2000's playlist by highlighting all files and pressing "Del"
    - enqueue (e.g. drag'n'drop) all samples from \temp to foobar2000's playlist
    - highlight all files
    - Rightclick -> convert -> Settings ...:
    -- Output directory: d:\test\B_samples
    -- Output filename formatting: %_filename%
    -- Output format: WAV (PCM 24bit dithered)
    -- [x] Use DSP
    -- [ ] Don't reset DSP between files
    -- [ ] Use replaygain
    -- Press "go to DSP settings" (DSP Manager) : Move *ONLY* Resampler (SSRC) to "Active DSPs" window
    -- Go to DSP Manager -> Resampler:
    --- Target sample rate: 96000 Hz
    --- [x] Slow mode
    --- close preferences
    - Rightclick -> convert -> Run conversion
    Finished. Now 96/24 versions of the source file(s) should be in d:\test\B_samples

    4. Try to hear differences and prove that you hear them by ABXing using ABCHR (***)
    - open ABCHR
    - File -> Setup Test -> Load "A" (=Original) and "B" file.
    - Press ABX...; choose "Select A" = Original and "Select B" = Sample 1
    - Listen to A, B, X (or parts of them) until you think you know if X is A or B and press next trial.
    - You can stop when the "Probability you were guessing" is < 1% (8/8 correct trials, 10/11, 12/14, 14/17, 16/20, ...) or give up.
    - If you ABXed successfully, provide the result, ideally with a detailed description about the difference you've heard.
    ____________________________
    (***)
    If you want to use DVD-A/V + hardware player:

    4.1. Use KikeG's fileABX to create randomized files from "A" and "B"
    - put fileABX.exe to test folder, e.g. d:\test
    - create a set of n test files (at least n=8; something like 20 would be better):
    -- given the name of the 1st sample is "sample1.wav" and n=20 the command line is: "d:\test\fileabx.exe d:\test\A_samples\sample1.wav d:\test\B_samples\sample1.wav 20" (copy + paste if you like)
    -- in \session01 subfolder you'll find the test files now. 01_A.wav and 02_B.wav are the reference files "A" and "B", 03_X01.wav, 04_X02.wav ... are randomized copys of either "A" or "B".
    - Print the file index.txt for later use.

    4.2. Author a DVD-A/V using the randomized files
    - Put the tracks in the same order as they are in \session01 folder.
    - If you want to try more than one sample just repeat 4.1 and put files from \session02, \session03 ... to the compilation.

    4.3. Try to ABX
    - play back with your hardware player and tell for each track if it's "A" or "B". You can listen to the reference tracks and the test tracks as often as you like, also to small parts of it if necessary.
    - When you have decided if one track is "A" or "B" write down the result.
    - When you're finished compare to results.txt and count how many results were correct. To prove that you hear a difference with > 99% security you need 8/8 correct trials or 10/11, 12/14, 14/17, 16/20, ...
    - If you ABXed successfully, provide the result, ideally with a detailed description about the difference you've heard._X_X_X_X_X_[small]AFTERDAWN FORUM RULES: http://forums.afterdawn.com/thread_view.cfm/2487[/small]
     
    Last edited: Sep 10, 2003
    tigre, Sep 10, 2003
    #1
  2. wilkes

    wilkes Regular member

    Joined:
    May 31, 2003
    Messages:
    922
    Likes Received:
    0
    Trophy Points:
    26
    Sorry, tigre, but this seems flawed to my mind. There's just way too much converting/reconverting and general buggering around going on for me. I may well be wrong, but for me a fair and impartial test would be involving dynamics as well. A big part of the difference in High Resolution isn't just the samplerate, but the massively increased dynamic ranges available - 144dB as opposed to CD's 96dB.
    How about this as a compromise?
    We need to mix and master a whole album, at the highest possible depths. (this is always the way I work, as do loads of other mix engineers. We try to keep the material at the highest resolution possible for as long as possible). Personally, I'd use 32 bit Floating Point and 96KHz. Mix the tracks, and when they are how they should be, export to a stereo master - still at the same word/sample rates. Master the album, and when doing the final export, have one export of each track to 16/44.1 and the other to 24/96.
    Go back to the multitracks, and create surround mixes. Prepare in the same way, then export as n-channel split files at working word/sample rate for any mastering.
    Export one set as 24/96 surround, and the other as 16/44.1 Surround.
    As a final note, during the mastering process and indeed the mixing process, it is extremely important NOT to use tools such as Waves L1 or L2, or Steinberg's Loudness Maximizer. For the test to be valid in the dynamics department, they must be mixed correctly - nothing over 0.01 dbFS, or else all we might be listening to is the plugin. Absolutely no brickwall stuff, as all we will succeed in doing will be destroying the dynamics, which is what we really need to preserve!

    If we do this sort of test, where you will have a whole album - one at 24/96 stereo, one at 16/44.1 - then we can do blind listening tests.
    Random samples, notes or tones are not going tp prove anything. We need to hear if the MUSIC sounds better. The double test using Surround files is, I believe, necessary so that on this test the exact same blank media is being used.
    When I have finished the album I'm currently mixing, I'll prepare 4 versions as detailed above, and send them to you so you can then hear for yourself.
    Is this an acceptable challenge?
     
    Last edited: Sep 15, 2003
    wilkes, Sep 15, 2003
    #2
  3. tigre

    tigre Moderator Staff Member

    Joined:
    Feb 18, 2003
    Messages:
    789
    Likes Received:
    0
    Trophy Points:
    26
    Hi Wilkes.

    Thanks for answering. I'm glad that you're willing to take the challenge.

    Obviously you haven't fully understood my test proposal.

    We'll test "A" vs "B".

    "A" = Original at 24bit/96kHz

    "B" = Original downsampled to CD audio resolution: 16bit/44.1kHz.

    BUT...: To create totally equal conditions for playback (-> no unfairnes possible due to e.g. resampling by player before D/A conversion) we resample to "B" to 24/96 again. So everything done by the player will happen at exactly the same conditions.

    I want to use foobar2000 for creating "B" because resampling and dithering is done at very high quality (e.g. 64bit floating point internal precision) and it's free - so everyone interested can verify the results easily.

    Exactly. Because of this "A" sample is native 24/96, not processed at all, "B" runs through resampling+dither to 16bit (-> 96dB dynamic range).

    No problem. Foobar2000 can process 32bit/96kHz files (and convert to 24/96 or 16/44.1 at high quality). If you want to export your 24/96 files (= "A" samples) using some other way, fine - just use foobar2000 for creating "B" samples from them.

    Maybe I haven't understood this correctly - but I thought the idea was to test 2 channel stereo 24/96 vs. 2 channel stereo 16/44.1 - As
    - CD Audio is just stereo
    - the test should be repeatable for as many people as possible (IMO), also for ppl with high resolution soundcard + decent headpones, no 5.1 or similar system should be needed.

    This is my opinion too. Let's do it like this. :)

    I'm not sure what you're talking about here ... brickwall lowpass filter? dynamic compression? ...?

    Blind tests are only "blind" if you don't know what you're listening to while you're listening. This won't be possible if you create on3 "A" and one "B" CD. Better create a CD with 10 x "A" track and 10 x "B" track, randomized by fileABX as described before. I think listening to a whole track (or maybe two in a row) instead of 20seconds samples should be acceptable for you, shouldn't it.

    Anyway, I'll still try to get some 24/96 samples from DVD-V's and post them at hydrogenaudio (and these will be 20-30 seconds, otherwise it would be piracy).

    Yes and no. Testing DVD-As with audio at different sampling rates could make the test flawed, so resampling 16/44.1 back to 24/96 would be necessary to call it "acceptable" as I explained above._X_X_X_X_X_[small]AFTERDAWN FORUM RULES: http://forums.afterdawn.com/thread_view.cfm/2487[/small]
     
    Last edited: Sep 15, 2003
    tigre, Sep 15, 2003
    #3
  4. wilkes

    wilkes Regular member

    Joined:
    May 31, 2003
    Messages:
    922
    Likes Received:
    0
    Trophy Points:
    26
    Sorry, but I still don't think that any upward resampling is any use. You cannot improve the quality like this. You can improve an original recording by going back to the original masters and resampling through high quality converters, and this would be fair IF you also did the 16/44.1 version at the same time.
    The other way I mentioned is the really fair way - a straight mixdown. 1 for DVDA at 24/96, and 1 for CD at 16/44.1.
    Any other type of test becomes dependant on a resampling process that may or may not introduce artifacts of it's own. Also, how does resampling from 16.44.1 to 24/96 constitute a fair test. There will be no difference, as you cannot upsample to increase quality. All you will do is make the filesizes bigger.
    I don't think I understand why my counter proposal is wrong? Please explain. Also, how can there be a CD with 24/96 and 16/44.1 files?
    The fair test is this: - 2 discs. 1 DVDA and 1 CD. Don't tell the subject which is which. What could be easier?
     
    Last edited: Sep 17, 2003
    wilkes, Sep 17, 2003
    #4
  5. listen

    listen Member

    Joined:
    Sep 11, 2003
    Messages:
    30
    Likes Received:
    0
    Trophy Points:
    16
    I don't think the point is to improve quality, but to ensure a fair test by eliminating variables. A playback system might produce a different sound purely because of the type of file (16/44.1 or 24/96) even if the content is the same (other than placeholders that is). Then again it might not, but it's still a good precaution. I haven't actually read this thread (it's late and long), so ignore this if it doesn't make sense.
     
    listen, Sep 17, 2003
    #5
  6. wilkes

    wilkes Regular member

    Joined:
    May 31, 2003
    Messages:
    922
    Likes Received:
    0
    Trophy Points:
    26
    That's the point! It's supposed to sound different on DVDA. It's supposed to sound better!
     
    wilkes, Sep 17, 2003
    #6
  7. tigre

    tigre Moderator Staff Member

    Joined:
    Feb 18, 2003
    Messages:
    789
    Likes Received:
    0
    Trophy Points:
    26
    That's the point.

    .
    Of course not. But upsampling using decent software like the one I've suggested won't decrease quality noticably either. You can test this software (e.g. using frequency analysis) and you'll know what you get.

    With hardware it's different. You'll never know
    - if the hardware you use for playback (e.g. DVD-A player) has a fixed-sample rate DAC, so it resamples internally, giving different results on different input sampling rates
    - if some filters (e.g. lowpass) are applied that are designed good enough not to change sound at 96kHz sampling rate but will change sound at lower sampling rates.
    - ...

    So if the files you finally test are identical in bit depth and sampling rate you'll know for sure that differences you might hear are not caused by the playback equipment.

    If the test would be performed the way you want, you would need to perform it with several different players. Only if you hear (and can ABX) the same difference everytime it would be a reliable proof that there IS a noticable difference between *formats*.

    My mistake. DVD-A(s) needed for the test of course.

    If you have someone who will help you no problem, you can use 2 DVD-As (one "A" and one "B") and let the "DJ" change them ...
     
    tigre, Sep 17, 2003
    #7
  8. tigre

    tigre Moderator Staff Member

    Joined:
    Feb 18, 2003
    Messages:
    789
    Likes Received:
    0
    Trophy Points:
    26
    http://www.hydrogenaudio.org/index.php?showtopic=13054
    24/96 samples available! So far I've uploaded 3, a few more will follow. Please only download if you want to perform the test. (What equipment required? -> See above: 24/96 capable soundcard OR DVD-A player OR DVD-V player capable of high resolution audio playback - the latter two alternatives require non-free DVD authoring software ...).

    I hope this will be an interesting experience to everyone participating. Please report your results here (and problems/questions).
     
    tigre, Sep 25, 2003
    #8
  9. wilkes

    wilkes Regular member

    Joined:
    May 31, 2003
    Messages:
    922
    Likes Received:
    0
    Trophy Points:
    26
    Were the samples recorded in 24/96?
    What converters were used - I need the THD/Noise floor information, make & model.
     
    wilkes, Sep 26, 2003
    #9
  10. tigre

    tigre Moderator Staff Member

    Joined:
    Feb 18, 2003
    Messages:
    789
    Likes Received:
    0
    Trophy Points:
    26
    It's a commercial recording from chesky records in DVD-V format with 24/96 LPCM stereo audio stream - I suggest to check the thread(s) I've linked to ...
    Yes it is 24/96 natively (music related information > 24kHz, especially on percussion, clearly visible in spectral view)
    Noise floor isn't totally constant; frequency analysis show something arround -108dB, sometimes down to -112dB.
    I don't know the abbreviation "THD". To get this information you probably need to contact chesky records. The webpage http://www.chesky.com has a forum where someone could know, support (email) is quite friendly too ...
    Please enlighten me for what reason you need this information.
     
    tigre, Sep 26, 2003
    #10
  11. listen

    listen Member

    Joined:
    Sep 11, 2003
    Messages:
    30
    Likes Received:
    0
    Trophy Points:
    16
    THD is total harmonic distortion.
    Anyway,
    I finally got a good sound card, and last night I downloaded lovely_1 from your hydrogenaudio thread.

    ABX results 12/12

    It was not easy, although maybe with more training and a quiet environment it would be easier.

    I'll post more details later and also on HA (when I remember my password), but before I bother I thought I should check I didn't do anything obvious wrong. I followed your suggestions list pretty much exactly, except I did the ABXing in foobar. I also replaygained the files.

    Is there such thing as dither artifacts? I was listening at fairly low volume so I don't think the noise floor would come into it at all, but perhaps the dither or the resampling introduces other problems? Otherwise, yes, I can hear the difference between 24/96 and 16/44.1

    how about a 192 test? ;)
     
    listen, Dec 28, 2003
    #11
  12. tigre

    tigre Moderator Staff Member

    Joined:
    Feb 18, 2003
    Messages:
    789
    Likes Received:
    0
    Trophy Points:
    26
    Hi listen.

    Thanks for participating. You're the 1st who reports although many have downloaded the samples.

    Something like "dither artifacts" could be possible if noise shaped dither (= most noise added at 16-22kHz) is reproduced distorted for some reason (e.g. aliasing, intermodulation distortion caused by amp or speakers), but if no such playback problems exist, dither (properly done) should do nothing else then adding low level noise (which can be audible if the volume of the music is very low).

    It would be interesting to try different dither methods (flat, triangular noise shaped, ATH noise shaped).

    Anyway, I'm looking forward to see details about your results (especially a description of the difference you heard - hopefully I'll be able to hear it myself).
     
    tigre, Dec 28, 2003
    #12
  13. listen

    listen Member

    Joined:
    Sep 11, 2003
    Messages:
    30
    Likes Received:
    0
    Trophy Points:
    16
    I'm about to go away for the new year, but here's the details so far..

    When I get back I'd be keen to try some different dithering like you say, and if that doesn't fix it I think it would be interesting to try 16/96 and 24/44.1, and then maybe 20/48 or something..

    I listened between 5.2 and 7.2 seconds. The most obvious difference is the drum thing that gets hit around 5.7 or so. The 24/96 one sounds more like it's being hit than the 16/44.1 one. They sound pretty much the same, but the high-def one is somehow more believable.
    Also, I'm fairly sure I could hear a difference in the tambourine (?) that follows it at 6s. One of the lower harmonics seems stronger in the 16/44.1 version, although it was very subtle and I wasn't getting perfect ABX results. Once I discovered 5.7s and then used 6s just to check my decision, it was much easier. Neither one is 100% obvious to me yet individually, but by checking both, I felt quite certain when I eventually picked the right one.
    I also checked with 3.7s, but I'm not sure if that helped me at all.

    My equipment is an M-Audio Revolution feeding straight to Sennheiser HD200 headphones. Volume is not very loud (I can hear my computer fan - just).

    Anway, thanks for setting up the test and good luck with the ABXing.
     
    listen, Dec 29, 2003
    #13
  14. tigre

    tigre Moderator Staff Member

    Joined:
    Feb 18, 2003
    Messages:
    789
    Likes Received:
    0
    Trophy Points:
    26
    Thanks alot for the information. I've just started trying to ABX myself.

    In case you're still arround: Could you please give me some more information:
    - What resampler did you use (foobar2000's - slow or fast mode)?
    - Did you resample back to 24/96 for the test or did you test 16/44.1 vs. 24/96
    - What output mode did you use for playback/ABXing (DirectSound, Waveout, Kernel Streaming), what bit depth?
     
    tigre, Dec 29, 2003
    #14
  15. maxg

    maxg Member

    Joined:
    Dec 16, 2003
    Messages:
    10
    Likes Received:
    0
    Trophy Points:
    11
    Judging by the complexity of this test procedure I think my own test must be deeply flawed.

    I have a Soundblaster Audigy USB sound card connected to my portable PC.

    Using a record as a source (Tracy Chapman's second album as it happens on a Project rpm4 very modified with shelter 901 cartridge and Accoustic signature Tango phono stage I took an RCA to din plug adapter cable (rather a good one - judging by the price and the weight) straight into the card and then recorded the whole album at 96/24.

    I then repeated the process at 44.1/16 and compared the outputs playing back from the computer directly into my main system (using the same cable connected from the front speaker out to one of the inputs on the pre-amp.

    Recording was done using the bundled Creative media source player.

    Before I divulge the findings - what was wrong with that procedure in creating 2 recordings identical in all but bit rates and word size?
     
    maxg, Dec 30, 2003
    #15
  16. wilkes

    wilkes Regular member

    Joined:
    May 31, 2003
    Messages:
    922
    Likes Received:
    0
    Trophy Points:
    26
    wilkes, Dec 30, 2003
    #16
  17. Prisoner

    Prisoner Guest

    I have a read a lot of posts between tigre and wilkes. I have learned a lot and not learned a lot. I have just come to the conclusion that even with a proven law on sound by a 4 time Nobel laurate in Physics on the study of Harmonics. I think there will still be debate.
    But I must admit, the banter is funny. I just hope that tigre and wilkes are too different people. the other case has been a thought, to test the waters on sound oppinions.
    Have a happy new year.
     
    Prisoner, Dec 31, 2003
    #17
  18. wilkes

    wilkes Regular member

    Joined:
    May 31, 2003
    Messages:
    922
    Likes Received:
    0
    Trophy Points:
    26
    Just for the record, Tigre is a top chap, and has certainly made me think a lot more. I believe that differing points of view are entirely healthy, especially in an area such as this where so much is still unsubstantiated in the real world.
    Remember too that the biggest areas we have to address are ignorance of the media and apathy. It speaks volumes to me when I realise just how many units the iPod has shifted. The average consumer - at this moment seems to prefer MP3 compression to any other format. I'm not even going to try and debate the whys and wherefores of this, just profess my amazement!
    Happy new year to all on the boards, and a prosperous one too.
     
    wilkes, Jan 1, 2004
    #18
  19. tigre

    tigre Moderator Staff Member

    Joined:
    Feb 18, 2003
    Messages:
    789
    Likes Received:
    0
    Trophy Points:
    26
    maxg wrote:
    The way you've done it you're testing several things at the same time:
    - Performance of A/D conversion you used at different resolutions
    - The capabilities of different resolutions themselves
    - Performance of D/A conversion of your equipment at different resolutions
    - Performance of your analog playback chain (amp, speakers) on music with > 22kHz content (intermodulation distortion etc.)
    - probably some more things I've missed

    So if you hear differences you won't know without further examination what of these things (or what combination) has caused them. Anyway, the results will be an interesting starting point._X_X_X_X_X_[small]AFTERDAWN FORUM RULES: http://forums.afterdawn.com/thread_view.cfm/2487[/small]
     
    Last edited: Jan 1, 2004
    tigre, Jan 1, 2004
    #19
  20. tigre

    tigre Moderator Staff Member

    Joined:
    Feb 18, 2003
    Messages:
    789
    Likes Received:
    0
    Trophy Points:
    26
    wilkes wrote
    Valuable information. Thanks. I've seen quotes from this often, but never the original. I think we've already agreed about the fact that at least a big part of the advantage of higher sampling rates is caused by filter design issues. Obviously a 24bit word length makes a lower noise level possible than 16bit (if dither is used). How much this is audible under real life conditions with good equipment (e.g. no aliasing, intermodulation distortion caused by noise shaped dither's high pitched noise) - and if there's something else audible than noise level related to word lenght - is one of the main purposes of this test.
     
    tigre, Jan 1, 2004
    #20
Page 1 of 2

Share This Page