Problem With Hitting The Targetsize With Xvid, Help needed please

Discussion in 'DivX / XviD' started by Plamen234, Apr 2, 2006.

  1. Plamen234

    Plamen234 Guest

    The situation:
    I want to backup my dvd9 movies in AVI Xvid format at very high bitrate on a 4.36GB DVD-R (dvd5)

    The procedure:
    1. dvddecrypter
    2. dvdshrink - to remove the unwanted material
    3. enconding in Xvid (targeting the 4.36GB avi)
    4. adding some subtitles in a separate .srt files

    Results: I actually succeeded in doing it twice - once in xvid and 2nd in DviX 6.1.1
    The movie: Heat (1995), Time=2h44min, frames 245382, pal 25.00, resolution 720x576 16:9
    Size in dvd = 6.36GB (only the movie with the ac3 audio, no menus nor other unnecessary stuff)

    Using dgmpgdec146 DGIndex: Heat-DGIndex.d2v (demux all tracks, honor pulldown flags)
    AVIsynth: Heat.avs
    LoadPlugin("D:\Programs\dgmpgdec146\DGDecode.dll")
    MPEG2Source("Heat-DGIndex.d2v")

    VirtualDub (original v1.6.11)
    No Audio
    Full Processing Mode
    Filters: Resize (720, 404)
    Nul Transform: Crop (12, 54, -4, -52)
    Video Compresion: variant1 Xvid encoding, Load Defaults, unconstrained profile, MPEG Quant, 2pass, Motion Search precision: 6-ultra-high, VHQ: 4-wide search, crhoma motion: ON, targetbitrate=3330kbps
    Resulting filesize: as expected, +ac3 audio -> fits perfectly on a 4.36GB dvd-r

    Video Compression: variant2 DivX 6.1.1 pro, High def profile, Encoding: Insane Quality, B-frames=OFF, Noise reduction=OFF, PVE=OFF, Quant=MPEG, bitrate=3400kbps
    Resulting filesize: as expected, +ac3 audio -> fits perfectly on a 4.36GB dvd-r (2MB smaller than Xvid)

    Considering the very wide dynamic range of the original ac3 audio on the dvd, I decided to add a second audio stream (with virtualdubmod) in mp3 CBR stereo format with corrected dynamic range and normalization. This demanded a new video compression at a bitrate about 2950kbps according DVtool. So I calculated the new video size like that
    dvd-r (4488MB) -18MB (for not reaching the outer edge) = 4470MB total (video+ac3+mp3+srt+overhead)
    4470 - 462MB (ac3 audio) - 188MB (mp3 2nd audio stream) = 3820MB
    So I chose 3800MB for the compressed video portion, leaving the rest 20MB for overhead and codec variation.
    Here I chose the Xvid codec for the final backup.

    The PROBLEM: the resulting AVI file with only the compressed video is with TOTALLY unpredicted size, exactly 4.7GB!!! Adding the ac3 stream and then the 2nd mp3 gives a 5GB AVI. Trying (other 12 times) with different Xvid options (with and without b-frames/ Q-pel, profile unconstrained / ASP@L5) the video portion varies from 4.4GB to 6.1GB - completely unsuitable for my intention.

    This is what I did the last time:
    1. removed the Divx 6.1.1 codec and the ffdshow-20051221-gcc4.0.2-sse-x264.nl.exe and restarted the PC. I haven't installed on my windows any codecs packs such as nimo, ace and so on - I use only the codec I need, XviD, DivX and ffdshow for x264 playback
    2. used the same Heat-DGIndex.d2v file
    3. this AVIsynth script (AVIsynth 2.56)
    LoadPlugin("D:\Programs\dgmpgdec146\DGDecode.dll")
    MPEG2Source("Heat-DGIndex.d2v")
    crop(12,76,-4,-74)
    Lanczos4Resize(704,304)
    4. VirtualDub v1.6.14 original stable
    4.1. no Audio
    4.2. Fast Recompress
    4.3. Video Compression - Xvid video codec, 20:18:22 Dec-30-2005, xvidcore.dll ver1.1.0 final, binary Koepi's vanilla CVS Build)
    4.3.1. Load defaulds for Xvid
    4.3.2. Profile@Level: ASP@L5, Quantanization Type=MPEG, Q-pel=ON
    4.3.3. Encoding Type = Twopass-first-pass (Full quality first pass = OFF; Discard first pass)
    4.3.4. Quality presets: user defined -> Motion: Motion Precision=6-ultra-high, VHQ=4-wide-search, use VHQ for bi-frames=ON, use chroma motion=ON)
    5. Save: Heat-Xvid-704-304-1pass.avi (=6MB) (added in pending jobs)
    6. video Compression: Xvid
    Encoding Type: Twopass-2nd-pass
    Target Size: 3893122 kbytes (=3800MB + overhead)
    7. Save: Heat-Xvid-704-304-2pass.avi (the desired one) added in jobs
    8. finally: Job control -> start

    So, please, tell me what's wrong that gives me such an unpredicted filesize???
    Any help is appreciated and quite welcome.

    p.s. Sorry about the long post, but tried to describe what I do and what I get.
     
  2. celtic_d

    celtic_d Regular member

    Joined:
    Jan 23, 2005
    Messages:
    3,352
    Likes Received:
    0
    Trophy Points:
    46
    Default settings means that quant 1's are allowed. Now if your first pass is smaller than the second, since the first pass is constant quant 2, XviD has to insert quant 1's to hit the size. With the default overflow settings this can mean oversized encodes. With the old XviD settings of 2-31 for quants it meant undersized encodes (the reason for the change).

    Not a good idea to have a 2nd pass larger than the first anyway since q1 is basically just padding. A higher quality quant matrix can help.

    So what is the size of your first pass? I guess using AVISynth and therefor avoiding converting to RGB24 could increase compression and drop the 1st pass size for your 2nd encode attempt? Seems a little unlikely, however larger 1st pass with default overflow settings and q1 is the usual cause for oversized XviD encodes.

    Leaving the encode anamorphic rather than resizing to 1:1 would also increase quality.
     
  3. Plamen234

    Plamen234 Guest

    Thank you for helping me

    (I'm still stack on the same movie)
    Last time I used these settings for the 1st pass and got a 6MB 1st-pass.avi file - this is very strange
    "Full quality first pass" = ON (checked)
    and "Discard first pass" = ON (checked)

    And the quantizer restrictions, loaded as defaults and unchanged for both passes, are
    Min I-frame quantizer=1 max=31
    Min P-frame quantizer=1 max=31
    Min B-frame quantizer=1 max=31

    The projected 2nd-pass file size is 4384MB on 44% (still encoding)

    As I see, your suggestion is to set the min quant values on 2, right?

    ... and the final size as predicted is 4.37GB, much above the desired 3.71GB

    Here are some more details from the XviD status window (after 2nd-pass encoding)
    Average bitrate=3822kbps
    I-vop: min quant=1 max=2 average=2.04
    P-vop: min quant=1 max=4 avg=1.44
    B-vop: min quant=2 max=7 avg=2.82
    Total: min quant=1 max=7 avg=2.19
     
    Last edited by a moderator: Apr 3, 2006
  4. celtic_d

    celtic_d Regular member

    Joined:
    Jan 23, 2005
    Messages:
    3,352
    Likes Received:
    0
    Trophy Points:
    46
    You need to look at the stats file to see how big the 1st pass would have been had you not discarded it.

    For the first pass you have I/P = Q2, B = Q4, so looking at your averages it would appear that your first pass was smaller. Capping the quants to 2 should therefor fix your oversize problem. You will just get an undersized output, which doesn't really matter since as I said Q1's are basically just padding. Encoding with full vertical resolution or using a higher quality CQM will give more quality gains than Q1's.
     
  5. Plamen234

    Plamen234 Guest

    Thank you for your attention.

    I'll do the encoding once again with "Discard first pass" = OFF (unchecked) to see that size.

    The resolution: original = 720x576 16:9 pal
    720/16=45*9=405 so I chose 720x404 before cutting the black fields
    croping them gives 704x298 #Crop (12, 54, -4, -52)

    About quants, is it better to leave the min value =1 for I-vops, and change the min=2 for the p/b ones?

    About CQM, in Koepi's Xvid binary pack v1.1.0 final there are 3 that catch my eye (hvs-best-picture.txt, MPEG.txt and Ultimate Matrix.txt), any suggestions, please?

    p.s.: with this particular version of xvid the value of compressibility test is 0.607 in Gknot which would give a size =3750MB (just 50MB under my target size =3800MB)
     
  6. celtic_d

    celtic_d Regular member

    Joined:
    Jan 23, 2005
    Messages:
    3,352
    Likes Received:
    0
    Trophy Points:
    46
    No need to re-encode. As I said the stats file contains info on how big the first pass was.

    I meant, crop, but don't resize (set the PAR accordingly so it plays at the correct resolution). Resizing like that just reduces vertical resolution. You might want to try one of Sharktooth's matricies.
     
  7. Plamen234

    Plamen234 Guest

    I can't the size in the stats file, it contains only (I hope the mistake is in me)
    # XviD 2pass stat file (core version 1.1.0)
    # Please do not modify this file

    i 2 836 0 0 2367 2367
    b 4 0 836 0 8 8
    b 4 0 836 0 8 8
    p 2 0 0 836 118 118
    b 4 0 836 0 8 8
    ... and so on.
    but anyway, I'll do a full quality twopass 1st-past at Q=2

    Ok, your advice is (PAR 16:9) vs (resize+PAR 1:1) - thank you (but the movies look unnatural on PC)

    I'm reading the material about Sharktooth's marices. The EHR one is said to be used for single pass encoding, so maybe I will use UHR or HR. Can a movie encoded with HR or UHR be played nornaly on a SAP?
     
  8. celtic_d

    celtic_d Regular member

    Joined:
    Jan 23, 2005
    Messages:
    3,352
    Likes Received:
    0
    Trophy Points:
    46
    Depends on the SAP. Try a test clip before wasting time encoding. If it looks weird then you don't have playback setup correctly because it isn't detecting the PAR and resizing like it should.

    Open the file with stats reader.

    The PAR for a 16:9 DAR won't be 16:9 by the way, so that could explain why it doesn't look right to. Also your DAR would be 2.35:1 since you are cropping. Should give you something like 704x432 with a PAR of 199:138.
     
  9. Plamen234

    Plamen234 Guest

    In fact I found these values in Heat-DGIndex-GK_firstpass.settings
    "desired_size"=dword:003ec85d (=4114525KB =4018MB)
    "target_size"=dword:0045d800 (=4577280KB =4470MB)

    and in Heat-DGIndex-GK_secondpass.settings
    "desired_size"=dword:003b6382 (=3892098KB =3800MB)
    "target_size"=dword:003b6000 (=3891200KB =3800MB)

    so the 2nd-pass file is smaller than the 1st-pass

    but what I got last was a 4470MB file (after the 2nd-pass)
     
  10. Plamen234

    Plamen234 Guest

    Thank you for your attention.

    In statsreader the size is 1490MB. Actually I saved all my .pass files (saved from vDub)and the size varies from 1401MB to 2187MB accoring to statsreader.

    After croping I get the resolution 704x426. crop(12,76,-4,-74)

    Can I Xvid encode a movie in vDub-original and a 20min clip from the same movie in vDubMod without bad interderence? (using the same .avs with DGDecode.dll loaded)
     

Share This Page