Tech Tip: Handbrake part 2

After over a hundred hours of testing, retesting, sleeping, then testing some more, I've finally come up with a configuration for Handbrake that I'm ready to share.  In the interest of full disclosure, however, I will say that I'm not entirely happy with the settings that I've found but I'll explain more about that later.

And, without much more ado, I shall begin part 2 of my Handbrake guide.

CONFIGURATION

Once again, I'll skip to the end and provide the settings I'm talking about before actually explaining them any.  The Encoder Options I'm suggesting are as follows:

aq-mode=2:bframes=5:b-adapt=2:direct=auto:me=umh:subme=9:merange=24:analyse=all:trellis=0

This has a few variations from my previous recommendation, with the first and most obvious being the complete lack of any AQ or Psy-RD strength tinkering.  Both of these settings I feel have some serious merit, but the problem I have with them is that you are forced to choose between how much "bad" you want - there doesn't really seem to be a "good" option without using some seriously large amounts of space and time.  The good news is I have found a way to work with those limits.

The other changes are perhaps less obvious but still important:

AQ-Mode 2, despite being considered as "experimental," doesn't really appear to affect the encoding time all that much and generally has a positive effect on the results, which is why it's being used.  This, at least in my opinion, makes it a good thing to use.

RC Lookahead has also been removed as an override going from the 10 I had before (which, in hindsight, I have no idea where it came from) to use the default 40.  This has a slightly negative effect on the encoding time but should result in a similar benefit to the results.  An acceptable trade-off.

Subme is now set to 9, instead of the default 7.  If you really want to eek out every last bit of speed than you can set it back to 7 but that's probably the lowest I'd go.  My experience suggests it doesn't make a massive difference to performance but it does result in a noticeable increase in quality on the output.

And, lastly, Trellis is turned off.  This setting probably has the biggest impact on the encode speed, cutting around 25% off in my experience without any significant negative effect on the output.  If having it turned off worries you than you may consider setting it to 1 (the default) but the benefits are negligible at best.

To go with these, I would also recommend using Loose anamorphic.  I've had a chance to really play around with it now and, unless you have a specific reason not to use it, it's probably a good idea to have it.  Encoding time and quality seem to be completely unaffected but you shave a few percent off the resulting filesize - and that's never a bad thing.

THE NEXT STEP

I'd be lying if I said the above is the settings that I'm using.  Those are very good settings but they're more of a compromise between speed and quality.  If you've got a faster machine or a lot of time (both of which I have) than you might consider the following:

no-fast-pskip=1:aq-mode=2:bframes=6:b-adapt=2:direct=auto:me=umh:subme=10:merange=24:analyse=all:trellis=2:no-dct-decimate=1:mbtree=0

The quick rundown of these settings in comparison is basically an increase in the Subme method (which requires Trellis to be set to 2), adding an extra B frame (my testing suggests 5 is the useful limit, so +1 should account for almost every situation) and disabling some of the speed and space optimizations in order to produce a more "true" picture.

These settings take longer.  A lot longer, in some cases - between 30% and 50% longer.  Generally they also won't result in a smaller file, either, assuming you leave the CRF the same.  What these settings do, however, is help combat one of the biggest flaws (as I see it) with how Handbrake handles x264 encoding: Banding.

Banding is the issue I mentioned in the previous article which affects "dark" areas.  As it turns out, it affects any area where there's only subtle gradients - it doesn't have to be dark, but it's more likely to occur when it's dark.  Banding basically occurs because of the "low" colour precision used in Handbrake without any kind of dithering being applied (dithering reduces compression potential) and the more you compress your source the worse it becomes.

These settings, unfortunately, are not a panacea when it comes to banding.  They reduce it but they don't eliminate it.  The only way to eliminate it without massively increasing the data rate (thus filesize) would be to introduce dithering or switch to a higher precision storage method (such as 10-bit x264, which is currently not an option within Handbrake and similar - mostly because very few devices outside of desktop PCs are capable of playing it back).

COMBATING BANDING

The above, however, doesn't mean that you have to accept banding for what it is if you want smaller files.  There are ways around it - some of which are very easy (but limited in their application) and some of which I'm still looking for a good way to do them.

The easy way is to use FFDShow.  This is a free codec for desktop PCs that can decode pretty much every format you throw at it - including x264.  It also includes a number of filters which can be applied on-the-fly - the one relevant to this discussion being the Deband filter.  When it's enabled it can work wonders, though you may want to play around with the threshold a little bit to get optimal results (I'm rather enjoying a threshold of 1.5 while I find a better alternative).

The more difficult way is to pre-process the video.  There are a number of AviSynth scripts available that introduce either light noise or dithering to a video which in turn causes Handbrake (and similar transcoders) to more appropriately preserve smooth gradients.  This naturally comes with an increase to filesize but it does mean that it will work equally well on a desktop PC or a stand-alone media player as nothing has to be done in post.  Unfortunately, Handbrake doesn't parse AviSynth scripts so this is fairly moot without another tool.

Unfortunately, I'm yet to find a suitably easy enough tool to do this processing.  I've been playing around with a couple (such as MeGUI) but ultimately it would be a lot easier if the Handbrake authors would implement a dithering filter based on something like GradFun3.

If anyone finds a more appropriate solution, please drop me a line - either through comments or e-mail.