lavc/dvdsubenc: improve color distance function.

Consider the color space as an hypercone with apex alpha=0
and base alpha=1 instead of an hypercube.
Make the encoder consider very transparent colors more similar
even if the hue is very different.
This corresponds roughly to using the alpha as a weight for the
color difference.
Only 4 bits of alpha are used, because this is what dvdsub uses,
and it avoids overflows.

Fix trac ticket #2005.