Mask all unused bits for packed pixel format instead of green and alpha mask only.

That fixes the case when converting 15-bit RGB/BGR to YUV and high bit is set
for input value(s).

Originally committed as revision 28916 to svn://svn.mplayerhq.hu/mplayer/trunk/libswscale