Fix broken 32-bit clipping, and write numbers in hex instead of decimal so

they are easier to understand. Also give the add a 'u' postfix to silence
a pre-c99 compiler warning.

Originally committed as revision 22965 to svn://svn.ffmpeg.org/ffmpeg/trunk