correct the custom coding mode alphabet, add some validation on the

motion vectors (which are not completely correct yet)

Originally committed as revision 1844 to svn://svn.ffmpeg.org/ffmpeg/trunk