Change variable types from int to enum PixelFormat.

Fixes icc warning #188: enumerated type mixed with another type

Originally committed as revision 27727 to svn://svn.mplayerhq.hu/mplayer/trunk/libswscale