Check rc_buffer_size value using integer arithmetic

Using floating-point here can cause erroneous rejection of
parameters due to rounding errors leading to a slightly too
large result.

This fixes the mxf regression test with gcc 4.5 on x86_32.

Originally committed as revision 25060 to svn://svn.ffmpeg.org/ffmpeg/trunk