Ensure that our total reference frame count does not exceed the SPS

max frame count, which is limited to less than the size of the
reference buffers, thereby preventing overflow.
Part of fix for issue 281.

Originally committed as revision 11216 to svn://svn.ffmpeg.org/ffmpeg/trunk