fix b frames + -mbd rd without -psnr (yes that was seriously buggy and...

fix b frames + -mbd rd without -psnr (yes that was seriously buggy and probably hasnt been noticed as everyone who tested stuff used -psnr)

Originally committed as revision 7148 to svn://svn.ffmpeg.org/ffmpeg/trunk