proresenc: Report buffer overflow

If the allocated size, despite best efforts, is too small, exit
with the appropriate error.

CC: libav-stable@libav.org

Signed-off-by: Diego Biurrun <diego@biurrun.de>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>