Also properly return AVERROR(ERANGE) in case of actual overflows.

Signed-off-by: Marton Balint <cus@passwd.hu>