avcodec/nvenc: use AVERROR_BUFFER_TOO_SMALL instead of ENOBUFS

Should fix compilation with mingw32

Reviewed-by: Timo Rothenpieler <timo@rothenpieler.org>
Signed-off-by: James Almer <jamrial@gmail.com>