avcodec/utils: report insane channel count errors

More than 64 is not *that* insane, so let's report the error at least.

Signed-off-by: Marton Balint <cus@passwd.hu>