avcodec/videotoolboxenc: fix undefined behavior with rc_max_rate=0

On macOS, a zero rc_max_rate cause an error from
VTSessionSetProperty(kVTCompressionPropertyKey_DataRateLimits).

on iOS (depending on device/version), a zero rc_max_rate cause invalid
arguments from the vtenc_output_callback after few frames and then a crash
within the VideoToolbox library.

Signed-off-by: Aman Gupta <aman@tmm1.net>