avcodec/dxtory: fix src size checks

Fixes integer overflow
Fixes out of array read
Fixes: d104661bb59b202df7671fb19a00ca6c-asan_heap-oob_d6429d_5066_cov_1729501105_dxtory_mic.avi
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>