qdm2: check array index before use, fix out of array accesses

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

(cherry picked from commit a7ee6281)

CC: libav-stable@libav.org

Signed-off-by: Reinhard Tartler <siretart@tauware.de>