This allows removing a special case for the fixed point decoder and will
make error checks simpler

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>