Make channel layout masks unsigned

It makes more sense for a bit mask to use an unsigned type.
The change should be source and binary compatible on all
supported systems, hence micro version bump.

Fixes a few invalid shifts.

Signed-off-by: Mans Rullgard <mans@mansr.com>