Skip to content
Snippets Groups Projects
Commit be1bafc3 authored by Ronald S. Bultje's avatar Ronald S. Bultje Committed by Mans Rullgard
Browse files

swscale: fix overflows in output of RGB48 pixels.


For certain types of filters where the intermediate sum of coefficients
can go above the fixed-point equivalent of 1.0 in the middle of a filter,
the sum of a 31-bit calculation can overflow in both directions and can
thus not be represented in a 32-bit signed or unsigned integer. To work
around this, we subtract 0x40000000 from a signed integer base, so that
we're halfway signed/unsigned, which makes it fit even if it overflows.
After the filter finishes, we add the scaled bias back after a shift.

We use the same trick for 16-bit bpc YUV output routines.

Signed-off-by: default avatarMans Rullgard <mans@mansr.com>
parent 5d8122db
No related branches found
No related tags found
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment