Fixes: signal_sigabrt_7ffff6ac8cc9_686_cov_1897408623_microsoft_new_way_to_shove_mpeg2_in_asf.dvr_ms

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

32bit is not sufficient for all cases

Fixes: signal_sigabrt_7ffff6ac8cc9_686_cov_1897408623_microsoft_new_way_to_shove_mpeg2_in_asf.dvr_ms

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: signal_sigabrt_7ffff6ac8cc9_686_cov_1897408623_microsoft_new_way_to_shove_mpeg2_in_asf.dvr_ms

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes: asan_heap-oob_26f6853_862_cov_585961513_sonic3dblast_intro-partial.avi

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Change-Id: I2f391ae912a079fb32f0703841dca86358aac72a
Signed-off-by: ZhouXiaoyong <zhouxiaoyong@loongson.cn>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

These should be re enabled once the issue is fixed

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

This commit reorders the coding tools such that they're doing what
the decoder does in reverse order. The very first thing the decoder
does is to decode M/S stereo if that's signalled, then prediction,
IS, and finally TNS and PNS in another function.
adjust_frame_information()'s application of IS and M/S was taken
out into two separate functions since prediction doesn't expect
to get the raw coefficients but rathe the coefficients at that
part of the encoding process.

The results show a much better PSNR when any combination of
Intensity Stereo, Mid/Side stereo and Prediction is used, which
is a sign of an increased encoder efficiency as well as the fact
that the decoder gets what it expects.

Otherwise, with only IS, PNS or prediction there are neither
regressions nor improvements except in the case of IS, which
now by itself (or with PNS) is less prone to artifacts. Enabling
M/S (using stereo_mode) as well will also reduce stereo artifacts
induced by IS, so in the very near future M/S may be enabled
by default.

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes assertion failure
Fixes: c6075771557e4f3b7b74e63d2d24fb01/signal_sigabrt_7ffff6ac8cc9_133_cov_2853689970_CREDITS.FST

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Found-by: ubitux
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

If the selected coder isn't twoloop, this commit temporarily
disables IS and PNS.
The problem is in the encode_window_bands_info() being confused
and setting invalid band_types for non-marked (normal) bands.

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>

Since the changes made a few week ago (which were done more than a
month ago) the quality and stability of intensity stereo has been
notably good. There were some requests and wishes to have in on by
default and therefore it has been enabled. Should any regressions
arise changes will be made to preferably keep it operating rather
than just disabling it by default again.

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>

It has been in the current encoder in its current implementation
for quite some time now, so enable it by default. Will increase
quality at all bitrates, especially at low ones.

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>

Purely a cosmetic change, most of the zeroing of encoder resources
should happen at the top of the main loop.

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>

This commit reworks the TNS implementation to a hybrid between what
the specifications say, what the decoder does and what's the best
thing to do.

The filter application function was copied from the decoder and
modified such that it applies the inverse AR filter to the
coefficients. The LPC coefficients themselves are fed into the
same quantization expression that the specifications say should
be used however further processing is not done, instead they're
converted to the form that the decoder expects them to be in
and are sent off to the compute_lpc_coeffs function exactly the
way the decoder does. This function does all conversions and will
return the exact coefficients that the decoder will generate, which
are then applied to the coefficients.
Having the exact same coefficients on both the encoder and decoder
is a must since otherwise the entire sfb's over which the filter
is applied will be attenuated.

Despite this major rework, TNS might not work fine on some audio
types at very low bitrates (e.g. sub 90kbps) as it can attenuate
some coefficients too much. Users are advised to experiment with
TNS at higher bitrates if they wish to use this tool or simply
wait for the implementation to be improved.

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>

Turns out autocorrelating more than 750 coefficients at once
will cause a segfault, despite there being enough space to
hold an entire frame of samples into the buffer.

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>

This commit adds a function to get the reflection coefficients on
floating point samples. It's functionally identical to
ff_lpc_calc_ref_coefs() except it works on float samples and will
return the global prediction gain. The Welch window implementation
which is more optimized works only on int32_t samples so a slower
generic expression was used.

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>

Not needed anymore, it was only used by the AAC TNS
encoder and was replaced with a more suitable function
in the following commit.

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>

Needed for following commits. Contains the starting sfb for
every samplerate and window type.

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>

Needed for the following commits.

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>

Fixes out of array access
Fixes: 87196d8bbc633629fc9dd851fce73e70/asan_heap-oob_26f6853_862_cov_585961513_sonic3dblast_intro-partial.avi

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes assertion failure
Fixes: 56dcafde14a8397161bb61a16c511179/signal_sigabrt_7ffff6ac8cc9_686_cov_1897408623_microsoft_new_way_to_shove_mpeg2_in_asf.dvr_ms

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes memleak
Fixes: efe937780e95574250dabe07151bdc23/unknown_unknown_351_849_cov_3187578556_shellymanne.mov

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Fixes integer overflow
Fixes: efe937780e95574250dabe07151bdc23/unknown_unknown_351_849_cov_3187578556_shellymanne.mov

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Donny Yang <work@kota.moe>

Reviewed-by: Ivan Uskov <ivan.uskov@nablet.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

It does not need to be non static

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

This was copied from the decoder, but is unneeded for the encoder.
tns_max_bands is unused and set to zero which zeroed out start, end
and size and thus no filter was actually applied.

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>

Since the coefficients are stepped up to order + 1 it was possible
that it went over TNS_MAX_ORDER. Also just return in case the only
coefficient is less than the threshold.

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>

The encoder-side filter isn't that important. The PSNR
shouldn't change so the FATE test should still be fine.

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>

The order should never go above TNS_MAX_ORDER (and thus cause
the context to be reinitialized) but this is just in case.

Also fix a comparison, since the coefficients are zero-indexed.

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>

It also made no sense to actually make the filter span the entire
window including the first band of the next window.

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>

Pulses are already on the way so expect to see the list
gone in the close future.

TNS is already of sufficiently high quality to be enabled
by default (but isn't yet, so you too can help by testing!).

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>

This commit abandons the way the specifications state to
quantize the coefficients, makes use of the new LPC float
functions and is much better.

The original way of converting non-normalized float samples
to int32_t which out LPC system expects was wrong and it was
wrong to assume the coefficients that are generated are also
valid. It was essentially a full garbage-in, garbage-out
system and it definitely shows when looking at spectrals
and listening. The high frequencies were very overattenuated.
The new LPC function performs the analysis directly.

The specifications state to quantize the coefficients into
four bit index values using an asin() function which of course
had to have ugly ternary operators because the function turns
negative if the coefficients are negative which when encoding
causes invalid bitstream to get generated.

This deviates from this by using the direct TNS tables, which
are fairly small since you only have 4 bits at most for index
values. The LPC values are directly quantized against the tables
and are then used to perform filtering after the requantization,
which simply fetches the array values.

The end result is that TNS works much better now and doesn't
attenuate anything but the actual signal, e.g. TNS removes
quantization errors and does it's job correctly now.

It might be enabled by default soon since it doesn't hurt and
helps reduce nastyness at low bitrates.

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>

This commit removes the array which was made redundant with
the last commit. The current prediction system gets the
quantization error directly (and without the single-frame delay)
in the search_for_pred function.

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>