Return proper error codes and print more descriptive error messages.

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>

Convert doxygen to multiline and express bitfields more simply.

Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>

The rationale is that coded_frame was only used to communicate key_frame,
pict_type and quality to the caller, as well as a few other random fields,
in a non predictable, let alone consistent way.

There was agreement that there was no use case for coded_frame, as it is
a full-sized AVFrame container used for just 2-3 int-sized properties,
which shouldn't even belong into the AVCodecContext in the first place.

The appropriate AVPacket flag can be used instead of key_frame, while
quality is exported with the new AVPacketSideData quality factor.
There is no replacement for the other fields as they were unreliable,
mishandled or just not used at all.

Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>

Allocating coded_frame is what most encoders do anyway, so it makes
sense to always allocate and free it in a single place. Moreover a lot
of encoders freed the frame with av_freep() instead of the correct API
av_frame_free().

This bring uniformity to encoder behaviour and prevents applications
from erroneusly accessing this field when not allocated. Additionally
this helps isolating encoders that export information with coded_frame,
and heavily simplifies its deprecation.

Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>

Partially based on the work of Timo Rothenpieler <timo@rothenpieler.org>

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>