avserver: Remove unused loadable module support

There is no record of this ever being used at all, anywhere,
since the feature was added in 2effd274.

This gets rid of extra linker tricks just to support a feature
that isn't used, simplifying portability to other platforms.

Signed-off-by: Martin Storsjö <martin@martin.st>