vaapi_encode: Check config attributes before creating config

This prevents attempts to use unsupported modes, such as low-power
H.264 mode on non-Skylake targets.  Also fixes a crash on invalid
configuration, when trying to destroy an invalid VA config/context.