The destination filter might expect the hw frames context to be already
set (this is the case e.g. for hwdownload).