Increase max resync size, fix demuxing of dvgrab-2009.03.28_19-07-22.m2t

Print error when max resync size is reached and return EAGAIN instead of
INVALIDDATA, so user can retry if wanted.

Originally committed as revision 20286 to svn://svn.ffmpeg.org/ffmpeg/trunk