asfenc: fix some possible integer overflows

Store the file duration in the same timebase it arrives (i.e.
milliseconds) and only convert it to the file duration units (100ns)
when it's actually written, thus simplifying some calculations. Also,
store the duration as unsigned, since it cannot be negative.

CC: libav-stable@libav.org
Bug-ID: CVE-2016-2326