win32: Make ff_win32_open more robust

- Make MultiByteToWideChar fail when it encounters invalid encoding.
  Without this, invalid characters might just be skipped
- When MultiByteToWideChar fails, assume the file name is in CP_ACP
  and open it via normal open function, even when the file will be
  written
- When malloc fails return error instead of crashing

Signed-off-by: Martin Storsjö <martin@martin.st>