use _wfopen instead of fopen on Windows

[cebtenzzre]

This PR uses _wfopen() instead of fopen() on Windows, to make it more convenient for downstream projects to use llama.cpp with UTF-8 paths.

(This doesn't fix llama_model_quantize, which uses std::ofstream, but AFAIK there is no way to open a std::ofstream with a wchar_t path on MinGW, and this should only really be used by the quantize example anyway.)

Supersedes #5927

Tested with llama-cpp-python. Without this change:

>>> from llama_cpp import Llama
>>> path = b'\xF0\x9F\x98\x8A'.decode()  # '😊'
>>> llm = Llama(model_path=path, verbose=True)
llama_model_load: error loading model: failed to open 😊: No such file or directory
llama_load_model_from_file: failed to load model
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\msys64\home\Jared\llama-cpp-python\llama_cpp\llama.py", line 314, in __init__
    self._model = _LlamaModel(
                  ^^^^^^^^^^^^
  File "C:\msys64\home\Jared\llama-cpp-python\llama_cpp\_internals.py", line 55, in __init__
    raise ValueError(f"Failed to load model from file: {path_model}")
ValueError: Failed to load model from file: 😊

With this change, the model loads successfully.

---

See the Wikipedia page Unicode in Microsoft Windows for historical context that explains why Unicode is such a headache on Windows.

On recent versions of Windows, one can set the "ANSI" code page used by the *A functions¹ to UTF-8 for the current process using setlocale(), for an executable or DLL using a manifest, or system-wide. However, for llama.cpp to be a portable and easy-to-use library, it shouldn't rely on this internally.

The approach used in this PR will work regardless of the above. The standard C wcstombs() depends on the process locale as set by setlocale(), but since this translation is only necessary on Windows, this PR directly uses MultiByteToWideChar with CP_UTF8.

NOTE: When the system code page is not UTF-8, the examples and tests will still fail on Unicode paths. The simplest fix for this that will also work on MinGW is for all of them to call setlocale(), and to use wmain or GetCommandLineW to get argv in UTF-8. If there is interest, this can be implemented in a future PR.

Footnotes

1. There is also the "OEM" code page used by console functions such as ReadConsoleA/WriteConsoleA - this is what chcp changes ↩

[cebtenzzre]

A related problem is that this doesn't work correctly on MSVC:

llama.cpp/llama.cpp

Lines 3974 to 3976 in 50ccaf5

	const std::vector<int> ids = llama_tokenize_internal(vocab, "\u010A", false);
	GGML_ASSERT(!ids.empty() && "model vocab missing newline token");
	vocab.linefeed_id = ids[0];

This warning is important:

  C:\msys64\home\Jared\llama-cpp-python\vendor\llama.cpp\llama.cpp(3974,69): warning C4566: character represented by universal-character-name '\u010A' cannot be represented in the current code page (1252) [C:\Users\Jared\AppData\Local\Temp\tmpapjgk6fk\build\vendor\llama.cpp\llama.vcxproj

MSVC uses the system ANSI code page to encode char* string literals by default, and Windows-1252 does not contain Ċ. This can be fixed by using at least /execution-charset:utf-8. Alternatively, we could explicitly encode the UTF-8 representation in the source using \x - this would be "\xC4\x8A".

[ggerganov]

Alternatively, we could explicitly encode the UTF-8 representation in the source using \x - this would be "\xC4\x8A".

Sounds good