Python 3.11 needs to install tokenizers from source

[lukas-blecher]

Discussed in #272

^{Originally posted by IlMancios May 13, 2023}
I'm new to this type of coding and practically a noob:

1. installed visual studio code
2. installed phyton
3. installed pytorch (it function)
4. tried to install pix2text with this command:
   pip install pix2tex[gui]

but I got this error in the output:

...copying py_src\tokenizers\trainers\__init__.pyi -> build\lib.win-amd64-cpython-311\tokenizers\trainers
      copying py_src\tokenizers\tools\visualizer-styles.css -> build\lib.win-amd64-cpython-311\tokenizers\tools
      running build_ext
      running build_rust
      error: can't find Rust compiler

      If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.

      To update pip, run:
          pip install --upgrade pip
      and then retry package installation.

      If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.      
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects'.

So I upgraded the pip version but nothing changed, then I installed Rust (on my PC) and rust-analyzer on VSC, but the output did not change.

Even tried to correctly set the path (in the environment variables) for rust but nothing happens, the problem is with the rust installation in fact if i try:

rustc --version
rustc: The term 'rustc' is not recognized as a name of a cmdlet, function, script file, or executable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

What is the problem? I only know that it is a stupid one!

[lukas-blecher]

Fixed in 511fc43