Windows (potentially transferable to other OSs) - dynamically linking and wrapper for virtual python environment or standalone environment on mainly Windows

[phlpphns]

Environment

Windows 11 (WSL ubuntu 24.04 for comparisons, see also here: #4841 (comment))

What I aspire to do:

I would like to ship an exe for windows that contains some custom code and takes the rest of the to be imported libraries from a virtual environment. I planned to have it dynamically linked and hoped that I then could "simply" provide a suited python environment; either an existing one or by means of an installation routine. I would prefer to be able to create an environment that is "separate from the rest of the system".

1. Create an executable with a dynamically linked python interpteter.
2. Set up a virtual environment upon first use of the distributed exe file or download and activate a fresh, isolated python installation.
3. Getting it to run with my shipped executable (I at least struggle with setting the path variables right, I guess).

At this point, I wonder if it is actually even a venv or rather a complete, separated, minimal python installation what I am searching for to be used. Initially, I wanted to have all necessay parts embedded in one file (or so), but for shipping and updating, e.g. utilizing the PyOxidizer project. - In the meanwhile, it appears to be more attractive to me to have only my main code included that I'll change and update, while numpy, pandas, etc would stay with the python installation on the respective machine. That means that the exe itself could easily be updated and small sized (+ I also want to understand what's happening under the hood of the linking process).

setting up venv-win on windows

https://github.com/pyenv-win/pyenv-win
(I downloaded a zip file and placed it as a "standalone").

[System.Environment]::SetEnvironmentVariable('PYENV', "C:\Users\pkv190\Desktop\py_env_win_tests\.pyenv\pyenv-win\","User")
[System.Environment]::SetEnvironmentVariable('PYENV_HOME
', "C:\Users\pkv190\Desktop\py_env_win_tests\.pyenv\pyenv-win\","User")
[System.Environment]::SetEnvironmentVariable('PYENV_ROOT', "C:\Users\pkv190\Desktop\py_env_win_tests\.pyenv\pyenv-win\","User")

[System.Environment]::SetEnvironmentVariable('path', "C:\Users\pkv190\Desktop\py_env_win_tests\.pyenv\pyenv-win\bin;" + "C:\Users\pkv190\Desktop\py_env_win_tests\.pyenv\pyenv-win\shims;" + [System.Environment]::GetEnvironmentVariable('path', "User"),"User")

python -m venv .venv
.venv\Scripts\activate

setting up conda

Installed conda for the current User having it as the standard python interpreter.
creating and activating according to e.g.

conda create -n py_310 python=3.10
conda activate base
conda activate py_310

Rust code

The following code (runs on windows only, I found part of it at #2338, #1554) is able to detect even multiple conda environments but upon run time we cannot change it.

Please note that the function initialize_python(); is essential to be able to access the different conda environments I have (base, py_310), but will fail with the system's standard python interpreter and vice versa. It will not work on linux because there one needs to return a 32bit pointer.

use std::io;

use pyo3::ffi::c_str;
use pyo3::prelude::*;

use which;

pub fn initialize_python() -> PyResult<()> {
    // Due to https://github.com/ContinuumIO/anaconda-issues/issues/11439,
    // we first need to set PYTHONHOME. To do so, we will look for whatever
    // directory on PATH currently has python.exe.
    let python_exe = which::which("python").unwrap();
    let python_home = python_exe.parent().unwrap();

    // The Python C API uses null-terminated UTF-16 strings, so we need to
    // encode the path into that format here.
    // We could use the Windows FFI modules provided in the standard library,
    // but we want this to work cross-platform, so we do things more manually.
    let mut python_home = python_home
        .to_str()
        .unwrap()
        .encode_utf16()
        .collect::<Vec<u16>>();
    python_home.push(0);
    unsafe {
        pyo3::ffi::Py_SetPythonHome(python_home.as_ptr());
    }

    // Once we've set the configuration we need, we can go on and manually
    // initialize PyO3.
    pyo3::prepare_freethreaded_python();

    Ok(())
}


#[pyfunction]
fn add_one(x: i64) -> i64 {
    let y = x + 1;
    println!("{y}");
    y
}

#[pyfunction]
fn print_my_name() -> i64 {
    println!("Hallo, my name is Philipp. :-)");
    let d = 4;
    println!("The value is {d}.");
    d
}

#[pymodule]
fn foo(foo_module: &Bound<'_, PyModule>) -> PyResult<()> {
    foo_module.add_function(wrap_pyfunction!(add_one, foo_module)?)?;
    Ok(())
}

#[pymodule]
fn phil(phil_module: &Bound<'_, PyModule>) -> PyResult<()> {
    phil_module.add_function(wrap_pyfunction!(print_my_name, phil_module)?)?;
    Ok(())
}

fn main() -> PyResult<()> {
    println!("the program starts.");

    pyo3::append_to_inittab!(foo);
    pyo3::append_to_inittab!(phil);

    let _ = initialize_python();

    // test_py();

    // pyo3::prepare_freethreaded_python();
    let _ = Python::with_gil(|py| py.run(pyo3::ffi::c_str!("print('Hello World')"), None, None));

    let _ =
        Python::with_gil(|py| Python::run(py, c_str!("import foo; foo.add_one(6)"), None, None));
    let _ = Python::with_gil(|py| {
        Python::run(py, c_str!("import phil; phil.print_my_name()"), None, None)
    });

    let _ = Python::with_gil(|py| Python::run(py, c_str!("import sys; print(sys.version)"), None, None));

    println!("Going to wait...");
    io::stdin().read_line(&mut String::new()).unwrap();

    Ok(())
}

Related questions and asking for support:

Connected with that, I now have a couple of problems/questions and search for advice:

1. In the pyo3-documentation (https://pyo3.rs/main/building-and-distribution) it's written

For distributing your program to non-technical users, you will have to consider including the Python shared library in your distribution as well as setting up wrapper scripts to set the right environment variables (such as LD_LIBRARY_PATH on UNIX, or PATH on Windows).

But what does this mean? - How would a wrapper script (or unsafe [?] code) to change the system's environment variables to correct locations look like?

I considered "emptying" the complete path and repopulating it with my environment and the exe related directories only. One idea could also be using an .env file; I could not accomplish that on windows yet.

2. How can I build the executable with pyO3 and use any python interpreter afterwards? Is there some "interpreter version lock"? <- It might be that different systems I want to ship the exe to offer different python versions. I wrote about some insights I got/tests I did in PYO3_PYTHON and LD_LIBRARY_PATH pointing to venv are ignored #4841 (comment)

3. Would you for example recommend to try using a full python installation, for example https://github.com/astral-sh/python-build-standalone/releases/tag/20250311 ?

4. Does it all finally boil down to activating the correct path settings (for and after compillation)? ---> Is it then possible to select at runtime dynamically the python interpreter including its version? For example, what if I want to use the python that's existing on the target machine and it's a different python interpreter version than the one the exe was compiled?

I might be trying PyOxidizer again nevertheless. (I tried this some years ago, with mixed success. One point that is good is that it seems to embed and link things properly for a bunch of modules.) But it's difficult then to update and components individually. From this perspective I am also interested in contributing #416 (comment), indygreg/PyOxidizer#751; I need more experience in Rust and support for that though.

observed errors:

----->>>> Using conda, using initialize_python without an activated conda venv, or not using initialize_python with an activated conda env:

Could not find platform independent libraries <prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'python'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = 'C:\Users\USER\SOFTWARE_DEVELOPMENT\RUST\pyo3-python-env-ignored\Lib'
  sys._base_executable = 'C:\\Users\\USER\\SOFTWARE_DEVELOPMENT\\RUST\\pyo3-python-env-ignored\\target\\debug\\pyo3-python-env-ignored.exe'
  sys.base_prefix = 'C:\\Users\\USER\\SOFTWARE_DEVELOPMENT\\RUST\\pyo3-python-env-ignored'
  sys.base_exec_prefix = 'C:\\Users\\USER\\SOFTWARE_DEVELOPMENT\\RUST\\pyo3-python-env-ignored'
  sys.platlibdir = 'DLLs'
  sys.executable = 'C:\\Users\\USER\\SOFTWARE_DEVELOPMENT\\RUST\\pyo3-python-env-ignored\\target\\debug\\pyo3-python-env-ignored.exe'
  sys.prefix = 'C:\\Users\\USER\\SOFTWARE_DEVELOPMENT\\RUST\\pyo3-python-env-ignored'
  sys.exec_prefix = 'C:\\Users\\USER\\SOFTWARE_DEVELOPMENT\\RUST\\pyo3-python-env-ignored'
  sys.path = [
    'C:\\Users\\USER\\SOFTWARE_DEVELOPMENT\\RUST\\pyo3-python-env-ignored\\python311.zip',
    'C:\\Users\\USER\\SOFTWARE_DEVELOPMENT\\RUST\\pyo3-python-env-ignored',
    'C:\\Users\\USER\\SOFTWARE_DEVELOPMENT\\RUST\\pyo3-python-env-ignored\\Lib',
    'C:\\Users\\USER\\SOFTWARE_DEVELOPMENT\\RUST\\pyo3-python-env-ignored\\target\\debug',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00005cbc (most recent call first):
  <no Python frame>
error: process didn't exit successfully: `target\debug\pyo3-python-env-ignored.exe` (exit code: 1)

----->>>> Having pyenv-win activated:

error: process didn't exit successfully: `target\debug\pyo3_dll_not_found.exe` (exit code: 0xc0000135, STATUS_DLL_NOT_FOUND)

copying python3.dll and python311.dll to venv\Scripts did not bring me success.

apparently related issues:

#1616
#1871
#1896
#3589
#4813
#4841
#4890
#4957

astral-sh/uv#11006
astral-sh/uv#12343

astral-sh/python-build-standalone#381

pypa/virtualenv#1050 (comment)

https://dev.to/douganderson444/rust-cargo-run-exit-code-0xc0000135-statusdllnotfound-203