ModuleNotFoundError: No module named 'supermodule.submodule'; 'supermodule' is not a package

[tcztzy]

🐛 Bug Reports

I am working for writing Python extension, but facing import submodule error.

ModuleNotFoundError: No module named 'supermodule.submodule'; 'supermodule' is not a package

What do I expected

Import submodule like usual Python package.

🌍 Environment

• Your operating system and version:
  ArchWSL version 19.11.16.0 on Windows Subsystem Linux 2 version Windows Insiders Slow ring 10.0.19041

$ uname -a
Linux xxxxxx 4.19.84-microsoft-standard #1 SMP Wed Nov 13 11:44:37 UTC 2019 x86_64 GNU/Linux

• Your python version:

$ python -V
Python 3.8.1

• How did you install python (e.g. apt or pyenv)? Did you use a virtualenv?:

$ pyenv virtualenv system xxxxxx

• Your rust version (rustc --version):

rustc 1.43.0-nightly (c9290dcee 2020-02-04)

• Are you using the latest pyo3 version? Have you tried using latest master (replace version = "0.x.y" with git = "https://github.com/PyO3/pyo3")?
  Yes.

💥 Reproducing

Please provide a minimal working example. This means both the rust code and the python.

Please also write what exact flags are required to reproduce your results.
src/lib.rs

use pyo3::prelude::*;
use pyo3::{wrap_pyfunction, wrap_pymodule};
use pyo3::types::IntoPyDict;

#[pyfunction]
fn subfunction() -> String {
    "Subfunction".to_string()
}

#[pymodule]
fn submodule(_py: Python, module: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pyfunction!(subfunction))?;
    Ok(())
}

#[pymodule]
fn supermodule(_py: Python, module: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pymodule!(submodule))?;
    Ok(())
}

Cargo.toml

[package]
name = "supermodule"
version = "0.1.0"
authors = ["Tang Ziya <tcztzy@gmail.com>"]
edition = "2018"

[package.metadata.maturin]
classifier = [
    "Development Status :: 3 - Alpha",
    "Programming Language :: Python",
    "Programming Language :: Rust",
]

[lib]
name = "supermodule"
crate-type = ["cdylib", "rlib"]

[features]
default = []

[dependencies]
pyo3 = { git = "https://github.com/PyO3/pyo3", features = ["extension-module"] }

[dev-dependencies]

$ maturin develop
$ python -c "from supermodule.submodule import subfunction"
ModuleNotFoundError: No module named 'supermodule.submodule'; 'supermodule' is not a package

[davidhewitt]

This is unfortunately a flaw in the way Python imports modules from native extensions. I think if you run import supermodule first, then from supermodule.submodule import subfunction will succeed.

[waymost]

I've run into this same issue with version 0.8.5 on Mac and no combination of imports that I've tried has worked, including the above suggestion.

[davidhewitt]

Thanks for the note. I must have mis-remembered that.

Based on stack overflow the solution looks like you might be able to create a symlink to the main library which has the name of the submodule: https://stackoverflow.com/questions/48706842/python-3-x-c-extension-module-and-submodule

[sunjay]

You can fix this in a somewhat hacky way by adding the module to sys.modules manually. From your Rust code, this can look like:

#[pymodule]
fn supermodule(py: Python, module: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pymodule!(submodule))?;

    py.run("\
import sys
sys.modules['supermodule.submodule'] = submodule
    ", None, Some(module.dict()))?;

    Ok(())
}

The documentation claims that sys.modules is writable, so this approach should not break anytime soon. The only caveat is that while this works when executed by a Python interpreter, it does not work with static analysis tools like pylint.

After a lot of trial and error, I am fairly convinced that the only "fully robust" solution that will also make static analysis tools happy is to have your entire module hierarchy represented in the file system. Whether you do this with Python files or with separate extension modules is up to you. I agree with the comments in PyO3/maturin#266 that we should look into how numpy, pandas, etc. handle this and see if there is a better way.

It would be great if in the meantime PyO3 could emit code to modify sys.modules automatically so from supermodule.submodule import ... would work out of the box.

[davidhewitt]

A solution for this came up in #1517 (comment)

We could try to support this pattern internally in PyO3 (or at least add documentation).

[chris-laplante]

Somewhat unrelated but maybe this will help someone. In my particular case I had a module on-disk with a number of submodules. I only wanted to import a subset of the submodules. Tried many solutions (e.g. PyModule::from_code, py_run! workarounds, etc) but this is what finally worked for me:

let sys: &PyModule = py.import("sys").unwrap();
let syspath: &PyList = sys.getattr("path").unwrap().try_into().unwrap();

syspath
    .insert(0, "/path/to/python/library/")
    .unwrap();

let my_module = PyModule::new(py, "my_module")?;
let my_mod_submodule_1 = py.import("my_module.submodule1").unwrap();
let my_mod_submodule_2 = py.import("my_module.submodule2").unwrap();
my_module.setattr("submodule1", my_mod_submodule_1)?;
my_module.setattr("submodule2", my_mod_submodule_2)?;

[vxgmichel]

For the record, an alternative to the py.run! solution is:

#[pymodule]
fn supermodule(py: Python, m: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pymodule!(submodule))?;

    py.import("sys")?
        .getattr("modules")?
        .set_item("supermodule.submodule", submodule)?;

    Ok(())
}

[davidhewitt]

NB it looks like submodules may also have complexity with things like pickling: #2469 (comment)