Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ModuleNotFoundError: The module 'protobuf' is included in the Pyodide distribution, but it is not installed. #831

Closed
drankush opened this issue Apr 2, 2024 · 7 comments

Comments

@drankush
Copy link

drankush commented Apr 2, 2024

I get this error trace inside the electron app when I run the dmg created by either npm run dist or yarn dist.
But before this npm run serve or yarn serve for preview does not result in this and the electron app runs fine.

Error during booting up

Traceback (most recent call last):
File "/lib/python311.zip/_pyodide/_base.py", line 573, in eval_code_async
await CodeRunner(
File "/lib/python311.zip/_pyodide/_base.py", line 393, in run_async
coroutine = eval(self.code, globals, locals)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 2, in
File "streamlit/init.py", line 69, in
File "streamlit/delta_generator.py", line 35, in
File "streamlit/cursor.py", line 20, in
File "streamlit/runtime/init.py", line 17, in
File "streamlit/runtime/runtime.py", line 26, in
File "streamlit/proto/BackMsg_pb2.py", line 5, in
ModuleNotFoundError: The module 'protobuf' is included in the Pyodide distribution, but it is not installed.
You can install it by calling:
await micropip.install("protobuf") in Python, or
await pyodide.loadPackage("protobuf") in JavaScript
See https://pyodide.org/en/stable/usage/loading-packages.html for more details.

And in the pop-up window in the electron app:

A JavaScript error occurred in the main process
Uncaught Exception:
PythonError: Traceback (most recent call last):
File "/lib/python311.zip/_pyodide/_base.py", line 573, in eval_code_async
await CodeRunner(
File "/lib/python311.zip/_pyodide/_base.py", line 393, in run_async
coroutine = eval(self.code, globals, locals)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 2, in
File "streamlit/init.py", line 69, in
File "streamlit/delta_generator.py", line 35, in
File "streamlit/cursor.py", line 20, in
File "streamlit/runtime/init.py", line 17, in
File "streamlit/runtime/runtime.py", line 26, in
File "streamlit/proto/BackMsg_pb2.py", line 5, in
ModuleNotFoundError: The module 'protobuf' is included in the Pyodide distribution, but it is not installed.
You can install it by calling:
await micropip.install("protobuf") in Python, or
await pyodide.loadPackage("protobuf") in JavaScript
See https://pyodide.org/en/stable/usage/loading-packages.html for more details.

at new_error (/Applications/TestApp.app/Contents/Resources/app.asar/build/pyodide/pyodide.asm.js:9:10014)
at wasm://wasm/02257296:wasm-function[300]:0x158e62
at wasm://wasm/02257296:wasm-function[454]:0x15ff9e
at _PyEM_TrampolineCall_JS (/Applications/TestApp.app/Contents/Resources/app.asar/build/pyodide/pyodide.asm.js:9:120223)
at wasm://wasm/02257296:wasm-function[1083]:0x1a3494
at wasm://wasm/02257296:wasm-function[3413]:0x28a2bd
at wasm://wasm/02257296:wasm-function[2063]:0x1e431e
at wasm://wasm/02257296:wasm-function[1090]:0x1a397d
at wasm://wasm/02257296:wasm-function[1093]:0x1a3c3e
at wasm://wasm/02257296:wasm-function[1094]:0x1a3ce0
at wasm://wasm/02257296:wasm-function[3226]:0x2689a4
at wasm://wasm/02257296:wasm-function[3227]:0x26e800
at wasm://wasm/02257296:wasm-function[1096]:0x1a3e08
at wasm://wasm/02257296:wasm-function[1091]:0x1a3a98
at wasm://wasm/02257296:wasm-function[442]:0x15f76b
at Module.callPyObjectKwargs (/Applications/TestApp.app/Contents/Resources/app.asar/build/pyodide/pyodide.asm.js:9:64297)
at Module.callPyObject (/Applications/TestApp.app/Contents/Resources/app.asar/build/pyodide/pyodide.asm.js:9:65365)
at Timeout.wrapper [as _onTimeout] (/Applications/TestApp.app/Contents/Resources/app.asar/build/pyodide/pyodide.asm.js:9:32855)
at listOnTimeout (node:internal/timers:569:17)
at process.processTimers (node:internal/timers:512:7)

Im running it on Mac 11.6
node -v && npm -v && yarn -v
v18.14.0
9.3.1
1.22.22.

devDependencies": {
"@stlite/desktop": "^0.51.0",
"cross-env": "^7.0.3",
"electron": "^28.2.1",
"electron-builder": "^24.9.1"

My project directory contains:

build
dist
node_modules
package.json
requirements.txt
streamlit_app -> streamlit_app.py
yarn.lock

I can see that, /node_modules/pyodide/protobuf-4.23.1-cp311-cp311-emscripten_3_1_46_wasm32.whl is present.

The requirements.txt contains:

st_files_connection
PyYAML
streamlit-tags

Also my package.json is modified for local file access (which I have modified by running the dump again)

{
// ...other fields...
"stlite": {
"desktop": {
"nodeJsWorker": true,
"nodefsMountpoints": {
"/mnt": "/Users/Shared/Appfiles/"
}
}
}
}

I there is no straight up permanent solution, if someone could suggest a hacky way to package it for now, as npm run serve or yarn serve for preview does not result in this error and the electron app works perfectly well with local directory access as desired, thats why I think its different from #648

@drankush
Copy link
Author

drankush commented Apr 5, 2024

@whitphx Would appreciate your insight in this issue.

@whitphx
Copy link
Owner

whitphx commented Apr 8, 2024

  • Can you show the entire package.json? Does it contain the "build" field as the example?
  • Can you provide the file tree in the build directory, e.g. the result of tree ./build? Does build/pyodide/protobuf-4.23.1-cp311-cp311-emscripten_3_1_45_wasm32.whl exist?

@drankush
Copy link
Author

drankush commented Apr 9, 2024

./build
├── asset-manifest.json
├── c43e2f2e6b6fd378c8e1.module.wasm
├── electron
│   ├── main.js
│   ├── preload.js
│   └── worker.js
├── index.html
├── pyodide
│   ├── Jinja2-3.1.2-py3-none-any.whl
│   ├── MarkupSafe-2.1.3-cp311-cp311-emscripten_3_1_46_wasm32.whl
│   ├── Pillow-10.0.0-cp311-cp311-emscripten_3_1_46_wasm32.whl
│   ├── PyYAML-6.0.1-cp311-cp311-emscripten_3_1_46_wasm32.whl
│   ├── attrs-23.1.0-py3-none-any.whl
│   ├── cachetools-5.3.1-py3-none-any.whl
│   ├── cramjam-2.6.2-cp311-cp311-emscripten_3_1_46_wasm32.whl
│   ├── fastparquet-2023.7.0-cp311-cp311-emscripten_3_1_46_wasm32.whl
│   ├── fsspec-2023.6.0-py3-none-any.whl
│   ├── jsonschema-4.17.3-py3-none-any.whl
│   ├── micropip-0.5.0-py3-none-any.whl
│   ├── numpy-1.26.1-cp311-cp311-emscripten_3_1_46_wasm32.whl
│   ├── packaging-23.1-py3-none-any.whl
│   ├── pandas-1.5.3-cp311-cp311-emscripten_3_1_46_wasm32.whl
│   ├── protobuf-4.23.1-cp311-cp311-emscripten_3_1_46_wasm32.whl
│   ├── pyodide-lock.json
│   ├── pyodide.asm.js
│   ├── pyodide.asm.wasm
│   ├── pyodide.mjs
│   ├── pyodide_http-0.2.1-py3-none-any.whl
│   ├── pyrsistent-0.19.3-cp311-cp311-emscripten_3_1_46_wasm32.whl
│   ├── python_dateutil-2.8.2-py2.py3-none-any.whl
│   ├── python_stdlib.zip
│   ├── pytz-2023.3-py2.py3-none-any.whl
│   ├── six-1.16.0-py2.py3-none-any.whl
│   ├── toolz-0.12.0-py3-none-any.whl
│   └── typing_extensions-4.7.1-py3-none-any.whl
├── requirements.txt
├── site-packages-snapshot.tar.gz
├── static
│   ├── css
│   │   ├── 8050.c24b25fa.chunk.css
│   │   ├── 8957.cd5b39ce.chunk.css
│   │   ├── 9449.1edfa457.chunk.css
│   │   └── main.fb5f4d99.css
│   ├── js
│   │   ├── 1046.19145034.chunk.js
│   │   ├── 1122.9b0e508e.chunk.js
│   │   ├── 1189.ff18a565.chunk.js
│   │   ├── 1261.e15a45e6.chunk.js
│   │   ├── 142.9be80baa.chunk.js
│   │   ├── 144.b10fb472.chunk.js
│   │   ├── 1579.418b1244.chunk.js
│   │   ├── 1729.3e10491c.chunk.js
│   │   ├── 1740.c4ec1606.chunk.js
│   │   ├── 1839.45bfc04d.chunk.js
│   │   ├── 2385.8cbb7ba4.chunk.js
│   │   ├── 2448.9a8682bf.chunk.js
│   │   ├── 2694.6d6de2d2.chunk.js
│   │   ├── 2694.6d6de2d2.chunk.js.LICENSE.txt
│   │   ├── 2753.9ea4d84f.chunk.js
│   │   ├── 2873.69bb3f79.chunk.js
│   │   ├── 2909.692a5576.chunk.js
│   │   ├── 2940.b9c9a565.chunk.js
│   │   ├── 3208.42c54f80.chunk.js
│   │   ├── 3685.39209b63.chunk.js
│   │   ├── 4004.05e297fd.chunk.js
│   │   ├── 4021.e04e7e55.chunk.js
│   │   ├── 4482.6da0ecb0.chunk.js
│   │   ├── 4494.b35991fe.chunk.js
│   │   ├── 4681.fc821bbf.chunk.js
│   │   ├── 4694.ead63044.chunk.js
│   │   ├── 4721.108179ed.chunk.js
│   │   ├── 4724.444c1588.chunk.js
│   │   ├── 4785.63597083.chunk.js
│   │   ├── 4831.595555ab.chunk.js
│   │   ├── 4994.37361da4.chunk.js
│   │   ├── 5299.478bd9da.chunk.js
│   │   ├── 5599.bc0ea1af.chunk.js
│   │   ├── 5783.1caefe8d.chunk.js
│   │   ├── 6093.31927157.chunk.js
│   │   ├── 6134.35028dd4.chunk.js
│   │   ├── 6449.98df60ee.chunk.js
│   │   ├── 6679.e1debb86.chunk.js
│   │   ├── 6679.e1debb86.chunk.js.LICENSE.txt
│   │   ├── 6802.615748ec.chunk.js
│   │   ├── 6865.93ac0107.chunk.js
│   │   ├── 698.096ccf39.chunk.js
│   │   ├── 7125.70a408b8.chunk.js
│   │   ├── 7135.0b3044f9.chunk.js
│   │   ├── 7579.ab4d23ff.chunk.js
│   │   ├── 7696.f57124bd.chunk.js
│   │   ├── 8050.8be50b25.chunk.js
│   │   ├── 8097.99be3737.chunk.js
│   │   ├── 8097.99be3737.chunk.js.LICENSE.txt
│   │   ├── 8186.18b3fb43.chunk.js
│   │   ├── 8186.18b3fb43.chunk.js.LICENSE.txt
│   │   ├── 865.6b1ec0ad.chunk.js
│   │   ├── 8801.d4ef7a00.chunk.js
│   │   ├── 8850.2182e134.chunk.js
│   │   ├── 8957.b697f7a2.chunk.js
│   │   ├── 9001.5422e085.chunk.js
│   │   ├── 9005.baeb8ab0.chunk.js
│   │   ├── 9166.e335c84b.chunk.js
│   │   ├── 9166.e335c84b.chunk.js.LICENSE.txt
│   │   ├── 9311.26e5e737.chunk.js
│   │   ├── 9320.3ed3fc02.chunk.js
│   │   ├── 9449.186aa58b.chunk.js
│   │   ├── 9449.186aa58b.chunk.js.LICENSE.txt
│   │   ├── 9813.ff50cf3b.chunk.js
│   │   ├── main.b686df03.js
│   │   └── main.b686df03.js.LICENSE.txt
│   └── media
│   ├── KaTeX_AMS-Regular.73ea273a72f4aca30ca5.woff2
│   ├── KaTeX_AMS-Regular.853be92419a6c3766b9a.ttf
│   ├── KaTeX_AMS-Regular.d562e886c52f12660a41.woff
│   ├── KaTeX_Caligraphic-Bold.7489a2fbfb9bfe704420.ttf
│   ├── KaTeX_Caligraphic-Bold.a1abf90dfd72792a577a.woff2
│   ├── KaTeX_Caligraphic-Bold.d757c535a2e5902f1325.woff
│   ├── KaTeX_Caligraphic-Regular.7e873d3833eb108a0758.ttf
│   ├── KaTeX_Caligraphic-Regular.d6484fce1ef428d5bd94.woff2
│   ├── KaTeX_Caligraphic-Regular.db074fa22cf224af93d7.woff
│   ├── KaTeX_Fraktur-Bold.354501bac435c3264834.woff
│   ├── KaTeX_Fraktur-Bold.4c761b3711973ab04edf.ttf
│   ├── KaTeX_Fraktur-Bold.931d67ea207ab37ee693.woff2
│   ├── KaTeX_Fraktur-Regular.172d3529b26f8cedef6b.woff2
│   ├── KaTeX_Fraktur-Regular.6fdf0ac577be0ba82a4c.woff
│   ├── KaTeX_Fraktur-Regular.ed305b5434865e06ffde.ttf
│   ├── KaTeX_Main-Bold.0c3b8929d377c0e9b2f3.woff
│   ├── KaTeX_Main-Bold.39890742bc957b368704.woff2
│   ├── KaTeX_Main-Bold.8169508bf58f8bd92ad8.ttf
│   ├── KaTeX_Main-BoldItalic.20f389c4120be058d80a.woff2
│   ├── KaTeX_Main-BoldItalic.428978dc7837d46de091.woff
│   ├── KaTeX_Main-BoldItalic.828abcb200061cffbaae.ttf
│   ├── KaTeX_Main-Italic.fa675e5e4bec9eb250b6.ttf
│   ├── KaTeX_Main-Italic.fd947498bc16392e76c2.woff
│   ├── KaTeX_Main-Italic.fe2176f79edaa716e621.woff2
│   ├── KaTeX_Main-Regular.4f35fbcc9ee8614c2bcc.woff
│   ├── KaTeX_Main-Regular.9eba1d77abcf2aa6e94e.ttf
│   ├── KaTeX_Main-Regular.f650f111a3b890d116f1.woff2
│   ├── KaTeX_Math-BoldItalic.3f07ed67f06c720120ce.woff
│   ├── KaTeX_Math-BoldItalic.bf2d440b3a42ea78a998.ttf
│   ├── KaTeX_Math-BoldItalic.dcbcbd93bac0470b462d.woff2
│   ├── KaTeX_Math-Italic.6d3d25f4820d0da8f01f.woff2
│   ├── KaTeX_Math-Italic.8a5f936332e8028c7278.ttf
│   ├── KaTeX_Math-Italic.96759856b4e70f3a8338.woff
│   ├── KaTeX_SansSerif-Bold.5b49f4993ae22d7975b4.ttf
│   ├── KaTeX_SansSerif-Bold.95591a929f0d32aa282a.woff2
│   ├── KaTeX_SansSerif-Bold.b9cd458ac6d5889ff9c3.woff
│   ├── KaTeX_SansSerif-Italic.7d393d382f3e7fb1c637.woff2
│   ├── KaTeX_SansSerif-Italic.8d593cfaa96238d5e2f8.woff
│   ├── KaTeX_SansSerif-Italic.b257a18c016f37ee4543.ttf
│   ├── KaTeX_SansSerif-Regular.02271ec5cb9f5b4588ac.woff
│   ├── KaTeX_SansSerif-Regular.2f7bc363fc5424ebda59.ttf
│   ├── KaTeX_SansSerif-Regular.cd5e231e0cc53b2cb2c0.woff2
│   ├── KaTeX_Script-Regular.073b3402d036714b4370.woff
│   ├── KaTeX_Script-Regular.c81d1b2a4b75d3eded60.woff2
│   ├── KaTeX_Script-Regular.fc9ba5249878cd8f8d88.ttf
│   ├── KaTeX_Size1-Regular.0108e89c9003e8c14ea3.woff
│   ├── KaTeX_Size1-Regular.6de7d4b539221a49e9e2.ttf
│   ├── KaTeX_Size1-Regular.6eec866c69313624be60.woff2
│   ├── KaTeX_Size2-Regular.2960900c4f271311eb36.woff2
│   ├── KaTeX_Size2-Regular.3a99e70aee4076660d38.woff
│   ├── KaTeX_Size2-Regular.57f5c1837853986ea1db.ttf
│   ├── KaTeX_Size3-Regular.7947224e8a9914fa332b.woff
│   ├── KaTeX_Size3-Regular.8d6b6822586eea3d3b20.ttf
│   ├── KaTeX_Size3-Regular.e1951519f6f0596f7356.woff2
│   ├── KaTeX_Size4-Regular.4ad7c7e8bb8d10a34bb7.ttf
│   ├── KaTeX_Size4-Regular.aeffd8025cba3647f1a6.woff
│   ├── KaTeX_Size4-Regular.e418bf257af1052628d8.woff2
│   ├── KaTeX_Typewriter-Regular.4c6b94fd1d07f8beff7c.woff
│   ├── KaTeX_Typewriter-Regular.c295e7f71970f03c0549.woff2
│   ├── KaTeX_Typewriter-Regular.c5c02d763c89380dcb4e.ttf
│   ├── SourceCodePro-Bold.17a68a0751a813474a0e.woff2
│   ├── SourceCodePro-BoldItalic.d45b7a3df103d441d78b.woff2
│   ├── SourceCodePro-Italic.ec122a420df4175e74f2.woff2
│   ├── SourceCodePro-Regular.84b900b88d09398d86b0.woff2
│   ├── SourceCodePro-SemiBold.17291d1c493cb25eb2c3.woff2
│   ├── SourceCodePro-SemiBoldItalic.f879ae27307c3926522b.woff2
│   ├── SourceSansPro-Bold.118dea98980e20a81ced.woff2
│   ├── SourceSansPro-BoldItalic.1d664be59d2eb5fef029.woff2
│   ├── SourceSansPro-Italic.8a9bfea74d43927d6eec.woff2
│   ├── SourceSansPro-Regular.0d69e5ff5e92ac64a0c9.woff2
│   ├── SourceSansPro-SemiBold.abed79cd0df1827e18cf.woff2
│   ├── SourceSansPro-SemiBoldItalic.befb0a2824eabc5ce36b.woff2
│   ├── SourceSerifPro-Bold.d7975b56594770699ae0.woff2
│   ├── SourceSerifPro-BoldItalic.913697a7178b128caa4c.woff2
│   ├── SourceSerifPro-Italic.d3529cb5797663ac5d88.woff2
│   ├── SourceSerifPro-Regular.e6c37aa3926474cc93e1.woff2
│   ├── SourceSerifPro-SemiBold.5c1d378dd5990ef334ca.woff2
│   ├── SourceSerifPro-SemiBoldItalic.249e948b885d0c7d30a8.woff2
│   ├── balloon-0.0e6e96ece0ea0719d6b6.png
│   ├── balloon-1.a503c271109e9a0fdd2d.png
│   ├── balloon-2.872250f3bf3a8965d655.png
│   ├── balloon-3.e772fafd3bfc55fb439b.png
│   ├── balloon-4.1a74162473d81c9cb9ee.png
│   ├── balloon-5.edb75c74c54165b29856.png
│   ├── checkmark.29851c8e9e6ef0c3d6c1e4efe3c1bb9e.svg
│   ├── fireworks.0906f02ea43f1018a6d2.gif
│   ├── flake-0.beded754e8024c73d9d2.png
│   ├── flake-1.8077dc154e0bf900aa73.png
│   ├── flake-2.e3f07d06933dd0e84c24.png
│   ├── icon_running.de1119daebe465fa5263.gif
│   ├── logo.83ae4f2fb87e38be7cbb8a5d2beb64d2.svg
│   └── rocket.b75b17d2b0a063c6cea230d1a9d77f1e.svg
├── stlite-manifest.json
└── streamlit_app
└── streamlit_app.py

The package.json

{
  "name": "TestApp",
  "version": "0.1.0",
  "main": "./build/electron/main.js",
  "scripts": {
    "dump": "dump-stlite-desktop-artifacts",
    "serve": "cross-env NODE_ENV=production electron .",
    "pack": "electron-builder --dir",
    "dist": "electron-builder",
    "postinstall": "electron-builder install-app-deps"
  },
  "build": {
    "files": ["build/**/*"],
    "directories": {
      "buildResources": "assets"
    }
  },
  "devDependencies": {
    "@stlite/desktop": "^0.51.0",
    "cross-env": "^7.0.3",
    "electron": "^28.2.1",
    "electron-builder": "^24.9.1"
  },
  "stlite": {
    "desktop": {
      "nodeJsWorker": true,
      "nodefsMountpoints": {
        "/mnt": "/Users/Shared/Appfiles"
      }
    }
  }
}

Yes build/pyodide/protobuf-4.23.1-cp311-cp311-emscripten_3_1_45_wasm32.whl does exist

@drankush
Copy link
Author

drankush commented Apr 26, 2024

Hey! @whitphx what do you think of this.

@whitphx
Copy link
Owner

whitphx commented Apr 28, 2024

Thank you, I could reproduce the error.
Looks like this happens if the NodeJS worker mode is enabled and at least one of st_files_connection, PyYAML, and streamlit-tags is installed.
Let me take more to investigate this.

@whitphx
Copy link
Owner

whitphx commented Apr 29, 2024

A quick workaround is to add "asar": false to the "build" field in your package.json and run yarn dist again, while it's not recommended in general.

As a dev note, the cause of this error is Pyodide's loadPackage() tries to write its package directory in NodeJS mode while the target dir is read-only in the case of the packaged Electron app as it's an ASAR archive.
Looks like this has to be fixed in Pyodide. Will work on it.

@whitphx
Copy link
Owner

whitphx commented May 29, 2024

Resolved by #934.
The patch is included in the 0.56.0 release.

@whitphx whitphx closed this as completed May 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants