File System Issues (sql)

[michaelweinold]

Currently, an error is raised when trying to query a SQLite database. This affects the use of peewee:

• SQLite Error: DatabaseError: database disk image is malformed jupyterlite/jupyterlite#1064
• Error when trying to use sqlite3 jupyterlite/pyodide-kernel#35

The present workaround is to not use /drive:

(...) I think it because the the current directory (/drive) is not a MEMFS (unlike the rest of the file system), but rather a DriveFS used, from what I understand, for notebook persistance. Probably it's even less POSIX compliant than MEMFS, which becomes an issue for sqlite. If you write in any other folder say /tmp/ the above example would work.

Ultimately, the issue seems to be with drivefs:

I think the code that needs to be checked is the drivefs, which I guess is ultimately backed by localForage:

[michaelweinold]

Current efforts to improve the JupyterLite file system through the use of iDrive:

• Switch to Contents.IDrive for the JupyterLite in-browser contents manager jupyterlite/jupyterlite#1041

and through the use of SharedArrayBuffer:

• Add docs about the use of SharedArrayBuffer jupyterlite/jupyterlite#1405
• Document the use of SharedArrayBuffer on GitHub Pages jupyterlite/jupyterlite#1409
• Update Vercel Setup Instructions for Node.js v20 jupyterlite/jupyterlite#1450
• Check use of `SharedArrayBuffer` jupyterlite/jupyterlite#1451

Unfortunately, these improvements did not affect the original SQL issue.

[michaelweinold]

There is a new "WASM-native" SQLite version. Unfortunately, this will not solve the SQL issue either:

This won't help us because we need to link it into an Emscripten shared libraray (or the main binary). They have their own loader code.

• Replace sqlite3 recipe with native WASM version? emscripten-forge/recipes#1254
• Replace current SQLite (2.6.0) with new WASM-native version? pyodide/pyodide#4981

[michaelweinold]

There might be potential for a collaboration, following the discussions here:

• Error when trying to use sqlite3 jupyterlite/pyodide-kernel#35
• [0.1.x] pyodide 0.24.0, widget shim patches jupyterlite/pyodide-kernel#62 (comment)

Both @psychemedia and @bollwyvl are interested in using SQL in a Pyodide/JupyterLite environment.

[michaelweinold]

• localForage itself seems to have no issues with SQL.
• "The Current State of SQlite persistence of the Web"