WasmFS JS API

[tlively]

Implement a JS API for WasmFS. It would be nice to make it similar to the API for the current FS, but it also might make sense to make some changes.

[kripken]

The old JS FS code is mainly here: https://github.com/emscripten-core/emscripten/blob/main/src/library_fs.js

WasmFS has a small amount of existing JS API code here:

emscripten/src/library_wasmfs.js

Line 24 in db35f20

$FS : {

The goal here is to implement the full API in the first link in the second link. For example, here is chdir in the old JS FS:

emscripten/src/library_fs.js

Lines 1315 to 1328 in db35f20

	chdir: (path) => {
	var lookup = FS.lookupPath(path, { follow: true });
	if (lookup.node === null) {
	throw new FS.ErrnoError({{{ cDefine('ENOENT') }}});
	}
	if (!FS.isDir(lookup.node.mode)) {
	throw new FS.ErrnoError({{{ cDefine('ENOTDIR') }}});
	}
	var errCode = FS.nodePermissions(lookup.node, 'x');
	if (errCode) {
	throw new FS.ErrnoError(errCode);
	}
	FS.currentPath = lookup.path;
	},

and here it is in the WasmFS JS code:

emscripten/src/library_wasmfs.js

Lines 133 to 138 in db35f20

	chdir: (path) => {
	return withStackSave(() => {
	var buffer = allocateUTF8OnStack(path);
	return __wasmfs_chdir(buffer);
	});
	},

Note how the old JS code implements everything in JS, while the new one just has some JS that calls into wasm: __wasmfs_chdir is a call to a wasm function implementing chdir, which is here:

emscripten/system/lib/wasmfs/js_api.cpp

Line 107 in db35f20

int _wasmfs_chdir(char* path) { return __syscall_chdir((intptr_t)path); }

That call is just a call further into the syscall implementation code inside the wasm. That is, WasmFS is implemented in wasm (in C++), and the goal of this project is to allow JS to use it, using the old JS API.

In general we want the entire old JS FS API implemented, but it is possible not all of it is needed. So one approach here could be to implement the entire API, and another is to focus on what is actually needed, as shown by what is used in the test suite. The process for going through those could look like this:

• Run some wasmfs tests, say ./test/runner.py wasmfs.test_fs_* (or test_a* for all tests starting with the letter a etc.), see what fails, and investigate those. (edit: fixed the command here, and removed a mention of changes to settings.js).

[kripken]

See also the main WasmFS docs and links there for background and other useful info.

[kripken]

A good starting point can be the TODOs in library_wasmfs.js, like these:

emscripten/src/library_wasmfs.js

Lines 216 to 224 in db35f20

	// TODO: mount
	// TODO: unmount
	// TODO: lookup
	// TODO: mknod
	// TODO: mkdev
	// TODO: rename
	// TODO: syncfs
	// TODO: llseek
	// TODO: ioctl

[Forchapeatl]

Hello , @kripken , please may I handle this issue ?

[kripken]

Hi @Forchapeatl , it looks like there is a new PR that begins to work on one of the functions here, mount, #18856 . So I would not look at that specific one right now. But feel free to look into any of the other functions (each can be done independently) and to ask any questions you might have.

[jameshu15869]

Hi @kripken, I've been working through this issue through GSOC and have a couple questions.

1. In the original issue post, you referenced the code below as the place where the call to the WasmFS function is.
   

   emscripten/src/library_wasmfs.js

   Lines 133 to 138 in db35f20

   	chdir: (path) => {
   	return withStackSave(() => {
   	var buffer = allocateUTF8OnStack(path);
   	return __wasmfs_chdir(buffer);
   	});
   	},

   According to your post, the above code should call the function below.
   

   emscripten/system/lib/wasmfs/js_api.cpp

   Line 107 in db35f20

   int _wasmfs_chdir(char* path) { return __syscall_chdir((intptr_t)path); }

   However, my code editor wasn't able to find a definition for the function "__wasmfs_chdir" in library_wasmfs.js. Does the use of 2 underscores in library_wasmfs.js and 1 underscore in js_api.cpp make a difference?

2. According to the GSOC post, this project mainly involves JavaScript and relatively little C++. However, as I am looking through js_api.cpp, it seems that many of the functions listed in the TODOs of library_wasmfs.js are not implemented (e.g. lookup, ioctl). I am still interested, but would like to clarify the scope of the project. Would the project also involve adding to the js_api.cpp file?

3. I used my editor's search feature to look for uses of "FS." in the "test" folder to see which methods are currently being used. Right now, it looks like a lot of functions aren't being tested (e.g. readlink). Right now it seems that a lot of these functions are intermediate helper functions to other FS functions. Would this GitHub issue also involve writing new tests for these intermediate methods?

[kripken]

@jameshu15869

However, my code editor wasn't able to find a definition for the function "__wasmfs_chdir" in library_wasmfs.js. Does the use of 2 underscores in library_wasmfs.js and 1 underscore in js_api.cpp make a difference?

That could be it. An extra underscore is used to prefix calls into wasm (compiled C code) from JS. That makes the name not match for the editor, I guess, and confuse it, unfortunately...

According to the GSOC post, this project mainly involves JavaScript and relatively little C++. [..] Would the project also involve adding to the js_api.cpp file?

Yes, this project will involve some C++. I don't expect that to be complex though. For some operations it'll just be a few lines. For some, like ioctl, we might want to translate the JS into C++, so it wouldn't mean writing some new C++ from scratch. Overall I'd say prior C++ knowledge is not necessary; as long as you're open to learning some C++ as you go, we can help with any language-specific questions.

Overall the goal here is to provide a JS API for WasmFS that functions as close as possible to the old JS API for the JS filesystem. I expect most of that work to be on the JS side. As the top comment here says, It would be nice to make it similar to the API for the current FS, but it also might make sense to make some changes - there are some open questions about how best to do that, which we'll all need to figure out together, so I can't specifically say what % of the work will be in JS and what in C++ (but I do expect more JS).

Would this GitHub issue also involve writing new tests for these intermediate methods?

Much should be tested already but good point about readlink and others that don't appear to be. I'd say the main priority is to get existing tests passing, but testing the others would be good too.

[jameshu15869]

@kripken
Thanks so much for answering my questions. I just have a few more follow up questions.

That makes the name not match for the editor, I guess, and confuse it, unfortunately...

Is there a specific development set up you would recommend?In addition to the underscore issue I also noticed that there are a lot of C/C++ preprocessing statements (Like #if, #endif) in the JS code that are confusing my code editor and giving me errors. I am currently using Visual Studio Code with the default settings for JavaScript and the Microsoft C/C++ extension and was wondering if there were any better ways to set up a development environment for Emscripten.

Overall the goal here is to provide a JS API for WasmFS that functions as close as possible to the old JS API for the JS filesystem. I expect most of that work to be on the JS side.

Based on my understanding of WASM, converting these functions from pure JS in the file library_fs.js to call the js_api.cpp wasm code in library_wasmfs.js would result in a performance improvement, right? So would the main motivation of the project be to improve the performance of the FS? I am asking this question to help refine my motivation/goal for a potential GSOC proposal.

Much should be tested already but good point about readlink and others that don't appear to be. I'd say the main priority is to get existing tests passing, but testing the others would be good too.

While looking through the Emscripten docs, I didn't see anything describing how to create tests. I have tried creating files such as "test_rmdir.c" in the "test/fs" folder, but have not been able to successfully run them with runner.py. Is there some documentation that I am missing? I noticed a lot of tests include a lot of different FS operations at once and was hoping to be able to write a few smaller tests to test out some of the individual functions as I work through them.

Finally, I've ran into a few errors while working through the project. I am currently trying to implement "open" in the library_wasmfs.js." After Googling, I learned that "__syscall_openat" might achieve what I wanted, but realized that js_api.cpp does not have a function for "_wasmfs_open".

emscripten/system/lib/wasmfs/syscalls.cpp

Lines 565 to 573 in 8778c72

	int __syscall_openat(int dirfd, intptr_t path, int flags, ...) {
	mode_t mode = 0;
	va_list v1;
	va_start(v1, flags);
	mode = va_arg(v1, int);
	va_end(v1);
	return doOpen(path::parseParent((char*)path, dirfd), flags, mode);
	}

After adding "_wasmfs_open", I still cannot call it from a test file. Currently, I know that _wasmfs_write_file is working, so I have tried to follow that model. I have tried the following approaches.

1. Directly calling "_wasmfs_open" from "library_wasmfs.js" - This approach results in a "_wasmfs_open" is not defined.
2. When I tried to add '_wasmfs_open' to the settings.REQUIRED_EXPORTS
   

   emscripten/emcc.py

   Lines 2305 to 2316 in 8778c72

   	settings.REQUIRED_EXPORTS += [
   	'_wasmfs_write_file',
   	'_wasmfs_mkdir',
   	'_wasmfs_unlink',
   	'_wasmfs_chdir',
   	'_wasmfs_symlink',
   	'_wasmfs_chmod',
   	'_wasmfs_identify',
   	'_wasmfs_readdir_start',
   	'_wasmfs_readdir_get',
   	'_wasmfs_readdir_finish',
   	]

   
   However, this gives an error saying "symbol exported via --export not found: _wasmfs_open"

Is there something else I'm missing here to add a new exported function?

[kripken]

Is there a specific development set up you would recommend?

Hmm, I don't have one to recommend. I think everyone uses something else around here 😄 I personally just use a simple text editor with syntax highlighting and little else.

Based on my understanding of WASM, converting these functions from pure JS in the file library_fs.js to call the js_api.cpp wasm code in library_wasmfs.js would result in a performance improvement, right? So would the main motivation of the project be to improve the performance of the FS?

It can be faster, yes. In general the WasmFS project moves as much logic from JS into wasm, which allows it to be faster and also use multithreading.

While looking through the Emscripten docs, I didn't see anything describing how to create tests. [..] Is there some documentation that I am missing?

The main docs are here: https://emscripten.org/docs/getting_started/test-suite.html

In general to create a test it's simplest to look at a similar existing one, and do what it does. For example, if you know you want a test like test_loop then you'd find that in test/*.py, since that is where tests are defined. From that python code you can see what it does (just a few lines in that case, but others need more).

After adding "_wasmfs_open", I still cannot call it from a test file.

If you've implemented that function in C++ but it complains on not finding it, then the issue might be that the wasmfs library needs to be rebuilt. Our system for automatic rebuilding of libraries does not notice changes to individual files, so you need to manually force a rebuild. You can do that with this:

./embuilder build libwasmfs libwasmfs-debug --force

(You may only need one of the normal and the debug versions, depending on if you run a test in normal or debug mode, so you can remove the other one from that line).

[tlively]

I think you'll actually need to rebuild libwasmfs-mt and libwasmfs-mt-debug since the tests are multithreaded. To really make sure everything is being rebuilt, you can do ./emcc --clear-cache. (Unless embuilder has gotten smarter about this recently?)

[jameshu15869]

@kripken
I've been stuck on getting the code to account for adding _wasmfs_open() to the js_api.cpp file. Here are a few things I've tried.

1. I tried running emcc on a regular file in hopes of triggering the auto build, according to this comment in embuilder.py.
   

   emscripten/embuilder.py

   Lines 7 to 13 in db51a47

   	"""Tool to manage building of system libraries and ports.
   	In general emcc will build them automatically on demand, so you do not
   	strictly need to use this tool, but it gives you more control over the
   	process (in particular, if emcc does this automatically, and you are
   	running multiple build commands in parallel, confusion can occur).
   	"""

   
   There was no change from this and _wasmfs_open() was still not found.

2. I have been running tests with EMCC_DEBUG=1, so I initially tried to run "embuilder build libwasmfs-debug" without --force. This didn't change anything and _wasmfs_open() was still not found.

3. When I ran any of the embuilder commands with --force, I ended up seeing an error with "Error: Missing C define S_IRUGO! If you just added it to struct_info.json, you need to ./emcc --clear-cache". This occured with and without -debug, with and without -mt.

   This is followed by the message "emcc: error: 'C:/Users/Documents/GitHub/emsdk/node/14.18.2_64bit/bin/node.exe C:\Users\Documents\GitHub\emsdk\emscripten\main\src\compiler.js C:\Users\AppData\Local\Temp\emscripten_temp\tmprfcuqfas.json --symbols-only' failed (returned 1)".

   After I see this message, no embuilder commands have been able to fix the problem - I've had to reclone the emscripten repo multiple times. I have searched for uses of S_IRUGO and none of the files I have edited should involve it. The console message that I should create a bug report for this issue. I am currently unsure if this bug is related to something I tried to implement in js_api.cpp or a bug in embuilder. I'm not currently sure about what steps to take next.

Just to summarize my process is as follows -

1. Add _wasmfs_open to js_api.cpp. Currently I just return 0 instead of a file descriptor/syscall as a placeholder.
   
2. Add an "open" method to library_wasmfs.js
   
3. Calling "FS.open()" in my test file.
   

Is there a step I'm missing? Could there be a missing import? One thing I have noticed is that when I add "_wasmfs_open" to the exports in the emcc.py file, the error changes.

emscripten/emcc.py

Lines 2305 to 2316 in db51a47

	settings.REQUIRED_EXPORTS += [
	'_wasmfs_write_file',
	'_wasmfs_mkdir',
	'_wasmfs_unlink',
	'_wasmfs_chdir',
	'_wasmfs_symlink',
	'_wasmfs_chmod',
	'_wasmfs_identify',
	'_wasmfs_readdir_start',
	'_wasmfs_readdir_get',
	'_wasmfs_readdir_finish',
	]

The error becomes "_wasmfs_open should be exported but is not found". Could this issue be related to not being able to rebuild the lib? I am currently on Windows 10, if that could be making the difference.

[curiousdannii]

Could we make each function stand alone rather than being methods of FS? Ie, prefix them like FS_chdir? This way you could export only the functions you need rather than the whole lot. That was one of the flaws of the original API.

On the other hand, if they become very thin wrappers of exported WASM functions then it wouldn't matter nearly so much.