Add Emscripten build to buildRelease workflow

[fghalasz]

This PR implements the automated builds for Emscripten maiko as part of the buildRelease workflow. The result is a maiko...emscripten.tgz file added to the release assets.

Note that this PR adds a new makeright target called wasm_nl (for wasm_nl) that just compiles the lde code to wasm without adding any preloaded files. The result is just two release files ldesdl.wasm and ldesdl.js.

The idea is that the Medley file system "preload" will be created in the Medley buildRelease workflow using the emscripten file packager. In the Medley repo there will also be a modified lde.html (probably called medley.html) that loads the separately package files appropriately (also allows choice of full.sysout or apps.sysout).

This scheme ensures that the Maiko automated builds don't depend on Medley automated builds - avoiding circular dependencies since Medley builds already depend on Maiko builds.

[nbriggs]

I'd like to see the medley side file packaging before I review this because it will all need to fit together.

[nbriggs]

I've written a script to build the file system, and what I'd suggest for this is that we not worry about creating a separate target emscripten-wasm_nl, and just replace emscripten-wasm. Do remove the --preload-file entries, and we should add -sLZ4 in the LDFLAGS so that we can check if packaging the file system with --lz4 gets better performance than not. We can still gzip an lz4 compressed file for about a 30% decrease in transmission size.
This was how I built a couple of files for the filesystem which I manually installed into my nginx data directory:

#!/bin/sh
 /usr/local/lib/emscripten/tools/file_packager /var/tmp/sysoutfs.data --js-output=/var/tmp/sysoutfs.js  --no-node --lz4 \
        --preload ./loadups/full.sysout@medley/loadups/full.sysout \
        --preload ./greetfiles/MEDLEYDIR-INIT.LCOM@usr/local/lde/site-init.lisp

 /usr/local/lib/emscripten/tools/file_packager /var/tmp/medleyfs.data --js-output=/var/tmp/medleyfs.js  --no-node --lz4 \
        --preload ./loadups/whereis.hash@medley/loadups/whereis.hash \
        --preload ./docs/@medley/docs \
        --preload ./doctools/@medley/doctools \
        --preload ./greetfiles/@medley/greetfiles \
        --preload ./internal/@medley/internal \
        --preload ./sources/@medley/sources \
        --preload ./library/@medley/library \
        --preload ./lispusers/@medley/lispusers \
        --preload ./fonts/@medley/fonts

and having the lde.html include the sysoutfs.js and medleyfs.js files before ldesdl.js.
I think we can probably speed things up in the "normal" case by building the compiled files and source files into different filesystem assets so that if you only load compiled files you don't get the overhead of bringing over the source package. They should be able to still appear in the same directory in the virtual file system.

[fghalasz]

@nbriggs You can see what I've done so far on the Medley side in the branch fgh_emscripten-01. The work is in installers/emscripten. Two files: em_build.sh and medley.html. embuild.sh is the file packager and release builder. It is designed to be called either locally or from a github action. medley.html is a revised version of lde.html.

1. I have separated the sysout(s) from the other Medley files in separate file packages. medley.html loads both the right sysout package and the 'rest' of the files package before loading/calling ldesdl.js.

2. I have packaged three sysouts separately - lisp, full and apps. medley.html is set up so that if you can invoke medley.html?lisp, medley.html?full, or medley.html?apps to get the corresponding sysout. medley.html with no parameters loads the full.sysout.

3. We can certainly extend the file factorization as you suggest and include just the compiled files in the "main" file package. We can then modify medley.html so that only if you include the "sources" parameter in the URL, then it will load the sources file package.

4. I thought about including the LZ4 stuff, but thought I'd get it all running and then put that in later. We can easily do it right away.

5. For the time-being I am thinking of deploying an Amazon Lightsail instance to host this stuff. The buildRelease github action would automatically upload each new release to this instance. Lightsail instances can be configured as basic nginx server so it should be able to support gzipping. This we can get more folks to try this out - without having to go thru the trouble of setting up a web server. (

6. BTW the emrun tool in the emscripten sdk is another easy way to test this all out on your local machine - similar to the hugo server stuff for our website).

7. I am still working on getting all of this plumbed into the github actions. I haven't begun to set up the Lightsail instance.

[fghalasz]

Oh and

8. I like the way you package the site init file with the sysout. This is more elegant than the way I handle it in the current medley.html. I will switch my builder script and medley.html over to this.

[nbriggs]

On the medley side, on the machine on which I have emscripten installed from standard packages, which is a FreeBSD 14.0 system:
(1)

flap% installers/emscripten/em_build.sh 
zsh: installers/emscripten/em_build.sh: bad interpreter: /bin/bash: no such file or directory

Here, bash is /usr/local/bin/bash. The only shells in /bin are csh, sh, tcsh. sh is a POSIX shell.
We probably should use /bin/sh for this script as we do for almost all the other scripts in the scripts directory excepting medley.sh, medley_vnc.sh, and medley.command.
(2) It looks as though tools like file_packager need to be located via em-config EMSCRIPTEN_ROOT in the tools/ subdirectory of that result.

[fghalasz]

@nbriggs New commit on Medley side in fgh_emscripten-01 branch.

1. Debashified em_build.sh. Now specifies /bin/sh and should be Posix compliant shellwise.

The calls to find (used to find lcoms and dfasls) are pretty vanilla but are not 100% posix (since posix find is kind of sparse). But I did check the freebsd find man page and I'm pretty confident that the finds will work ok for you.

2. Fixed the file_package location issue using EMSCRIPTEM_ROOT as you suggested

3. Added LZ4 to file packaging (and also in Maiko-side makeright wasm_nl in the fgh_embuild-01 PR)

4. Separated sources and compiled files into separate file packages. Sources are only loaded if the medley.html url has a "source" parameter as in "medley.html?full&source".

[nbriggs]

Looking good. I made everything and installed on my FreeBSD/nginx setup, gzipped every file and it was still happy. Having looked at it in the non-source case, the only thing I think I'd change is putting the (TEdit/text) documentation files in the compiled files package. Longer term it might be worth considering the lazy file loading option, so we'd transmit all the file names and only get the contents, on a per-file basis, if you actually open them - but that looks a bit complicated to set up.