WebIDE for development

[FWeinb]

As discussed in #102 it would be great to have the ability to change/create/extend the script that is running on the device using some kind of WebIDE. It was suggested there that it would be possible to compile the JS code on the device itself.

I experimented with a different solution but don't know how useful it would be. I successfully compiled xsc/xsl using emscripten and am able to compile JS in the browser now. I did not look into the whole architecture of this project, my initial idea is to transfer the generated buffer to the device and then execute them there.

That is the point where I am currently stuck I don't know how I would go and execute the payload compiled by the browser.

Any pointers to how/if this could work?

[phoddie]

Very cool that you have xsc and xsl running in the browser. Delivering byte code to the embedded device is preferred over delivering source code for several reasons. It saves flash by avoiding the need to deploy the parser to the device, and it allows larger scripts to be downloaded (as the parser is contained by free RAM on the embedded device).

From XSL you should have a .xsb file (XS binary). That needs to be transferred to the device flash, for example using an HTTP PUT. I assume that's easy from the browser? After that, you'll need a small shell app to install and execute the binary. I can prepare that for you. It isn't very complicated, but I may not have time until the weekend.

[FWeinb]

Yeah making a PUT/POST request isn’t the problem. Would be great if you could provide me with an example of how I would startup the VM running the *.xsb file on an ESP.

[phoddie]

Of course. I'll put together a small project.

[dtribble]

@FWeinb Do you have a pointer to your approach to compiling with emscripten? We got something similar working at https://github.com/Agoric/moddable/tree/wasm-spike to get XS to compile to wasm. We would like other eyes and usage on it, getting tests to run within that runtime, and to eventually get it into a pull request, so any info on related projects is welcome.

[phoddie]

@FWeinb - one question: what device target are you using? I was assuming ESP8266 but your comment didn't say.

[FWeinb]

@phoddie ESP8266/ESP32 would be great.
@dtribble i will have a look at your work. Don’t have that much time this week, has to wait till the weekend.

[phoddie]

@FWeinb - I put together an example for you. It consists of a host application to run on the ESP8266 and three sample mods you can run on it. The only only tools needed are xsc, xsl, and some way to do an HTTP PUT. The details are a bit involved, so I also wrote a document explaining it. All the code and documentation is available here Building and Installing JavaScript Mods.

The ability to dynamically install modules compiled to byte code is definitely an advanced feature, one that we are just starting to use. That's why there wasn't an example or documentation already in place. If you run into problems or have questions, please let me know. I've very curious to see this working from the browser. ;)

[FWeinb]

@phoddie Sorry for the slow feedback. I was sick during the weekend, could not look into it.

Will have a look at it this weekend.

[FWeinb]

@phoddie
Just had some time and need your feedback, sorry for double posting here.
I got xsc to compile and it will generate two files code.xs.h and code.xs.c based on an code.js, you can test this here: Basic WebIDE

Do I really need to compile these two files using an C-Compiler to then use xsl to link it? That would be a deal breaker for me. I don't think I want to use gcc in the browser.

Would it be possible to extract xsSymbols and xsCode and send these to an ESP without having to compile/link it?

If that is the case I could write a simplified version of xsc that reads from stdin and writes to stdout without generating all these files, would reduce the js code size a lot.

[phoddie]

There's no need to run GCC to build JavaScript byte code binaries. The instructions posted here show the xsc and xsl options to use. The only output of xsc in this case is a .xsb file.

[FWeinb]

@phoddie sorry total missed that, thanks a lot.

[FWeinb]

@phoddie
Thanks for your help. Got it working. I had to modify your runmod a little. Adding some headers to allow JS to access the endpoint.

In main.js:89

-return {body: "done\n"}
+return {headers: ['Access-Control-Allow-Origin', '*', 'Access-Control-Allow-Methods', 'GET, PUT'],  body: "done\n"};

I made a little video showing it in action (click to open on vimeo):

To run it yourself go here.

Because netlify forces https you need to allow loading resources over http after hitting the Install/Uninstall button (little shield icon in the address bar).
All of this is currently only tested in Chrome 72

Going forward I think it would be best to write a tool that simplifies the xsc/xsl tooling in c so the emscripten overhead is reduced.

[phoddie]

@FWeinb - that is awesome indeed. Congratulations!

I will apply your HTTP header changes to the runmod repository for consistency.

Going forward I think it would be best to write a tool that simplifies the xsc/xsl tooling in c so the emscripten overhead is reduced

In watching your video and running the previous version of your page, performance doesn't seem to be a concern. Changes to xsc and xsl are going to be a significant undertaking to put in place, and then will need to be separately maintained as XS evolves. I would be hesitant to head down that road unless there are very significant potential benefits.

Would be you be wiling to share your code and build scripts? I'd like to be able to reproduce your build to be able to experiment further. Once we have some first-hand experience with XS on WAssm we will be in a better position to contribute to discussion of optimization.

[FWeinb]

@phoddie
Thanks!

I will need to clean this up a lot. I have no idea of the codebase and just wrapped stuff in #ifndef/#ifdef until it compiled. I will work on that in the coming days. Currently it's a big mess.

But here is the basic idea I had:

• Installed emscripten using brew (macOS only)
• I added another architecture to xs/makefiles called js (maybe change that to wasm)
• Modified the linux xsc.mk/xsl.mk to add emscripten specific stuff
• Wrapped stuff in #ifndef/#ifdef. The emscripten build is very similar to the linux build. With the exception that signal isn't working so I just #ifndef it out.

One problem for building xsl is that it needs a platform dependent xsc to compile. That is currently hardcoded to use the mac build in the makefile.

I hope I can clean this up tomorrow so you can have a look at what I did.