Explain the current state of our SEA work

[jviotti]

This document is a bit like blog post, in that in documents the state at
this point in time and will likely be out of date at some point. For
this reason, I put this document inside a blog/ directory, prefixed
with its date.

Signed-off-by: Juan Cruz Viotti jv@jviotti.com

[saswatds]

In point 2 which talks about supporting every binary format supported by Node.js, by support I am assuming it means the ability to inject. So in point 6 does it mean that the scope to go beyond that?

[jviotti]

Point 2 talks about the capability of injecting data in each platform. For example, using Windows to inject the data into an EXE, using macOS to inject the data into a Mach-O, etc.

Point 6 is about being able to do any injection on any operating system. For example, being able to use macOS to produce a Windows SEA, etc.

We support this at the moment on Postject thanks to LIEF, and its pretty handy for i.e. continuous integration purposes, as you can generate all binaries for your app through a single Linux runner, etc.

I'll try to clarify this better now.

[saswatds]

So considering both point 4 and 5 together, we are looking at two different components? One in javascript which performs the injecting and the other is native code which forms the native API? I probably don't understand this really well.

[jviotti]

Yeah, they are two sides of the same coin.

• Injecting a new section into a binary can be done (in theory) with just fs (like PKG does), or even by compiling something like LIEF to web assembly. The main motivator for wanting a pure-JS implementation of the injector is that injecting is something that developers would run on their own systems. Adding a native add-on dependency means they need a full blown compiler toolchain installed, etc which makes the setup more complicated and would likely result in many people submitting issues when they cannot make it work
• Reading from the section at runtime is a C/C++ feature only, and it is useful for integrators of i.e. Postject (like Node.js itself)

While we could argue they are different components, both read and write access are important and must be always kept in sync.

[saswatds]

Is the uses of only javascript for the solution a hard requirement?

[jviotti]

It was framed as a hard requirement by @jesec based on his experience managing PKG. A lot of people are likely to face troubles setting things up locally (i.e. mainly on Windows), resulting in unnecessary maintenance overhead.

Maybe "pure JavaScript" is not the right term... Something like WebAssembly would do as well. I'll clarify

[guest271314]

Node.js executable may seem daunting at first, but its not!

"its" should be "it's".

First, we start with the assets that make a typical Node.js application. These
typically consist of JavaScript files from the application code

"typically" is redundant in 2d sentence, where "typical" is used in 1st sentence.

Suggest substituting "These consist of one or more JavaScript files from the application code".

Most interesting Node.js programs are composed by more than one file.

Is this opinion or fact? There should not be a difference from the user viewpoint between creating a SEA with 1 or 100 input files.

• Require maintaining custom Node.js patches. These patches typically touch
  on the initialization logic of Node.js and are different across Node.js
  versions. These patches result in high maintenance burden and make it
  difficult for SEA implementations to support new Node.js versions as soon as
  they are released.

This is what I was alluding to in my posts in #1. That is, the process should be as simple as a user visisting the Node.js downloads Website, uploading their existsing Node.js-specific JavaScript code and the site producing out an executable with tip-of-tree (nightly) node executable for download. That way the SEA is always up to date, and/or is capable of being re-compiled daily/nightly. Perhaps that is a point that needs to be coordinated with the team/WG that published node nightly.

No support for non-JavaScript assets. Some SEA implementations
concatenate JavaScript assets and inject the resulting "single" file into the
executable. While this approach works for simple programs, many Node.js
applications require application resources that are not typically
concatenated as part of the JavaScript code, such as Node.js native add-ons,
text files, executable scripts and more. These assets are often dynamically
consumed using fs and child_process.

I'll point out here that QuickJS has its own compiler, qjsc https://bellard.org/quickjs/quickjs.html#qjsc-compiler.

[Trott]

Node.js executable may seem daunting at first, but its not!

"its" should be "it's".

Please use GitHub's review comment feature for comments like this. Even better if you use their suggestion feature so someone can simply commit your suggestion within the interface.

[guest271314]

I thought I did. Then couldn't find the same commit again until someone a collaborator/member replied to my suggestion to remove the bit about "interesting" 2f5c6ac. Done.

[jviotti]

@guest271314 Thanks for the suggestions!

Is this opinion or fact? There should not be a difference from the user viewpoint between creating a SEA with 1 or 100 input files.

Very true. What I wanted to exemplify is that a notion of a VFS is often needed for apps consisting of more than one file (which might not even be require()-able files or are dynamically loaded). A VFS is not very useful for an app built as a single JS file.

This is what I was alluding to in my posts in #1. That is, the process should be as simple as a user visisting the Node.js downloads Website, uploading their existsing Node.js-specific JavaScript code and the site producing out an executable with tip-of-tree (nightly) node executable for download. That way the SEA is always up to date, and/or is capable of being re-compiled daily/nightly. Perhaps that is a point that needs to be coordinated with the team/WG that published node nightly.

Exactly. Projects like PKG involve significant maintenance overhead to produce Node.js binaries that can injected into. If we can provide that capability with upstream Node.js builds, then this entire complexity goes away! I tried to clarify this better now on the "Problems with current SEAs" section.

I'll point out here that QuickJS has its own compiler, qjsc https://bellard.org/quickjs/quickjs.html#qjsc-compiler.

Thanks for sharing! We should definitely document this

[RaisinTen]

This document is a bit like blog post

Should we place this in a directory called blog then?

[jviotti]

Should we place this in a directory called blog then?

Works for me! Done!

[guest271314]

Doesn't Native Client do something like what is being worked on here?

I tried to reverse-engineer a .nexe file; contains the entire program in one file.

[jviotti]

I don't know a lot about Native Client. Can you open a GitHub Discussion on this repo about it so we get more eyes on it? It'd be great if you can explain in more detail what it does (for people who are not super familiar) and even show some of the results you've seen from reverse engineering that .nexe file.

According to https://developer.chrome.com/docs/native-client/welcome-to-native-client/, it is deprecated technology for anything other than ChromeOS, but there might definitely be some lessons we can learn from it.

[RaisinTen]

LGTM in its current state. Would anyone else from @nodejs/single-executable like to drop your LGTMs / request changes on this one?

[jviotti]

Awesome. I'll merge now so we can get the ball rolling, but happy to address other comments on further PRs.