Mini- Summit Nov 18 -2021

[mhdawson]

Next-10 Mini summit

When

November 18th
10-2 Eastern

Where

zoom: https://zoom.us/j/99950131676

Agenda

* Suitable types for end-users - 90 mins
* Break 15 mins
* Single Executable Applications - 90 mins
* Break 15 mins
* Wrap up and capture next actions

Misc

Will look to promote through

• TSC agenda
• post a collaborators discussion
• Tierney will see if somebody from Types Team
• Tweet from Node.js account

[mhdawson]

@nodejs/tsc FYI hope to see you there.

[mhdawson]

Flagged to collaborators in discussion - nodejs/node#40637

[mhdawson]

PR for tweet - https://github.com/nodejs/tweet/pull/53/files

[mhdawson]

@bnb FYI, if you can confirm once you've reached out to the Types team that would be great.

[joesepi]

@mhdawson is this event 4hrs long (guessing based on agenda)? If so, I think the time (10-12 eastern) is incorrect and perhaps should be 10-2 eastern?

[mhdawson]

@joesepi thanks for catching that. Fixing.

[SimonSchick]

Hey I'm interested in attending this as I've made a lot of contributions to the NodeJS types on DT and am familiar with a lot of it's current issues/gotchas/constraints that we've had to overcome over the last ~3 years.

I will do my best to show up :)

[arcanis]

Single Executable Applications - 90 mins

Are there prior issues/PR one should read to get initial context? We distribute Yarn as a single binary (via bundling), but the parsing/startup cost is becoming a problem and we'd be interested to know if you have something in mind that could address that (related issue I opened at the beginning of the year: nodejs/node#36812).

[mhdawson]

@arcanis we did not get into any depth in terms of solutions for single binary, the only solution that was mentioned as one of the existing options we should look at is the one the mongodb used.

[mhdawson]

Added zoom link as I see I missed that : https://zoom.us/j/99950131676

[mhdawson]

@addaleax FYI, if you have time/interest I think you would add a lot to the discussion on Single Executable Applications

[styfle]

@guybedford Would be great to have your feedback on the Single Executable Applications discussion as well 🙂

[mhdawson]

@SimonSchick hope to see you there.

[mhdawson]

@igorklopov, @jesec vercel/pkg came up in the discussions of Single Executable Applications. I thought I'd give you an FYI in case you have any interest in joining.

[jesec]

@igorklopov, @jesec vercel/pkg came up in the discussions of Single Executable Applications. I thought I'd give you an FYI in case you have any interest in joining.

I am super interested. However, it would be difficult for me to clear my schedules. If there are related discussions in the future, please let me know in advance.

Back to technical stuff. Some info for a more constructive discussion. (I would do a slide and present myself if I have the chance)

Here is a rough description on the design of pkg:

• patches to Node.js (e.g. node.v16.13.0.cpp.patch@vercel/pkg-fetch)

  • bootstrap script that executes a script located in a certain offset of the executable (L141-L188)
  • allows to "bake in" Node options (L394-L502)
  • lets V8 execute bytecode without source (L12-L129) (not needed if user does not need "sourceless")
  • miscellaneous (disable inspector, enable LTO) (optional)

• pkg project or any other project that implements the same functionalities

  • dependency tracer that generates a list of files required by the user's project.
  • produces bytecodes of JavaScript files if requested.
  • generates a snapshot that contains metadata of files, offset range of the executable.
  • prelude script that's baked into the executable and executed upon launch.
  • virtual file system that reads files in snapshot with the offset of the executable. (by overriding Node fs functions)

The layout of pkg executable:

------------------------------------------------
Node.js runtime (patched)
- bootstrap script inside
- PRELUDE_POSITION
- PRELUDE_SIZE
- PAYLOAD_POSITION
- PAYLOAD_SIZE
- BAKERY[]
- baked in by pkg (injectPlaceholder)
------------------------------------------------
Prelude script (start at offset PRELUDE_POSITION)
- VFS snapshot/dictionary object
------------------------------------------------
Payload (start at offset PAYLOAD_POSITION)
- files of user's project, in offsets specified
------------------------------------------------

We would greatly appreciate help from the upstream. Specifically, it would greatly reduce the maintenance burden if we don't have to patch and compile Node.js ourselves.

I am thinking about a more generic implementation of the bootstrap script. Currently pkg simply bakes in the numbers and options with simple string replacement. We can instead design a header. Node.js runtime, upon launch, can scan itself for this header. If the header is found, a prelude script is executed. The proposed layout:

------------------------------------------------
Node.js runtime (generic/official builds)
- C++ routines that scan the executable for the header, if found, parse and
- use the specified options;
- pass the offset of prelude script and other info to JavaScript world
- a bootstrap script that executes a script in specified offset
------------------------------------------------
Header
- Starts with a magic string
- Header size
- Baked-in Node.js command line arguments
- Other info?
------------------------------------------------
Prelude script
------------------------------------------------
Payload
------------------------------------------------

[mhdawson]

@jesec thanks for chiming in. A couple of questions:

1. Can you point us to the patches that you need to apply?

2. I don't think I have enough context to understand what you mean by "We can instead design a header." Can you expand a bit on that?

[mcollina]

Unfortunately I won't be able to partecipate this time.

[jesec]

@jesec thanks for chiming in. A couple of questions:

1. Can you point us to the patches that you need to apply?

https://github.com/vercel/pkg-fetch/blob/b895b85fa43f8bcd0c17c19c44c957317c28bf36/patches/node.v16.13.0.cpp.patch

2. I don't think I have enough context to understand what you mean by "We can instead design a header." Can you expand a bit on that?

See my illustration of current and proposed binary layout of generated single executable application.

[mhdawson]

Markdown file to take notes in - https://hackmd.io/mW7rTTYvREOwGRpOYkXc4w

[ErickWendel]

I'll be there watching!!

[mhdawson]

@SimonSchick if you can still make it the meeting is starting now.

[mhdawson]

PR for minutes - #102