Executable Versions + Script Runner

[jeff-hykin]

AFAIK, there isn't a standard place for mentioning versions of binaries. For example, what version of powershell is being used for the CI scripts, and versions of python are needed for node-gyp for building native modules.

Ideally there would be a standard way to list versions of these binaries (outside of atom) but I'm unaware of any standard. So, in substitution of that, could the versions be added to the package.json as a "bin" or "binaries" or "executables" object?

I think this would be helpful since it's already a struggle trying to build atom.

[DeeDeeG]

You mean to specify some versions in the project somewhere, so it will warn or prevent you proceeding if you've got a build setup that won't work?

An interesting thought, although I think we should focus on testing a set of supported build setups and documenting which are supported.

I just did a very large amount of research and testing as to what works for building Atom today that you may be interested in: atom/flight-manual.atom.io#630 (comment)

• npm 3.10.7 (!!!) and newer are technically supported.
  • Npm 6.12.0 and newer recommended for simplicity of use and broad compatibility with versions of Python and Visual Studio.
• Node 10.12.0 and newer are supported, up to Node 12.16.x.
• Python 2.7 and 3.5+ are supported with npm 6.12+
  • Older versions of npm work with only Python 2.7 (and Python 2.6, but ignore that. Please don't install Python 2.6, just install Python 2.7 if you need Python 2.)
• Visual Studio 2015, 2017 and 2019 are fully supported with npm 6.10.1 and newer.
  • The timeline of changing support for Visual Studio is longish and somewhat complex. See my comment liked above, but also I strongly encourage you to just use npm 6.12.0 or newer. It's been out for a while and supports VS 2015. 2017 and 2019 automatically, no hassle, no fuss.

[DeeDeeG]

We already have a good way of detecting Node, npm and Python and stopping you from bootstrapping if you've got the wrong versions.

See: https://github.com/atom/atom/blob/2f550cf0d38693598bbba1d2abf34e176a9e8ece/script/lib/verify-machine-requirements.js

(Edit: This is a bit out of date for Node version, though, as of atom#20879. (electron-mksnapshot 9.0.2 and electron-chromdriver 9.0.0, in script/package.json, have an indirect dependency that requires Node 10.12+). And it hasn't been updated to deny Node 12.17 or newer, which was released at the end of May this year with a breaking change for babel.)

It's surprisingly difficult to properly detect a working Visual Studio install. And node-gyp will tell you if you've got no working Visual Studio installs. I'd like to leave detecting Visual Studio to node-gyp.

(Long aside: We have a fix for apm that makes apm get out of the way so node-gyp can properly detect a working Visual Studio install, but this only really helps in a niche situation (broken Visual Studio 2015 or older, installed alongside working Visual Studio 2017 or newer). It's already mostly working in upstream apm.)

I would be shocked and surprised if powershell version or cmd.exe version had any impact on a successful build. Okay, I've seen a lot of things, so it wouldn't be that surprising. It's my sense that that isn't the case though, absent specific testing that shows otherwise.

[aminya]

I think this would be helpful since it's already a struggle trying to build atom.

@jeff-hykin

#10 will solve this issue partially. You will not need anything to start hacking the core using the pre-bootstrapped repository.

To build the Atom, I just install windows-build-tools, and I use the node that is provided by apm. I think we can add a script to find that node version automatically. After the unification that we made, this step is necessary.

One other way is to use a docker file. We have a docker file but it is out of date.

See: https://github.com/atom/atom/blob/2f550cf0d38693598bbba1d2abf34e176a9e8ece/script/lib/verify-machine-requirements.js

I need to update this script. This is missing a couple of things.

[jeff-hykin]

Docker would be nice. I'm on Mac but I triple boot and I'd like to generate a redistributable app for all three OS's.

so it will warn or prevent you proceeding if you've got a build setup that won't work

Idk if that can be expected to be reliable. I've tried node 14, 12, 10, 11, python 2 and 3.6, 3.7, 3.8, disabling the GitHub extension, upgrading downgrading npm packages, deleting node_modules and the other usual suspects, resulting in different build failures. I've got the full install of xcode and the other prerequisites on the flight manual.

Just now (after changing nothing inside the folder for a week), the build script completed for the first time. But as luck would have it, the Dev Atom immediately fails on opening with a Cannot find module '../build/Release/nslog.node'.

A package-lock style of guaranteed-to-work versions of node, python, etc would just be nice so that I can rule those out when trying to debug why the build is failing.

To build the Atom, I just install windows-build-tools, and I use the node that is provided by apm.

@aminya
How do you use the node that is provided by apm? (edit the build script or PATH?)

[jeff-hykin]

(Thanks for the help btw)

[aminya]

To build the Atom, I just install windows-build-tools, and I use the node that is provided by apm.

@aminya
How do you use the node that is provided by apm? (edit the build script or PATH?)

Yes, you can edit your path so the Node provided by apm is the first.

Yes, I said, I need to change the scripts so that Node version is automatically found! The script that we have which checks the versions is out of date and does not reflect the actual requirements!

---

For now, you can download and use the following 😄. Use the absolute path to this to run the scripts. Like:

C:/node.exe ./script/bootstrap

node.zip

To make this Node version yourself, run npm install in the atom/apm path. A node.exe will appear in the bin folder (node_modules/atom-package-manager/bin)

[DeeDeeG]

How old is your mac? Are we talking recent like Mojave or Catalina? Older than that? (Should still work, but it would be nice to know.)

Also, having cd'ed into the Atom repo, can you do git remote update and git status? Are you up to date with the latest master? Are you building upstream or this fork? Are you on a custom branch?

Edit to add: It's a good habit to do script/clean then script/bootstrap and script/build --no-bootstrap if you want a clean build environment. If you haven't found a reliable-to-build commit of the repo yet, try this on upstream's master branch, or maybe even better, a release tag like v1.50.0.

Lastly: How much RAM and free disk space do you have? ~4GB is enough RAM, but could cause issues if you also have a lot of other things open. Especially web browsers, even moreso with lots of tabs open. If you're almost 100% out of disk space, things could get dicey during install, files not ending up where they're supposed to, etc.

I am on Linux and Windows, and builds there are fairly reliable on my real machines on command-line. (As opposed to CI, which is its own separate thing)

How do you use the node that is provided by apm? (edit the build script or PATH?)

Technically you could copy that somewhere and add it to your PATH... But, uh... on Windows I just download the x64 .zip from here: https://nodejs.org/dist/ extract to somewhere and add it to my PATH. And on Linux I use n. n works on macOS as well. Latest Node v10.x is a safe bet for now. Otherwise Node v12.16.x. Node 12.4.0 is the version you should use.

I've got the full install of xcode and the other prerequisites on the flight manual.

For macOS, you should just need these on your PATH:

• Clang (from Xcode)
• Git (usually mac users get this on their PATH via Xcode)
• Python (2.7 or 3.5+) (Python is shipped with macOS, no install needed!)
• Node & npm

Nothing else should be required whatsoever in terms of system requirements (on macOS). But willing to troubleshoot with you.

[jeff-hykin]

Latest Node v10.x is a safe bet for now. Otherwise Node v12.16.x. Node 12.4.0 is the version you should use

Haha, yeah thats why I think they need to be locked to a specific version.

I'm on node 12.0.0 which is probably why the build worked this time, I was previously on 12.16.x

At the moment I'm testing on a Hackintosh, 48Gb ram, running Sierra, but I've got mac books with Mojave and High Sierra I can build on as well. I run a lot of projects with a lot of different versions so switching isn't too bad, I use asdf, nvm, and nix for managing versions.

Note: npm install atom-package-manager quickly fails with node v12.0.0, on node v12.4.0 it fails on node-gyp (even after clearing node_modules deleted both times), but I decided to try on node 12.18.3 and sure enough npm install atom-package-manager works on that. Now I'll see if I can build atom with the 12.4.0 node that I had to get using 12.18.3 haha.

Note the first Node is 12.4.0, then the second one is Node 10.20.1 x64 which is weird

Okay, I've seen a lot of things, so it wouldn't be that surprising. It's my sense that that isn't the case though

I don't think this is a good policy. It takes 1 minute to add a powershell version to a package.json or somewhere else. The "Node.js 6.x or later" thats still on the Flight manual was probably also a "It's my sense that that isn't the case" recommendation, which has cost me and others days of debugging. I'd bet most people trying just give up and don't contribute. I don't mean to sound attacking; you guys are helping me out and I see you contributing everywhere. But the CI tests are already spotty right? pinning down the versions seems like the best action to start making builds more reliable. It takes 5 min to record all the working versions you have, it can even be automated; but, on my end, checking all the possible combinations of versions is nearly impossible. Why would you not just preemptively record the versions?

[jeff-hykin]

Hey @aminya your trick worked, Atom Dev actually opened without crashing 🎊 🎉 Thanks 👍

[aminya]

Thank you Jeff. I will update the scripts such that the versions are pinned. Currently, the only version of Node that I support is 12.4.0. It should build everything from Atom to apm itself.

I don't mean to sound attacking; you guys are helping me out and I see you contributing everywhere. But the CI tests are already spotty right? pinning down the versions seems like the best action to start making builds more reliable. It takes 5 min to record all the working versions you have, it can even be automated; but, on my end, checking all the possible combinations of versions is nearly impossible. Why would you not just preemptively record the versions?

That is totally true. I have had this issue myself. That's why I created #10. Using that PR you can edit the JavaScript code right away! It is like a prebuilt docker image!

Additionally, I am also trying to pin and unify the versions everywhere. A single pinned version makes sure that everything works. For example, different node versions in Atom and APM created a lot of issues for us.

Currently, this file includes all the versions I recommend:
https://github.com/atom-ide-community/atom/blob/master/script/vsts/platforms/templates/preparation.yml

I am thinking of making a preparation.ps1 and preparation.sh that installs all the dependencies (for different OS) similar to our CI.

Using a container is another solution, but not all people have experience with using those. A preparation.ps1 is more accessible than a docker container.

[jeff-hykin]

That's why I created #10. Using that PR you can edit the JavaScript code right away! It is like a prebuilt docker image!

That sounds fantastic, I'll have to look into that further.

Additionally, I am also trying to pin and unify the versions everywhere. A single pinned version makes sure that everything works. For example, different node versions in Atom and APM created a lot of issues for us.

When I propose changes, I'll try to help with that. I noticed just now that oginumura and the tree sitter are using native modules even though there's WASM builds for both so I'll be trying to swap those out to lessen the dependency hell.

[aminya]

When I propose changes, I'll try to help with that. I noticed just now that oginumura and the tree sitter are using native modules even though there's WASM builds for both so I'll be trying to swap those out to lessen the dependency hell.

That's awesome! I am looking forward to your contributions!

Currently, Atom does not tree-sitter by default, and you should enable it in the settings. I was thinking about making them the default. I'll wait until you bring those changes, and we can switch to tree-sitter as the default language mode.

[DeeDeeG]

Hi again, folks. I do not doubt your personal experiences, but if there are specific failures on various Node versions, we should document them, as they may be specific bugs we can solve. The reason most projects do NOT pint to an exact version is that you simply don't have to. The code builds on many Node versions. It is bordering on superstitious to pin to an exact Node version. Most Node code is Node version agnostic to a certain degree forward and back. It is certainly less informative (and less proactively healthy for the code) to pin a Node version, rather an tracking down and solving these bugs. Though admittedly, Atom runs on fixed versions of Node/Electron, so the (direct) impact to end-users is none. It's more of a "looking out for developers' sanity, long-term" thing. My two cents. Glad to see you are having more success now building Atom, Jeff.

[DeeDeeG]

Also please be careful doing npm install with no flags in the apm/ folder. That can actually break the install.

It should be npm install --global-style, just for the apm folder. Otherwise apm can't find its bundled node and things like that.

(This is due to a weird workaround for using apm with npm@5, as apm was first written a few major versions of npm ago before npm learned how to hoist and dedupe repeated dependencies, and somewhat flatten the node_modules dependency tree as extracted to disk. Which changes the place that various things get installed to relative to the atom-package-manager module dir. Some paths are hard-coded inapm and can't handle things moving around. Ideally the apm scripts would be re-written to be smarter about finding the bundled npm and apm launcher scripts, and the bundled node and so on. I haven't thought of the right way to do this, but it had been a goal of mine to make that happen if possible, eventually.)

[aminya]

The reason most projects do NOT pint to an exact version is that you simply don't have to.

if the code is only JavaScript that is not a problem because Node is not breaking most of the time, but native modules are required to be built against the same Node versions that run them! There is no way you can build a native module on a different Node version and run it.

See this document so you get familiar with this:
https://www.electronjs.org/docs/tutorial/using-native-node-modules

That is why we shall use the same Node version in both apm and Atom.

The specific projects that use prebuild or node-pre-gyp may run in Electron without rebuilding from the source.