Proposal: remove .gz downloads for some platforms

[rvagg]

I'd really like to explore ways to simplify our deliverables. The complexity of our infra constantly gets us into trouble.

The Linux kernel stopped offering .gz downloads long ago, so perhaps its time to follow suit? Unfortunately xz isn't on many minimal distributions, like some popular Docker images. e.g. it's not on any debian or ubuntu images, but it is on centos:7. Of course it's a pretty trivial install almost everywhere. But tar does come with everything and -J is natively supported so I'm pretty sure where you have a modern distro, you can unpack .tar.xz files. On alpine:latest you can even do wget https://nodejs.org/download/release/latest-v11.x/node-v11.2.0-linux-x64.tar.xz -O - | tar -Jxv without anything extra installed, none of the Ubuntu or Debian images come with either curl or wget so you have to install stuff anyway.

AIX and macOS are the awkward platforms for .xz files. We don't even offer .xz for AIX and you have to install a third-party package to even use them on macOS. So we could keep offering them there.

Headers are also awkward because they're consumed across all platforms and gzip is easy everywhere. node-gyp even consumes them in native Node with just a gzip stream. Full source packages are similar. So we could keep headers and source as both.

Here's our current list of deliverables:

node-v11.2.0-aix-ppc64.tar.gz
node-v11.2.0-darwin-x64.tar.gz
node-v11.2.0-darwin-x64.tar.xz
node-v11.2.0-headers.tar.gz
node-v11.2.0-headers.tar.xz
node-v11.2.0-linux-arm64.tar.gz
node-v11.2.0-linux-arm64.tar.xz
node-v11.2.0-linux-armv6l.tar.gz
node-v11.2.0-linux-armv6l.tar.xz
node-v11.2.0-linux-armv7l.tar.gz
node-v11.2.0-linux-armv7l.tar.xz
node-v11.2.0-linux-ppc64le.tar.gz
node-v11.2.0-linux-ppc64le.tar.xz
node-v11.2.0-linux-s390x.tar.gz
node-v11.2.0-linux-s390x.tar.xz
node-v11.2.0-linux-x64.tar.gz
node-v11.2.0-linux-x64.tar.xz
node-v11.2.0.pkg
node-v11.2.0-sunos-x64.tar.gz
node-v11.2.0-sunos-x64.tar.xz
node-v11.2.0.tar.gz
node-v11.2.0.tar.xz
node-v11.2.0-win-x64.7z
node-v11.2.0-win-x64.zip
node-v11.2.0-win-x86.7z
node-v11.2.0-win-x86.zip
node-v11.2.0-x64.msi
node-v11.2.0-x86.msi

We could trim that down to:

node-v11.2.0-aix-ppc64.tar.gz
node-v11.2.0-darwin-x64.tar.gz
node-v11.2.0-darwin-x64.tar.xz
node-v11.2.0-headers.tar.gz
node-v11.2.0-headers.tar.xz
node-v11.2.0-linux-arm64.tar.xz
node-v11.2.0-linux-armv6l.tar.xz
node-v11.2.0-linux-armv7l.tar.xz
node-v11.2.0-linux-ppc64le.tar.xz
node-v11.2.0-linux-s390x.tar.xz
node-v11.2.0-linux-x64.tar.xz
node-v11.2.0.pkg
node-v11.2.0-sunos-x64.tar.xz
node-v11.2.0.tar.gz
node-v11.2.0.tar.xz
node-v11.2.0-win-x64.7z
node-v11.2.0-win-x64.zip
node-v11.2.0-win-x86.7z
node-v11.2.0-win-x86.zip
node-v11.2.0-x64.msi
node-v11.2.0-x86.msi

Thoughts @nodejs/build & @nodejs/version-management?

(edit: to be clear, this would be something we do from a particular version forward, not impacting existing release lines. So maybe Node 12+).

[refack]

+1 as a server major change

[gdams]

+1 as a server major change

[refack]

node-v11.2.0-win-x64.7z
node-v11.2.0-win-x64.zip
node-v11.2.0-win-x86.7z
node-v11.2.0-win-x86.zip

Since we're revamping the list, for Windows we could create "self-extracting" 7z archives (add -sfx to command line), and eliminate the *.zip (this files can be executed or extracted with a client)

P.S. if we switch from .msi to NSIS, those files can be easily extracted without running the installer logic, and/or we could add a "just extract" option to the installer.

[ljharb]

How does removing an archive format for “not all” platforms reduce complexity?

[refack]

How does removing an archive format for “not all” platforms reduce complexity?

I was just about to ping you to ask that.
Well, as I see it it reduces complexity for us, and for users who pick a package from the list. But yeah it pushed that complexity to the tools (sorry).

[ljharb]

When you say “us”, I’m not sure who you mean - don’t the tools manage creating those archives in the first place?

[refack]

When you say “us”,

I meant the BuildWG (and maybe the Releasers). We would have less need to make sure tools are available and properly configured on all platforms, and that pack & deploy scripts are correct and run to completion for less artifacts.

P.S. we could also delegate GZIP compression to the HTTP layer, since AFAIK all (reasonable) HTTP clients have builtin GZIP decompression built in, and if not the protocol degrades gracefully (for example that's what S3 does).

[refack]

P.P.S. We could also go for the lower common denominator and pull the .xz and .7z packs ¯_(ツ)_/¯

[rvagg]

or we could just offer one compressed bundle per platform, ditch xz on darwin, ditch 7z on windows, only offer .gz source and headers, it'd be really nice to have fewer items to check to confirm a valid release and fewer requirements for a release build machine

[ljharb]

I guess I'm unclear on why all of this isn't a single automated tool, such that it's any easier to check 1 format over 50.

[refack]

I guess I'm unclear on why all of this isn't a single automated tool, such that it's any easier to check 1 format over 50.

We have 10 ~20 platforms, each one with it's own edges. Just this week we hit one of those edges, and it's always at the 11th hour.
I think what @rvagg is trying to raise is the question of the ROI of this.
Maybe we should looks at some download stats.

[ljharb]

Certainly if the cost of a certain format and/or platform is too high, then it's reasonable to explore removing it - but I'd hope that prior to doing so, all efforts to automate it would be exhausted (or deemed impractical).

[refack]

but I'd hope that prior to doing so, all efforts to automate it would be exhausted (or deemed impractical).

Ack.

[rvagg]

This isn't going to be a massive reduction in complexity, fairly minimal in the scheme of things, but it's directional. From the beginning we've expanded our complexity rapidly because we can, some of it for the challenge or apparent convenience, a lot of it just because we can and have the resources. But then when the crunch comes we feel it. The Build WG is regularly on the receiving end of criticism because when stuff doesn't work, everyone feels it. But of course when stuff's working it's taken for granted. The breakage happens because of complexity, I suspect if we did a root cause analysis of our failures in the last couple of years we'd blame complexity for most of them.

So what I'm proposing here is not a huge bite out of complexity, it's a slight reduction but it gets us in the right direction. Lots of small steps like this, and a bit of resistance to expanding complexity will go a long way to making our resources (infra and people) more resilient.

Cost/benefit critique of this proposal in isolation is welcome of course.

[rvagg]

Oh, and I forgot to say also that I consider calls for getting the Foundation to pay people to maintain our infrastructure as a sign that we've bitten off more than we can chew and we should scale back. If we've really grown that complex that we need to pay people to maintain it then we're doing it wrong.