The future of GYP in Node.js

[rvagg]

Inspired by a discussion with @indutny, but this has been brewing for a long time now:

It's clear that Chromium has effectively deprecated GYP and stopped active development on it, with effort moving over to GN. @domenic reports that this is being finished up "this month".

GYP is stuck on Python ^2.7 and we get a regular stream of issues filed against node-gyp because of the pain people feel when using a default installed Python 3. In fact, the strict limited range of Python support (must be at least 2.7) causes pain on our build infra, where very old EL (EL5 / CentOS 5) comes with an older version and much of its system tools are tied to that version so introducing a new Python becomes a bit hacky.

We don't "support" GYP, we regularly get pull requests against node-gyp and nodejs/node that adjust the GYP source and our normal response is to take it upstream, where it sometimes succeeds through the Chromium gauntlet and is sometimes dropped. We've been (generally) unwilling to float patches, which puts us, and therefore our users, entirely at the mercy of Chromium developers. This is obviously an increasing concern as they move to GN and GYP becomes a legacy concern that they need to detach themselves from.

So, we need to come up with a plan for how to deal with GYP, or at least a general goal that we can encourage effort on where appropriate. If we are not proactive about this we're going to be left with a lot of pain, in fact, we've probably left it far too late as it is.

There are two very distinct issues to discuss and we need to make sure we keep them both in mind when discussing:

• Ecosystem: we have a very large ecosystem that depends on node-gyp, it's built into every version of npm, there's a ton of knowledge about how to wrangle it for Node.js addons and it's vital (IMO, up for discussion of course), that we consider longer-term support for GYP and node-gyp to ensure ecosystem stability—even if we plan on migrating to another tool.
• Core: we use GYP in core (libuv does too, but it's optional). One of the primary reasons for doing so is that our most complex dependency (by far), V8, uses it so it's straightforward to extend that into the rest of the project. When Chromium's support for GN starts to progress further than GYP or GYP is completely dropped, we need to be prepared for it. This doesn't necessarily mean dropping GYP as our primary tool but there are obvious ramifications if we're supporting it while Chromium isn't.

Some beginning talking points:

• @chrisdickinson has collected some useful links on GN @ collecting useful links chrisdickinson/nojs#3 and that project uses GN
• This thread on Twitter discusses how difficult it is to extract GN from the Chromium codebase and reuse it elsewhere, it's not a separate project we can just pick off the shelf although @chrisdickinson is doing some work on that.
• We could consider taking over GYP from Chromium, either officially or by forking it. This would be particularly helpful with ecosystem fork and maybe we can finally get Python 3 support into it! Maybe we can even make it more pleasant for our needs. I know we have some quality Python hackers on the team that could step up.
• CMake often gets proposed as an alternative, I'm not a huge fan but I also don't have a ton of experience so can't be objective. It's worth discussing since it's a mature solution.
• There have been multiple experiments in creating something new, including a bunch of JavaScript-based tools (@indutny proposes bootstrapping with duktape for this purpose). This feels a little like the Jenkins-replacement effort, it's a lot of work that someone needs to do and we may not have anything to show for a long time.

[saghul]

require('this_comment');

I think replacing GYP (the build file format thingy) would break things too much. We can keep GYP (the syntax) and replace the project with a pure JS thing which just generates Ninja files. GYP and GN files are similar enough that even if Google drops it Node can keep up. And be independent.

We could consider taking over GYP from Chromium, either officially or by forking it. This would be particularly helpful with ecosystem fork and maybe we can finally get Python 3 support into it! Maybe we can even make it more pleasant for our needs. I know we have some quality Python hackers on the team that could step up.

I did that, and called it GYN and I was able to compile iojs with it. Original post. I still volunteer to finish the work, if that's the approach everyone agrees upon.

Note that this is not orthogonal to replacing GYP with a pure JS thing. Since it's the same codebase, GYP could be replaced with GYN, which gives us the advantage of working on Python 3 and switching to Ninja. Then if GYN is replaced with a pure JS implementation, the Python dependency would go away.

/saghul's 2 cents

[seishun]

Ecosystem: we have a very large ecosystem that depends on node-gyp, it's built into every version of npm, there's a ton of knowledge about how to wrangle it for Node.js addons and it's vital (IMO, up for discussion of course), that we consider longer-term support for GYP and node-gyp to ensure ecosystem stability—even if we plan on migrating to another tool.

We've already survived migration from waf to gyp. Migrating the ecosystem from gyp to gn is going to painful, but in the long term it's better to use a build system that's actually used outside of Node.js.

Core: we use GYP in core (libuv does too, but it's optional). One of the primary reasons for doing so is that our most complex dependency (by far), V8, uses it so it's straightforward to extend that into the rest of the project. When Chromium's support for GN starts to progress further than GYP or GYP is completely dropped, we need to be prepared for it. This doesn't necessarily mean dropping GYP as our primary tool but there are obvious ramifications if we're supporting it while Chromium isn't.

So in the end it depends on what V8 does, doesn't it? I'm not sure how much V8 team cares about third-party users, but if they do, I find it unlikely that they would switch to gn in its current state.

There have been multiple experiments in creating something new, including a bunch of JavaScript-based tools (@indutny proposes bootstrapping with duktape for this purpose). This feels a little like the Jenkins-replacement effort, it's a lot of work that someone needs to do and we may not have anything to show for a long time.

Not a huge fan of introducing another prerequisite for building Node.js.

[bnoordhuis]

We've already survived migration from waf to gyp.

The ecosystem was a lot smaller those days, though.

Migrating the ecosystem from gyp to gn is going to painful, but in the long term it's better to use a build system that's actually used outside of Node.js.

But only in one other project (Chromium), just like GYP in 2012.

There was debate back then about whether to move to GYP or CMake. I supported GYP but, with the benefit of hindsight, that was the wrong move.

[seishun]

But only in one other project (Chromium), just like GYP in 2012.

What about V8?

There was debate back then about whether to move to GYP or CMake. I supported GYP but, with the benefit of hindsight, that was the wrong move.

Do you mean addons or Node.js itself? I'm not sure what would be worse, using different build systems for addons and Node.js itself, or using a different build system for Node.js than for V8.

[indutny]

I'm working on a JS port of GYP at this very moment. Once it is done - it should be easier for the contributors to support it, and we could possibly modify it to fit our needs better.

[ofrobots]

Some other discussions on this: nodejs/node#6089 and nodejs/NG#24.

[eljefedelrodeodeljefe]

For reference I was working on this week and have a working compiler abstraction here: https://github.com/eljefedelrodeodeljefe/platform-tools

I am able to build C/C++ source files, native addons (haven't pushed win support yet) across all platforms and will head to compiling node itself after the weekend.

Also I disagree that we need a port of GYP or integrating Ninja or GN, since the task of building sources gets overcomplicated and overabstracted in all of those tools, imo, after looking at their sources and having been able to build a prototype really quick. There is a little bit more justification for a build tool on windows, since VS hides a lot of information like include paths in macros, which you have to dig out really. After that a callout to cl.exe is as easy as one to gcc.

[saghul]

@eljefedelrodeodeljefe Thanks for working on this! What I suggested was always based on the assumption that the migration would be slow, and changing one component at a time, trying to limit the risk.

I'm happy to hear you are able to compile addons calling the compiler directly, that's great news! FWIW Python does the same, you just create an instance of Extension, list some sources, call python setup.py build_ext and magic happens.

Keep up the good work!

[indutny]

FWIW, https://github.com/indutny/gyp.js

[indutny]

Update from gyp.js: node.js builds on windows with ninja and gyp.js, it also builds on mac/linux/freebsd as well, and possible on SmartOS with some tweaks to the .gyp file.

Should we consider investing some love into this project and/or using it in node-gyp?

[eljefedelrodeodeljefe]

I hope not. It's good engineering, don't get me wrong, but it is really bad management, sorry. This is 5000+ sloc of a port of a build systems that is by all means mediocre and chromium specific.

I am really unemotional about it, but ...sorry... this .... this is really bad news.

[Fishrock123]

Fwiw, with a JS port of GYP, people will likely no longer require python to build node addons if we use it in node-gyp, could be a big win, there's a good amount of non-compiler python issues that crop up there.

[Fishrock123]

I am really unemotional about it, but ...sorry... this .... this is really bad news.

What's bad news? Supporting an entire existing ecosystem without a another required language dependancy that we can better maintain ourselves? Perhaps not perfect, but that's a lot of wins IMO.

[evanlucas]

I'm pretty +1 on gyp.js. My only concern is what will we do once v8 switches to gn? IIRC that was happening this year sometime?