What about gyp?

[jbergstroem]

Edit: For all people landing here because you have issues after updating from nodejs to io.js or getting compile errors while building native modules: Your issue is about v8 being updated, not gyp itself. Please file issues at the module's issue trackers about having issues with newer v8 versions instead

Some months ago, I brought up the fact that gyp is pretty much abandonware on the nodejs mailing-list. People chipped in with concerns and questions, but it pretty much stayed dormant after that. I think the outcome of "ignoring" it – especially since the io.js (and/or node) community would now have to cater to gyp development if needed – is not what I'd call planning for a stable future.

I feel that the road to 1.0.0 would be best spent on trying to come to a conclusion as to what role gyp should play in the future of io.js. I'm inclined to summarise what's been said in the previous topic, but will refrain from doing so since it could come off as biased (go read it, now). If gyp will stay around, a decision to keep it instead of "for now" would be optimal. The idea of python requiring ruby to build, the same way io.js requires python is somewhat silly to me though.

[rvagg]

I think the logical path is to move to gn in sync with Chromium, it's something we could start experimenting with already here.

[jbergstroem]

@rvagg: The problem is that the gn devs has shown little to no interest in making gn accessible outside the chromium sources. I've yet to revisit this, but at least last time I was looking at it there was a very tight coupling with headers and includes left and right. Replacing gyp with gn would also be my first suggestion, but if it requires some forking action we might not be much better off than sticking with gyp (besides sans Python) from a maintainer point of view.

[indutny]

I'm a bit afraid of GN, honestly saying. I think we should either stay with GYP, or move to something completely new and public. Probably that thing would compile to ninja, so it'll be a GN alternative in our control :)

[bnoordhuis]

I agree with @indutny: we should either fork gyp (and gut everything from it except make/ninja/VS support) or switch to another build system. It pains me to say it but perhaps the gyp critics were right in 2011 and we should have switched to cmake instead.

@jbergstroem Thanks for filing that issue. Now we know what the position of the GN developers is. We could perhaps lobby hard for a standalone build but there is no guarantee of success and the burden of maintenance would fall upon us. Maintaining a gyp fork where we don't have to care about other users is much easier.

[bnoordhuis]

A tangentially related issue: I think StrongLoop would be willing to donate developer time to replace gyp and node-gyp with a native application that doesn't require a python install. Windows users without python and RHEL 5 users with ancient python versions are the bane of our support department.

[jbergstroem]

@bnoordhuis That's great news. I think that having python as a buildtime requirement for io.js but not within the npm ecosystem would be an interesting part-goal. As for the bigger picture, getting something to work seamlessly over multiple os:es is a major headache, so starting fresh should barely even be considered.

I guess one option I haven't seen mentioned yet (for npm/gyp modules) could be falling back to basic Makefiles with something similar to how pg_config work for the postgres community. It's tried and tested.

I think the guys behind npm should weigh in here as well if possible - their input would be very valuable. gyp is somewhat stuck between io/node.js and npm and both sides could benefit from a well-defined surface (as well as who is responsible for what).

[bnoordhuis]

I guess one option I haven't seen mentioned yet (for npm/gyp modules) could be falling back to basic Makefiles with something similar to how pg_config work for the postgres community. It's tried and tested.

I'm unsure how that would work on Windows. Bundling make.exe with npm is probably not a big deal but writing a Makefile that works with cl.exe on Windows and g++/clang++ on UNIX is likely not a pleasant experience. We could predefine recipes but then we're back to creating our own ad hoc build system.

I think the guys behind npm should weigh in here as well if possible - their input would be very valuable. gyp is somewhat stuck between io/node.js and npm and both sides could benefit from a well-defined surface (as well as who is responsible for what).

That would be @isaacs and @othiym23 and perhaps @mmalecki too now.

Maybe @TooTallNate as well, he maintains node-gyp.

[maxkorp]

Rather than looking at suddenly having to maintain a standalone GN for the global community, what about just forking it, the same way everyone is talking about forking GYP? That way it's still an io.js specific build system, but there's already got a much faster system than gyp as a foundation.

[indutny]

I have an idea, could we write this tool in subset of javascript that is capable of compiling to some C code? Or in C that could be easily transpiled to the JS?

This would simplify life of node-gyp significantly. Otherwise after @bnoordhuis proposal, it'll need to download platform and arch -specific binaries on every install, or build them.

[bnoordhuis]

@maxkorp Two reasons that I can think of:

1. GN stands for 'Generate Ninja'. It doesn't support Visual Studio or Xcode or even make.
2. GN is closely integrated with the massive chromium-base source tree. It's very hard to try and extract (and maintain) a slimmed down version from that.

I tried and didn't exactly fail but it was no picnic either (3.5 MB source-only is the smallest I could trim it down to, in case you're wondering.) I would dread every upgrade and those are bound to happen frequently because of its young age. Forking and maintaining gyp is almost certainly much less work.

@indutny

it'll need to download platform and arch -specific binaries on every install, or build them.

Yes, but I don't see a problem with that. We can include it as part of a release.

Another approach is to remake gyp in pure JS but that would preclude the possibility of building node with that tool because of the chicken and egg problem. A native binary would still need to bootstrap itself first, of course, but that's less insurmountable.

For the record, I'm not married to the idea of a native binary but it would certainly make life easier for some users.

[maxkorp]

@bnoordhuis Regarding CN, a cursory look around doesn't seem to indicate anything contrary to your experience, and I'm inclined to say you're right in wanting to avoid it.

A question to consider then, Is ninja notably slower on windows/mac than msbuild/xcode? If so, that grants a gyp fork a significant edge in my mind. If not, I'd say the the simplicity of a single type of build file is definitely something to consider regardless of the system generating them, be it something new and open as indutny suggests, or a fork of gyp, or any of the other existing options.

Regarding the pureJS option, what about bundling some minimalist interpreter like flathead.js (windows already has jsc.exe) and defaulting to that when io isn't available? You then have something capable enough for an initial build, but something more optimized for subsequent builds.

Regardless, I'll echo the sentiment that dropping the dependency on python (without introducing any others) would be a massive plus.

[jbergstroem]

@bnoordhuis I'm not well versed in Windows, but wouldn't visual studio's nmake.exe be the equivalent of make, not cl.exe?

Built on that, either io.js or npm would grow functionality similar to pg_config (pass linker, compile and version info) and possibly some other magic. Nmake should be able to tackle the Makefiles.

Having python part of iojs bootstrap/compile would be acceptable to me. Platforms like Windows would most likely not build io.js anyway - rather use a binary. Even if gyp went away, there's still ./configureand stuff in tools/ that needs to be taken care of to remove Python completely

@maxkorp Ninja and gyp are two different things, often used together (gyp generating build information that ninja executes). GN is now replacing gyp in the chromium ecosystem.

[bnoordhuis]

@maxkorp

A question to consider then, Is ninja notably slower on windows/mac than msbuild/xcode? If so, that grants a gyp fork a significant edge in my mind.

I don't know if it's faster or slower but GN doesn't know how to generate VS project files and that would likely upset Windows developers. I think the best GN can do is drive cl.exe or msbuild.exe through ninja.

Regarding the pureJS option, what about bundling some minimalist interpreter like flathead.js (windows already has jsc.exe) and defaulting to that when io isn't available? You then have something capable enough for an initial build, but something more optimized for subsequent builds.

It's an option. The interpreter would still have to be bootstrapped first but that's doable for a simple project like that. I'll have to think about it.

@jbergstroem

wouldn't visual studio's nmake.exe be the equivalent of make, not cl.exe?

Sorry, I meant driving cl.exe from make.exe. nmake is not a drop-in replacement for GNU make; forcing the use of nmake would make supporting both Windows and UNIX more complicated for add-on developers.

[rlidwka]

Bundling make.exe with npm is probably not a big deal but writing a Makefile that works with cl.exe on Windows and g++/clang++ on UNIX is likely not a pleasant experience.

Is it possible to write a substitute for make + /bin/sh + builtins in javascript? Just for windows users?

[seishun]

I'm more curious about V8's stance on this. It seems unlikely to me that they would drop gyp in favor of a Chromium-dependent build system. I would rather expect them to either take over gyp or switch to something completely different. I think we shouldn't jump the gun here and instead just wait and see what V8 does.