Repository navigation
Proposal: gypkg for GYP dependency management #7240
Description
Hello @nodejs/build @nodejs/collaborators @nodejs/ctc !
I have been recently working on gypkg, which is something that I have craved for a very long time. What do you think about trying it out for Node.js core, and maybe libuv (cc @saghul) ?
This will allow us to put every GYP dependency to a separate repo, and manage them with a semver in a node.gyp file. Should be pretty neat, and should make the upgrade process easier and cleaner.
Thank you,
Fedor.
Activity
indutny commentedon Jun 9, 2016
Oh, and also it will reduce future size of the Node.js repo tree, since we won't be adding way too much changes on each dep update anymore.
[-]Use gypkg for GYP dependencies management[/-][+]Proposal: gypkg for GYP dependencies management[/+][-]Proposal: gypkg for GYP dependencies management[/-][+]Proposal: gypkg for GYP dependency management[/+]eljefedelrodeodeljefe commentedon Jun 9, 2016
I like the work vey much, but as posted on twitter I am very strongly opposed to introducing another js-foreign dependency or something that increasing organizational dependency to one.
I think it's rather time to build our own tooling since we are big enough and can agree that most stuff out there isn't really what we are looking for out of the box and we are just piggy-backing on the Google stuff.
Since node is proliferated really much nowadays I think we would also do the software world a favor and build more cross platform tooling that anyone could use for other purposes too without having things like python installed correctly. Gyp seems orthogonal to that goal.
I am working on something wrapping shelling out to
gcc,cl.exeandclangand to be honest that seems very easy to me and I don't even see the need for more advanced build system features. Everything else, like dependency management, change detection, target management, scripting can be done very easily if not much better with JS as opposed to declarative configs and declarative Makefile scripts.saghul commentedon Jun 9, 2016
Ohai! Here are my 2 cents, FWIW.
This project is nice in that it reduces bolireplate and makes it easier to use GYP for Node and other projects. Alas, it still requires Google's GYP, which doesn't change much of the status quo.
If we backtrack a little, it seems that the dependency on Python was one of the things people don't seem to like. Not to mention the fact that GYP doesn't work on Python 3, which adds more to the confusion. Google itself it transitioning to GN, which just generates Ninja buildfiles, and lets Ninja do the work. Personally, I think this a good approach.
My little contribution to help solve this problem was GYN a while ago. It's basically Google's GYP, with a single generator (Ninja) and ported to Python 3. I could compile Node itself. It was met with some resistance since it's still Python, which I do understand.
That said, I think a good way forward would be to port GYP to JS but make it just generate only Ninja files. We'd still be using GYP files, which Node is kinda heavily invested on it by now, and Ninja is super small and self-contained, so bundling it in shouldn't be much of a problem.
I briefly mentioned this last part to @eljefedelrodeodeljefe on Twitter. The idea being that Ninja could be replaced later, saving that stage and going directly from GYP files to binaries if need be. Personally, I'd take the Unix approach of one tool one job and let Ninja do the compilation.
One thing I wonder though, is that if this looks like a bigger problem than it is because we don't have a good story for binary packages. If we did then only the people who build the package would need the build tools, and (specially those on Windows) others wouldn't have much trouble.
seishun commentedon Jun 9, 2016
I don't think this is a good thing. Currently, Node.js repo is self-contained. This guarantees that even 10 years from now you can easily check out an ancient commit and build it. If it had dependencies in other repos, then it would depend on whether these repos still exist.
jbergstroem commentedon Jun 9, 2016
I generally share @saghul's point of view. Adding on top: I now and then attempt to disconnect gn from the rest of the google stack which is perfectly possible but very hard in reality without hard forking; it's just moving too quick. It's relatively simple to build a standalone version of it; and I really like the idea of using gn-files/ninja as the "future" native compilation stack but I have a hard time seeing it happen if we stick with python.
rvagg commentedon Jun 9, 2016
Some of this discussion can move to nodejs/CTC#2 which is attempting to take a broader perspective on what to do with GYP in core and the ecosystem.
kzc commentedon Jun 9, 2016
@saghul
It's a chicken and egg problem. I think people should be able to build
nodewithout anodebinary. So not using JS to build is ironically preferable IMO.saghul commentedon Jun 9, 2016
@kzc We could use Duktape to bootstrap the process. It's a single C file, so maybe that would be acceptable? (just as a fallback) Or write the replacement in C++.
kzc commentedon Jun 9, 2016
@saghul
Very interesting idea. The issue is that Duktape doesn't support most node standard modules like
fsAFAIK, although it does have someBuffercompatibility. Could be confusing maintaining JS code that is not compatible with node.saghul commentedon Jun 9, 2016
Yes, it doesn't support any modules out of the box. It does have a
Bufferimplementation, though it has the 0.12 API. We really only need to read and write a file, so adding a couple of builtins should do. At any rate, I don't think the bootstraping is a big deal, comparing it with the rest.indutny commentedon Jun 9, 2016
FWIW, I'm working on JS port of GYP right now. Will keep you updated.
eljefedelrodeodeljefe commentedon Jun 9, 2016
@kzc chicken-and-egg problem does not apply in this case. It's a common misconception. That is what cross compilation and downloading binaries is for. Go can build itself, so can gcc. We are no different.
kzc commentedon Jun 9, 2016
@eljefedelrodeodeljefe Comparisons with Go don't apply as node can't be built with node alone. A C++ compiler is really the key prerequisite of building node. That's what builds v8, libuv, openssl and node itself. That's what I like about @saghul's proposal bootstrapping via Duktape - it eliminates the python dependency in favor of the C++ dependency that node can't avoid.
eljefedelrodeodeljefe commentedon Jun 9, 2016
@kzc that's wrong. Any build tool needs a C/C++ compiler. In the end, no matter what tool you are using, you are calling out to your platforms compiler binary. So why using a dependency for making a call to
gccwhen you can do it withchild_process?eljefedelrodeodeljefe commentedon Jun 9, 2016
@kzc fun fact: the reason why you want to use some toolchain is to avoid writing something like the below just to compile an
addon.cc. But you can do it. Same goes for runningmake -j8and copy pasting and output line when you compile node in order to mirror gyps (and any other build tool's behaviour)kzc commentedon Jun 9, 2016
Great. Then we're in agreement on that point.
Using node to bootstrap node is a needless complication. It can be done, but it's not worth the trouble when other ways are simpler.
eljefedelrodeodeljefe commentedon Jun 9, 2016
-.- I bail the conversation here.
jbergstroem commentedon Jun 9, 2016
@kzc the relevant question here isn't bootstrapping node itself, rather being able to compile node native modules on a system where node is installed.
kzc commentedon Jun 9, 2016
@jbergstroem Ah, okay. My mistake.
But perhaps this JS port of GYP initiative could have applications to bootstrapping node itself if successful.
indutny commentedon Jun 12, 2016
Port of GYP (WIP): https://github.com/indutny/gyp.js
Trott commentedon Jul 7, 2017
This issue has been inactive for sufficiently long that it seems like perhaps it should be closed. Feel free to re-open (or leave a comment requesting that it be re-opened) if you disagree. I'm just tidying up and not acting on a super-strong opinion or anything like that.