Repository navigation
npm_install requires external build tools for installing native modules #506
Description
Activity
Do you hit this issue by running npm outside of Bazel? I suppose no?
I understand that this violates the hermeticity but it might be a reasonable compromise so that we do not couple
rules_nodejswith the external tooling involved in building native node modules.- What do you suggest? That we have every developer install make and various other tools needed (we stumbled on to two more IIRC)? I’m not asking sarcastically (though am a bit worried). Have you encountered this in the past? What did you do? I’d be surprised if @alexeagle never faced this dilemma…On Fri, 18 Jan 2019 at 23:42 Minko Gechev ***@***.***> wrote: Do you hit this issue by running npm outside of Bazel? I suppose no? I understand that this violates the hermeticity but it might be a reasonable compromise so that we do not couple rules_nodejs with the external tooling involved in building native node modules. cc @alexeagle <https://github.com/alexeagle> @gregmagolan <https://github.com/gregmagolan> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#506 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIF3b45WEA-dundMFR1tY4DQNIbNmNks5vEj-7gaJpZM4aIk_c> .
@gregmagolan can you update about the toolchain approach we discussed? Thanks!
Yes. @ittaiz @alexeagle and I discussed this at the bazel rules summit last week. The thinking is that we should be able to resolve this with bazel toolchains. We're planning to migrate nodejs to bazel toolchains in the next month so we'll look at this at that point.
Reacted by Guy TregerDoesn't this break determinism as well? The
npmprocess for building native modules seems to break caching for me.- Greg, Any chance you have an update?…On Sun, 3 Mar 2019 at 5:57 Rohan Singh ***@***.***> wrote: Doesn't this break determinism as well? The npm process for building native modules isn't deterministic or hermetic, and breaks caching. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#506 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIFyQj9zcaSOSASoUQG8iYYQGr3vRkks5vS0gNgaJpZM4aIk_c> .
Flyby comment: I think the long term solution for this one is getting projects that publish native code to instead publish WASM
Reacted by Patrick Rice and Edvard Hübinette- added 2 commits that reference this issue
on Oct 17, 2020 Is there any planned next-steps on this? Or is it just a big question-mark still?
We have similar issues where we can't use
npm_installdue to failures with modules installing themselves vianode-gyp. We live in a weird mixed world at the moment (we have hacked around to get theBUILDfile generation but leave module installation outside of bazel), but we'd much prefer to get to the standard patterns expected by the library.*clarification, our issue with node modules and
node-gypis explicitly when executing in our Linux docker container on CI, however it works fine on our local MacOS machines.for now, I think we've found a workaround with using
npm_command = "install", as it's only whenbazelrunsnpm ciin these environments that it seems to fail to installnode-gyp-buildappropriately (?)*edit: that only worked due to
symlink_node_modulesbeing set toTrue, however we'll probably want to use the default once we can updaterules_nodejs(nowFalse). That sets us back to the same issue as before :/Ok, so our issue isn't purely with
node-gyp-buildbut rather that, when it executes, it can't findnode-gyp. Which shouldn't be a thing seeing thatnode-gypis a pack-in withnpmitself.So, if I get onto the machine and use
npmoutside ofbazelto globally installnode-gyp(shouldn't be needed, asnode-gypis a pack-in withnode), then we get past this asbazelis finding that secondary installednode-gyp.Poking deeper, I patched our version of the
_npm.shgenerated here to include a callnpm ls -g node-gypimmediately before the call tonpm ci. It returned:`-- npm@8.1.0 +-- @npmcli/run-script@2.0.0 | `-- node-gyp@8.2.0 deduped `-- node-gyp@8.2.0yet still failed on the very next line with:
npm ERR! command failed npm ERR! command sh -c node-gyp-build npm ERR! node:events:368 npm ERR! throw er; // Unhandled 'error' event npm ERR! ^ npm ERR! npm ERR! Error: spawn node-gyp ENOENT npm ERR! at Process.ChildProcess._handle.onexit (node:internal/child_process:282:19)which has me extra flummoxed...
I was hoping this was maybe due to something with symlinked paths, assuming it was executing the
node-gypfrom withinutf-8-validate's build (that's who's usingnode-gyp-build), and during that install thecwdis a different place, but various attempts to get--preserve-symlinksfor node and/or--scripts-prepend-node-path=truefornpminto the command line have also failed to produce any results.At the moment, we're solving this with a forced global install of
node-gyp(i.e.npm i -g node-gyp), so that when the other modules get installed it can findnode-gypwithout issue. This isn't great though.Ok, one last update: this isn't Linux specific. We're seeing the same issue on our MacOS containers as well.
phooey, the "globally install" trick isn't working on our MacOS containers :/
We never found a solution under the old versions of
rules_nodejs. We now use the latest version ofrules_nodejsalong withaspect-build/rules_jsand don't have this specific issue.We did however recently run into an issue with
node-gyp-build, which was solved by apnpm.packageExtensionsblock in ourpackage.json(forcingnode-gypas a dependency of packages which depended onnode-gyp-build).Reacted by Eddie MashayevNo longer in scope for rules_nodejs
🐞 bug report
Affected Rule
The issue is caused by the rule: npm_installIs this a regression?
Not that I have noticed.Description
When I use npm_install on a package.json that includes, in its list of dependencies, a native node module, the run of the rule attempts to use build tools external to bazel (e.g. **make**) and fails when such build tools are not present in the linux environment where bazel is being run.After installing the required tools (i.e. running
apt-get install build-essential), running the rule succeeds. This violates bazel's proclaimed hermeticity to some extent.🔬 Minimal Reproduction
Partial content of WORKSPACE file:
Content of pacakge.json:
🔥 Exception or Error
🌍 Your Environment
Operating System:
Output of
bazel version:Rules version (SHA):
Encountered as part of using Bazel at Wix.com @ittaiz