MacOS Runner Nix Integration for Platform-Specific Build

[CMCDragonkai]

Specification

The Homebrew system that is currently being used to install our packages. It is taking quite some time even with all the caching.

This is partly because the node version we want and the homebrew formula that is available isn't already available on the macos VM.

Homebrew ends up installing alot of other dependencies, and although it's not that much, it's quite a bit slower than Nix.

So that means a job that takes 3 minutes on our Linux runner, is now taking 23 minutes on our macos runner on gitlab.

Now I'm tried installing Nix on macos on the M1 mac mini. There are some notes to consider.

1. The installation process requires sudo, thankfully both Gitlab and Github have passwordless sudo for their mac runers.
2. Therefore sudo sh <(curl -L https://nixos.org/nix/install) should be sufficient to ensure an unattended installation. However this is not confirmed (it has to do a bunch of disk manipulation too to get /nix/store up and running). Otherwise look at the installation script to see what env variables or flags must be set: https://github.com/NixOS/nix/blob/master/scripts/install-darwin-multi-user.sh
3. The installation changes /etc/bashrc and /etc/zshrc. In order to ensure you can get everything Nix provides on-path, you need to source their script. . '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'.
4. While homebrew is slow. Nix has to install alot of things to get setup on our project, and not everything is compatible. On my first setup of TypeScript-Demo-Lib, I had to download 250 MiB worth of packages, and 1 GiB unpacked. Even then things like multiStdenv isn't compatible, and node2nix currently has a derivation that is limited to x86_64-linux. I'm pretty sure node2nix is incorrect, the derivation has no requirement on x86_64-linux (https://github.com/svanderburg/node2nix/blob/68f5735f9a56737e3fedceb182705985e3ab8799/nix/node-env.nix#L212). There's also some potential configuration required in ~/.config/nixpkgs/config.nix to allow packages that is considered "deprecated".

Since it has to do so much and also compile packages, I'm not confident that using Nix will be appreciably faster than homebrew. It may be marginally faster since with our private Nix cache, any compiled packages will end up there.

Trying to ensure compatibility between the darwin-arm64 and linux-x64 is going to be difficult, and it most likely will require us to replace node2nix with our own nix system that understands javascript/typescript properly.

BTW, it turns out that the nix installation on MacOS doesn't use NIX_PATH. This is because the nix on mac is not a "nixos", it's just a Mac with the nix program.

It seems that the best solution would to have our own Mac runners with the appropriate setup already done. We can contribute to this project: https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/tree/main to get Nix into it. This means continuing to use gitlab runners within the gitlab platform.

Even GitHub actions would have similar problems.

At any-case, I was not successful in running a nix-shell atm due to node2nix.

Additional context

• WIP: Demonstrating Native Addons TypeScript-Demo-Lib#38 - big work that brought in native-addons and CI/CD build pipeline involving macos and windows
• https://discourse.nixos.org/t/nix-macos-monthly/12330/21 - discussion on macos and nix usage
• Status of macOS NixOS/nixpkgs#116341 - macos support in nixpkgs
• https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners - github comparison
• Note that even though Homebrew installs multi-user in /opt/homebrew/bin, its sourcing of eval "$(/opt/homebrew/bin/brew shellenv)" is put into ~/.zprofile. Which means other users won't be able to get the PATH until they set that up. In comparison, nix puts it into the /etc/bashrc and /etc/zshrc. This means, Nix PATH would be overwritten by homebrew PATH. However this won't matter when using nix-shell.
• Cannot run on macos aarch64-darwin svanderburg/node2nix#297 - Node2nix issue about how addintegrityfields.js.drv should not need x86_64-linux.
• If you want mac to be like nixos, then you actually need this third party system: https://github.com/LnL7/nix-darwin (which is installed separately from installing official nix)

Tasks

1. ...
2. ...
3. ...