-
-
Notifications
You must be signed in to change notification settings - Fork 343
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NixOS module support: Separate NUR nixpkgs from repos nixpkgs, avoid callPackages #27
Conversation
I like the idea of having a separate expression source for fetchgit and co. I try to think about the most straightforward syntax the next days or so. Documentation should be definitely added before merging any changes. |
One thing that sucks about |
👍 to channel-like behavior, and if not anything that can fallback when remote resource is unavailable (server down, you're on a plane, etc.) would be great. Kinda lame to have eval fail in those cases. I filed an issue on Nix about this and one solution suggested was to basically (temporarily) bump ttl to infinity (or some high value). Hope this helps and regardless glad your considering this with NUR! |
In the latest commit I added support for still being able to use the deprecated callPackage syntax, but it issuing a warning. Now when you try to use such a repository for a NixOS module you'll get this:
Also, I added support for specifying an attrset directly in the repos default.nix, so this works too now: {
lib = import ./lib;
modules = import ./modules;
} I'll put the logic to evaluate a repository into a separate Nix file in future commits, such that the update script won't have to duplicate it. |
default.nix
Outdated
|
||
{ pkgs ? import <nixpkgs> {} }: | ||
{ nurpkgs ? import <nixpkgs> {} # For nixpkgs dependencies used by NUR itself | ||
, pkgs ? import <nixpkgs> {} # Dependencies to call NUR repos with |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we make this argument mandatory and add a error message if not supplied by the user?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The error message could then also include the canonical syntax to import NUR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure, making the pkgs
argument mandatory would mean that import <nur> {}
wouldn't work, and you couldn't evaluate it anymore directly, e.g. nix-instantiate '<nur>' -A repos.infinisil.foo
wouldn't work anymore.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should it then default to one import
expression for example:
let fallback = import <nixpkgs> {} in {
nurpkgs ? fallback,
pkgs ? fallback
}
I am not sure if this make any difference in terms of performance.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
According to @cleverca22 and @LnL7 in #nixos, what you proposed should be faster, so I'll change it to that.
+1 for adding warnings. For the sake of simplicity I would prefer if all repo root expression take with import <nixpkgs> {};
{
} because they don't understand the implication of it yet.
is too much boiler code to ask for and makes all modules looks the same (also simpler documentation). |
Yeah that sounds like a good idea. Should we also disallow defaulted arguments though? E.g. Edit: Yeah I think default arguments should be allowed, they can be used to do stuff like |
I need some help with python (I don't know python well at all): The latest commit needs the path to the new evalRepo.nix file in python, but I'm not sure how to get it. |
The update script is now working again with the changes I made. (For some reason eeva's repo is being evaluated every time, but that was like this before my changes as well). |
nur/update.py
Outdated
@@ -169,7 +169,11 @@ def eval_repo(spec: RepoSpec, repo_path: Path) -> None: | |||
with open(eval_path, "w") as f: | |||
f.write(f""" | |||
with import <nixpkgs> {{}}; | |||
callPackages {repo_path.joinpath(spec.nix_file)} {{}} | |||
import /home/infinisil/src/NUR/evalRepo.nix {{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This here, I don't think it can stay like this :P
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The following should do it:
import {ROOT.joinpath("evalRepo.nix")} {{
evalRepo.nix
Outdated
@@ -0,0 +1,20 @@ | |||
{ name |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think to establish best practices, this could go to lib/
nur/update.py
Outdated
@@ -184,6 +188,7 @@ def eval_repo(spec: RepoSpec, repo_path: Path) -> None: | |||
"-I", f"nixpkgs={nixpkgs_path()}", | |||
"-I", str(repo_path), | |||
"-I", str(eval_path), | |||
"-I", str("/home/infinisil/src/NUR/evalRepo.nix") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If everything would be in lib
then "-I", str(ROOT.joinpath("lib"))
would cover it.
If you have trouble get it working, I can take tomorrow a look at it again. |
Thanks! Got it figured out, there was a problem of ROOT being just I'll add some docs in future commits, and will squash commits into reasonable changes when ready for merge |
README.md
Outdated
nur = pkgs.callPackage (import (builtins.fetchGit { | ||
url = "https://github.com/nix-community/NUR"; | ||
})) {}; | ||
nur = import (builtins.fetchGit "https://github.com/nix-community/NUR") { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it is also time to switch to fetchTarball
, which is apparently faster then fetchgit
- even for incremental updates - since the repository is rather small.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, will be a bit longer though, but I think that's okay
README.md
Outdated
@@ -135,13 +135,13 @@ in stdenv.mkDerivation rec { | |||
You can use `nix-shell` or `nix-build` to build your packages: | |||
|
|||
```console | |||
$ nix-shell -E 'with import <nixpkgs>{}; (callPackage ./default.nix {}).inxi' | |||
$ nix-shell -E '(import ./default.nix { pkgs = import <nixpkgs>{}; }).inxi' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is the import required here? There is a default nixpkgs import in the file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, sorry this is for the repository, not for NUR.
README.md
Outdated
nix-shell> inxi | ||
nix-shell> find $buildInputs | ||
``` | ||
|
||
```console | ||
$ nix-build -E 'with import <nixpkgs>{}; (callPackage ./default.nix {})' | ||
$ nix-build -E '(import ./default.nix { pkgs = import <nixpkgs>{}; }).inxi' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With the new repository format the following is also possible:
nix-build --arg pkgs "import <nixpkgs> {}" -A inxi
Each repository should return a set of Nix derivations: | ||
|
||
```nix | ||
{ callPackage }: | ||
{ pkgs }: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This might be a bit more convenient for nix-build/nix-shell:
{ pkgs ? import <nixpkgs> {} }:
Nix-shell would then shorten too:
$ nix-shell -E '(import ./default.nix {}).inxi'
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also this might encourage people again to import nixpkgs themselves?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It doesn't matter as long as import <nixpkgs> {}
is only used as the default value, since NUR will pass its own pkgs to override it. (I added a section for that in the newest commit).
I added the This PR is more-or-less ready now, probably needs some tweaks here and there. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nice
README.md
Outdated
modules = ./import modules; | ||
{ pkgs, config, lib, ... }: | ||
let | ||
nur = import (builtins.fetchTarball "https://github.com/nix-community/NUR/archive/master.tar.gz") {}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
{ inherit pkgs; };
at the end?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, I'm not sure, because pkgs
isn't actually needed when you just use NUR for modules/overlays/library functions.
Oh I think what would be neat is to change the default pkgs
argument used by NUR from import <nixpkgs> {}
to throw "NUR didn't receive a pkgs argument"
. Then usage without a pkgs
argument is just fine as long as you don't evaluate it (which is the case when you only need NixOS modules), but as soon as you try to use a package, you get the throw.
README.md
Outdated
|
||
```nix | ||
{ lib }: | ||
{ pkgs, lib }: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it necessary to pass lib
if it's accessible through pkgs.lib
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using lib
without a dependency on pkgs
means that it's possible to use library functions (which are built only via lib
) in your NixOS config without having to worry about infinite recursion, e.g. when using such a function in an overlay (using pkgs.lib
would give inf rec because pkgs depends on overlays).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In an overlay lib
from super
is supposed to be used and for nixos modules, lib
from the function argument is preferred. Would the current approach bypass overlays by reimporting nixpkgs
? Are there any other use cases that I have not covered yet?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would the current approach bypass overlays by reimporting
nixpkgs
Not sure what you mean by that, overlays should still work as one would expect. A definition would be like
{ pkgs, lib }: {
overlays.foo = self: super: {
bar = super.hello.override ...;
# super.lib.dosomethingwithlib
};
}
Yeah, nixpkgs would get reimported if you were using the lib
argument from NUR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To clarify my previous comment: By reading the writeup on overlays it recommends to use the super
argument of the overlay to get access to library functions:
Quote:
super one overlay down in the stack (and base nixpkgs for the first overlay). Use it to access the package recipes you want to customize, and for library functions.
I therefore conclude that using a dedicated lib
from our own import in overlays and modules is neither necessary nor best practice. I think this is a pitfall worth documenting though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm yeah, actually the lib
argument really doesn't have a lot of uses other than for using it for library functions, and even then, it will only be a problem in NixOS in certain not even common cases. So maybe it would be best to just not provide lib
for now, only add it later if problems arise.
let | ||
inherit (pkgs) fetchgit fetchzip callPackages lib; | ||
inherit (nurpkgs) fetchgit fetchzip lib; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe add a note that the builtins have issues with TTL
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Probably not at this line of code, but yeah I can add that to the readme :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is possible to prevent garbage collection by linking used tarballs in a profile?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think so
71e8b18
to
0d3917b
Compare
I rebased on master, and squashed all commits into 3, one for docs, one for nix and one for python. Noteworthy changes are:
|
}; | ||
} | ||
``` | ||
|
||
### Pinning | ||
|
||
Using `builtins.fetchTarball` without a sha256 will only cache the download for 1 hour by default, so you need internet access almost every time you build something. You can pin the version if you don't want that: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This section is new
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Update I have not yet understand the full consequence of a fixed-input derivation: I have a non-stable URL in configuration.nix that I gave a checksum and nix is testing it from time to time and complains if the checksum changes. On the other hand I never had evaluation errors because the file is not available. Does this only works in my case because the referenced file is part of system profile closure or does this also apply to imported nix expression?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For fixed output derivations (ones you give a hash), Nix will first check if a file with that hash is already in the store, and use that if there is. It will only try to fetch it when no such hash could be found. After the fetch it will only succeed in adding the download if the hash you gave matches the one it calculated. If it doesn't match, it fails. The fetchers from nixpkgs will also output the correct hash on a mismatch.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How does nix decide, when to delete fixed-input derivations in this case?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the only way it deletes them is garbage collection.
default.nix
Outdated
@@ -1,7 +1,10 @@ | |||
{ nurpkgs ? import <nixpkgs> {} # For nixpkgs dependencies used by NUR itself | |||
# Dependencies to call NUR repos with | |||
, pkgs ? throw "NUR call didn't receive a pkgs argument, but the evaluation requires it." |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is new, I think this should be okay, there's no need to provide an implicit pkgs
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also coincides with requiring a pkgs
before this PR, as it was pkgs.callPackage (fetchGit ...)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This change is great because it will help users when using NUR wrong. I would be cool if it also contains a small example how the fix the error.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mhm. It might be not only the user to blame, the repo author could also use it incorrectly... tricky stuff.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can add a note that says it could be an error on either side.
# Get the revision by choosing a version from https://github.com/nix-community/NUR/commits/master | ||
url = "https://github.com/nix-community/NUR/archive/3a6a6f4da737da41e27922ce2cfacf68a109ebce.tar.gz"; | ||
# Get the hash by running `nix-prefetch-url --unpack <url>` on the above url | ||
sha256 = "04387gzgl8y555b3lkz9aiw9xsldfg4zmzp930m62qw8zbrvrshd"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Future work (not in the scope of this pr) - automate pinning updates.
README.md
Outdated
```nix | ||
{ pkgs, config, lib, ... }: | ||
let | ||
nur = import (builtins.fetchTarball "https://github.com/nix-community/NUR/archive/master.tar.gz") {}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is easy to shadow nur by using environment.systemPackages = [ nur.repos.mic92.inxi ];
instead of environment.systemPackages = [ pkgs.nur.repos.mic92.inxi ];
.
In this case pkgs
in NUR will be undefined and the error message you added is thrown.
Maybe the example should not called it nur
for overlays/modules to avoid this namespace collision?
The alternative could be also to make nur
a nixos module (also it is probably not possible to dynamically inject imports and overlays?) - maybe a function that returns a nixos module could work.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The alternative could be also to make nur a nixos module
Yeah I thought of that! Would actually solve this problem rather elegantly: Users could add something like imports = [ (import <nur/modules> {}) ]
(replace nur with the fetch command) exactly once to their nixos config.
To make this work, NUR should wrap every repositories modules under an enable option. So e.g. when I now want to use your foo module (which you declared in modules.foo
), I'd do nur.mic92.foo.enable = true
. This would work like this, and I might implement this in a future PR. For now this more hacky solution should be fine, especially since it can be forwards compatible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh and regarding the nur
attribute name, I think changing it to nur_for_modules
would work well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
True, the nixos module can be also implemented in future. I also intentionally put repos into sub namespace so we can put more into the top-level hierarchy.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tracked here: #43
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/nur_for_modules/nur-modules/g
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe nur-no-pkgs
? Because the example there also uses overlays with it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed.
Sorry for being extra nitpicky and critical on this pull request, but since it requires changes for both users and maintainers I would like to keep the revisions on the API low. |
No problem, I'm also always very much like this on non-trivial nixpkgs PR reviews :) |
Added a super descriptive error message which includes the repository the error originates from. |
Feel free to merge this, if you think it is ready. Please also notify all repo users so they can update accordingly. |
This is essential to get modules in NUR to work. By taking a separate argument for NUR's nixpkgs (for fetchgit, fetchzip and lib), we don't need to evaluate the nixpkgs used for repos. This also implies that you won't be able to `callPackage` NUR anymore, and instead you'll have to use `import (builtins.fetchGit ".../NUR") { inherit pkgs; }` instead. Doing this also prevents the evaluation of pkgs. In case of NixOS, this pkgs depends on your whole config, which is the source of the recursion. Evaluating this at the last possible moment is key. This also means that you won't be able to take package arguments in a repo definition, you instead get just `pkgs`, also to avoid evaluation of it. An error will be thrown when pkgs was required for evaluation but wasn't passed to the NUR import The old callPackage syntax will still be supported albeit with a warning Also repos receive a lib argument, Using this lib instead of pkgs.lib makes it possible to define library functions that use other library functions without depending on pkgs -> should prevent some infinite recursion cases for NixOS module usage.
- changes for new style of invoking NUR (passing pkgs argument) - Using fetchTarball instead of fetchGit for speed - Add more sections to readme
Made some minor adjustments and fixes:
I tested everything now and I think it's ready. I'll merge this today/tomorrow and open an issue mentioning all people that need to update their repo. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! It's taken shape rather nicely compared to the initial version
nur = pkgs.callPackage (import (builtins.fetchGit { | ||
url = "https://github.com/nix-community/NUR"; | ||
})) {}; | ||
nur = import (builtins.fetchTarball "https://github.com/nix-community/NUR/archive/master.tar.gz") { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry I missed why using callPackage was an issue. Both are fine for me
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Because when you do pkgs.callPackage
, this needs to evaluate pkgs
(to get the attribute), which requires evaluating all overlays and the config (because they're passed to the nixpkgs import), but those overlays can be defined in all modules. So if you want to imports = [ nur.repos.infinisil.modules.foo ]
, this gives infinite recursion, because to evaluate the modules it needs to evaluate pkgs!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, that makes a lot of sense
Sorry for saying this after the merge, but I think packages should have been in a separate attribute too, since now there's no way to only get packages from a NUR without doing this |
@fgaz Yeah I thought of that too. Doesn't really have anything to do with this PR though, it was the same before. Maybe open an issue regarding that |
@fgaz I have a pkgs attribute in my nur-packages, additional to having them as top-level attributes. I use that to overlay just my packages locally (lib would cause problems if overlayed naively). Do you intend for something like this to become convention, or do you want all top-level packages to be gone? |
NixOS module support: Separate NUR nixpkgs from repos nixpkgs, avoid callPackages
…hub_actions/actions/checkout-v2.3.3 Bump actions/checkout from v2.3.2 to v2.3.3
This is essential to get modules in NUR to work. By taking a separate
argument for NUR's nixpkgs (for fetchgit, fetchzip and lib), we don't
need to evaluate the nixpkgs used for repos.
This also implies that you won't be able to
callPackage
NUR anymore,and instead you'll have to use
import (builtins.fetchGit ".../NUR") { inherit pkgs; }
instead. Doing this also prevents the evaluation ofpkgs. In case of NixOS, this pkgs depends on your whole config, which is
the source of the recursion. Evaluating this at the last possible moment
is key.
This also means that you won't be able to take package arguments in a
repo definition, you instead get just
pkgs
, also to avoid evaluationof it.
New install instructions would be
To get NixOS modules to work you'll have to use something like
Preferably one would set
nur=https://github.com/nix-community/NUR
in their NIX_PATH, which allows one to doAlso the pkgs argument should be made optional, so you don't even need to pass it when importing a module.
Docs and stuff would need to be updated, I might do this later.