pkgs/by-name

[lorenzleutgeb]

Inspired by NixOS/rfcs#140 I'd like to propose: In order to minimize the size of /all-packages.nix let's agree on a convention for registering packages simply by their location in the tree.

My suggestion would be to use /pkgs/by-name/${pname}/package.nix. As of today, the following packages could be moved:

• euclid3
• flarum
• gnunet-messenger-cli
• kikit
• lcrq
• lcsync
• libgnunetchat
• librecast
• pcbnew-transition
• pybars3
• pymeta3
• rosenpass
• rosenpass-tools

I expect that most packages in this repo would fit pkgs/by-name. Further, I do not expect that shards (like in nixpkgs) will be required anytime soon. If that point is ever reached, they can easily be introduced automatically.

I expect the glue code required to produce a package from such a structure to be very similar to https://github.com/tweag/nixpkgs/blob/77d50b03e4388f22e1f36a2621a9287a12a138be/pkgs/top-level/by-name-overlay.nix and I'd be happy to implement it.

Such a convention might speed up things like #12 slightly.

[alejandrosame]

The by-name-overlay.nix expression is not reusable as is due to a hardcoded local import to lib.

Since last week I was working on adapting the pkgs/by-name pattern for a personal project I have, so I'm leaving some breadcrumbs in case ngipkgs wants to adopt the pattern in some capacity before it stabilizes in nixpkgs.

I arrived for the time being to this by-name-overlay.nix file:

# Adapted from https://github.com/NixOS/nixpkgs/commit/f6467c357419d70d8f32816fe68b9bde6278f8b0
# Reference: https://discourse.nixos.org/t/namespacing-scoping-a-group-of-packages/13782/10
# Type: String -> String -> Path -> Overlay
namespace: scope: baseDirectory: self: super:
let
    newScope = extra: super.lib.callPackageWith (super // defaults // extra);
    defaults = {};

    byNameMapper = let
      inherit (builtins)
        readDir
        ;

      inherit (self.lib.attrsets)
        mapAttrs
        mapAttrsToList
        mergeAttrsList
        ;

      # Package files for a simple package (not sharding but scoping)
      # Type: String -> String -> AttrsOf Path
      namesForScope = name: type:
        if type != "directory" then
          { }
        else
          mapAttrs
            (name: _: baseDirectory + "/${scope}/${name}/package.nix")
            (readDir (baseDirectory + "/${scope}"));

      packageFiles = mergeAttrsList (mapAttrsToList namesForScope (readDir baseDirectory));
    in
      self': mapAttrs (name: file:
        self'.callPackage file { }
      ) packageFiles;
in
  {
    ${namespace}.${scope} = self.lib.makeScope newScope byNameMapper;
  }

The packages (a bunch of bash scripts in this case) to be imported by-name are structured in the following way:

.
├── by-name-overlay.nix
├── flake.lock
├── flake.nix
└── pkgs
    └── by-name
        └── pkgs
            ├── pkg1
            │   ├── package.nix
            │   └── script.sh
            ...
            └── pkgn
                ├── package.nix
                └── script.sh

In the flake.nix I can then generate and use the overlay with

pkgs = import nixpkgs {
  inherit system;
  overlays = [
    (import ./by-name-overlay.nix "myNamespace" "pkgs" ./pkgs/by-name)
  ];
};
myNamespacePackages = pkgs.myNamespace.pkgs;

[infinisil]

  {
    ${namespace}.${scope} = self.lib.makeScope newScope byNameMapper;
  }

Watch out, applying multiple such overrides will clear ${namespace} every time. Something like this is necessary:

{ 
  ${namespace} = super.${namespace} or {} // {
    ${scope} = ...;
  };
}