nvfetcher

nvfetcher is a tool to automate nix package updates. It's built on top of shake, integrating nvchecker. nvfetcher CLI program accepts a TOML file as config, which defines a set of package sources to run.

Overview

For example, feeding the following configuration to nvfetcher:

# nvfetcher.toml
[rustdeck]
src.github = "rustdesk/rustdesk"
fetch.github = "rustdesk/rustdesk"
# Compute outputHashes for git dependencies in cargo lock
cargo_lock = ["Cargo.lock"]

You would get _sources/generated.nix:

# This file was generated by nvfetcher, please do not modify it manually.
{
  fetchgit,
  fetchurl,
  fetchFromGitHub,
  dockerTools,
}:
{
  rustdesk = {
    pname = "rustdesk";
    version = "1.4.2";
    src = fetchFromGitHub {
      owner = "rustdesk";
      repo = "rustdesk";
      rev = "1.4.2";
      fetchSubmodules = false;
      sha256 = "sha256-b/KuWECsmEcz9nPiuPqCcGQ1p+GdbaELScaIXnUTGv8=";
    };
    cargoLock."Cargo.lock" = {
      lockFile = ./. + "/sha256-b_KuWECsmEcz9nPiuPqCcGQ1p+GdbaELScaIXnUTGv8=/Cargo.lock";
      outputHashes = {
        "filedescriptor-0.8.2" = "sha256-zXTt9eHAVdCPipWAMFeiqk1JteX+7IZcXZ7uvxkzDjQ=";
         # ...
      };
    };
  };
}

and _sources/generated.json:

{
    "rustdesk": {
        "cargoLock": {
            "Cargo.lock": [
                "sha256-b_KuWECsmEcz9nPiuPqCcGQ1p+GdbaELScaIXnUTGv8=/Cargo.lock",
                {
                    "android-wakelock-0.1.0": "sha256-09EH/U1BBs3l4galQOrTKmPUYBgryUjfc/rqPZhdYc4=",
                    // ...
                }
            ]
        },
        "date": null,
        "extract": null,
        "name": "rustdesk",
        "passthru": null,
        "pinned": false,
        "src": {
            "deepClone": false,
            "fetchSubmodules": false,
            "leaveDotGit": false,
            "name": null,
            "owner": "rustdesk",
            "repo": "rustdesk",
            "rev": "1.4.2",
            "sha256": "sha256-b/KuWECsmEcz9nPiuPqCcGQ1p+GdbaELScaIXnUTGv8=",
            "sparseCheckout": [],
            "type": "github"
        },
        "version": "1.4.2"
    }
}

We tell nvfetcher how to get the latest version number of packages and how to fetch their sources given version numbers, and nvfetcher will help us keep their version and prefetched SHA256 sums up-to-date, producing ready-to-use nix expressions in _sources/generated.nix.

nvfetcher reads generated.json to produce version change messages, such as feeluown: 3.8.1 → 3.8.2. Versions of packages are always checked during each run, but only do prefetch and further operations when needed.

Live examples

How to use the generated sources file? Here are several examples:

• My flakes repo

Add your own flakes repo here if you want to share it with others!

Installation

nvfetcher package is available in nixpkgs, so you can try it with:

$ nix-shell -p nvfetcher

This repo also has flakes support:

$ nix run github:berberman/nvfetcher

To use it as a Haskell library, the package is available on Hackage. If you want to use the Haskell library from flakes, there is also a shell ghcWithNvfetcher:

$ nix develop github:berberman/nvfetcher#ghcWithNvfetcher
$ runghc Main.hs

where you can define packages in Main.hs. See Haskell library for details.

Usage

Basically, there are two ways to use nvfetcher, where the difference is how we provide package sources definitions to it.

CLI

To run nvfetcher as a CLI program, you'll need to provide package sources defined in TOML.

Usage: nvfetcher [--version] [--help] [-o|--build-dir DIR] [--commit-changes] 
                 [-l|--changelog FILE] [-j NUM] [-r|--retry NUM] [-t|--timing] 
                 [-v|--verbose] [-f|--filter REGEX] [-k|--keyfile FILE] 
                 [--keep-old] [--keep-going] [TARGET] [-c|--config FILE]

  generate nix sources expr for the latest version of packages

Available options:
  --version                Show version
  --help                   Show this help text
  -o,--build-dir DIR       Directory that nvfetcher puts artifacts to
                           (default: "_sources")
  --commit-changes         `git commit` build dir with version changes as commit
                           message
  -l,--changelog FILE      Dump version changes to a file
  -j NUM                   Number of threads (0: detected number of processors)
                           (default: 0)
  -r,--retry NUM           Times to retry of some rules (nvchecker, prefetch,
                           nix-build, etc.) (default: 3)
  -t,--timing              Show build time
  -v,--verbose             Verbose mode
  -f,--filter REGEX        Regex to filter packages to be updated
  -k,--keyfile FILE        Nvchecker keyfile
  --keep-old               Don't remove old files other than generated json and
                           nix before build
  --keep-going             Don't stop if some packages failed to be fetched
  TARGET                   Three targets are available: 1.build 2.clean (remove
                           all generated files) 3.purge (remove shake db)
                           (default: build)
  -c,--config FILE         Path to nvfetcher TOML config
                           (default: "nvfetcher.toml")

Each package corresponds to a TOML table: name is encoded as table key, with two required fields and three optional fields in each table. You can find an example of the configuration file, see nvfetcher_example.toml.

Keyfile

You can specify nvchecker keyfile via command line option. For the format of this file, please refer to nvchecker documentation.

Nvchecker

Version source -- how do we track upstream version updates?

• src.github = owner/repo - the latest github release
• src.github_tag = owner/repo - the max github tag, usually used with list options (see below)
• src.pypi = pypi_name - the latest pypi release
• src.git = git_url (and an optional src.branch = git_branch) - the latest commit of a repo
• src.archpkg = archlinux_pkg_name -- the latest version of an archlinux package
• src.aur = aur_pkg_name -- the latest version of an aur package
• src.manual = v -- a fixed version, which never updates
• src.repology = project:repo -- the latest version from repology
• src.webpage = web_url and src.regex -- a string in webpage that matches with regex
• src.httpheader = request_url and src.regex -- a string in http header that matches with regex
• src.openvsx = publisher.ext_name -- the latest version of a vscode extension from open vsx
• src.vsmarketplace = publisher.ext_name -- the latest version of a vscode extension from vscode marketplace
• src.cmd = cmd -- the version from a shell command (e.g. echo Meow)
• src.container = owner/name - the latest tag of a container from the Docker registry

Optional list options for some version sources (src.github_tag, src.webpage, and src.httpheader and src.container), see the corresponding nvchecker documentation for details.

• src.include_regex
• src.exclude_regex
• src.sort_version_key
• src.ignored

Optional global options for all kinds of version sources, see the corresponding nvchecker documentation for details. You can tweak obtained version number using this option, e.g. stripping the prefix v or transforming the result by regex.

• src.prefix
• src.from_pattern
• src.to_pattern

Nix fetcher

How do we fetch the package source if we have the target version number? $ver is available in string, which will be set to the result of nvchecker.

• fetch.github = owner/repo
• fetch.pypi = pypi_name
• fetch.git = git_url
• fetch.url = url
• fetch.openvsx = publisher.ext_name
• fetch.vsmarketplace = publisher.ext_name
• fetch.tarball = tarball_url
• fetch.docker = owner/name

Optional config for nix-prefetch-url, applies when the fetcher equals to fetch.url. $ver is available in string, just like for the fetch config.

• url.name = file_name

Optional config for nix-prefetch-git, applies when the fetcher equals to fetch.github or fetch.git.

• git.deepClone
• git.fetchSubmodules
• git.leaveDotGit

Optional config for fetch.docker/dockerTools.pullImage:

• docker.os
• docker.arch
• docker.finalImageName
• docker.finalImageTag
• docker.tlsVerify

Extract source files

Optional source extracting config, files are extracted into build directory.

• extract = [ "glob_1", "glob_2", ...] - globs are relative to the source root

Each glob will be resolved to a list of files matching the pattern. For each matching file, nvfetcher copies it to the build directory, and generates extract.<file_name> = ./. + "<path>" entry in the nix expr.

Recursive globs are supported, e.g. one can write extract = [ "**/*" ] to pull all files in a package to build directory. See Glob for details.

Note: Directories are ignored.

Rust support

rustPlatform.buildRustPackage now accepts an attribute cargoLock to vendor dependencies from Cargo.lock, so we can use this instead of TOFU cargoSha256 for Rust packaging. nvfetcher supports automating this process, extracting the lock file to build and calculating cargoLock.outputHashes, as long as you set the config. There can be many lock files in one source.

• cargo_lock = [ "cargo_lock_glob_1", "cargo_lock_glob_2", ...] - globs are relative to the source root

Similar to extract, cargo locks will be pulled into build directory.

Passthru

passthru config, an additional set of attrs to be generated.

• passthru = { k1 = "v1", k2 = "v2", ... }

Note: currently the values can only be strings

Pinned

If a package is pinned, we call nvchecker to check the new version iff there's no existing version.

• pinned = true

Git commit date

If the version source of a package is git, nvfetcher can finds out the commit date of this revision, in the format of %Y-%m-%d by default. You can provide your own strftime format:

• git.date_format = "strftime_format"
• git.date_tz = "timezone"

You can use the git.date_tz option to specify the time zone for the commit date. Accepts IANA time zone names, e.g. git.date_tz = "America/New_York". Plus, you can use git.date_tz = "local" to refer to the local time zone.

Force fetching

Always fetch the package, even if its version (nvchecker output) doesn't change. This is useful when the file to be downloaded cannot not be determined by the url.

• fetch.force = true

Note: In such case, nvfetcher will not produce a version change log, since the version doesn't change at all. Only sha256 in generated files will be updated.

Haskell library

nvfetcher itself is a Haskell library as well, whereas the CLI program is just a trivial wrapper of the library. You can create a Haskell program depending on it directly, by using the runNvFetcher entry point. In this case, we can define packages in Haskell language, getting rid of TOML constraints.

You can find an example of using nvfetcher in the library way, see Main_example.hs.

Documentation

For details of the library, documentation of released versions is available on Hackage, and of master is on our github pages.

Contributing

Issues and PRs are always welcome. _(:з」∠)_

Building from source:

$ git clone https://github.com/berberman/nvfetcher
$ nix develop
$ cabal update
$ cabal build