Skip to content

Install from cache

Jump to bottom
Eugene Lazutkin edited this page Jun 12, 2022 · 4 revisions

install-from-cache retrieves a previously saved artifact, tests if it works properly, and rebuilds a project from sources in the case of failure.

How to use

In your package.json (pseudo-code with comments):

{
  // your custom package.json stuff
  // ...
  "scripts": {
    // your scripts go here
    // ...

    // saves an artifact
    "save-to-github": "save-to-github-cache --artifact build/Release/ABC.node",

    // installs using pre-created artifacts
    "install": "install-from-cache --artifact build/Release/ABC.node",

    // used by "install" to test the artifact
    "verify-build": "node scripts/verify-build.js"

    // used by "install" to rebuild from sources
    "rebuild": "node-gyp rebuild"
  }
}

When a project, which uses install-artifact-from-github, is being installed, it does the following actions:

  • Acquiring an artifact from the cache (an appropriate GitHub release).
    1. If the environment variable DEVELOPMENT_SKIP_GETTING_ASSET set to a non-empty value ⇒ it builds from sources.
    2. If the file .development is present in the project folder ⇒ it builds from sources.
    3. It tries to download an appropriate artifact compressed by brotli, if it is available. If it succeeds ⇒ it checks if it works.
    4. It tries to download an appropriate artifact compressed by gzip. If it succeeds ⇒ it checks if it works.
    5. If all downloads have failed ⇒ it builds from sources.
  • In order to check that the downloaded artifact works:
    • It runs npm run verify-build. You may provide the script verify-build to do the checking.
      • If it returns with 0 exit code, we are done. Otherwise ⇒ it builds from sources.
    • If there is no verify-build, it tries npm test.
      • If it returns with 0 exit code, we are done. Otherwise ⇒ it builds from sources.
    • If both scripts are missing ⇒ it builds from sources.
  • If it was determined to build the artifact from sources, it runs npm run rebuild, which should be provided.

Environment variables

  • DEVELOPMENT_SKIP_GETTING_ASSET — if it is set to a non-empty value, it forces the build from sources. It is useful for development and testing.
  • DEVELOPMENT_SHOW_VERIFICATION_RESULTS — if it is non-empty, it shows the verification output. Otherwise, the output is suppressed to avoid scaring unsuspecting users with possible errors. It is useful for development and testing.
  • DOWNLOAD_HOST — if set, its value is used instead of https://github.com.
  • (Since 1.3.0) DOWNLOAD_SKIP_PATH — if set to a non-empty value, it allows to skip a project-specific part of path. See below for more details.
  • (Since 1.3.0) DOWNLOAD_SKIP_VER — if set to a non-empty value, it allows to skip a version-specific part of path. See below for more details.

This script is meant to be run using npm run. It relies on npm environment variables to learn about the project.

Command-line parameters

  • --artifact path — points to the location of the downloaded artifact. It is a required parameter.
  • --prefix prefix — provides a prefix for the generated artifact name. Default: ''.
  • --suffix suffix — provides a suffix for the generated artifact name. Default: ''.
  • --host host — provides a prefix for the download host. It should not end with /. Example: --host https://sample.com/repo.
    • If specified and non-empty, its value sets the host with an optional path prefix.
    • The default: https://github.com.
  • --host-var ENVVAR — provides the name of an environment variable, which value will specify the download host. Example: --host-var RE2_DOWNLOAD_MIRROR.
    • Used only if --host is not specified.
    • If it is not specified, DOWNLOAD_HOST name is assumed.
    • If the specified environment variable is empty, https://github.com will be used.
  • --skip-path — if present, allows skipping a project-specific part of the path.
    • If present, a project-specific part of the path. See below for more details.
    • The default: don't skip.
  • --skip-path-var ENVVAR — provides the name of an environment variable, which value will specify if a project-specific part of the path should be skipped. Example: --skip-path-var RE2_SKIP_PATH.
    • Used only if --skip-path is not specified.
    • If it is not specified, DOWNLOAD_SKIP_PATH name is assumed.
    • If the specified environment variable is non-empty, a project-specific part of the path should be skipped.
  • --skip-ver — if present, allows skipping a version-specific part of the path.
    • If present, a version-specific part of the path. See below for more details.
    • The default: don't skip.
  • --skip-ver-var ENVVAR — provides the name of an environment variable, which value will specify if a version-specific part of the path should be skipped. Example: --skip-ver-var RE2_SKIP_VER.
    • Used only if --skip-ver is not specified.
    • If it is not specified, DOWNLOAD_SKIP_VER name is assumed.
    • If the specified environment variable is non-empty, a version-specific part of the path should be skipped.

Ultimately, the downloadable file name has the following format:

`${host}/${user}/${repo}/releases/download/${tag}/${prefix}${platform}-${arch}-${abi}${suffix}.${compression}`

Where:

  • host is a hostname with an optional path prefix. See above --host and --host-var for more details.
  • user is a repository user name. It is deduced from package.json.
  • repo is a repository name. It is deduced from package.json.
  • tag is a tag that triggered the build. It is deduced from package.json.
  • prefix is a user-supplied value (see above).
  • platform is based on process.platform.
    • Because Linux has different implementations of the C standard library, a special case is made for musl used by distributions like Alpine. Such platforms have a code linux-musl.
  • arch is based on process.arch.
  • abi is based on process.versions.modules.
  • suffix is a user-supplied value (see above).
  • compression can be br or gz.

Since 1.3.1: platform, arch and abi can be manually overridden for cross-platform builds. It allows specifying a different pre-compiled module to be fetched. In this case, no build verification is done. See "Recognized NPM parameters" below to learn how to override platform-specific parameters.

Example with default values: https://github.com/uhop/node-re2/releases/download/1.15.2/linux-x64-83.br.

If --skip-path is specified or --skip-path-var environment variable values is non-empty the following part of the path will be skipped:

`/${user}/${repo}/releases/download`

If --skip-ver is specified or --skip-ver-var environment variable values is non-empty the following part of the path will be skipped:

`/${tag}`

Both those variables are used mostly to organize local mirrors.

The former example with DOWNLOAD_HOST=https://local.mirror, DOWNLOAD_SKIP_PATH=1, DOWNLOAD_SKIP_VER=1: https://local.mirror/linux-x64-83.br.

Recognized NPM parameters

Some parameters can be specified when using NPM like this:

npm install re2 --platform=linux

The full list of such parameters:

  • Since 1.3.1: Parameters used to specify platform-specific values for cross-platform builds to specify manually what pre-compiled binary to fetch:
    • --platform=XXX — platform value such as darwin, windows, linux, linux-musl. The default is the current platform based on process.platform.
    • --platform-arch=XXX — platform CPU architecture such as x64. The default is the current CPU architecture based on process.arch.
    • --platform-abi=XXX — platform ABI version such as 108. The default is the ABI of the currently running Node based on process.versions.modules.

Example

The realistic complex example can be found in uhop/node-re2:

Notes

NPM 7

Update on 11/12/2020: npm 7 has changed completely what data it exposes through environment variables. Instead, it provides a path to package.json. Arguably it is a less messy alternative. Support for npm 7 is implemented starting with version 1.15.9. Previous versions fail gracefully and build locally.

Fixed in NPM

These notes are kept here for posterity:

  • Unfortunately, NPM 7 introduced an error: npm ci does not build a binary extension. If you encountered this problem, please use npm i instead. The bug was filed with NPM: https://github.com/npm/cli/issues/2169
Clone this wiki locally