Standard way to download assets out of the NPM Registry to support LLM Models and general software

[pavelai]

NPM needs to solve issue with external assets being downloaded with a package

Problem

There are packages which requires big files (binaries, data files, LLM models, etc) being downloaded with the package itself and there is no standard solution to download these files with NPM. Such packages use install-scripts to do so. But developers disables install-scripts, because of security concerns. The problem is that it's hard (and sometimes impossible) to audit what has been downloaded and there could be malicious packages which doesn't download malicious payloads every time to hide their presence in repository. It make the whole process of installation unreliable and the infrastructure vulnerable

1. Package managers like PNPM doesn't know about this external dependencies and can not optimize how they are stored on the disk
2. Security tools can not tell for sure what would be downloaded with the scripts
3. There is no way to check whether the software is installed properly, because there is no common source of truth and every installation is potentially unique
4. Potential errors in installation scripts could be non-intentionally harmful

So developers should just trust to the package owners to be honest and careful to store only required files without bloating free space

Electron's RFC

For example Electron requires 200+ Mb to be with their package. They have pretty sophisticated installation system and release scheme. In a few words: they have different binaries and drivers for many platforms and bunch of temporal builds which they don't want to upload to NPM Registry. Now as most of developers disables installation scripts they want to change their installation to make things work for them without ruining DX for Electron users. It highlights the problem with the current NPM installation architecture

More here: electron/rfcs#22

Requirements

The solution should be:

1. Reproducible
2. Verifiable
3. Predictable
4. Deterministic
5. Flexible

There should be some deterministic way to specify and select URLs to download and then to verify them and to verify the final outcome of the installation

Solution

To solve this package.json should include "assets" directive for external downloadable resources. Assets directive allows to:

1. Specify URLs provided as an extra assets with checksums to verify the installation
2. Specify custom files depending on the users' platform, os, JS runtime, etc.
3. Specify which files should be unpacked after download

The asset directive is an object or array of objects. If the asset is an object it contains list of files to be downloaded. If it's an array it contains installation variants. Each installation variant has conditions, like os, architecture and a runtime. This conditions are used by package manager to select suitable installation variant. After selecting suitable installation variant it would select the assets from provided list of files.

NPM and other package managers should download and check these files against checksums out of the box without the need to run installation scripts

Example of a regular assets files list:

{
  "assets":  {
      "bin/app": {
        "urls": ["https://cdn.developer.example/v1/bin/app"],
        "checksum": "sha-512:09af..."
      },
      "models/llm-1": {
        "urls": ["https://cdn.models.example/llms/releases/llm-1/llm.tar.gz"],
        "archive": "tar-gz",
        "checksum": "sha-512:09af..."
      }
    }
}

Example of assets field with installation variants:

{
  "assets": [
    {
      "os": "darwin",
      "arch": "arm64",
      "osVersion": "",
      "files": {
        ".": {
          "urls": ["https://github.com/electron/electron/releases/download/v39.2.4/electron-v39.2.4-darwin-arm64.zip"],
          "archive": "zip",
          "checksum": "sha-512:09af..."
        }
      }
    },
    {
      "os": "linux",
      "arch": "i686",
      "files": {
        ".": {
          "urls": ["https://github.com/electron/electron/releases/download/v39.2.4/electron-v39.2.4-linux-x64.zip"],
          "checksum": "sha-512:09af..."
        }
      }
    },
  ]
}

Benefits of such solution

1. Reproducible builds
2. No need to run installation scripts for many scenarios
3. Provide statically analyzable information for package managers for better DX
4. Package managers could store this files in a local or remote cache
5. Provide analyzable information for security tools and npm's audit command

[bcomnes]

The standard way is to put them in the package tarball. Most native builds with external downloads can actually just do this (see https://github.com/prebuild/prebuildify). As with many problems in the npm ecosystem, people opt-in to many of the complexities and problems they encounter.

[pavelai]

Prebuilding can solve only part of the addressed issues for relatively small packages. But mostly developers would not publish 2, 20, or 120 Gb LLM model in a tarball at NPM. I'm not sure the registry would support such big files in the future

[pavelai]

And as an example I've added link to Electron RFC and their discussion, where they express concerns about practicality of such a solution for them. Here is the link for the particular comment: electron/rfcs#22 (comment)

• The Electron binary is an order of magnitude larger than the esbuild one. We also publish nightly releases under electron-nightly (tbh this should be mentioned within the RFC). This will ultimately lead to a lot being uploaded to the npm registry.
  On package-time, Electron uses @electron/get to download the binary from GitHub Releases.
• If the development binary gets downloaded from npm instead, we now have two sources of truth for the binary instead of one. I think this point isn't very clear in the current RFC and could be improved (it was briefly discussed rfc: lazy electron download, remove postinstall electron/rfcs#22 (comment)).

This is what they are concerned about and it seems a reasonable problem to solve