fetch-github-folder 🦎

^{(Don't judge me for my obsession with badges)}

Caution

This project currently is not stable, but I'm actively working on writing tests and handling edge cases, such as support for Git-LFS and other. Check out the dev branch to see the latest updates! 😉 (except that I currently don't care and just push to main)

What this library+CLI does?

It allows you to download any folder or a file from a repo on github.

Requirements

1. Latest Node.js (You can install it easily via mise)
2. Git for development

Installation

We support various installation options. Regardless of what you choose, if you want to make CLIs written in JS run faster, I highly recommend adding this line to your .bashrc:

export NODE_COMPILE_CACHE=~/.cache/nodejs-compile-cache

Install package with CLI and functions from default NPM registry

npm i fetch-github-folder

Install package with only functions from JSR

Unfortunately JSR doesn't support publishing executables yet, so you can install only script library with functions that will allow you to fetch github folder from other scripts.

npx jsr add @nikelborm/fetch-github-folder

Install package with CLI and functions from GitHub's NPM registry

1. Generate Personal access token (classic) with read:packages scope

2. Login to Github's NPM registry (yes you need to do it, even if the package is public):

   1. Run the following command (Info about --auth-type=legacy here):

      npm login --scope=@nikelborm --auth-type=legacy --registry=https://npm.pkg.github.com

   2. Enter your username when asked

   3. Paste the access token as password value

3. Install the package by executing:

   npm i @nikelborm/fetch-github-folder

Install package with CLI and functions from Github Releases

PACKAGE=fetch-github-folder

# Either set specific tag
TAG=0.1.28 && npm i https://github.com/nikelborm/$PACKAGE/releases/download/$TAG/$PACKAGE.tgz
# or download the latest
npm i https://github.com/nikelborm/$PACKAGE/releases/latest/download/$PACKAGE.tgz

Install only the CLI directly into the system from Github Releases

set -euo pipefail
PACKAGE=fetch-github-folder

path_to_the_script=/usr/bin/$PACKAGE

# Either set specific tag
TAG=0.1.28 && sudo curl -sLo $path_to_the_script https://github.com/nikelborm/$PACKAGE/releases/download/$TAG/$PACKAGE.js
# or download the latest
sudo curl -sLo $path_to_the_script https://github.com/nikelborm/$PACKAGE/releases/latest/download/$PACKAGE.js

sudo chmod +x $path_to_the_script

Setup the repo for local development

git clone -b main git@github.com:nikelborm/fetch-github-folder.git
cd fetch-github-folder
npm install
cp template.env .env
read -sp 'Enter github access token: ' gh_token; echo;
sed -i "s/\(GITHUB_ACCESS_TOKEN\)='.*'/\1='$gh_token'/" .env

Usage

EcmaScript module

import {
  downloadEntityFromRepo,
  OctokitLayer,
  FailedToParseGitLFSInfoError,
  GitHubApiBadCredentialsError,
  type InputConfig,
  type OutputConfig,
  type SingleTargetConfig,
  repoNameCLIOptionBackedByEnv,
  repoOwnerCLIOptionBackedByEnv,
  // etc...
} from 'fetch-github-folder';
// or '@nikelborm/fetch-github-folder' for non-default installation methods

Execution of CLI installed with NPM

The easiest way to execute the CLI (preliminary installation is not required) is like this:

npx fetch-github-folder --repoOwner apache --repoName superset

Also there's a shorter form available (preliminary installation is required):

npx fgf --repoOwner apache --repoName superset

Non-interactive CLI execution on the fly from Github Releases

If you already know the supported arguments (e.g. --help to print them all), you can pipe the bundled and minified script version into node directly and pass your arguments after node -:

set -euo pipefail

# Either set specific tag
TAG=0.1.28 && curl -sL https://github.com/nikelborm/$PACKAGE/releases/download/$TAG/$PACKAGE.js | node - --repoOwner apache --repoName superset
# or download the latest
curl -sL https://github.com/nikelborm/$PACKAGE/releases/latest/download/$PACKAGE.js | node - --repoOwner apache --repoName superset

Interactive CLI execution from Github Releases

The script also supports interactive mode (--wizard), where you will be asked to pass arguments sequentially and interactively. Since it requires user input, it can't be piped and needs to be saved to a temporary file:

set -euo pipefail
tmp_js=$(mktemp --suffix .js)

# Either set specific tag
TAG=0.1.28 && curl -sLo $tmp_js https://github.com/nikelborm/$PACKAGE/releases/download/$TAG/$PACKAGE.js
# or download the latest
curl -sLo $tmp_js https://github.com/nikelborm/$PACKAGE/releases/latest/download/$PACKAGE.js

node $tmp_js --wizard
rm $tmp_js

Execution of CLI installed directly into the system

fetch-github-folder --repoOwner apache --repoName superset

Environment Variables

If you often use the CLI, you can permanently set the arguments via env variables, and they will be used as a backup if arguments weren't provided explicitly.

• GITHUB_ACCESS_TOKEN: This is your personal access token from GitHub. It is used to authenticate your requests to the GitHub API. You can generate one here.
• REPO_OWNER: This is the username of the owner of the repository you are trying to download from. For example, if the repository's URL is https://github.com/apache/superset, the owner is apache.
• REPO_NAME: This is the name of the repository you are trying to download from. In the example above, the repository name is superset.
• PATH_TO_ENTITY_IN_REPO: This is the path to the directory you want to download. It can be directory that lies inside root of repo like docker or it can be some nested directory like docker/nginx.
• GIT_REF: This is the commit SHA hash, branch name, or tag name you want to download from. If you don't specify it, the default branch in the repository will be used.
• DESTINATION_PATH: If entity at PATH_TO_ENTITY_IN_REPO is a file, then destination path is a path to downloaded file. If it's a directory, then all files and directories from target directory of remote repository at PATH_TO_ENTITY_IN_REPO will be put into a directory having DESTINATION_PATH path. If the directory doesn't exist, it will be automatically created.