[BUG] CLI (and API) is broken when used with "Personal access tokens"

[fprochazka]

Short reproduce

I'm unable to use netlify-cli (or API for that matter) with "Personal access token" - all commands end with error.

1. install netlify-cli npm install netlify-cli -g
2. create personal access token https://app.netlify.com/user/applications#personal-access-tokens
3. run NETLIFY_AUTH_TOKEN=secret-token netlify sites:list

environment:

  System:
    OS: Linux 4.15 Ubuntu 18.04.3 LTS (Bionic Beaver)
    CPU: (12) x64 Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
  Binaries:
    Node: 12.13.1 - /usr/bin/node
    Yarn: 1.19.1 - /usr/bin/yarn
    npm: 6.12.1 - /usr/bin/npm
  Browsers:
    Chrome: 78.0.3904.70
    Firefox: 70.0.1

status:

$ NETLIFY_AUTH_TOKEN=secret-token netlify status --verbose 
──────────────────────┐
 Current Netlify User │
──────────────────────┘
FetchError: invalid json response body at https://api.netlify.com/api/v1/accounts reason: Unexpected end of JSON input
    at ~/.config/yarn/global/node_modules/node-fetch/lib/index.js:272:32
    at async NetlifyAPI.listAccountsForUser (~/.config/yarn/global/node_modules/netlify/src/open-api/index.js:160:20)
    at async StatusCommand.run (~/.config/yarn/global/node_modules/netlify-cli/src/commands/status/index.js:29:22)
    at async StatusCommand._run (~/.config/yarn/global/node_modules/@oclif/command/lib/command.js:44:20)
    at async Config.runCommand (~/.config/yarn/global/node_modules/@oclif/config/lib/config.js:151:9)
    at async Main.run (~/.config/yarn/global/node_modules/@oclif/command/lib/main.js:21:9)
    at async Main._run (~/.config/yarn/global/node_modules/@oclif/command/lib/command.js:44:20)

api:

curl -v -H 'User-Agent: Test (filip@prochazka.su)' 'https://api.netlify.com/api/v1/sites?access_token=secret-token'

curl -v -H 'User-Agent: Test (filip@prochazka.su)' -H "Authorization: Bearer secret-token" 'https://api.netlify.com/api/v1/sites'

all my attempts end with

HTTP/1.1 401 Unauthorized

which I had to find out using direct API calls and the netlify-cli did not tell me that, instead, all got was a cryptic "Unexpected end of JSON input"

When I run netlify login and use the oauth workflow of opening browser and confirming the login, all the netlify-cli commands work as expected. Which is obviously not what I would want to do in CI.

Full reproduce

I'm trying to integrate netlify-cli into our Gitlab-CI build (gitlab and runners are on-premises).

I've created a custom docker image node-js/netlify-cli:2-alpine, built like this:

FROM node:12.13-alpine

RUN set -ex \
 && apk add --no-cache curl git \
 && npm install netlify-cli@2.21.0 -g

This image is then used to run netlify tasks.

stages:
  - build
  - deploy

variables:
  JEKYLL_VERSION: '3.8'

job-build:
  stage: build
  tags: ['docker', 'small']
  image: jekyll/jekyll:$JEKYLL_VERSION
  variables:
    JEKYLL_DATA_DIR: $CI_PROJECT_DIR
  before_script:
    - bundle install
  script:
    - jekyll build -d dist
  artifacts:
    paths:
      - dist/*

job-deploy-prod:
  stage: deploy
  tags: ['docker', 'small']
  image: node-js/netlify-cli:2-alpine
  script:
    - DEBUG=* netlify deploy --site $NETLIFY_SITE_ID --message "$(git show -s --format=%s)" --prod
  dependencies:
    - job-build
  only:
    - master

the job is configured with NETLIFY_SITE_ID and NETLIFY_AUTH_TOKEN ENV variables

[DavidWells]

Hey @fprochazka

I just tested this out and was able to use a personal access token to run netlify sites:list

Is the account you are creating the personal access token under a user of the sites you are trying to operate on?

---

The token generated from the netlify login command is stored in ~.netlify/config.json file. If that token works, you can use that in the CI process with the NETLIFY_AUTH_TOKEN env variable or via the auth flag like netlify deploy --auth xyz

---

I've opened an issue to make the API error messages more developer friendly. netlify/js-client#84

[fprochazka]

I've ended up using the oauth token from my local machine, but I'm not sure if that's ideal.

---

Is the account you are creating the personal access token under a user of the sites you are trying to operate on?

Well... I'm not sure I understand... I have an account... I've created a personal token under that account... Then I've tried to list sites with NETLIFY_AUTH_TOKEN=secret-token netlify sites:list and an error was returned instead of the sites. Why should it matter what account is it, if it doesn't return anything and simply errors?

I don't actually need to list the sites, I'm trying to deploy from Gitlab-CI. The sites:list command was mentioned just to illustrate that nothing works for me with the personal access token. Not cli, not even direct API calls.

It looks like you're a netlify developer? Maybe your account is different? My account filip@prochazka.su has few personal sites and is also a member of an organization - and the tokens I create do not work - neither locally, nor in CI.

Would it be better if I raise this directly with Netlify support?

[DavidWells]

It looks like the token is expired or malformed in the requests. This could very well be an issue with the underlying netlify API.

Can you try to create a new personal access token and run a GET request against https://api.netlify.com/api/v1/sites https://open-api.netlify.com/#tag/site

The token generated by the CLI will also work in the CI context

[fprochazka]

I've created a new token, used it like this NETLIFY_AUTH_TOKEN=secret-token netlify sites:list and now it works...

I can no longer reproduce the problem... There must have been something wrong with the tokens I've created yesterday, but I have no way to debug that now.

Thanks for your help!