Proxy Support

[danieladams456]

I know Node can sort of be a pain with adding proxy support, but it would be great if we could use the Netlify CLI behind a corporate proxy. Thanks!

[bcomnes]

Open to the idea. What are the requirements? Any suggestions on how to best support this?

[danieladams456]

My use case is sort of extreme, so I was initially thinking I'd put this out there to start keeping track of the feedback and then think about implementation when it hit critical mass. If we didn't have this, we could probably just set up a Jenkins job on a server with a firewall rule.

Requirements:

• Parse $http_proxy in the form of http://user:pass@host:port and do basic auth
• Be able to support additional certificates like AWS_CA_BUNDLE since SSL intercepting proxy

I see you're using node-fetch. If this was a normal package and not a CLI you could just allow someone to pass in an HTTP agent in the constructor config, which could still work for the js-client repo.

One sort of far-out idea for implementation was having a separate package with the proxy code as a peer dependency. That way people would only have to install it if they needed it. The CLI would inject it into the client if it existed (pseudocode below). That way the vast majority of people could just use the normal package and you all wouldn't have to worry about testing the proxy code to make sure it didn't break things for everyone else.

const netlifyCliProxy = require('netlify-cli-proxy');
const agent = netlifyCliProxy ? netlifyCliProxy.agent : null;
const client = new NetlifyAPI('1234myAccessToken', {agent});

[bcomnes]

Its possible we could swap the agent in https://github.com/netlify/js-client. Does Node have built in system proxy support we could leverage? I've never looked into that before, I'll look around.

[danieladams456]

Unfortunately node doesn't have native support. It uses http[s].Agent objects to do any proxy customization. Here is a specific issue about that in node-fetch.

The proxy implementations are third party and have various levels of functionality. I had used tunnel before, but it looks like https-proxy-agent supports proxy auth. I will try it out.

Update: proxy auth works for https-proxy-agent, still working through custom CA, but it looks like npm transitively uses it and figured it out here.

Update 2: looks like custom CA working with core https module because of this commit. I think this could be done cleanly enough to not need the optional peer dependency thing. I will write up some code.

const HttpsProxyAgent = require('https-proxy-agent');
const opts = url.parse(process.env.http_proxy);
opts.ca = fs.readFileSync(process.env.REQUESTS_CA_BUNDLE);  // works with my Python bundle
const agent = new HttpsProxyAgent(opts);

[danieladams456]

How does this look? The only additional dependency is https-proxy-agent. fs and uri are core node modules, but with only including them just in the cli package we don't have to worry about browser compatibility. The only change to the client-js package would be allowing an agent to be specified in the constructor opts. If proxy environment variables are not set, then agent is undefined and there is no change to the behavior.

const fs = require('fs');
const url = require('url');
const fetch = require('node-fetch');
const HttpsProxyAgent = require('https-proxy-agent');


const demoRequestUrl = 'https://open-api.netlify.com';
const proxyUrl = process.env.https_proxy || process.env.http_proxy;
// follow the naming convention of REQUESTS_CA_BUNDLE and AWS_CA_BUNDLE
const caFilename = process.env.NETLIFY_CA_BUNDLE;
let agent; // only set if configured via env vars

if (proxyUrl) {
  const agentOpts = url.parse(proxyUrl);
  if (caFilename) {
    try {
      agentOpts.ca = fs.readFileSync(caFilename);
    } catch (error) {
      // this should be a real logger
      console.log('WARN: could not read CA file');
    }
  }
  agent = new HttpsProxyAgent(agentOpts);
}

// now we can inject agent into client-js opts and it can pass into fetch
fetch(demoRequestUrl, { agent })
  .then(res => res.text())
  .then(text => console.log(text));

[bcomnes]

@danieladams456 thank you for the thorough research! Yeah, this sounds good to me.

• Add an agent option to js-client
• Add some flags / ENV var support for passing in / loading urls and certs via CLI.

This sounds like something that may be a good fit for the global config object, supplemented with ENV Vars / flags as needed. Do you see yourself needing to toggle a proxy config depending on where you are running it?

[danieladams456]

Thanks! I have setproxy and setoffnetwork functions in my bash profile that I use to toggle the 6 environment variables that work across all the languages/tools. I find that more streamlined than having to pass flags to each command.

I could see the use for a global config object being helpful as more of these type of requests come in. I know some packages for that, but they are more targeted at server side (env vars + layered config files.) I'm sure there are some CLI config packages out there. We use traefik at work for our internal stuff, and its config system flaeg seems really solid. Go doesn't really help in this case though. 😄

[nolessafool]

also requested by netlify customer in this conversation: https://app.intercom.io/a/apps/q245f50x/inbox/inbox/unassigned/conversations/20015116786
https://community.netlify.com/t/netlify-deploy-from-china-is-blocked-is-there-any-way-to-use-proxy-for-netlify-deploy/16327

[bcomnes]

I'm hoping to refactor the code paths soon, and I can try to add this in then.

[TheDropZone]

Wanted to jump in and see if this ever got implemented into the CLI tool. Our entire company is fully required to operate behind a corporate proxy, so I don't have many other options. Thanks guys!

[dfang]

yes, proxy support is really helpful in China

it does not even open a browser tab, when i run netlify login

netlify login
Logging into your Netlify account...
    FetchError: request to https://api.netlify.com/api/v1/oauth/tickets?client_id=d6f37de6614df7ae58664cfca524744d73807a377f5ee71f1a254f78412e3750 failed,
    reason: connect ETIMEDOUT 35.201.69.186:443
    Code: ETIMEDOUT

dig api.netlify.com +short
35.201.69.186

nali 35.201.69.186
HTTP/1.1 200 OK
Date: Sun, 02 Aug 2020 13:31:09 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 295
Access-Control-Allow-Origin: *
X-Ttl: 60
X-Rl: 44

{
    "status": "success",
    "country": "United States",
    "countryCode": "US",
    "region": "MO",
    "regionName": "Missouri",
    "city": "Kansas City",
    "zip": "",
    "lat": 39.0997,
    "lon": -94.5785,
    "timezone": "America/Chicago",
    "isp": "Google LLC",
    "org": "Google Cloud (asia-south1)",
    "as": "AS15169 Google LLC",
    "query": "35.201.69.186"
}

this ip is blocked in china...

and netlify cli does not read HTTP_PROXY env.

i use the latest version netlify.

netlify version
netlify-cli/2.58.0 darwin-x64 node-v14.6.0