Add support to the client (CLI, http client) for sending additional basic auth credentials (aka MFA)

[Kami]

This pull request adds support for new credentials.basic_auth = username:password StackStorm CLI config option and -basic-auth CLI argument.

This argument allows client to use additional set of basic auth credentials when talking to the StackStorm API endpoints (api, auth, stream) - that is, in addition to the token / api key native StackStorm auth.

Background, Context

I'm running a StackStorm instance which I want to put behind additional auth wall (think second factor auth). I could probably achieve that using SSO and handling multi factor auth on the SSO side, but I'm not using SSO nor want to use it (and even with SSO used, if there is a bug in StackStorm auth code for SSO, dual factor auth on SSO side would not help in this case while having additional auth in another layer like nginx would).

The easiest way to add second factor to any kind of application which runs behind some kind of http proxy (nginx, Apache, etc.) is to simply utilize existing basic auth primitives of the proxy - this PR does exactly that.

Notes, Limitations

1. Current CLI and http client code is very messy and hard to work it. We don't have good abstractions for it (a lot of kwargs hacks all over the place) so it's a PITA to update it and work with it (something which has been a known issue for a long time already).

2. If this second factor auth is enabled and user wants to use webhook integrations, they will also need to send basic auth credentials with webhooks (which shouldn't be an issue since most 3rd party service support that and if they don't, they could also add exceptions in http proxy layer to not require it for webhooks API endpoint or similar).

3. Currently /v1/auth/tokens API endpoint also utilizes basic auth for authentication which means you should disable proxy based basic auth wall for /v1/auth/tokens API endpoint. This is of course not ideal since it means actually auth endpoint won't be behind MFA wall (aka user may be able to auth and obtain a valid st2 auth token without additional basic auth auth, but other operations and endpoints won't work if valid basic auth header is not provided), but all other endpoints will be.

In nginx proxy, this would look something like this:

...
# Require additional basic auth credentials for all StackStorm API endpoints except /auth/tokens one
location / {
   auth_basic "off";
   auth_basic	"Auth required";
   auth_basic_user_file	/etc/nginx/htpasswd/foo.bar.example;
  ...
}
...

location /auth/tokens {
   auth_basic "off";
  ...
}

TODO

• Unit tests
• Verify it works with all the endpoints (so far I verified all the basic one + event stream aka execution tail one)
• Verify st2web (may require changes in case the current client doesn't simply send existing basic auth credentials browser has cached)

[blag]

If this second factor auth is enabled and user wants to use webhook integrations, they will also need to send basic auth credentials with webhooks (which shouldn't be an issue since most 3rd party service support that and if they don't, they could also add exceptions in http proxy layer to not require it for webhooks API endpoint or similar).

I haven't ever run across a third party service that supports basic auth for outgoing webhooks. Jira doesn't, GitHub doesn't. So allowing the base webhook endpoint, and all children, to bypass this would definitely be required. Edit: I'm wrong, they both do.

I'm not hugely happy with this. I think putting our efforts into an SSO backend that handles MFA itself would be better.

Also, I'm afraid that this change would give users more undeserved confidence in running StackStorm exposed to the internet, which I don't think has been our security model/mindset so far. I'm already aware of one such instance, and I hope it's the only one.

[Kami]

@blag

I haven't ever run across a third party service that supports basic auth for outgoing webhooks. Jira doesn't, GitHub doesn't. So allowing the base webhook endpoint, and all children, to bypass this would definitely be required.

I'm pretty sure Github and lot of other services support https://<credentials>@url notation for the webhook URLs and handle credentials correctly.

But if that's not possible in some situations, user can also handle that in nginx config level (similar as tokens auth endpoint) so that can be documented.

I'm not hugely happy with this. I think putting our efforts into an SSO backend that handles MFA itself would be better.
Also, I'm afraid that this change would give users more undeserved confidence in running StackStorm exposed to the internet, which I don't think has been our security model/mindset so far. I'm already aware of one such instance, and I hope it's the only one.

Security is all about layers and I think it's good to give users a choice of adding additional layer on top StackStorm auth (+optionally SSO with MFA).

SSO + MFA is definitely a good choice, but for some more security conscious users, it's better to have another layer somewhere outside of the app layer.

There were definitely cases in the past where it was possible to bypass login + mfa due to the bug in the app code.

[blag]

I'm pretty sure Github and lot of other services support https://@url notation for the webhook URLs and handle credentials correctly.

Oh, right, yes they do, nevermind.

[CLAassistant]

All committers have signed the CLA.

[nzlosh]

@Kami Would you mind rebasing the branch against master so we can merge it for the 3.7 release?

[Kami]

@nmaludy I've added changelog entry and synced it up with latest master. Will go ahead and merge it once CI passes - thanks.