[Actions] document `NODE_EXTRA_CA_CERTS` et al as a way to enable extra https certs for 3rd party services

[pmuellr]

For customers that need to add additional security certificate settings for use with 3rd party services making https requests, we should document all the ways this can be done. @romain-chanu noted that he used NODE_EXTRA_CA_CERTS in one customer environment successfully.

We should document all these somewhere in the actions documentation. We should also look into whether this is good enough:

• Are env vars good enough? Should we have config yml settings?
• Are there cases where we need to have different sets of cert info available to different actions?

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[mikecote]

++ we should update the docs in the interim.

I'm wondering more long term if we should support this in the UI where the connectors are configured. Maybe an upload of the certificate which gets encrypted in secrets or something like that..

[pmuellr]

My guess would be that if they need special certs to access a service, there may be other software they're using that also needs these certs, so if there's any way to reuse some kind of "standard" discovery for these, that would be useful. Eg, env var names.

Uploading certs to ES is an interesting approach, not sure there's enough bang for the buck for it though, considering the extra code we'd have to write to manage these. Making use of the keystore would be a similar sort of story, probably easier for everyone to deal with given it's a set of CLI commands, and no new key management code would need to be written.

[pmuellr]

BTW, cloud can't currently set this option, not to mention allow the customer to upload CA cert files, so we'll need some sort of whitelisted kibana config to handle all this SSL/TLS stuff for cloud. Probably a fairly big piece of work, and the CA cert file thing in particular may be a different sort of beast if we're going to need customers to upload certs to a cloud deployment.

[mikecote]

I think you're right and the issue is a bit bigger than simply updating the docs.

@legrego I'm reaching out to get some thoughts on this issue in case something like this has crossed your path. Some customers have custom certs they need to set before they can use some connectors. They can be set via NODE_EXTRA_CA_CERTS environment variable which is easy to do on prem but not as much on Cloud. Do you think going the environment variable path would make sense and we explore how to make this work on cloud? Or do you think having some UI in Kibana to allow customers to upload their certificates would be a better path to explore? (per connector upload or global)

[legrego]

I'm personally not a fan of using NODE_EXTRA_CA_CERTS, as this will apply to all TLS connections by default, unless they explicitly provide their own set of trusted CAs when constructing the client. It's also inconsistent with how we specify CAs everywhere else.

We've instead favored explicit configuration via kibana.yml for these settings for each use case. For example, we have settings which allow you to specify the set of trusted CAs for your connections to ES, and another set for certificates used to make Kibana available over https://. You can use the same CAs for both, but you absolutely don't have to.

I'd rather see us add this to kibana.yml for now. Watcher requires yml configuration for SSL today, so I presume that we have a mechanism for allowing custom certificates on ESS. We should obviously verify this though.

[pmuellr]

Ya, I think we can certainly handle CAs specifically for use with actions vs across all https connections Kibana is making - the next question is do we need per-connection CAs - eg, only use CA xyz for this particular webhook connector. It's likely we'd only need this for webhook, actually, since most/all of the other actions talk to 3rd party services on the web, so are likely already using CAs acceptable to node.

[pmuellr]

We may want to factor in the http proxy stuff here as well. Would a customer have an https proxy that required special certs? Note that cloud is unlikely to be affected here, I doubt there's a requirement for a cloud customer to have to use http proxies for actions. None-the-less, that's still a cert we'd have to deal with, for on-prem, so it would be nice to have a somewhat consistent story.

[pmuellr]

Watcher requires yml configuration for SSL today, so I presume that we have a mechanism for allowing custom certificates on ESS. We should obviously verify this though.

The list of allowed ES settings for cloud only lists xpack.http.ssl.cipher_suites as allowed, whereas we'd be looking at the equivalent of watcher's xpack.http.ssl.certificate_authorities (List of paths to PEM encoded certificate files that should be trusted). Note there is an alternative way to use cert files in ES via xpack.http.ssl.truststore.path - you just provide a path and not the file names, which doesn't seem like it would be useful in cloud in any case.

So it doesn't look like uploading certs to the cloud is currently supported for watcher.

[pmuellr]

So, seems like a couple of things to work on here - I think it's worth documenting the NODE_EXTRA_CA_CERTS and other node settings, with the caveats from Larry that these are globally applied. It will let customers access services they wouldn't otherwise be able to, if they are willing to accept the risks, before we have a better solution.

Then we need a better solution, which. is presumably a new action-based config specifying cert files/path, however we want to shape that (files / paths, for all action types, action-type specific, or connector-specific).

Then if we need this for cloud, we'll need to figure out the cloud story, which is that presumably customers would be uploading certs to their cloud account | deployments.

[pmuellr]

A customer also reported using NODE_TLS_REJECT_UNAUTHORIZED with a webhook that got their action working; presumably the service they were calling was using a self-signed certificate. Makes sense - we should probably have that as an config param on webhook, along with any other tls-related things that come out of this.