Secret management for monitor webhooks

[daniel-rolls-sky]

Is your feature request related to a problem? Please describe.
There is no clear way to reference secrets from monitor webhook templates and to keep these secrets stored securely outside of the templates. For us the secrets are needed for basic auth passwords but others may need them for other forms of authentication.

Describe the solution you'd like
We wish to be able to reference secrets in an appropriate store (e.g. secrets manager). Users with access to the monitor config should not be able to see the secrets unless they have explicitly been granted access to those secrets.

Describe alternatives you've considered
none

Additional context
none

[anasalkouz]

@opensearch-project/admin Can someone move this to dashboard repository.

[ashwin-pc]

Speaking to @lezzago, this appears to be a feature the security plugin needs to first support before the notification plugin can build such a feature. Since the first change is needed from the security team, @opensearch-project/admin Can someone move this to security repository?

[lezzago]

Yes, there needs to be a feature to store secrets securely through the security plugin that the plugins can utilize. The only current way to store secrets securely is through opensearch-keystore, but it creates for a poor UX as the secrets need to be configured on every node.

[peternied]

I'm not sure this is a good fit for the security plugin, before we transfer can we get more context around the scenario?

I believe there are scenarios for webhooks around notifications, these are stored 'in the clear'? If there were secrets passed as part of the path/query parameters they are also in the clear.

There needs to be a place to manage these secrets and associate them with a runtime replacement.

Considering the effort to decouple Dashboards from OpenSearch, the security plugin might or might not be the right place to build this feature. @lezzago @ashwin-pc do you have thoughts on how this would be approached medium to long term?

Further note, might be good to file another issue around the secrets management system separate from this specific scenario.

[ashwin-pc]

@peternied My medium to long term vision here is similar to what you have called out in opensearch-project/security#3140, to have a distributed keystore management utility that plugins can tap into. Based on the comments there, it looks like bringing it into the security plugin is the path forward. I dont know the long term plan for decoupling OSD from Core, but whatever the approach we take there, I assume that it will also need to work with all the existing security features that we currently support. @zengyan-amazon can provide more context here about decoupling.

[peternied]

@ashwin-pc / @zengyan-amazon Could you come up with an interface for what you'd want this key management system to do, we can use that to work backwards in what/how its supported with security systems? Might make sense to do that in a new issue.