Encrypted saved-object attributes cause issues for import/export

[kobelb]

Encrypted saved-object attributes were originally implemented to store end-user provided secrets that only needed to be consumed by the Kibana server. Since the secrets are only consumed by the Kibana server, we intentionally prevented them from being disclosed to end-users because they contain sensitive information. For example, Alerting's connectors store credentials for third-party services, and even though we allow them to be provided via Kibana's UI, we didn't want to disclose them to other end-users who should only be consuming the configured connector.

Since their original implementation, a change was made to allow the developer to opt-out of this protection and allow the secrets to be retrieved by end-users. In those situations, there aren't any complications with import/export. However, we still have quite a few situations where the attributes are being stripped from the responses.

When the encrypted saved-object attributes are excluded from the responses, this causes complications for import/export. When a user exports one of these saved-objects, the encrypted attributes are excluded from the response and this will cause complications on import.

Potential Solution

Import the saved-objects without these attributes

If we allow the saved-objects to be imported without these attributes, it can cause complications within the consuming code because the code must accommodate for the fact that these attributes are undefined. This can also convey the wrong message to our users, and make them think that the import was completed successfully and there's nothing else that needs to be done. Any functionality that is dependent on the existence of these attributes will continue to fail until the end-user re-provides the secrets. For example, Alerting's actions will fail to execute until the user specifies the credentials for the third-party services.

Prompt the user to provide the secrets on import

During the import process, we can detect when there are encrypted attributes that are missing and prompt the end-user to provide these credentials. However, the import process is already rather complicated, so adding another step potentially makes this process even more confusing.

Include the attributes in the export

We currently exclude the encrypted attributes from all saved-object operation responses. It's possible for us to only include these secrets from the export operation. We already have a "Saved-Object Management" feature that we could use to provide this elevated level of access. However, we'll want to make sure that we aren't violating the end-users expectations of secrecy by doing so.

Use-case specific import/export

It's also possible for us to not try to solve this at that platform level, and require a use-case specific import/export. This would allow the consumers of encrypted saved-object attributes to make this decision and implement one of the aforementioned solutions, or one we haven't outlined yet. This would create a rather fragmented experience, and make it difficult for end-users to import/export all of their saved-objects.

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[kobelb]

Pinging @elastic/kibana-alerting-services

[mikecote]

From an alerting perspective, I have a few notes below (sorry if they talk outside encrypted saved objects specifically, though there are other issues alerting would have with import / export. See #60053 (comment)):

In regards to connectors, we are exploring the option of exposing encrypted attributes to the end user (#78704). If ever we go that approach, it would play nicely with the Include the attributes in the export option proposed.

More specifically to import / export, the @elastic/kibana-alerting-services team would need to confirm if ever these connector configurations do change when the end user is moving them between environments (ex: from staging to production). Is moving between environments the primary use case for import / export? Would the connectors all connect to the same endpoints? We could always alternatively add an enable / disable functionality and disable the connectors on import..

In regards to alerts, we encrypt the API key which will not exist when importing into a different environment. The option of Include the attributes in the export is most likely not going to work for this scenario. We could in theory create new API keys for each alert on import but this will change the expected privileges the alert would run as. Here as well, we could explore importing the alerts as disabled since the enable / disable feature exists already.

[mikecote]

Use-case specific import/export

It's also possible for us to not try to solve this at that platform level, and require a use-case specific import/export. This would allow the consumers of encrypted saved-object attributes to make this decision and implement one of the aforementioned solutions, or one we haven't outlined yet. This would create a rather fragmented experience, and make it difficult for end-users to import/export all of their saved-objects.

This is sort of the path I've been thinking for a while and we've been tracking it here: #50266. I agree about the fragmented experience and hopefully we can come up with something better.

[legrego]

Include the attributes in the export

We currently exclude the encrypted attributes from all saved-object operation responses. It's possible for us to only include these secrets from the export operation. We already have a "Saved-Object Management" feature that we could use to provide this elevated level of access. However, we'll want to make sure that we aren't violating the end-users expectations of secrecy by doing so.

It's not possible today due to licensing restrictions, but it's technically possible to model this with sub-feature privileges. We could add a new sub-feature privilege to export encrypted secrets, and not grant this to the "primary" feature privileges by default. One downside with this approach is that it would require explicit action from an administrator to grant this privilege, which adds friction to the export process. That might be tolerable, but it's worth keeping in mind as we consider these options.

Use-case specific import/export

It's also possible for us to not try to solve this at that platform level, and require a use-case specific import/export. This would allow the consumers of encrypted saved-object attributes to make this decision and implement one of the aforementioned solutions, or one we haven't outlined yet. This would create a rather fragmented experience, and make it difficult for end-users to import/export all of their saved-objects.

I'm not opposed to this approach...it might be more palatable to think of this as "solution-specific import/export", as those are somewhat more contained than features or apps, and the solutions might be in a better position to make informed decisions about which objects/attributes to include in the export. There's admittedly a lot of open questions about exporting from the shared services such as Alerts, though.

[pgayvallet]

Import the saved-objects without these attributes

Forces the consuming code to be aware of such limitation and to handle them accordingly. Extremely error prone imho.

Prompt the user to provide the secrets on import

Yea, as you said, this would be a pain to implement. Not doable at the moment at least, as there is no way for plugins to hook or add additional logic to the export/import process. I feel like this export 'enhancement' feature is going to be required to solve most of the issues we currently have with SO export/import, so long term, that might be an option? Still, allowing the client-side import workflow to be modified from plugins (like, to add this 'prompt secret' step) will be all but trivial.

Include the attributes in the export

It's not possible today due to licensing restrictions

@legrego Could you elaborate on that?

One downside with this approach is that it would require explicit action from an administrator to grant this privilege, which adds friction to the export process. That might be tolerable, but it's worth keeping in mind as we consider these options.

Also, that would still force to be able to support the first point (Import the saved-objects without these attributes), as a user exporting without this privilege would have the attributes removed.

Use-case specific import/export

Ideally at some point, the SO import/export APIs (note: not necessarily the SO management section, but at least the APIs) would be the single entry point to perform such operations (even if we are aware that in current state, it is not possible). But as a short term, I guess it would work. That's basically doing nothing on our side though, so not sure it should be considered an option.

[azasypkin]

Still, allowing the client-side import workflow to be modified from plugins (like, to add this 'prompt secret' step) will be all but trivial.

Just an idea, alternatively we could just proceed with import as usual and notify subscribers (e.g. Alerts subscribes to import events for the alert SO type) that these particular objects were imported so that plugin can analyze them and warn/guide users afterwards (or do anything it wishes with these objects). Not sure if it's technically feasible for the large imports though.

@legrego Could you elaborate on that?

I believe @legrego meant that sub-feature privileges is a gold+ Kibana feature right now.