[Ingest] Hash processor - require keyed hash ?

[jakelandis]

A new ingest processor, hash, is near completion. See #31087. As is, it is an excellent addition to the set of processors, but I wanted to continue the discussion started here if we should require key based hashing.

The current implementation requires a key for the hash function, which requires the key to exist in the keystore. For the key to exist in the keystore a human would need to add it.

This step could prove to be a barier to adoption since it requires configuration that lives outside the ingest pipeline configuration in order for the processor to be functional. The user wishing to use the new hash processor would need to coordinate with the user responsible for the elasticsearch deployment to ensure that this key exists. (not always the same the person / org). This extra step may also be cumbersome when used with Beats modules. Also, this could prove even more challenging on cloud (disclaimer: I am not familiar with how the keystore and cloud interop).

@jkakavas makes some excellent points about why a keyed hash is better, but I would like to consider removing the requirement to use keys for hashes.

This is a classic case of usability vs. security for the default values.

While I agree with @jkakavas that keyed hashes are better, I believe that non-keyed modern cryptogphic hashes are sufficient for the default case. IMO, allowing non-keyed hashes would remove that barier to adoption without introducing too much risk. Documentation can be provided to encourage the user to provide a key for the hash.

Also in the original thread @jkakavas mentions:

Another alternative would be the use of generated keys for HMACs if the user doesn't set one in the settings ( similar to what Kibana does for its auth cookie encryption key )

Thoughts ?

[elasticmachine]

Pinging @elastic/es-core-infra

[jasontedor]

Pinging @rjernst.

[jakelandis]

This has been discussed in person. To the general question as logged - should we allow non-keyed hashes ? The general consensus is yes, keyless hashes can provide value for example: #31087 (comment).

However, we need to be careful to NOT position a keyless hash as either a security feature, nor a GDPR feature. Only keyed hashes should be be positioned for security and GDPR. We also shouldn't release a Hashing processor without the a keyed hash, and likely require a user to opt-out of using a key (as opposed to opt-in) as a security first approach.

This issue did raise this concern (#31087 (comment)) such that we need some means to ensure that key is in sync between multiple nodes before we can release the hashing processor.

[jakelandis]

A keyless hash option is a nice to have, however this is blocked until #34085 is resolved since a keyed hash is must have.

Removing team-discuss label and will keep discussing internally how to best move forward on #34085.

[dakrone]

This has been open for quite a while, and we haven't made much progress on this due to focus in other areas. For now I'm going to close this as something we aren't planning on implementing. We can re-open it later if needed.