Better error messages for common TLS problems

[tvernum]

We have a goal of making TLS easier (see #41182).

One of the concrete ways that we can do that is to improve the error messages that users get when TLS fails.
This is a meta issue to track the various scenarios that we would like to improve.

Overall goals

• Failures should be as early as possible. If we can detect a configuration problem at node start-up, then we should.
• Messages should explain the cause of the problem in terms a human can understand.
• Messages should indicate how the problem can be resolved in as specific terms as possible (e.g. By indicating which ES settings control the behaviour).
• Messages should not hide the technical details that would be useful to ES engineers or other people who understand TLS.
• Warnings can be written to the logs if we detect something that seems to be wrong, but we should not assume that the logs will be consulted unless something breaks, and we should avoid "spamming" log files with repeated or recurring messages.

Glossary

In the scenarios below...

• Resources refers to Keystores, Truststores, Keys & Certificates as appropriate.
• TLS refers to any version of SSL/TLS unless it explicitly indicates a version

Scenarios where good messages will help resolve problems:

Note: Some of these scenarios overlap.

• Resources that are missing (typo in settings, or not relative to config directory) Improve errors when TLS files cannot be read #44787
• Resources that cannot be read due to file permissions Improve errors when TLS files cannot be read #44787
• Resources that cannot be read due to the security manager (wrong directory) Improve errors when TLS files cannot be read #44787
• Resources that cannot be read due to incorrect password
• Resources that cannot be read due to missing password (password required, but not provided)
• Resources in an incorrect format (e.g. PKCS#12 file in certificate_authorities)
• Resources that are corrupt or malformed (random data in a .crt file)
• Resources configured, but not enabled (e.g. xpack.security.transport.ssl.key is set, but xpack.security.transport.ssl.enabled is not). This should be a warning in 7.x, but 8.0 could require an explicit value for ssl.enabled if any other ssl.* key is configured. Reject misconfigured/ambiguous SSL server config #45892
• TLS enabled but not configured (e.g. xpack.security.transport.ssl.enabled is true, but no resources have been configured). Reject misconfigured/ambiguous SSL server config #45892
• Configured certificate, but no key
• Configured key, but no certificate
• Mismatched certificate & key
• Configured truststore (or certificate_authorities) does not contain any trusted CAs
• Configured keystore does not contain any key-pairs.
• Client: Handshake failed due to no cipher suites in common
• Server: No cipher suites in common: unrecognised cipher suites
• Server: No cipher suites in common: client's suites are recognised but disabled
• No TLS version in common (server & client)
• Provided certificate is signed by a CA that is unknown to us
• Provided certificate is signed by a CA that is known to us but not trusted in this context (e.g. JRE's cacerts)
• A full certificate chain was provided, but we don't trust it
• A partial certificate chain was provided, and we don't recognise any of it
• A certificate was provided the matches the DN of a certifcate we know/trust, but is not the same certificate (e.g. user incorrectly generates multiple CAs with CN=Elastic Certificate Tool Autogenerated CA)
• Certificate has expired.
• Certificate is not yet valid.
• Hostname verification is enabled (verification_mode: full) but server certificate does not contain any SANs.
• The server certificate does not match the hostname we used.
• The server certificate only contains DNS SANs but we are connecting via IP (typical solution is to configure publish_host)
• Server requires a client certificate, but we don't have one
• Server rejected our client certificate (Unknown CA)
• Client does not trust this server's certificate
• A certificate has been configured as a CA, but is not a valid issuer
• A certificate has been configured as a server/client certificate, but is a CA (warning only)
• A certificate has an Extended Key Usage of server-only, but is being used on a client.
• A certificate has an Extended Key Usage of client-only, but is being used on a server.
• Plaintext on an TLS connection (Fix plaintext on TLS port logging #45852, for transport and HTTP)
• TLS on a plaintext connection

[elasticmachine]

Pinging @elastic/es-security

[jpcarey]

related #32688