[libbeat] Enable TLS 1.3

[faec]

Closes #10226

Add TLS 1.3 support in libbeat SSL configurations.

[urso]

We will also need to update the list of configurable cihper suites + docs.

By default TLS 1.3 won't be available, unless one enables it via GODEBUG. I wonder if we want to introduce an environment variable named BEATS_TLS13 and update GODEBUG on init if this one is set. Also print a warning that TLS 1.3 is experimental. Until TLS 1.3 will be enabled by default in the stdlib. WDYT?

[faec]

I've added a BEATS_TLS13 environment variable (and confirmed that it works as expected; messing with global environment vars on startup makes me nervous).

I also removed the 1.3 cipher suites from the config again after realizing that they can't be used in tls.Config.CipherSuites (see the docs) -- this seems odd to me, but since that's the only place we used that setting I removed them from our enums and documented their absence.

[ph]

@faec adding an environment variable is always tricky, I suggest we add two simple python test that check that it works as expected and we don't remove that behavior, I believe we could use the filebeat TCP input as the candidate and we can pass the environment variable to the python runner.

[urso]

Do we need to add cipher suites as well?

[faec]

Do we need to add cipher suites as well?

I did in a previous iteration, but then removed them (see the previous comment / doc link, Go doesn't allow individual configuration of TLS 1.3 cipher suites)

[urso]

Change looks good. But CI doesn't seem to like it :)

[urso]

fyi; we are switching master to go1.13: #14335

[urso]

Some tests are failing due to outdated certificates. This has been fixed recently in master.

[faec]

Unfortunately the important failure is unrelated to the certificate bug: test_tcp_tls.Test.test_tcp_over_tls_mutual_auth_fails fails, i.e. auth with an incorrect cert does not raise an SSL exception. My suspicion / hope is that this is because the mechanism the test uses to detect unsuccessful auth doesn't work in 1.3, or because enabling 1.3 allows a fallback auth when the intended one fails, but I'm still trying to confirm.

[faec]

@andrewkroh helped me get a local repro of the CI problem this week, which led to a diagnosis / fix:

The failure in test_tcp_tls.Test.test_tcp_over_tls_mutual_auth_fails was "real" (i.e. not-flaky); the appearance that it passed was because local runs were using an old (pre-TLS 1.3) openssl in the python installation.

The problem was that authentication failures in TLS 1.3 are detected later in the connection process than earlier versions, so a simple connection and handshake is not enough (at least not with the way Python's SSL library reports errors) -- we must also explicitly wait on a response from the server so that the failure can be reported as an exception when it arrives.

[urso]

This sounds like the test outcome depends on the system we run the test on. Yay on unwanted surprises.

Can we replace the test (and maybe other TLS tests) with unit tests in go?

For the python tests (given we have unit tests), we can consider to selectively skip them, based on the tls libs version in use.

[faec]

Oh, sorry to be clear, there's a fix now that should work on any platform (connect-handshake-read throws an SSLError for all versions)... though before finding the fix I was starting to lean towards a go unit test instead (I just wanted to make sure it wasn't somehow exposing a misconfiguration of Filebeat itself). I'd be happy to get these migrated to go soon...

[faec]

Manual release testing: Basic connectivity rules are tested automatically, but considering the security implications of this update let's double check the following end-to-end scenarios:

• Filebeat with default SSL enabled successfully indexes events in an Elasticsearch that requires TLS 1.3 (see ssl.supported_protocols in the Elasticsearch security settings)
• Filebeat with TLS 1.3 disabled cannot connect to an Elasticsearch that requires TLS 1.3 (TLS 1.3 can also be disabled in Filebeat by setting ssl.supported_protocols in filebeat.yml)
• Filebeat with TLS 1.3 required (supported_protocols = [TLSv1.3]) cannot connect to an Elasticsearch that doesn't enable TLS 1.3.

[urso]

The TLS handshake has been verified with wireshark for all tests.

• ES default ssl settings, beats default ssl setting. OK
• ES default ssl settings, beats TLS1.3 only. OK
• ES TLS1.3 only, beats no TLS 1.3, can not connect. OK
• ES TLS1.3 only, beats default can connect. OK
• ES TLS1.3 only, beats TLS1.3 only. OK
• ES no TLS1.3, beats no TLS1.3, can connect with TLS1.2. OK
• ES no TLS1.3, beats default can connect. OK
• ES no TLS1.3, beats TLS1.3 only, can not connect. OK
• ES + FB with default TLS settings should use TLS 1.3. OK
• Default configuration mentions correct default versions. FAIL
• Documentation mentions correct default versions. OK

I will create follow up PR fixing the default configuration files.

[urso]

Fix for default configuration file: #15760