Fix / clarify capabilities API

[faec]

Refactor the capabilities manager to address #2951 and associated issues, allowing dramatic simplification of capability logic. There are two types of capability invocation, described in #2951. They are currently called on semi-opaque config trees, but in reality each type has a simple explicit variable dependency: upgrade checks need the version and sourceURI, and input/output checks need the name of the input/output type. This refactor makes that dependency explicit in the API, removing the rest of the config tree and requiring only the specific fields the capability needs.

After talking with @cmacknz, I've also included the fix for #2949 and updated the relevant tests.

The high-level changes are:

• Instead of a catchall Apply function, the Capabilities interface now defines three calls, AllowInput(inputType string), AllowOutput(outputType string), and AllowUpgrade(version string, upgradeURI string).
• Coordinator is now responsible for calling the appropriate API, instead of invoking Apply at every potential call point. In particular, Coordinator now filters input/output types itself using AllowInput and AllowOutput -- Coordinator already needs to traverse these component structures, and this means that all capabilities code that previously needed to traverse an untyped interface{} tree can be removed. This also fixes Input/output capabilities miss input types defined by variables #2950 since the new checks are run on the final input/output names instead of the raw configuration tree.
• The capabilities tests preserve all logical checks from the previous version (and add a couple extra), but removed all extraneous work involving proper encoding/decoding of opaque config trees since capabilities no longer need to do that.
• The input/output capabilities were line-for-line identical except for the name of their defining YAML field, so they've been merged into a single helper, stringMatcher, and the input vs output distinction is now owned by capabilitiesManager which has separate arrays for the two cases.
• Upgrade capabilities now follow the same semantics as input/output constraints, by applying the "allow" and "deny" rules in the order they're encountered when one of the conditions passes (see Upgrade capability semantics conflict with other types #2949).

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Related issues

• Fixes Upgrade capability semantics conflict with other types #2949
• Fixes Input/output capabilities miss input types defined by variables #2950
• Fixes Capabilities use secret internal type matching to avoid a defined API #2951

[mergify]

This pull request does not have a backport label. Could you fix it @faec? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-06-29T15:15:12.931+0000

• Duration: 21 min 49 sec

Test stats 🧪

Test	Results
Failed	0
Passed	5899
Skipped	19
Total	5918

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages.

• run integration tests : Run the Elastic Agent Integration tests.

• run end-to-end tests : Generate the packages and run the E2E Tests.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	98.684% (75/76)	👍
Files	69.112% (179/259)	👎 -0.119
Classes	67.288% (325/483)	👎 -0.606
Methods	54.204% (1025/1891)	👎 -0.351
Lines	40.255% (11727/29132)	👎 -0.31
Conditionals	100.0% (0/0)	💚

[ycombinator]

Since this change fixes a bug, any reason we shouldn't backport it?

[ycombinator]

LGTM.

[faec]

Since this change fixes a bug, any reason we shouldn't backport it?

Hmm, it does fix a bug but it's also downstream of the big Coordinator refactor (which also technically fixes bugs but is a quite large change to merge after feature freeze)... I will check how big the merge conflict is but I'm not sure it's worth much patch wrangling since the bugs were mostly in upgrade capabilities which seem quite rarely used, the real motivation was code cleanup and clarifying Coordinator internals for future work

[ycombinator]

Looks like TestDiagnosticComponentsExpected needs to be updated with the changes in f261717?

[ycombinator]

I really love how simple and readable capabilities_test.go has become now! Reading through it, it would be nice to add a couple more test scenarios there to ensure we've covered all capabilities configurations and haven't introduced any regressions:

1. Allowing/denying any metrics inputs, i.e. */metrics.
2. Allowing/denying upgrading to a certain version.

[ycombinator]

LGTM. Thanks @faec for making the capabilities APIs more explicit and simplifying the tests in the process as well.