[Fleet] Implement new transform installation mechanism

[joshdover]

Official support for transforms was added to the package-spec in elastic/package-spec#307. We now need to update Fleet to support installing transforms defined in the official format.

Scope

The spec defines transforms to be contained in a package with the following file structure:

• package-version/elasticsearch/transform/<name>/transform.yml - required, contains the transform definition itself
• package-version/elasticsearch/transform/<name>/manifest.yml - optional, contains index template settings and whether or not the transform should be started on installation
• package-version/elasticsearch/transform/<name>/fields.yml - optional, contains index mappings using the same format as data streams mapping defintions

You can find some examples of valid transforms that match the new spec here: https://github.com/elastic/package-spec/tree/main/test/packages/good/elasticsearch/transform

The transform installation code needs to be updated to:

• Handle creating an index template when either the fields.yml file is present or manifest.yml contains destination_index_template:
  • Generate the mappings from the fields.yml file. This can be done by leveraging loadFieldsFromYaml and generateMappings
  • Combine the mappings and other index template settings from manifest.yml into a single index template
  • Create the index template and track the template in EsAssetReferences
• Create the destination index if it doesn't already exist
  • Open question: how will new mappings and settings get applied across upgrades if the destination index is not deleted?
• After the index template / destination index has been created and initialized, create the transform and start it. This largely already exists, we just need a couple adjustments:
  • Update the logic to read the transform from the transform.yml file rather than a json file.
  • Do not start the transform if the manifest.yml file contains start: false (the default is to start the transform if this isn't specified).
• Handle creating the transforms in the correct order. For instance, if one transform's src index is the dest index of another transform, the second transform and index need to be created first.

The transform uninstallation code should not need to be updated. Our existing code for deleting transforms and index templates should already work as long as the EsAssetReferences are updated correctly to track the assets that were created by the install code.

Things to ensure still work:

• We need to ensure the old endpoint packages can be uninstalled
• We need to be able to install the old endpoint packages which used a different specification format for the transform. Support for this can be removed in the next major version.

Out of scope

• For the time being, we will also continue to support transforms defined in the legacy/illegal format that is currently used by the Endpoint package and supported by Fleet but not allowed by the package-spec. This will allow the Endpoint package to be updated for the new format separate from this change on their own schedule. It will also allow older versions of the package to be installed in newer Kibana versions, if necessary.
• Changing which privileges are used to install transforms. That is a larger effort which is still being discussed in [Fleet] Support installing transforms without granting kibana_system index privileges #137278
• Changing the upgrade strategy. This is also a larger effort that needs to be defined. For now, we'll continue to delete the existing transform and create a new one.
• [Discuss] Support stored scripts in Fleet packages package-spec#202
• Space-specific transforms

Blocks:

• Move transforms from the CSP plugin to the CSP integration package #130086

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[joshdover]

@kevinlog Please provide any feedback on the requirements around the Endpoint package. Will we need to be able to install older versions of the package that define transforms in the legacy way that was not implemented in the spec or can we get away with only supporting the new format?

[kevinlog]

@joeypoon @pzl can you take a look at this and comment?

[pzl]

I believe the ability to install legacy transforms is still required. There are package rollbacks and other behaviors that could still task a relatively new (8.3,8.4) kibana and fleet plugin with installing an older endpoint package (rather than just a stack upgrade scenario with a pre-existing old package). There are a few edge cases of the actual installation of an older package still needed.

[joshdover]

Makes sense, but I want to be sure we don't need to support this for a long period of time. Would it be feasible to drop support for the legacy transforms in the following minor release? So if we were to implement this in 8.4 (for example) could we remove support for legacy in 8.5?

[pzl]

I'm sure on some timescale we can drop it, but I think it might need a few release cycles. Package upgrading still remains somewhat detached from stack upgrades, which are historically fraught with user issues and considered "difficult". I want to avoid contributing to that reputation for as long as reasonable.