[APM] Document custom ingest pipelines

[bmorelli25]

Related dev issue

• [Ingest Pipelines][discuss] Support custom pipelines  package-spec#129

Background

As of 8.0: The Fleet-managed ingest policy will be overwritten on reinstall/upgrade. Users can define their own custom ingest pipeline if needed, but they need to make sure that they also call the package-provided pipeline.

Details

1. Fleet adds ingest pipelines for the APM integration using index templates that include pipeline index settings.
2. Elasticsearch uses the data view in these index templates to match pipelines to APM data streams.
   As an example, the index template traces-apm, matches traces-apm-* data streams and assigns the "traces-apm-7.16.0" pipeline:

3. This Fleet-managed index template ⬆️ will be overwritten on reinstall/upgrade.

Assuming the above is correct, then I see three scenarios:

Workarounds

1. A user can edit the Fleet-managed index template to point to a new pipeline, ensuring the following are true:
   a. ensure the package-provided pipeline is also called
   b. know that the index template will be overwritten on install/upgrade

2. A user can edit the default pipeline directly, like the "traces-apm-7.16.0" pipeline, ensuring the following are true:
   a. the package-provided pipelines are still included (first!)
   b. know that their edits will be overwritten on install/upgrade

Documentaiton ideas

The following snippet was removed from elastic/apm-server#6940.

[float]
[id="custom-pipelines{append-legacy}"]
=== Add a custom ingest pipeline

IMPORTANT: Fleet-managed index templates are overwritten on reinstall/upgrade.
Any customizations to index pipelines are not persisted when upgrading.
When editing {fleet}-managed index templates, ensure that the package-provided pipeline is still called.

// tutorial would be something like this:
. Find the pipeline **Stack Management** > **Index Management** > **Index Templates** > `traces-apm`.
. **Settings** >
+
[source,json]
----
{
  "index": {
    "default_pipeline": "traces-apm-{version}"
  }
}
----

// Edit default
. **Stack Management** > **Ingest Pipelines** > Search for +traces-apm-{version}+.
. **Manage** > **Edit** to edit the default pipeline.

// Add custom pipeline
. **Stack Management** > **Ingest Pipelines** > **Create pipeline**
. Create it
. **Stack Management** > **Ingest Pipelines** > Search for +traces-apm-{version}+.
. **Manage** > **Edit** to edit the default pipeline. Add the new pipeline as the final step.

Links 'n stuff

• More context: Add content to APM User guide apm-server#6378 (comment).

[bmorelli25]

I'm wondering if we should be documenting this. Not being able to edit pipelines in a way that persists is a known issue, but we don't know how much of a pain that is yet. By documenting the workaround so prominently we may not learn that.

That was Andrew, with a thumbs up from Silvia in elastic/apm-server#6940 (comment). Holding off on documenting this for now.

[bmorelli25]

When we're ready to document custom ingest pipelines, we also need to remove the commented-out content added in elastic/apm-server#8043.