[Fleet] Alternative Index pattern in ES templates

[jalvz]

When installing integrations, Fleet creates templates with index patterns following the indexing strategy: <type>-<dataset>-*

In the APM package, the datasets defined are only a prefix, and a service name will be appended during runtime. We therefore need that the APM templates are created with matching index patterns, like <type>-<dataset>*-*

Fleet should know whether to create templates with this alternative index pattern based on the dataset_is_prefix boolean field in the data stream manifest file.

Spec PR: elastic/package-spec#102

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ruflin]

@jalvz Is it <type>-<dataset>*-* on purpose or should it be <type>-<dataset>.*-*?

[jalvz]

The . is kind of a convention in apm server, I wasn't intending to force it on everybody. But both would work, so I'm fine with either.

[ruflin]

I kind of like the idea of the . because it prevents "accidental" applying of the pattern. Lets assume build the service apmish and now the dataset has the same name. Now without the . the pattern would match, but with the doc it is not matching.

[ph]

@skh Can you point me to the code? I would expect it to be a few lines of changes is that the case?

[skh]

Can you point me to the code?

https://github.com/elastic/kibana/tree/master/x-pack/plugins/fleet/server/services/epm/elasticsearch/template

I would expect it to be a few lines of changes is that the case?

Possibly, but the lines are probably not trivial. I don't think we parse the datastream manifest at this point, and we'll need to in order to decide which pattern to use.

[ph]

Thanks, @skh, I thought that the surface area was limited and could have been done by another team with a little guidance. But like you have said in slack if it's a few hours. It's probably less problematic.

[skh]

Minor correction: we don't need to add manifest parsing but we do need elastic/package-registry#674 .

[skh]

I have opened a draft PR at #89870, but I ran into an issue that needs further specification (copied from the PR description):

If the package contains, at the same time e.g. a data stream with this manifest:

title: APM application metrics
type: metrics
dataset: apm   # <- no .something here
dataset_is_prefix: true

and another one with this manifest

title: APM internal metrics
type: metrics
dataset: apm.internal

the first data stream will use metrics-apm.*-* as index pattern in its template, and the second one metrics-apm.internal-*, which conflict. ES returns this error:

{"statusCode":400,"error":"Bad Request","message":"[illegal_argument_exception] index template [metrics-apm] has index patterns [metrics-apm.*-*] matching patterns from existing templates [metrics-apm.internal] with patterns (metrics-apm.internal => [metrics-apm.internal-*]) that have the same priority [200], multiple index templates may not match during index creation, please use a different priority response from /_index_template/metrics-apm: {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"index template [metrics-apm] has index patterns [metrics-apm.*-*] matching patterns from existing templates [metrics-apm.internal] with patterns (metrics-apm.internal => [metrics-apm.internal-*]) that have the same priority [200], multiple index templates may not match during index creation, please use a different priority\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"index template [metrics-apm] has index patterns [metrics-apm.*-*] matching patterns from existing templates [metrics-apm.internal] with patterns (metrics-apm.internal => [metrics-apm.internal-*]) that have the same priority [200], multiple index templates may not match during index creation, please use a different priority\"},\"status\":400}"}

So if the data stream containing dataset_is_prefix in its manifest is meant as a catch-all for indices following the same naming pattern, but without their "own" data stream defining them, we will probably need to give different priorities here.

If this is not meant to happen, maybe we need to add some package validation so that this situation can't arise.

cc @ruflin @jalvz

[ruflin]

@jalvz I would assume in the actual APM package, this conflict should not happen?

In any case, we still need to resolve it. My suggestion would be to change priority on the dataset_is_prefix. I think apm.* should have lower priority then apm.internal. I don't remember what the exact priorities are we have at the moment but in general if dataset_is_prefix should always be set x-1 or something similar.

[skh]

I don't remember what the exact priorities are we have at the moment but in general if dataset_is_prefix should always be set x-1 or something similar.

For the built-in index templates we use priority 100, see https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html

Elasticsearch has built-in index templates for the metrics-*-*, logs-*-*, and synthetics-*-* index patterns, each with a priority of 100. The Elastic Agent uses these templates to create data streams. If you use the Elastic Agent, assign your index templates a priority lower than 100 to avoid overriding the built-in templates.

Otherwise, to avoid accidentally applying the built-in templates, use a non-overlapping index pattern or assign templates with an overlapping pattern a priority higher than 100.

For example, if you don’t use the Elastic Agent and want to create a template for the logs-* index pattern, assign your template a priority of 200. This ensures your template is applied instead of the built-in template for logs-*-*.

The templates we currently generate in EPM have priority 200. So for the dataset_is_prefix: true case, would 150 be a good value? Also, should we stay below 200 in all cases, given that the docs recommend this for customers who don't want to use our templates at all?

[jalvz]

This should be fixed in APM from next version on (elastic/apm-server#4669), we don't need to change priorities.
Not sure what is best for other cases, I think we are the first ones (accidentally) running into this.

[ruflin]

@skh 150 sounds like a good pick. Good point about the "conflicting" messaging for 200. Lets follow up on this separately. Either we should update ES docs or adjust our priority.

[skh]

Lets follow up on this separately. Either we should update ES docs or adjust our priority.

#90299