Docs: Changing index pattern as described in docs does not work

[roncohen]

Documentation describes setting the index pattern like this:

output.elasticsearch:
  index: "apm-%{[app.name]}-%{[beat.version]}-%{+yyyy.MM.dd}"

here https://www.elastic.co/guide/en/apm/server/current/configuring.html#index-pattern

but changing it make APM Server print:
2017/11/28 14:03:39.259288 beat.go:635: CRIT Exiting: setup.template.name and setup.template.pattern have to be set if index name is modified.

We should update the documentation to explain that these also need to be set and how to set these.

[graphaelli]

@roncohen I started working on this but ran into issues. With this configuration (yes it's slightly different than the docs, more on that later):

 output.elasticsearch:
  index: "apm-%{[beat.version]}-%{[app.name]}"

setup.template.name: "apm-%{[beat.version]}"
setup.template.pattern: "apm-%{[beat.version]}-*"

The onboarding event:

{
  "@timestamp": "2017-12-01T21:21:24.418Z",
  "@metadata": {
    "beat": "apm-server",
    "type": "doc",
    "version": "7.0.0-alpha1"
  },
  "listening": "localhost:8200",
  "beat": {
    "name": "local",
    "hostname": "local",
    "version": "7.0.0-alpha1"
  }
}

fails to index:

client.go:280: DBG [elasticsearch] PublishEvents: 1 events have been  published to elasticsearch in 2.141084ms.
client.go:469: DBG [elasticsearch] Bulk item insert failed (i=0, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}

Same issue with pushing a typical event.

Modifying that config slightly to:

 output.elasticsearch:
  index: "apm-%{[beat.version]}-%{[context.app.name]}"

allowed the payload to load property but the onboarding document still fails, as expected, since it does not have a context.app.name (or even an app.name). Is context.app.name acceptable for the configuration? Is there a way to make app.name reference context.app.name?

Back to the setup.template.pattern - do you know if that's prefix match only? The same question was asked here but the docs haven't been updated and I haven't tracked down the code that uses it yet or figured out how to just try it out yet.

[roncohen]

those are great questions. I suspect this hasn't been revisited since app.name was moved to context.app.name and has not been tested since the onboarding document was introduced. A possible solution would be to allow the onboarding document to go to a separate index. This could prove non-trivial to do. Let's discuss at our weekly. Could you put it on the agenda?

[graphaelli]

Per discussion, we'll hold off on supporting splitting events into per-application indexes. I'll propose updates to the documentation shortly to reflect that.