Improve handling of fields.yml and templates

[ruflin]

Currently each beat is shipped with the fields.yml as a separate file which requires the fields.yml to be in the right place for the template to be loaded. To make the setup less error prone the goal is to ship the fields.yml with each beat but allow to overwrite the fields.yml or the template in the config.

This issue is to track the progress of the implementation steps.

• Add fields.yml to the beats binary Make fields.yml part of the binary #4834
• Add config option to load fields.yml from a file
• Add command export fields.yml by the beat
• Remove fields.yml from being packaged with the beat
• Add config option to allow index template file to be loaded from file
• Add command to export beat template file based on fields.yml and ES version

[y0299]

@ruflin do you have some documents to explain the elements in fields.yml, or how to override it in practise?

[ruflin]

@y0299 Not a general one yet. There are snippets in our dev guide like https://www.elastic.co/guide/en/beats/devguide/master/metricset-details.html#_fields_yml but nothing "general" that explains it. I wanted to write a guide for some time :-( Anything specific I can help with?

[tsg]

Adding a point here: We should add (back) the option of loading the template from a simple JSON file. That's because fields.yml doesn't support all/arbitrary mapping features, so for some situations it might still make sense to fallback to writing JSON. Also, when loading e.g. JSON fields with keys_under_root, a new fully template file might be easier to produce than the equivalent fields.yml, because we don't yet document fields.yml.

[ruflin]

Instead of letting users load there specific template I'm hoping to mitigate the issue with #6024 The problem I see with users loading their own template is that it could break a lot of things in our Dashboards, ML jobs etc.

If someone wants to load their own template the workflow would be as following:

• Export json template
• Change it to your own need
• Upload it with curl

The above should make it more obvious that this is not necessarly a supported workflow.

For the docs especially with #6024 it's really time to document it.

There is still the issue with features not supported in fields.yml. I'm thinking by now it has become pretty to add them if needed based on a feature request.

[urso]

@ruflin I think we can close this issue for now. Feel free to reopen if there is still something we should work on.