Remove generated fields.yml from git repositories #2931
Conversation
ruflin
force-pushed
the
remove-generated-fieldsyml
branch
2 times, most recently
from
0d3df51 to
6677fb4
Compare
| mkdir -p etc/kibana | ||
| -cp -r module/*/_meta/kibana etc/ | ||
|
|
||
| # Collects all module and metricset fields | ||
| .PHONY: fields | ||
| fields: | ||
| mkdir -p etc/ | ||
| cat ${ES_BEATS}/metricbeat/etc/_meta/fields_base.yml > etc/fields.yml | ||
| . ${PYTHON_ENV}/bin/activate; python ${ES_BEATS}/metricbeat/scripts/fields_collector.py >> etc/fields.yml | ||
| cat ${ES_BEATS}/metricbeat/etc/fields.yml > etc/fields.generated.yml |
There was a problem hiding this comment.
@ruflin I am not sure I understand the reason for replacing _meta/fields_base.yml with fields.yml. It supposed to contain common fields that are exported by all Metricbeat modules, right? If yes, then wouldn't make more sense to call it something more obvious like etc/fields_common.yml?
There was a problem hiding this comment.
fields.yml contains field which are common for all metricbeat events. The reason I called it fields.yml that in filebeat or winlogbeat the file with all the common options is also called fields.yml. fields_common.yml would also be an option, but my idea was to call all non generated files which contain fields.yml, fields.yml.
There was a problem hiding this comment.
I like fields.common.yml for the Beats where it only contains a subset of the fields. I like this name because it makes it clear that it's not the complete fields.yml.
There was a problem hiding this comment.
The common fields for libbeat are placed under libbeat/_meta/fields_base.yml. I think we should be a bit consistent here, and have the same approach in all the Beats, including libbeat. Maybe an option would be to have for Metricbeat the file under metricbeat/_meta/fields_common.yml (not sure, why we need to have it under etc/).
There was a problem hiding this comment.
Both works for me. I renamed it to fields.common.yml
ruflin
force-pushed
the
remove-generated-fieldsyml
branch
4 times, most recently
from
01d1c85 to
e5ee0fc
Compare
As some of the generated fields are also needed for the tests, make collect must be run in some cases before running the tests. Testsuite does this automatically. It was not added to each test as it would run multiple times and causes some issue with docker container access rights. * Adapt script to rely on generated fields.generated.yml file if it exists * Add fields.generate.yml to gitignore * Make update dependent on collect call instead of the other way around as update requires now collect. * Make system-tests use fields.generated.yml * Remove obsolete metricbeat.json file NOTE: More fields.yml generated fields can be removed in the future. This PR provides the basic logic for it.
ruflin
force-pushed
the
remove-generated-fieldsyml
branch
from
e5ee0fc to
f29f363
Compare
As some of the generated fields are also needed for the tests, make collect must be run in some cases before running the tests. Testsuite does this automatically. It was not added to each test as it would run multiple times and causes some issue with docker container access rights.
NOTE: More fields.yml generated fields can be removed in the future. This PR provides the basic logic for it.