Input packages don't require to define fields #994
Conversation
With OTel inputs we have seen that all fields required are already defined in the built-in templates, and requiring to add mappings can lead to conflicts with them. For ECS inputs, required fields are also defined in ecs@mappings, so there would be no need to require them. These packages can still define fields if needed. Test package for OTel is updated for current recommendations.
| elasticsearch: | ||
| index_template: | ||
| mappings: | ||
| subobjects: false |
There was a problem hiding this comment.
subobjects: false at the root level conflicts with built-in mappings for OTel.
| type: folder | ||
| name: fields | ||
| required: true | ||
| required: false |
There was a problem hiding this comment.
For ECS input packages, there could be issues when running system tests if the ECS fields are not defined.
Mappings and dynamic templates defined in ecs@mappings are not taken into account in that validation (in elastic-package). So, I think there will be fields reported as not defined for those input packages.
As those packages could still define the fields folder, there should not be any problem.
It needs to be tested also how elastic-package behaves when there is no fields folder in those input packages.
There was a problem hiding this comment.
Currently, elastic-package create package command to create an input package creates that fields folder:
$ ls test/
agent changelog.yml _dev docs fields img LICENSE.txt manifest.yml
$ ls test/fields/
base-fields.ymlIf one of these packages created with elastic-package configures an otelcol input without removing that default fields folder, IIUC those mappings would be created as usual. This is the expected behavior here, isn't it ?
There was a problem hiding this comment.
For ECS input packages, there could be issues when running system tests if the ECS fields are not defined.
Mappings and dynamic templates defined in ecs@mappings are not taken into account in that validation (in elastic-package). So, I think there will be fields reported as not defined for those input packages.
As those packages could still define the fields folder, there should not be any problem.
Yes, for these packages we still expect them to define @timestamp, ecs.version and so on. Though this wouldn't be currently required.
If the package has system tests, elastic-package will complain if these fields are not defined, if there are no system tests and no defined fields, this won't be detected, but shouldn't cause any issue (because of missing the usually required fields, there might be other issues undetected by the lack of tests...).
We may need to review this in elastic-package.
It needs to be tested also how elastic-package behaves when there is no fields folder in those input packages.
Tests work with the sql_input test package in elastic-package, but there are test failures caused by missing fields definitions (what would be expected).
Currently,
elastic-package create packagecommand to create an input package creates thatfieldsfolder:
We need to review elastic-package create package for OTel packages in general. For ECS input packages it would be fine if it creates the fields folder.
💚 Build Succeeded
History
cc @jsoriano |
3 participants
With OTel inputs we have seen that all fields required are already defined in the built-in templates, and requiring to add mappings can lead to conflicts with them.
For ECS inputs, required fields are also defined in
ecs@mappings, so there would be no need to require them.By their generic nature, it makes sense to think that they should not be required to define any field.
These packages can still define fields if needed.
Test package for OTel is updated for current recommendations.
Follow up to elastic/kibana#237165.