Error when using usage-example with generator.py

[faudebert]

Description of the problem including expected versus actual behavior:

Trying script/generator.py for the first time, I stumble upon a stack trace throwing KeyError: 'error'. I initially though I maybe passed wrong parameters but it looks like example documented in USAGE.md also have the same issue.

Thanks for writing this tool, I'm eager to use it since it can greatly help me to generate/maintain configuration for custom Beats indices. Let me know if I can somehow help you further.

Steps to reproduce:

1. git clone https://github.com/elastic/ecs && cd ecs
2. Running generator.py as documented in USAGE.md:

$ python3 scripts/generator.py --ref v1.6.0 \
  --subset            usage-example/fields/subset.yml \
  --include           usage-example/fields/custom/ \
  --out               usage-example/ \
  --template-settings usage-example/fields/template-settings.json \
  --mapping-settings  usage-example/fields/mapping-settings.json

Provide logs (if relevant):

Loading schemas from git ref v1.6.0
Running generator. ECS version 1.6.0
Loading user defined schemas: ['usage-example/fields/custom/']
/Users/florent/Sources/ecs/scripts/schema/cleaner.py:185: UserWarning: Example value for field `header_flags` contains an object or array which must be quoted to avoid YAML interpretation.

This will cause an exception when running in strict mode.
Warning check:
  check_example_value(field, strict=strict_mode)
/Users/florent/Sources/ecs/scripts/schema/cleaner.py:185: UserWarning: Example value for field `resolved_ip` contains an object or array which must be quoted to avoid YAML interpretation.
[...]
Traceback (most recent call last):
  File "/Users/florent/Sources/ecs/scripts/generator.py", line 106, in <module>
    main()
  File "/Users/florent/Sources/ecs/scripts/generator.py", line 60, in main
    es_template.generate_legacy(flat, ecs_version, out_dir, args.template_settings, args.mapping_settings)
  File "/Users/florent/Sources/ecs/scripts/generators/es_template.py", line 109, in generate_legacy
    generate_legacy_template_version(6, ecs_version, mappings_section, out_dir, template_settings_file)
  File "/Users/florent/Sources/ecs/scripts/generators/es_template.py", line 115, in generate_legacy_template_version
    template = template_settings(es_version, ecs_version, mappings_section, template_settings_file)
  File "/Users/florent/Sources/ecs/scripts/generators/es_template.py", line 202, in template_settings
    error_stack_trace_mappings = mappings_section['properties']['error']['properties']['stack_trace']
KeyError: 'error'

Any additional context:

$ git rev-parse HEAD
83ced31c9da6b3e5599b01e319a827a7941e6439

$ python3 --version
Python 3.9.0

$ pip3 list | grep -i -E '(yaml|pep|mock|git|jinja)'
autopep8      1.4.4
gitdb         4.0.5
GitPython     3.1.2
Jinja2        2.11.2
mock          4.0.2
PyYAML        5.3b1
yamllint      1.19.0

[lwille]

What a coincidence, I just ran into the same error! However, I only used a --subset and get to the same error:

$ python scripts/generator.py --out ~/tmp --subset subset.yaml --oss
Loading schemas from local files
Running generator. ECS version 2.0.0-dev
Traceback (most recent call last):
  File "scripts/generator.py", line 106, in <module>
    main()
  File "scripts/generator.py", line 60, in main
    es_template.generate_legacy(flat, ecs_version, out_dir, args.template_settings, args.mapping_settings)
  File "/var/home/lwille/src/ext/ecs/scripts/generators/es_template.py", line 109, in generate_legacy
    generate_legacy_template_version(6, ecs_version, mappings_section, out_dir, template_settings_file)
  File "/var/home/lwille/src/ext/ecs/scripts/generators/es_template.py", line 115, in generate_legacy_template_version
    template = template_settings(es_version, ecs_version, mappings_section, template_settings_file)
  File "/var/home/lwille/src/ext/ecs/scripts/generators/es_template.py", line 202, in template_settings
    error_stack_trace_mappings = mappings_section['properties']['error']['properties']['stack_trace']
KeyError: 'error'

subset.yaml:

---
name: json_filebeat
fields:
  base:
    fields:
      "@timestamp": {}

$ git rev-parse HEAD
83ced31c9da6b3e5599b01e319a827a7941e6439
$ python3 --version
Python 3.8.1
$  pip3 list | grep -i -E '(yaml|pep|mock|git|jinja)'
aspy.yaml        1.3.0
autopep8         1.4.4
gitdb            4.0.5
githubrelease    1.5.8
GitPython        3.1.2
Jinja2           2.11.2
mock             4.0.2
PyYAML           5.3b1
yamllint         1.19.0

[lwille]

@faudebert you might want to try checking out the v1.7.0 release tag; at least that works much better for me!

[ebeahan]

Thanks to both @faudebert for the great issue writeup and @lwille for chiming in with the guidance!

You've uncovered a bug introduced recently with some feature work, but a fix will be merged soon (PR #1191).

As @lwille pointed out, using the latest release tag, git checkout v1.7.0, may help work around the issue until a fix is merged.