Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: improve error messages for package field validation checks #994

Open
efd6 opened this issue Sep 29, 2022 · 3 comments
Open

feature: improve error messages for package field validation checks #994

efd6 opened this issue Sep 29, 2022 · 3 comments
Labels
Team:Ecosystem Label for the Packages Ecosystem team

Comments

@efd6
Copy link
Contributor

efd6 commented Sep 29, 2022

I have recently been doing a large update of packages in integrations and have found in that process that the errors that are printed could be improved.

Two example cases:

  1. Invalid field type
    1. file "/Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/gcp-2.12.0.zip/data_stream/audit/fields/fields.yml" is invalid: field 0.fields.2.type: 0.fields.2.type must be one of the following: "alias", "histogram", "constant_keyword", "text", "match_only_text", "keyword", "long", "integer", "short", "byte", "double", "float", "half_float", "scaled_float", "date", "date_nanos", "boolean", "binary", "integer_range", "float_range", "long_range", "double_range", "date_range", "ip_range", "group", "geo_point", "object", "ip", "nested", "flattened", "wildcard", "version", "unsigned_long"
    
  2. Duplicated field definition
    1. field "cloud.instance.id" is defined multiple times for data stream "storage", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/gcp-2.12.0.zip/data_stream/storage/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/gcp-2.12.0.zip/data_stream/storage/fields/ecs.yml
    

In both cases there is unnecessary information that reduces the readability of the error, particularly when there are many lines of errors. So I would suggest that the file paths be made relative to the package at least, and probably to the data stream.

In the case of the invalid field, the value that is invalid is not displayed, only the values that it may take. This in conjunction with the fact that the name of the field is not printed, only its position in the AST, make it difficult to find the offending value. In the case above it is not particularly hard since it is early in the file, but some are quite deep. So I would suggest that the offending value be printed, the name of the field (i.e. the value of the name field in the yml) and the file position also be provided (e.g. audit/fields/fields.yml:21:13 in the case above) to allow direct movement to the error in an editor.

The impact of the current output format can be clearly demonstrated with the following detail.

Error: building package failed: invalid content found in built zip package: found 87 validation errors:
   1. field "host.ip" is defined multiple times for data stream "rfb", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/rfb/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/rfb/fields/ecs.yml
   2. field "container.id" is defined multiple times for data stream "rfb", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/rfb/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/rfb/fields/beats.yml
   3. field "container.id" is defined multiple times for data stream "smb_cmd", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_cmd/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_cmd/fields/beats.yml
   4. field "host.ip" is defined multiple times for data stream "smb_cmd", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_cmd/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_cmd/fields/ecs.yml
   5. field "container.id" is defined multiple times for data stream "smb_files", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_files/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_files/fields/beats.yml
   6. field "host.ip" is defined multiple times for data stream "smb_files", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_files/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_files/fields/ecs.yml
   7. field "host.ip" is defined multiple times for data stream "smb_mapping", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_mapping/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_mapping/fields/ecs.yml
   8. field "container.id" is defined multiple times for data stream "smb_mapping", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_mapping/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smb_mapping/fields/beats.yml
   9. field "container.id" is defined multiple times for data stream "dnp3", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dnp3/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dnp3/fields/beats.yml
  10. field "host.ip" is defined multiple times for data stream "dnp3", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dnp3/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dnp3/fields/ecs.yml
  11. field "host.ip" is defined multiple times for data stream "ntlm", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ntlm/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ntlm/fields/ecs.yml
  12. field "container.id" is defined multiple times for data stream "ntlm", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ntlm/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ntlm/fields/beats.yml
  13. field "container.id" is defined multiple times for data stream "rdp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/rdp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/rdp/fields/beats.yml
  14. field "host.ip" is defined multiple times for data stream "rdp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/rdp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/rdp/fields/ecs.yml
  15. field "host.ip" is defined multiple times for data stream "http", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/http/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/http/fields/ecs.yml
  16. field "container.id" is defined multiple times for data stream "http", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/http/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/http/fields/beats.yml
  17. field "host.ip" is defined multiple times for data stream "known_certs", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_certs/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_certs/fields/ecs.yml
  18. field "container.id" is defined multiple times for data stream "known_certs", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_certs/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_certs/fields/beats.yml
  19. field "host.ip" is defined multiple times for data stream "modbus", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/modbus/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/modbus/fields/ecs.yml
  20. field "container.id" is defined multiple times for data stream "modbus", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/modbus/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/modbus/fields/beats.yml
  21. field "host.ip" is defined multiple times for data stream "mysql", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/mysql/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/mysql/fields/ecs.yml
  22. field "container.id" is defined multiple times for data stream "mysql", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/mysql/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/mysql/fields/beats.yml
  23. field "host.ip" is defined multiple times for data stream "ntp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ntp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ntp/fields/ecs.yml
  24. field "container.id" is defined multiple times for data stream "ntp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ntp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ntp/fields/beats.yml
  25. field "container.id" is defined multiple times for data stream "dhcp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dhcp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dhcp/fields/beats.yml
  26. field "host.ip" is defined multiple times for data stream "dhcp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dhcp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dhcp/fields/ecs.yml
  27. field "host.ip" is defined multiple times for data stream "dns", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dns/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dns/fields/ecs.yml
  28. field "container.id" is defined multiple times for data stream "dns", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dns/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dns/fields/beats.yml
  29. field "host.ip" is defined multiple times for data stream "files", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/files/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/files/fields/ecs.yml
  30. field "container.id" is defined multiple times for data stream "files", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/files/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/files/fields/beats.yml
  31. field "host.ip" is defined multiple times for data stream "socks", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/socks/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/socks/fields/ecs.yml
  32. field "container.id" is defined multiple times for data stream "socks", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/socks/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/socks/fields/beats.yml
  33. field "host.ip" is defined multiple times for data stream "tunnel", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/tunnel/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/tunnel/fields/ecs.yml
  34. field "container.id" is defined multiple times for data stream "tunnel", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/tunnel/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/tunnel/fields/beats.yml
  35. field "host.ip" is defined multiple times for data stream "ftp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ftp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ftp/fields/ecs.yml
  36. field "container.id" is defined multiple times for data stream "ftp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ftp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ftp/fields/beats.yml
  37. field "host.ip" is defined multiple times for data stream "radius", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/radius/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/radius/fields/ecs.yml
  38. field "container.id" is defined multiple times for data stream "radius", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/radius/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/radius/fields/beats.yml
  39. field "host.ip" is defined multiple times for data stream "smtp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smtp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smtp/fields/ecs.yml
  40. field "container.id" is defined multiple times for data stream "smtp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smtp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/smtp/fields/beats.yml
  41. field "host.ip" is defined multiple times for data stream "signature", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/signature/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/signature/fields/ecs.yml
  42. field "container.id" is defined multiple times for data stream "signature", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/signature/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/signature/fields/beats.yml
  43. field "container.id" is defined multiple times for data stream "ssl", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ssl/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ssl/fields/beats.yml
  44. field "tls.version_protocol" is defined multiple times for data stream "ssl", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ssl/fields/ecs.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ssl/fields/ecs.yml
  45. field "host.ip" is defined multiple times for data stream "ssl", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ssl/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ssl/fields/ecs.yml
  46. field "container.id" is defined multiple times for data stream "syslog", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/syslog/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/syslog/fields/beats.yml
  47. field "host.ip" is defined multiple times for data stream "syslog", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/syslog/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/syslog/fields/ecs.yml
  48. field "host.ip" is defined multiple times for data stream "traceroute", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/traceroute/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/traceroute/fields/ecs.yml
  49. field "container.id" is defined multiple times for data stream "traceroute", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/traceroute/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/traceroute/fields/beats.yml
  50. field "host.ip" is defined multiple times for data stream "intel", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/intel/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/intel/fields/ecs.yml
  51. field "container.id" is defined multiple times for data stream "intel", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/intel/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/intel/fields/beats.yml
  52. field "container.id" is defined multiple times for data stream "known_hosts", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_hosts/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_hosts/fields/beats.yml
  53. field "host.ip" is defined multiple times for data stream "known_hosts", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_hosts/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_hosts/fields/ecs.yml
  54. field "host.ip" is defined multiple times for data stream "ocsp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ocsp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ocsp/fields/ecs.yml
  55. field "container.id" is defined multiple times for data stream "ocsp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ocsp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ocsp/fields/beats.yml
  56. field "container.id" is defined multiple times for data stream "irc", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/irc/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/irc/fields/beats.yml
  57. field "host.ip" is defined multiple times for data stream "irc", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/irc/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/irc/fields/ecs.yml
  58. field "container.id" is defined multiple times for data stream "kerberos", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/kerberos/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/kerberos/fields/beats.yml
  59. field "host.ip" is defined multiple times for data stream "kerberos", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/kerberos/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/kerberos/fields/ecs.yml
  60. field "container.id" is defined multiple times for data stream "sip", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/sip/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/sip/fields/beats.yml
  61. field "host.ip" is defined multiple times for data stream "sip", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/sip/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/sip/fields/ecs.yml
  62. field "container.id" is defined multiple times for data stream "connection", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/connection/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/connection/fields/beats.yml
  63. field "host.ip" is defined multiple times for data stream "connection", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/connection/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/connection/fields/ecs.yml
  64. field "container.id" is defined multiple times for data stream "dpd", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dpd/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dpd/fields/beats.yml
  65. field "host.ip" is defined multiple times for data stream "dpd", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dpd/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dpd/fields/ecs.yml
  66. field "container.id" is defined multiple times for data stream "known_services", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_services/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_services/fields/beats.yml
  67. field "host.ip" is defined multiple times for data stream "known_services", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_services/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/known_services/fields/ecs.yml
  68. field "host.ip" is defined multiple times for data stream "pe", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/pe/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/pe/fields/ecs.yml
  69. field "container.id" is defined multiple times for data stream "pe", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/pe/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/pe/fields/beats.yml
  70. field "container.id" is defined multiple times for data stream "ssh", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ssh/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ssh/fields/beats.yml
  71. field "host.ip" is defined multiple times for data stream "ssh", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ssh/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/ssh/fields/ecs.yml
  72. field "host.ip" is defined multiple times for data stream "snmp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/snmp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/snmp/fields/ecs.yml
  73. field "container.id" is defined multiple times for data stream "snmp", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/snmp/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/snmp/fields/beats.yml
  74. field "container.id" is defined multiple times for data stream "software", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/software/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/software/fields/beats.yml
  75. field "host.ip" is defined multiple times for data stream "software", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/software/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/software/fields/ecs.yml
  76. field "container.id" is defined multiple times for data stream "stats", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/stats/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/stats/fields/beats.yml
  77. field "host.ip" is defined multiple times for data stream "stats", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/stats/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/stats/fields/ecs.yml
  78. field "host.ip" is defined multiple times for data stream "weird", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/weird/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/weird/fields/ecs.yml
  79. field "container.id" is defined multiple times for data stream "weird", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/weird/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/weird/fields/beats.yml
  80. field "host.ip" is defined multiple times for data stream "x509", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/x509/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/x509/fields/ecs.yml
  81. field "container.id" is defined multiple times for data stream "x509", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/x509/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/x509/fields/beats.yml
  82. field "container.id" is defined multiple times for data stream "capture_loss", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/capture_loss/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/capture_loss/fields/beats.yml
  83. field "host.ip" is defined multiple times for data stream "capture_loss", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/capture_loss/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/capture_loss/fields/ecs.yml
  84. field "host.ip" is defined multiple times for data stream "dce_rpc", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dce_rpc/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dce_rpc/fields/ecs.yml
  85. field "container.id" is defined multiple times for data stream "dce_rpc", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dce_rpc/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/dce_rpc/fields/beats.yml
  86. field "container.id" is defined multiple times for data stream "notice", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/notice/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/notice/fields/beats.yml
  87. field "host.ip" is defined multiple times for data stream "notice", found in: /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/notice/fields/agent.yml, /Users/xxxxxxxx/go/src/github.com/elastic/integrations/build/packages/zeek-2.6.0.zip/data_stream/notice/fields/ecs.yml

This would be made much clearer if the path were made relative to the data stream, file positions were emitted and the outputs were made to be consistent between runs (preferably grouped by file in sections in lexical order of datastream name then file name grouped and then by position in file). A sketch of how this might look is in the following detail.

Error: building package failed: invalid content found in built zip package: found 87 validation errors:
in data stream "capture_loss"
	"container.id" is defined multiple times
		capture_loss/fields/agent.yml:123:34
		capture_loss/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		capture_loss/fields/agent.yml:123:34
		capture_loss/fields/ecs.yml:123:34
in data stream "connection"
	"container.id" is defined multiple times
		connection/fields/agent.yml:123:34
		connection/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		connection/fields/agent.yml:123:34
		connection/fields/ecs.yml:123:34
in data stream "dce_rpc"
	"host.ip" is defined multiple times
		dce_rpc/fields/agent.yml:123:34
		dce_rpc/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		dce_rpc/fields/agent.yml:123:34
		dce_rpc/fields/beats.yml:123:34
in data stream "dhcp"
	"container.id" is defined multiple times
		dhcp/fields/agent.yml:123:34
		dhcp/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		dhcp/fields/agent.yml:123:34
		dhcp/fields/ecs.yml:123:34
in data stream "dnp3"
	"container.id" is defined multiple times
		dnp3/fields/agent.yml:123:34
		dnp3/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		dnp3/fields/agent.yml:123:34
		dnp3/fields/ecs.yml:123:34
in data stream "dns"
	"host.ip" is defined multiple times
		dns/fields/agent.yml:123:34
		dns/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		dns/fields/agent.yml:123:34
		dns/fields/beats.yml:123:34
in data stream "dpd"
	"container.id" is defined multiple times
		dpd/fields/agent.yml:123:34
		dpd/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		dpd/fields/agent.yml:123:34
		dpd/fields/ecs.yml:123:34
in data stream "files"
	"host.ip" is defined multiple times
		files/fields/agent.yml:123:34
		files/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		files/fields/agent.yml:123:34
		files/fields/beats.yml:123:34
in data stream "ftp"
	"host.ip" is defined multiple times
		ftp/fields/agent.yml:123:34
		ftp/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		ftp/fields/agent.yml:123:34
		ftp/fields/beats.yml:123:34
in data stream "http"
	"host.ip" is defined multiple times
		http/fields/agent.yml:123:34
		http/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		http/fields/agent.yml:123:34
		http/fields/beats.yml:123:34
in data stream "intel"
	"host.ip" is defined multiple times
		intel/fields/agent.yml:123:34
		intel/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		intel/fields/agent.yml:123:34
		intel/fields/beats.yml:123:34
in data stream "irc"
	"container.id" is defined multiple times
		irc/fields/agent.yml:123:34
		irc/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		irc/fields/agent.yml:123:34
		irc/fields/ecs.yml:123:34
in data stream "kerberos"
	"container.id" is defined multiple times
		kerberos/fields/agent.yml:123:34
		kerberos/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		kerberos/fields/agent.yml:123:34
		kerberos/fields/ecs.yml:123:34
in data stream "known_certs"
	"host.ip" is defined multiple times
		known_certs/fields/agent.yml:123:34
		known_certs/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		known_certs/fields/agent.yml:123:34
		known_certs/fields/beats.yml:123:34
in data stream "known_hosts"
	"container.id" is defined multiple times
		known_hosts/fields/agent.yml:123:34
		known_hosts/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		known_hosts/fields/agent.yml:123:34
		known_hosts/fields/ecs.yml:123:34
in data stream "known_services"
	"container.id" is defined multiple times
		known_services/fields/agent.yml:123:34
		known_services/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		known_services/fields/agent.yml:123:34
		known_services/fields/ecs.yml:123:34
in data stream "modbus"
	"host.ip" is defined multiple times
		modbus/fields/agent.yml:123:34
		modbus/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		modbus/fields/agent.yml:123:34
		modbus/fields/beats.yml:123:34
in data stream "mysql"
	"host.ip" is defined multiple times
		mysql/fields/agent.yml:123:34
		mysql/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		mysql/fields/agent.yml:123:34
		mysql/fields/beats.yml:123:34
in data stream "notice"
	"container.id" is defined multiple times
		notice/fields/agent.yml:123:34
		notice/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		notice/fields/agent.yml:123:34
		notice/fields/ecs.yml:123:34
in data stream "ntlm"
	"host.ip" is defined multiple times
		ntlm/fields/agent.yml:123:34
		ntlm/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		ntlm/fields/agent.yml:123:34
		ntlm/fields/beats.yml:123:34
in data stream "ntp"
	"host.ip" is defined multiple times
		ntp/fields/agent.yml:123:34
		ntp/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		ntp/fields/agent.yml:123:34
		ntp/fields/beats.yml:123:34
in data stream "ocsp"
	"host.ip" is defined multiple times
		ocsp/fields/agent.yml:123:34
		ocsp/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		ocsp/fields/agent.yml:123:34
		ocsp/fields/beats.yml:123:34
in data stream "pe"
	"host.ip" is defined multiple times
		pe/fields/agent.yml:123:34
		pe/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		pe/fields/agent.yml:123:34
		pe/fields/beats.yml:123:34
in data stream "radius"
	"host.ip" is defined multiple times
		radius/fields/agent.yml:123:34
		radius/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		radius/fields/agent.yml:123:34
		radius/fields/beats.yml:123:34
in data stream "rdp"
	"container.id" is defined multiple times
		rdp/fields/agent.yml:123:34
		rdp/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		rdp/fields/agent.yml:123:34
		rdp/fields/ecs.yml:123:34
in data stream "rfb"
	"host.ip" is defined multiple times
		rfb/fields/agent.yml:123:34
		rfb/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		rfb/fields/agent.yml:123:34
		rfb/fields/beats.yml:123:34
in data stream "signature"
	"host.ip" is defined multiple times
		signature/fields/agent.yml:123:34
		signature/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		signature/fields/agent.yml:123:34
		signature/fields/beats.yml:123:34
in data stream "sip"
	"container.id" is defined multiple times
		sip/fields/agent.yml:123:34
		sip/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		sip/fields/agent.yml:123:34
		sip/fields/ecs.yml:123:34
in data stream "smb_cmd"
	"container.id" is defined multiple times
		smb_cmd/fields/agent.yml:123:34
		smb_cmd/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		smb_cmd/fields/agent.yml:123:34
		smb_cmd/fields/ecs.yml:123:34
in data stream "smb_files"
	"container.id" is defined multiple times
		smb_files/fields/agent.yml:123:34
		smb_files/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		smb_files/fields/agent.yml:123:34
		smb_files/fields/ecs.yml:123:34
in data stream "smb_mapping"
	"host.ip" is defined multiple times
		smb_mapping/fields/agent.yml:123:34
		smb_mapping/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		smb_mapping/fields/agent.yml:123:34
		smb_mapping/fields/beats.yml:123:34
in data stream "smtp"
	"host.ip" is defined multiple times
		smtp/fields/agent.yml:123:34
		smtp/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		smtp/fields/agent.yml:123:34
		smtp/fields/beats.yml:123:34
in data stream "snmp"
	"host.ip" is defined multiple times
		snmp/fields/agent.yml:123:34
		snmp/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		snmp/fields/agent.yml:123:34
		snmp/fields/beats.yml:123:34
in data stream "socks"
	"host.ip" is defined multiple times
		socks/fields/agent.yml:123:34
		socks/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		socks/fields/agent.yml:123:34
		socks/fields/beats.yml:123:34
in data stream "software"
	"container.id" is defined multiple times
		software/fields/agent.yml:123:34
		software/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		software/fields/agent.yml:123:34
		software/fields/ecs.yml:123:34
in data stream "ssh"
	"container.id" is defined multiple times
		ssh/fields/agent.yml:123:34
		ssh/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		ssh/fields/agent.yml:123:34
		ssh/fields/ecs.yml:123:34
in data stream "ssl"
	"container.id" is defined multiple times
		ssl/fields/agent.yml:123:34
		ssl/fields/beats.yml:123:34
	"tls.version_protocol" is defined multiple times
		ssl/fields/ecs.yml:123:34
		ssl/fields/ecs.yml:123:34
	"host.ip" is defined multiple times
		ssl/fields/agent.yml:123:34
		ssl/fields/ecs.yml:123:34
in data stream "stats"
	"container.id" is defined multiple times
		stats/fields/agent.yml:123:34
		stats/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		stats/fields/agent.yml:123:34
		stats/fields/ecs.yml:123:34
in data stream "syslog"
	"container.id" is defined multiple times
		syslog/fields/agent.yml:123:34
		syslog/fields/beats.yml:123:34
	"host.ip" is defined multiple times
		syslog/fields/agent.yml:123:34
		syslog/fields/ecs.yml:123:34
in data stream "traceroute"
	"host.ip" is defined multiple times
		traceroute/fields/agent.yml:123:34
		traceroute/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		traceroute/fields/agent.yml:123:34
		traceroute/fields/beats.yml:123:34
in data stream "tunnel"
	"host.ip" is defined multiple times
		tunnel/fields/agent.yml:123:34
		tunnel/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		tunnel/fields/agent.yml:123:34
		tunnel/fields/beats.yml:123:34
in data stream "weird"
	"host.ip" is defined multiple times
		weird/fields/agent.yml:123:34
		weird/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		weird/fields/agent.yml:123:34
		weird/fields/beats.yml:123:34
in data stream "x509"
	"host.ip" is defined multiple times
		x509/fields/agent.yml:123:34
		x509/fields/ecs.yml:123:34
	"container.id" is defined multiple times
		x509/fields/agent.yml:123:34
		x509/fields/beats.yml:123:34

It is easier to read, follows the ordering of the files in an editor's file pane and the fields are presented in the order that you would find them in the files as they are being edited (not done in the example here but is straighforward to do mechanically).

@jlind23 jlind23 added the Team:Ecosystem Label for the Packages Ecosystem team label Sep 29, 2022
@efd6
Copy link
Contributor Author

efd6 commented Sep 29, 2022

Having slept on this, I think the paths to files should be relative to the package since that is where you are likely to be and so would be able to $EDITOR path/to/errored/file:line:column and end up in the right place.

@pzl
Copy link
Member

pzl commented Oct 10, 2023

To add a +1 to the problem of identifying which field is the offending one (it's AST position is printed, instead of trying to parse a possibly valid name), I am sorting through a large list of errors (200+) and my output looks like this:

   9. file "/home/dan/dev/elastic/endpoint-package/package/endpoint/data_stream/alerts/fields/fields.yml" is invalid: field 3.fields.0: object_type is required
  10. file "/home/dan/dev/elastic/endpoint-package/package/endpoint/data_stream/alerts/fields/fields.yml" is invalid: field 3.fields.1: object_type is required
  11. file "/home/dan/dev/elastic/endpoint-package/package/endpoint/data_stream/alerts/fields/fields.yml" is invalid: field 3.fields.2: object_type is required
  12. file "/home/dan/dev/elastic/endpoint-package/package/endpoint/data_stream/alerts/fields/fields.yml" is invalid: field 3.fields.3: object_type is required
  13. file "/home/dan/dev/elastic/endpoint-package/package/endpoint/data_stream/alerts/fields/fields.yml" is invalid: field 3.fields.8: object_type is required
  14. file "/home/dan/dev/elastic/endpoint-package/package/endpoint/data_stream/alerts/fields/fields.yml" is invalid: field 6.fields.9: object_type is required
  15. file "/home/dan/dev/elastic/endpoint-package/package/endpoint/data_stream/alerts/fields/fields.yml" is invalid: field 7.fields.0: object_type is required
  16. file "/home/dan/dev/elastic/endpoint-package/package/endpoint/data_stream/alerts/fields/fields.yml" is invalid: field 7.fields.8: object_type is required
  17. file "/home/dan/dev/elastic/endpoint-package/package/endpoint/data_stream/alerts/fields/fields.yml" is invalid: field 7.fields.38: object_type is required
  18. file "/home/dan/dev/elastic/endpoint-package/package/endpoint/data_stream/alerts/fields/fields.yml" is invalid: field 7.fields.48: object_type is required
  19. file "/home/dan/dev/elastic/endpoint-package/package/endpoint/data_stream/alerts/fields/fields.yml" is invalid: field 7.fields.77: object_type is required

sorting through this part repeatedly is challenging to identify what to update:

invalid: field 3.fields.0: object_type is
invalid: field 3.fields.1: object_type is
invalid: field 3.fields.2: object_type is
invalid: field 3.fields.3: object_type is
invalid: field 3.fields.8: object_type is
invalid: field 6.fields.9: object_type is
invalid: field 7.fields.0: object_type is
invalid: field 7.fields.8: object_type is
invalid: field 7.fields.38: object_type is
invalid: field 7.fields.48: object_type is
invalid: field 7.fields.77: object_type is

@efd6
Copy link
Contributor Author

efd6 commented Mar 18, 2024

Another example of where this could be improved is in the context for syntax failures. Recently I was working on a package and due to a edit error, I was told

2024/03/15 09:06:48  WARN CommitHash is undefined, in both /home/daniel/.elastic-package/version and the compiled binary, config may be out of date.
Build the package
README.md file rendered: /home/daniel/src/github.com/elastic/integrations/packages/qualys_vmdr/docs/README.md
2024/03/15 09:06:48  INFO License text found in "/home/daniel/src/github.com/elastic/integrations/LICENSE.txt" will be included in package
2024/03/15 09:06:48  INFO Import ECS mappings into the built package (technical preview)
Error: building package failed: adding dynamic mappings: yaml: line 34: did not find expected key

This doesn't tell me which file is problematic (there are two manifest files and three fields definition files that could arguably be considered to be the object of the error) and does not show a context.

Looking at the files I finally found the error (a single space addition, which was why it was hard to find).

01: title: Collect Asset Host Detection data from Qualys VMDR platform.
02: type: logs
03: streams:
04:   - input: cel
05:     title: Asset Host Detection data
06:     description: Collect Asset Host Detection data from Qualys VMDR platform.
07:     template_path: input.yml.hbs
08:     vars:
09:       - name: url
10:         type: text
11:         title: URL
12:         description: Base URL of the Qualys Server API.
13:         required: true
14:         show_user: true
15:       - name: interval
16:         type: text
17:         title: Interval
18:         description: >-
19:           Interval between two REST API calls. User can choose interval as per their plan mentioned in
20:           [Qualys API Limits](https://www.qualys.com/docs/qualys-api-limits.pdf). Supported units for this parameter are h/m/s.
21:         default: 4h
22:         multi: false
23:         required: true
24:         show_user: true
25:       - name: batch_size
26:         type: integer
27:         title: Batch Size
28:         description: >-
29:           Batch size for the response of the Qualys Server API. This parameter specifies the truncation limit for the response.
30:           Specify 0 for no truncation limit.
31:         default: 1000
32:         multi: false
33:         required: true
34:         show_user: false
35:       - name: http_client_timeout
36:         type: text
37:         title: HTTP Client Timeout
38:         description: >-
39:           Duration before declaring that the HTTP client connection has timed out. Give a timeout of more than 1 minute when
40:           retrieving data which is more than 15 days old. Supported time units are ns, us, ms, s, m, h. Requests may take
41:           significant time, so short timeouts are not recommended.
42:          multi: false ← HERE
43:         required: true
44:         show_user: false
45:         default: 10m
<snip>

Note that the error message points to line 34, but the actual error is on line 42. The disparity appears to be due to the error pointing to the last line of the last valid list element.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:Ecosystem Label for the Packages Ecosystem team
Projects
None yet
Development

No branches or pull requests

3 participants