Metricbeat reinstalls dashboards on every restart. Overwrites user changes to dashboards

[LeeDr]

• Version: 7.17.0 (but issue started in 7.16)
• Operating System: CentOS 7

I'm labeling this a bug because I don't think it was an intentional change back in 7.16 to always load the dashboards when metricbeat starts. I haven't checked this yet on a tar.gz install, only the rpm package install. I'm filing this on metricbeat but I think all beats are doing this.

• Steps to Reproduce:

1. Install metricbeat 7.17.0-SNAPSHOT from rpm
2. see config pasted below;
3. start metricbeat service
4. see in logs that metricbeat is loading all the saved objects with overwrite
5. modify a metricbeat dashboard
6. restart metricbeat service
7. watch metricbeat log shows its loading dashboards again
8. reopen the dashboard and see your changes are gone

I verified metricbeat 7.15 doesn't load saved objects unless you call the setup step. But 7.16 and 7.17 do.

metricbeat.yml:

logging.level: debug
logging.to_files: true

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.period: 10s
  reload.enabled: false
metricbeat.max_start_delay: 10s
metricbeat.modules:
- module: system
  enabled: true
  period: 10s
  metricsets:
    - cpu
    - load
    - memory
    - network
    - process
    - process_summary
  processes: ['.*']
- module: system
  enabled: true
  period: 1m
  metricsets:
    - filesystem
    - fsstat
- module: kibana
  metricsets: ["status"]
  enabled: false
  period: 10s
  hosts: ["localhost:5601"]
- module: elasticsearch
  metricsets: ["node", "node_stats"]
  enabled: false
  period: 10s
  hosts: ["localhost:9200"]


output.elasticsearch:
  hosts: ["localhost:9200"]
  protocol: https
  username: "beats_internal"
  password: "changeme"
  ssl.enabled: true
  ssl.certificate_authorities: ["/etc/metricbeat/ca.crt"]
  ssl.certificate: /etc/metricbeat/elasticsearch.crt
  ssl.key: /etc/metricbeat/elasticsearch.key

setup.dashboards.snapshot: true

setup.kibana:
  protocol: https
  host: localhost:5601
  username: "beats_internal"
  password: "changeme"
  ssl.enabled: true
  ssl.certificate_authorities: ["/etc/metricbeat/ca.crt"]
  ssl.certificate: /etc/metricbeat/elasticsearch.crt
  ssl.key: /etc/metricbeat/elasticsearch.key

xpack.monitoring:
  enabled: false
  elasticsearch:
    url: "https://localhost:9200"
    username: elastic
    password: changeit
http.enabled: true
http.port: 5068

I'll attached the debug log which shows the dashboards loading.

[LeeDr]

metricbeat.log

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[kvch]

It is not a change/bug in Beats behaviour. If you configure any options under setup.dashboards.*, Beats will automatically enable dashboard loading without the user explicitly setting setup.dashboards.enabled. This option group behaves similarly to option groups that have an enabled setting, to e.g. ssl.* or kerberos.*. Given that you added setup.dashboard.snapshot to the configuration, Metricbeat loads the dashboards on every run.

This might not be straightforward, but this is how Beats work. The problem is rather that we missing configuration for this option. For SSL settings we have a note that clearly lets users know about this behaviour: https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-ssl.html#enabled We should add the same clarifications instead of breaking existing behaviour.

[joshdover]

I verified metricbeat 7.15 doesn't load saved objects unless you call the setup step. But 7.16 and 7.17 do.

I'm curious about this, is this true that this behavior wasn't present prior to 7.16?