Skip to content

Commit b3c5397

Browse files
ruflinjsoriano
ruflin
authored and
jsoriano
committed
Unify Kibana dashboard directory logic across Beats (#7265)
Currently all dashboards end up under `_meta/kibana` in each Beat. The problem with this directory is that for some Beats it contains the automatically collected and generated files and for others the original. This prevents us from having decoded JSON files for all dashboards across Beats. An other benefit of having the `kibana.generated` directory is that now `_meta/kibana` can become the place for dashboards across modules in one Beat without having to worry about overwriting / removing these dashboards. Community Beats must convert their dashboards with the script under #7224. Further changes: * Decode Packetbeat and Winlogbeat dashboards * Move encoding step to libbeat so it is applied automatically to all dashboards in the kibana.generated directory. * Remove _meta/kibana from .gitignore as it can now be used to contain dashboards in all Beats. * Adjust Makefiles for the new logic. * Adjust tests to depend on kibana.generated directory
1 parent ee0dc3e commit b3c5397

29 files changed

+9990
-2538
lines changed

‎CHANGELOG-developer.asciidoc

+1
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ The list below covers the major changes between 6.3.0 and master only.
2222
- The beat.Pipeline is now passed to cfgfile.RunnerFactory. Beats using libbeat for module reloading or autodiscovery need to be adapted. {pull}7018[7017]
2323
- Moving of TLS helper functions and structs from `output/tls` to `tlscommon`. {pull}7054[7054]
2424
- Port fields.yml collector to Golang {pull}6911[6911]
25+
- Dashboards under _meta/kibana are expected to be decoded. See https://github.com/elastic/beats/pull/7224 for a conversion script. {pull}7265[7265]
2526

2627
==== Bugfixes
2728

‎auditbeat/.gitignore

-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
build
2-
_meta/kibana
32
_meta/beat.yml
43
_meta/beat.reference.yml
54
module/*/_meta/config.yml

‎auditbeat/Makefile

+3-5
Original file line numberDiff line numberDiff line change
@@ -79,8 +79,6 @@ collect-docs: python-env
7979
# Collects all module dashboards
8080
.PHONY: kibana
8181
kibana:
82-
@-rm -rf _meta/kibana
83-
@mkdir -p _meta/kibana
84-
@-cp -pr module/*/_meta/kibana _meta/
85-
@# Convert all dashboards to string
86-
@python ${ES_BEATS}/libbeat/scripts/unpack_dashboards.py --glob="./_meta/kibana/6/dashboard/*.json"
82+
@-rm -rf _meta/kibana.generated
83+
@mkdir -p _meta/kibana.generated
84+
@-cp -pr module/*/_meta/kibana/* _meta/kibana.generated

‎auditbeat/tests/system/test_base.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ def test_dashboards(self):
6363

6464
dirs = [self.temp_dir("auditbeat_test")]
6565
with PathCleanup(dirs):
66-
kibana_dir = os.path.join(self.beat_path, "_meta", "kibana")
66+
kibana_dir = os.path.join(self.beat_path, "_meta", "kibana.generated")
6767
shutil.copytree(kibana_dir, os.path.join(self.working_dir, "kibana"))
6868

6969
es = Elasticsearch([self.get_elasticsearch_url()])

‎filebeat/.gitignore

-1
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,6 @@
55
filebeat
66

77
build
8-
_meta/kibana
98
_meta/module.generated
109
_meta/beat.yml
1110
_meta/beat.reference.yml

‎filebeat/Dockerfile

+1-1
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ MAINTAINER Nicolas Ruflin <ruflin@elastic.co>
44
RUN set -x && \
55
apt-get update && \
66
apt-get install -y --no-install-recommends \
7-
netcat python-pip virtualenv && \
7+
netcat python-pip rsync virtualenv && \
88
apt-get clean
99

1010
RUN pip install --upgrade setuptools

‎filebeat/Makefile

+3-5
Original file line numberDiff line numberDiff line change
@@ -17,11 +17,9 @@ before-build:
1717
# Collects all module dashboards
1818
.PHONY: kibana
1919
kibana:
20-
@rm -rf _meta/kibana
21-
@mkdir -p _meta/kibana
22-
@-cp -r module/*/_meta/kibana _meta/
23-
@# Convert all dashboards to string
24-
@python ${ES_BEATS}/libbeat/scripts/unpack_dashboards.py --glob="./_meta/kibana/6/dashboard/*.json"
20+
@rm -rf _meta/kibana.generated
21+
@mkdir -p _meta/kibana.generated
22+
@-cp -pr module/*/_meta/kibana/* _meta/kibana.generated
2523

2624
# Collects all modules files to be packaged in a temporary folder
2725
.PHONY: modules

‎filebeat/tests/system/test_modules.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ def init(self):
6363
"/../../../../module")
6464

6565
self.kibana_path = os.path.abspath(self.working_dir +
66-
"/../../../../_meta/kibana")
66+
"/../../../../_meta/kibana.generated")
6767

6868
self.filebeat = os.path.abspath(self.working_dir +
6969
"/../../../../filebeat.test")

‎heartbeat/.gitignore

-2
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,5 @@
44

55
heartbeat
66

7-
_meta/kibana
8-
97
build
108
/tests/load/logs

‎heartbeat/Makefile

+3-5
Original file line numberDiff line numberDiff line change
@@ -31,8 +31,6 @@ create-monitor:
3131
# Collects all module dashboards
3232
.PHONY: kibana
3333
kibana:
34-
@rm -rf _meta/kibana
35-
@mkdir -p _meta/kibana
36-
@-cp -r monitors/active/*/_meta/kibana _meta/
37-
@# Convert all dashboards to string
38-
@python ${ES_BEATS}/libbeat/scripts/unpack_dashboards.py --glob="./_meta/kibana/6/dashboard/*.json"
34+
@rm -rf _meta/kibana.generated
35+
@mkdir -p _meta/kibana.generated
36+
@-cp -r monitors/active/*/_meta/kibana/* _meta/kibana.generated

‎libbeat/scripts/Makefile

+4-1
Original file line numberDiff line numberDiff line change
@@ -342,10 +342,13 @@ endif
342342
@mkdir -p docs
343343
@${PYTHON_ENV}/bin/python ${ES_BEATS}/libbeat/scripts/generate_fields_docs.py $(PWD) ${BEAT_TITLE} ${ES_BEATS}
344344

345+
@mkdir -p $(PWD)/_meta/kibana.generated
345346
@# Generate Kibana index pattern and copy dashboard files
346347
if [ -d $(PWD)/_meta/kibana ]; then \
347-
cp -r $(PWD)/_meta/kibana/ $(PWD)/_meta/kibana.generated; \
348+
cp -pr $(PWD)/_meta/kibana/* $(PWD)/_meta/kibana.generated ; \
348349
fi
350+
@# Convert all dashboards to string
351+
@python ${ES_BEATS}/libbeat/scripts/unpack_dashboards.py --glob="./_meta/kibana.generated/6/dashboard/*.json"
349352
@mkdir -p $(PWD)/_meta/kibana.generated/5/index-pattern
350353
@mkdir -p $(PWD)/_meta/kibana.generated/6/index-pattern
351354
@go run ${ES_BEATS}/dev-tools/cmd/kibana_index_pattern/kibana_index_pattern.go -index '${BEAT_INDEX_PREFIX}-*' -beat-name ${BEAT_NAME} -beat-dir $(PWD) -version ${BEAT_VERSION}

‎metricbeat/.gitignore

-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
build
2-
_meta/kibana
32
_meta/beat.yml
43
_meta/beat.reference.yml
54

‎metricbeat/Makefile

+3-6
Original file line numberDiff line numberDiff line change
@@ -18,12 +18,9 @@ include ${ES_BEATS}/libbeat/scripts/Makefile
1818
# Collects all module dashboards
1919
.PHONY: kibana
2020
kibana:
21-
@rm -rf _meta/kibana
22-
@mkdir -p _meta/kibana
23-
@-cp -r module/*/_meta/kibana _meta/
24-
@-cp -r module/*/_meta/kibana _meta/
25-
@# Convert all dashboards to string
26-
@python ${ES_BEATS}/libbeat/scripts/unpack_dashboards.py --glob="./_meta/kibana/6/dashboard/*.json"
21+
@rm -rf _meta/kibana.generated
22+
@mkdir -p _meta/kibana.generated
23+
@-cp -pr module/*/_meta/kibana/* _meta/kibana.generated
2724

2825
# Collects all module docs
2926
.PHONY: collect-docs

‎metricbeat/tests/system/test_base.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ def test_dashboards(self):
5959
Test that the dashboards can be loaded with `setup --dashboards`
6060
"""
6161

62-
kibana_dir = os.path.join(self.beat_path, "_meta", "kibana")
62+
kibana_dir = os.path.join(self.beat_path, "_meta", "kibana.generated")
6363
shutil.copytree(kibana_dir, os.path.join(self.working_dir, "kibana"))
6464

6565
es = Elasticsearch([self.get_elasticsearch_url()])

‎packetbeat/.gitignore

-1
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,6 @@
1010
_obj
1111
_test
1212
_output
13-
_meta/kibana/*/index-pattern
1413

1514
# Architecture specific extensions/prefixes
1615
*.[568vq]

‎packetbeat/_meta/kibana/6/dashboard/Packetbeat-cassandra.json

+938-196
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)