When contributing to Beats development, you may want to add new dashboards or
customize the existing ones. To get started, you can
import the Kibana dashboards that come with the official
Beats and use them as a starting point for your own dashboards. When you’re done
making changes to the dashboards in Kibana, you can use the export_dashboards
script to export the dashboards, along with all
dependencies, to a local directory.
To make sure the dashboards are compatible with the latest version of Kibana and Elasticsearch, we recommend that you use the virtual environment under beats/testing/environments to import, create, and export the Kibana dashboards.
The following topics provide more detail about importing and working with Beats dashboards:
The official Beats come with Kibana dashboards, and starting with 6.0.0, they are part of every Beat package.
You can use the Beat executable to import all the dashboards and the index pattern for a Beat, including the dependencies such as visualizations and searches.
To import the dashboards, run the setup command.
./metricbeat setupThe setup phase loads several dependencies, such as:
-
Index mapping template in Elasticsearch
-
Kibana dashboards
-
Ingest pipelines
-
ILM policy
The dependencies vary depending on the Beat you’re setting up.
For more details about the setup command, see the command-line help. For example:
./metricbeat help setup
This command does initial setup of the environment:
* Index mapping template in Elasticsearch to ensure fields are mapped.
* Kibana dashboards (where available).
* ML jobs (where available).
* Ingest pipelines (where available).
* ILM policy (for Elasticsearch 6.5 and newer).
Usage:
metricbeat setup [flags]
Flags:
--dashboards Setup dashboards
-h, --help help for setup
--index-management Setup all components related to Elasticsearch index management, including template, ilm policy and rollover alias
--pipelines Setup Ingest pipelinesThe flags are useful when you don’t want to load everything. For example, to
import only the dashboards, use the --dashboards flag:
./metricbeat setup --dashboardsStarting with Beats 6.0.0, the dashboards are no longer loaded directly into Elasticsearch. Instead, they are imported directly into Kibana.
Thus, if your Kibana instance is not listening on localhost, or you enabled
{xpack} for Kibana, you need to either configure the Kibana endpoint in
the config for the Beat, or pass the Kibana host and credentials as
arguments to the setup command. For example:
./metricbeat setup -E setup.kibana.host=192.168.3.206:5601 -E setup.kibana.username=elastic -E setup.kibana.password=secretBy default, the setup command imports the dashboards from the kibana
directory, which is available in the Beat package.
|
Note
|
The format of the saved dashboards is not compatible between Kibana 5.x and 6.x. Thus, the Kibana 5.x dashboards are available in
the 5.x directory, and the Kibana 6.0 dashboards, and older are in the default directory.
|
In case you are using customized dashboards, you can import them:
-
from a local directory:
./metricbeat setup -E setup.dashboards.directory=kibana
-
from a local zip archive:
./metricbeat setup -E setup.dashboards.file=metricbeat-dashboards-6.0.zip
-
from a zip archive available online:
./metricbeat setup -E setup.dashboards.url=path/to/url
See Kibana dashboards configuration for a description of the
setup.dashboardsconfiguration options.
You can make use of the Magefile from the Beat GitHub repository to import the dashboards. If Kibana is running on localhost, then you can run the following command from the root of the Beat:
mage dashboardsThe configuration file (*.reference.yml) of each Beat contains the setup.dashboards section for configuring from where to get the Kibana dashboards, as well as the name of the index pattern.
Each of these configuration options can be overwritten with the command line options by using -E flag.
setup.dashboards.directory=<local_dir>-
Local directory that contains the saved dashboards and their dependencies. The default value is the
kibanadirectory available in the Beat package. setup.dashboards.file=<local_archive>-
Local zip archive with the dashboards. The archive can contain Kibana dashboards for a single Beat or for multiple Beats. The dashboards of each Beat are placed under a separate directory with the name of the Beat.
setup.dashboards.url=<zip_url>-
Zip archive with the dashboards, available online. The archive can contain Kibana dashboards for a single Beat or for multiple Beats. The dashboards for each Beat are placed under a separate directory with the name of the Beat.
setup.dashboards.index <elasticsearch_index>-
You should only use this option if you want to change the index pattern name that’s used by default. For example, if the default is
metricbeat-, you can change it tocustombeat-.
|
Note
|
If you want to modify a dashboard that comes with a Beat, it’s better to modify a copy of the dashboard because the Beat overwrites the dashboards during the setup phase in order to have the latest version. For duplicating a dashboard, just use the Clone button from the top of the page.
|
Before building your own dashboards or customizing the existing ones, you need to load:
-
the Beat index pattern, which specifies how Kibana should display the Beat fields
-
the Beat dashboards that you want to customize
For the Elastic Beats, the index pattern is available in the Beat package under
kibana/*/index-pattern. The index-pattern is automatically generated from the fields.yml file, available in the Beat package. For more details
check the generate index pattern section.
All Beats dashboards, visualizations and saved searches must follow common naming conventions:
-
Dashboard names have prefix
[BeatName Module], e.g.[Filebeat Nginx] Access logs -
Visualizations and searches have suffix
[BeatName Module], e.g.Top processes [Filebeat Nginx]
|
Note
|
You can set a custom name (skip suffix) for visualization placed on a dashboard. The original visualization will stay intact. |
The naming convention rules can be verified with the the tool mage check. The command fails if it detects:
-
empty description on a dashboard
-
unexpected dashboard title format (missing prefix
[BeatName ModuleName]) -
unexpected visualization title format (missing suffix
[BeatName Module])
After creating your own dashboards in Kibana, you can export the Kibana dashboards to a local directory, and then archive the dashboards in order to be able to share the dashboards with the community.
The index-pattern defines the format of each field, and it’s used by Kibana to know how to display the field.
If you change the fields exported by the Beat, you need to generate a new index pattern for your Beat. Otherwise, you can just use the index pattern available under the kibana/*/index-pattern directory.
The Beat index pattern is generated from the fields.yml, which contains all
the fields exported by the Beat. For each field, besides the type, you can configure the
format field. The format informs Kibana about how to display a certain field. A good example is percentage or bytes
to display fields as 50% or 5MB.
To generate the index pattern from the fields.yml, you need to run the following command in the Beat repository:
make updateTo export all the dashboards for any Elastic Beat or any community Beat, including any new or modified dashboards and all dependencies such as
visualizations, searches, you can use the Go script export_dashboards.go from
dev-tools.
See the dev-tools readme for more info.
Alternatively, if the scripts above are not available, you can use your Beat binary to export Kibana 6.0 dashboards or later.
The dev-tools/cmd/export_dashboards.go script helps you export your customized Kibana dashboards until the v7.14.x release.
You might need to export a single dashboard or all the dashboards available for a module or Beat.
It is also possible to use a Beat binary to export.
From 7.15, your Beats version must be the same as your Kibana version to make sure the export API required is available.
After you updated your Kibana instance to at least 7.15, you have to
export your dashboards again with either export_dashboards.go tool or
with your Beat.
To export a single dashboard for a module you can use the following command inside a Beat with modules:
MODULE=redis ID=AV4REOpp5NkDleZmzKkE mage exportDashboard./filebeat export dashboard -id 7fea2930-478e-11e7-b1f0-cb29bac6bf8b -folder module/redisThis generates an appropriate folder under module/redis for the dashboard, separating assets into dashboards, searches, vizualizations, etc. Each exported file is a JSON and their names are the IDs of the assets.
|
Note
|
The dashboard ID is available in the dashboard URL. For example, in case the dashboard URL is
app/kibana#/dashboard/AV4REOpp5NkDleZmzKkE?_g=()&_a=(description:'Overview%2…, the dashboard ID is AV4REOpp5NkDleZmzKkE.
|
Each module should contain a module.yml file with a list of all the dashboards available for the module. For the Beats that don’t have support for modules (e.g. Packetbeat),
there is a dashboards.yml file that defines all the Packetbeat dashboards.
Below, it’s an example of the module.yml file for the system module in Metricbeat:
dashboards:
- id: Metricbeat-system-overview
file: Metricbeat-system-overview.ndjson
- id: 79ffd6e0-faa0-11e6-947f-177f697178b8
file: Metricbeat-host-overview.ndjson
- id: CPU-slash-Memory-per-container
file: Metricbeat-containers-overview.ndjsonEach dashboard is defined by an id and the name of ndjson file where the dashboard is saved locally.
By passing the yml file to the export_dashboards.go script or to the Beat, you can export all the dashboards defined:
go run dev-tools/cmd/dashboards/export_dashboards.go -yml filebeat/module/system/module.yml -folder dashboards./filebeat export dashboard -yml filebeat/module/system/module.ymlIf you are using the Kibana Spaces feature and want to export dashboards from a specific Space, pass the Space ID to the export_dashboards.go script:
go run dev-tools/cmd/dashboards/export_dashboards.go -space-id my-space [other-options]In case of running export dashboard of a Beat, you need to set the Space ID in setup.kibana.space.id.
To export only some Kibana dashboards for an Elastic Beat or community Beat, you can simply pass a regular expression to
the export_dashboards.py script to match the selected Kibana dashboards.
Before running the export_dashboards.py script for the first time, you
need to create an environment that contains all the required Python packages.
make python-envFor example, to export all Kibana dashboards that start with the Packetbeat name:
python ../dev-tools/cmd/dashboards/export_dashboards.py --regex Packetbeat*To see all the available options, read the descriptions below or run:
python ../dev-tools/cmd/dashboards/export_dashboards.py -h--url <elasticsearch_url>-
The Elasticsearch URL. The default value is http://localhost:9200.
--regex <regular_expression>-
Regular expression to match all the Kibana dashboards to be exported. This argument is required.
--kibana <kibana_index>-
The Elasticsearch index pattern where Kibana saves its configuration. The default value is
.kibana. --dir <output_dir>-
The output directory where the dashboards and all dependencies will be saved. The default value is
output.
The output directory has the following structure:
output/
index-pattern/
dashboard/
visualization/
search/The Kibana dashboards for the Elastic Beats are saved under the kibana directory. To create a zip archive with the
dashboards, including visualizations and searches and the index pattern, you can run the following command in the Beat
repository:
make package-dashboardsThe Makefile is part of libbeat, which means that community Beats contributors can use the commands shown here to
archive dashboards. The dashboards must be available under the kibana directory.
Another option would be to create a repository only with the dashboards, and use the GitHub release functionality to create a zip archive.
Share the Kibana dashboards archive with the community, so other users can use your cool Kibana visualizations!
When you’re done with your own Beat dashboards, how about letting everyone know? You can create a topic on the Beats forum, and provide the link to the zip archive together with a short description.