Directory layout and data path

[tsg]

This a meta issue for the tasks required to support a "data path" in Beats.

The end goal is to have:

• A clear location for the Filebeat/Winlogbeat registry files. This fixes the issue where the registry file was created in the home directory of the user.
• Make it possible to load the template when running as a service in Windows. Fixes issue: Loading the template by default doesn't work on Windows when running as a serivce #1315
• Provide a path for a persistent UUID, which we will need when implementing configuration management.
• Have a clear location for storing the Kibana dashboards

The following configurable "paths" are introduced:

	Description	Default
path.home	It’s the default folder for everything that doesn't fit in the categories before	{binary directory}
path.data	Contains things that are expected to change often during normal operations (“registry” files, UUID file, etc.)	{path.home}/data
path.config	Configuration file location	{path.home}
path.logs	Default path for log files.	{path.home}/logs

This is modeled after Elasticsearch directory layout with some differences:

• Not all path types are present. We might introduce more as we feel the need.
• path.config has the same default as path.home. This is because in beats we don't have a config/ folder, and introducing one would be difficult because we have a config package.

Directory layouts

Illustrating these with Filebeat.

ZIP / TAR.GZ / TGZ

When running directly under the tarball archives, everything is relative to the binary location. This also matches the organization of the source code repository, so things should work the same when running from a source checkout.

    filebeat      
    filebeat.yml
    filebeat.template.json
    data/
       registry.json
       filebeat.uuid
    kibana/
      load.sh
      searches/
      visualisations/
      dashboards/
      index-patterns/

To run, you typically just start with: ./filebeat

DEB/RPM

Debian/Centos packages have their home path set to /usr/share/filebeat and their data path set to /var/lib/filebeat, to match current installations.

/usr/share/filebeat/
   bin/
     filebeat-god
     filebeat
   kibana/
        load.sh
        searches/
        visualisations/
        dashboards/
        index-patterns/
/etc/filebeat/
    filebeat.yml
    filebeat.template.json
/var/lib/filebeat/
    registry.json
    filebeat.uuid

The init scripts, start filebeat with:

   /usr/share/filebeat/filebeat --path.data /var/lib/filebeat
                                             --path.config /etc/filebeat/

The downside of moving the binary out of /usr/bin is that it gets a bit more complex to run in foreground from an installed system. This is because the binary is not in the system PATH anymore and because the data path and config paths need to be set correctly. We could mitigate this by providing a shell script in /usr/bin.

Windows

On Windows there are two options. Either run directly from the unzipped directory, in which case everything works as for the tar/zip case, or you install as a service. If you install as a service, the data path is set to c:\ProgramData\filebeat, so the we have the following structure:

{extractpath}/
   filebeat
   filebeat.template.json
   filebeat.yml
   dashboards/
        load.sh
        searches/
        visualisations/
c:\programdata\filebeat
   registry.json
   filebeat.uuid

The service will start filebeat with:

  $extractpath/filebeat -path.data c:\programdata\filebeat

This is not ideal yet, but matches pretty much what we have so far, fixes the automatic template loading and can be improved when we get a real Windows installer.

Required changes

• Add CLI option -path.home. Added home, data, config paths #1373
• Add CLI option -path.data. Added home, data, config paths #1373
• Add CLI option -path.config Added home, data, config paths #1373
• Add CLI option -path.logs (lower prio). Added path.logs and made logging to files the default #1437
• Change the default of filebeat registry location from .filebeat to {data_path}/registry Added home, data, config paths #1373
• Provide a libbeat function to open files from one of the location above ("data", "config", "log", or by default from "home") Added home, data, config paths #1373
• Use the config path to look for the template. Added home, data, config paths #1373
• Change winlogbeat to have use the data path for it's registry file. Look for the .winlogbeat.yml in the data path #1386
• Update DEB/RPM scripts to deploy files under /usr/share/{beatname}. Update packaging #1385
• Update packer to install the dashboards unde the kibana folder for all platforms. Install the etc/kibana dir in the home directory #1399
• Update init scripts and unit files to use the new CLI flags. Update packaging #1385
• Document new CLI flags and configuration options. Added documentation for the new directory layout #1441
• Add namebeat.sh scripts to /usr/bin to make it easy to run beats in foreground. Added beatname.sh in /usr/bin #1446
• Make the confidir option from Filebeat yml file, relative to the config path. If confdir is relative, make it relative to the config path #1435

[ruflin]

I like the proposal. You state path.config has the same default as path.data, this should be path.home, right?

Could it make sense to add these also as configuration option to the config file? This would allow to run beat -c path/config.yml and the other paths would be read from the config file.

For the data directory, should we add it to the git repository but ignore all files inside?

[tsg]

I like the proposal. You state path.config has the same default as path.data, this should be path.home, right?

Yep, fixed :-).

Could it make sense to add these also as configuration option to the config file? This would allow to run beat -c path/config.yml and the other paths would be read from the config file.

Yeah, they are configuration options, see #1373. The plan is that after elastic/go-ucfg#15 is implemented, they will only be configuration file options, which can be overwritten from the command line.

For the data directory, should we add it to the git repository but ignore all files inside?

I'll add /data/ to .gitignore. The beats are creating the path automatically if it doesn't exist.