Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Load configuration from modules.d when running setup subcommand #12340

Merged
merged 8 commits into from
Jun 4, 2019

Conversation

kvch
Copy link
Contributor

@kvch kvch commented May 29, 2019

Previously, when running the subcommand setup, configuration from modules.d was not loaded. This lead to Filebeat ignoring settings of the module system and loading incorrect pipelines.

Now when running setup configuration from the modules folder are read and pipelines are loaded.

TODO

  • more testing to check corner cases
  • add automated test
  • add changelog entry

Closes #9747

@kvch kvch requested a review from a team as a code owner May 29, 2019 11:57
@kvch kvch added Filebeat Filebeat review labels May 30, 2019
@kvch kvch force-pushed the fix-filebeat-setup-modules.d branch 2 times, most recently from 4a536de to 6d22781 Compare May 31, 2019 12:23
@kvch kvch force-pushed the fix-filebeat-setup-modules.d branch from 6d22781 to 03a3388 Compare May 31, 2019 15:16
@kvch kvch requested review from ruflin, faec and jsoriano and removed request for faec May 31, 2019 16:59
Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this! We have been removing the convert_timezones conditions from the pipeline templates (#12253), but I guess we should continue supporting templates in pipelines for other cases by now.

Co-Authored-By: Jaime Soriano Pastor <jaime.soriano@elastic.co>
@codecov
Copy link

codecov bot commented Jun 4, 2019

Codecov Report

Merging #12340 into master will decrease coverage by 0.22%.
The diff coverage is 0%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #12340      +/-   ##
==========================================
- Coverage   58.02%    57.8%   -0.23%     
==========================================
  Files         933      934       +1     
  Lines       57454    57485      +31     
==========================================
- Hits        33340    33231     -109     
- Misses      21050    21192     +142     
+ Partials     3064     3062       -2
Impacted Files Coverage Δ
filebeat/fileset/setup.go 0% <0%> (ø)
libbeat/cfgfile/reload.go 0% <0%> (ø) ⬆️
filebeat/beater/filebeat.go 0% <0%> (ø) ⬆️
filebeat/fileset/pipelines.go 64.1% <0%> (-17.1%) ⬇️
libbeat/publisher/pipeline/acker.go 0% <0%> (-14.37%) ⬇️
filebeat/fileset/modules.go 36.88% <0%> (-14.35%) ⬇️
libbeat/publisher/queue/memqueue/ringbuf.go 55.96% <0%> (-5.51%) ⬇️
metricbeat/module/docker/memory/helper.go 63.15% <0%> (-5.03%) ⬇️
...etricbeat/module/rabbitmq/connection/connection.go 68.75% <0%> (-4.59%) ⬇️
metricbeat/module/rabbitmq/exchange/exchange.go 68.75% <0%> (-4.59%) ⬇️
... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4319c31...f491b71. Read the comment docs.

@kvch kvch merged commit 5d7c1fa into elastic:master Jun 4, 2019
andrewvc pushed a commit to andrewvc/beats that referenced this pull request Jun 12, 2019
…tic#12340)

Previously, when running the subcommand `setup`, configuration from `modules.d` was not loaded. This lead to Filebeat ignoring settings of modules which had special template directive in their pipelines. Thus, uploading incorrect pipelines.

Now when running setup configuration from the modules folder are read and pipelines are loaded.

Closes elastic#9747
@adriansr
Copy link
Contributor

adriansr commented Jul 17, 2019

@kvch as this is marked as a bug in CHANGELOG, can you backport it to 7.2? Or is there a reason not to?

@kvch
Copy link
Contributor Author

kvch commented Jul 17, 2019

No reason. Backporting it.

kvch added a commit to kvch/beats that referenced this pull request Jul 17, 2019
…tic#12340)

Previously, when running the subcommand `setup`, configuration from `modules.d` was not loaded. This lead to Filebeat ignoring settings of modules which had special template directive in their pipelines. Thus, uploading incorrect pipelines.

Now when running setup configuration from the modules folder are read and pipelines are loaded.

Closes elastic#9747
(cherry picked from commit 5d7c1fa)
kvch added a commit to kvch/beats that referenced this pull request Jul 17, 2019
…tic#12340)

Previously, when running the subcommand `setup`, configuration from `modules.d` was not loaded. This lead to Filebeat ignoring settings of modules which had special template directive in their pipelines. Thus, uploading incorrect pipelines.

Now when running setup configuration from the modules folder are read and pipelines are loaded.

Closes elastic#9747
(cherry picked from commit 5d7c1fa)
kvch added a commit that referenced this pull request Jul 17, 2019
…ing setup subcommand (#12941)

* Load configuration from modules.d when running setup subcommand (#12340)

Previously, when running the subcommand `setup`, configuration from `modules.d` was not loaded. This lead to Filebeat ignoring settings of modules which had special template directive in their pipelines. Thus, uploading incorrect pipelines.

Now when running setup configuration from the modules folder are read and pipelines are loaded.

Closes #9747
(cherry picked from commit 5d7c1fa)

* add missing function to libbeat/beat/beat.py
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
…en running setup subcommand (elastic#12941)

* Load configuration from modules.d when running setup subcommand (elastic#12340)

Previously, when running the subcommand `setup`, configuration from `modules.d` was not loaded. This lead to Filebeat ignoring settings of modules which had special template directive in their pipelines. Thus, uploading incorrect pipelines.

Now when running setup configuration from the modules folder are read and pipelines are loaded.

Closes elastic#9747
(cherry picked from commit 8f27fe8)

* add missing function to libbeat/beat/beat.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Filebeat system module pipelines setup doesn't honor module settings
4 participants