Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IBM MQ Filebeat module #8782

Merged
merged 43 commits into from
Jul 29, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
63c61b6
Initial commit for the IBM MQ Filebeat module
felix-lessoer Oct 29, 2018
599fa45
Updated manifest to fit for windows files
felix-lessoer Oct 29, 2018
c2447d2
Add example log data
felix-lessoer Oct 30, 2018
95aa65c
add expected files
ruflin May 10, 2019
016baa1
update docs
ruflin May 21, 2019
ecc341d
added kibana export
felix-lessoer Jun 4, 2019
93363c2
Initial commit for the IBM MQ Filebeat module
felix-lessoer Oct 29, 2018
c4003d8
Updated manifest to fit for windows files
felix-lessoer Oct 29, 2018
8d5cc45
Add example log data
felix-lessoer Oct 30, 2018
abc3390
add expected files
ruflin May 10, 2019
9349f86
update docs
ruflin May 21, 2019
a7d9da0
added kibana export
felix-lessoer Jun 4, 2019
49f0f3c
add screenshot and screenshot link to docs
ruflin Jun 7, 2019
21bc961
update error.id and timestamp field
ruflin Jun 7, 2019
a4a585b
update fields to be ECS compabtible.
ruflin Jun 7, 2019
36ff43c
adjust json formatting
ruflin Jun 7, 2019
0e7a5cb
update docs
ruflin Jun 7, 2019
0273f92
update reference docs
ruflin Jun 7, 2019
94a4c9b
update fields by removing example entry
ruflin Jun 7, 2019
811bacd
update pipeline to combine arithinsert and commentinsert
ruflin Jun 11, 2019
983b378
update queue manager definition
ruflin Jun 11, 2019
9cee42e
update installation definition
ruflin Jun 11, 2019
e349c05
update fields
ruflin Jun 11, 2019
fe06d5a
update error code placing
ruflin Jun 11, 2019
ad01321
update error code
ruflin Jun 11, 2019
ceb92a8
update fields
ruflin Jun 13, 2019
985c411
cleanup code indentation
ruflin Jun 13, 2019
b3697d6
revert change as error is not used anymore
ruflin Jun 13, 2019
a56f235
update ecs files
ruflin Jun 13, 2019
eef1ed5
Merge branch 'mq' with local changes
felix-lessoer Jul 18, 2019
1a3e8b9
Update pipeline to improve error handling + add @timestamp and message
felix-lessoer Jul 19, 2019
cd3d613
Updated Dashboard to fit to ECS
felix-lessoer Jul 19, 2019
ee63457
Updated Dashboard to fit to the right index pattern
felix-lessoer Jul 24, 2019
cd027bc
Bugfixing dashboard
felix-lessoer Jul 24, 2019
8a7620f
Merge branch 'master' into felix-lessoer-mq
Jul 24, 2019
0439eed
Add compatibility note
Jul 24, 2019
7938cc0
Update changelog
Jul 24, 2019
6e294c6
Dashboard is now based on Errorlog Saved search
felix-lessoer Jul 24, 2019
d58f8bd
Update expected files
Jul 25, 2019
c0b5726
Use common error handling
Jul 25, 2019
f911b5f
Use generic error reporting
Jul 25, 2019
14b07a6
Fix grok pattern
Jul 25, 2019
b26b443
Fix pipeline remove processor
Jul 26, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Add netflow dashboards based on Logstash netflow. {pull}12857[12857]
- Parse more fields from Elasticsearch slowlogs. {pull}11939[11939]
- Update module pipelines to enrich events with autonomous system fields. {pull}13036[13036]
- Add module for ingesting IBM MQ logs. {pull}8782[8782]

*Heartbeat*

Expand Down
97 changes: 97 additions & 0 deletions filebeat/docs/fields.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ grouped in the following categories:
* <<exported-fields-googlecloud>>
* <<exported-fields-haproxy>>
* <<exported-fields-host-processor>>
* <<exported-fields-ibmmq>>
* <<exported-fields-icinga>>
* <<exported-fields-iis>>
* <<exported-fields-iptables>>
Expand Down Expand Up @@ -6178,6 +6179,102 @@ example: stretch

--

[[exported-fields-ibmmq]]
== ibmmq fields

ibmmq Module



[float]
=== ibmmq




[float]
=== errorlog

IBM MQ error logs


*`ibmmq.errorlog.installation`*::
+
--
This is the installation name which can be given at installation time.
Each installation of IBM MQ on UNIX, Linux, and Windows, has a unique identifier known as an installation name. The installation name is used to associate things such as queue managers and configuration files with an installation.


type: keyword

--

*`ibmmq.errorlog.qmgr`*::
+
--
Name of the queue manager. Queue managers provide queuing services to applications, and manages the queues that belong to them.


type: keyword

--

*`ibmmq.errorlog.arithinsert`*::
+
--
Changing content based on error.id

type: keyword

--

*`ibmmq.errorlog.commentinsert`*::
+
--
Changing content based on error.id

type: keyword

--

*`ibmmq.errorlog.errordescription`*::
+
--
Please add description

type: text

example: Please add example

--

*`ibmmq.errorlog.explanation`*::
+
--
Explaines the error in more detail

type: keyword

--

*`ibmmq.errorlog.action`*::
+
--
Defines what to do when the error occurs

type: keyword

--

*`ibmmq.errorlog.code`*::
+
--
Error code.

type: keyword

--

[[exported-fields-icinga]]
== Icinga fields

Expand Down
Binary file added filebeat/docs/images/filebeat-ibmmq.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
62 changes: 62 additions & 0 deletions filebeat/docs/modules/ibmmq.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
////
This file is generated! See scripts/docs_collector.py
////

[[filebeat-module-ibmmq]]
:modulename: ibmmq

== IBM MQ module

This is the ibmmq module.

include::../include/what-happens.asciidoc[]

[float]
=== Compatibility

This module has been tested with IBM MQ v9.1.0.0, but it should be compatible with older versions.

include::../include/running-modules.asciidoc[]

[float]
=== Example dashboard

This module comes with a sample dashboard. For example:

[role="screenshot"]
image::./images/filebeat-ibmmq.png[]


include::../include/configuring-intro.asciidoc[]


The following example shows how to set paths in the +modules.d/{modulename}.yml+
file to override the default paths for IBM MQ errorlog:

["source","yaml",subs="attributes"]
-----
- module: ibmmq
errorlog:
enabled: true
var.paths: ["C:/ibmmq/logs/*.log"]
-----
:fileset_ex: errorlog

include::../include/config-option-intro.asciidoc[]

[float]
==== `errorlog` log fileset settings

include::../include/var-paths.asciidoc[]

:fileset_ex!:

:modulename!:


[float]
=== Fields

For a description of each field in the module, see the
<<exported-fields-ibmmq,exported fields>> section.

2 changes: 2 additions & 0 deletions filebeat/docs/modules_list.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ This file is generated! See scripts/docs_collector.py
* <<filebeat-module-envoyproxy>>
* <<filebeat-module-googlecloud>>
* <<filebeat-module-haproxy>>
* <<filebeat-module-ibmmq>>
* <<filebeat-module-icinga>>
* <<filebeat-module-iis>>
* <<filebeat-module-iptables>>
Expand Down Expand Up @@ -46,6 +47,7 @@ include::modules/elasticsearch.asciidoc[]
include::modules/envoyproxy.asciidoc[]
include::modules/googlecloud.asciidoc[]
include::modules/haproxy.asciidoc[]
include::modules/ibmmq.asciidoc[]
include::modules/icinga.asciidoc[]
include::modules/iis.asciidoc[]
include::modules/iptables.asciidoc[]
Expand Down
10 changes: 10 additions & 0 deletions x-pack/filebeat/filebeat.reference.yml
Original file line number Diff line number Diff line change
Expand Up @@ -200,6 +200,16 @@ filebeat.modules:
# Filebeat will choose the paths depending on your OS.
#var.paths:

#-------------------------------- Ibmmq Module --------------------------------
- module: ibmmq
# All logs
errorlog:
enabled: true

# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:

#-------------------------------- Icinga Module --------------------------------
#- module: icinga
# Main logs
Expand Down
1 change: 1 addition & 0 deletions x-pack/filebeat/include/list.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions x-pack/filebeat/module/ibmmq/_meta/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
- module: ibmmq
# All logs
errorlog:
enabled: true

# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
49 changes: 49 additions & 0 deletions x-pack/filebeat/module/ibmmq/_meta/docs.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
:modulename: ibmmq

== IBM MQ module

This is the ibmmq module.

include::../include/what-happens.asciidoc[]

[float]
=== Compatibility

This module has been tested with IBM MQ v9.1.0.0, but it should be compatible with older versions.

include::../include/running-modules.asciidoc[]

[float]
=== Example dashboard

This module comes with a sample dashboard. For example:

[role="screenshot"]
image::./images/filebeat-ibmmq.png[]


include::../include/configuring-intro.asciidoc[]


The following example shows how to set paths in the +modules.d/{modulename}.yml+
file to override the default paths for IBM MQ errorlog:

["source","yaml",subs="attributes"]
-----
- module: ibmmq
errorlog:
enabled: true
var.paths: ["C:/ibmmq/logs/*.log"]
-----
:fileset_ex: errorlog

include::../include/config-option-intro.asciidoc[]

[float]
==== `errorlog` log fileset settings

include::../include/var-paths.asciidoc[]

:fileset_ex!:

:modulename!:
9 changes: 9 additions & 0 deletions x-pack/filebeat/module/ibmmq/_meta/fields.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
- key: ibmmq
title: "ibmmq"
description: >
ibmmq Module
fields:
- name: ibmmq
type: group
description: >
fields:
Loading