[Filebeat] Add Pensando DFW Module (elastic#21063)

* Add Pensando module init

* explicitly define the ECS version per testing

* updates to docs from make update

* updates for pensando module

* updates to documentation and db screenshot

* add dashboard export to repo

* update to add pensando beat

* Update filebeat/module/pensando/dfw/config/dfw.yml

Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com>

* Update pipeline.yml

Condensed all "remove" fields to 1 list of fields.

* Update pipeline.yml

Do not remove the payload_raw field.

* Update filebeat/module/pensando/_meta/docs.asciidoc

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

* Update config.yml

Added syslog_host and syslog_port values as suggested.

* Update docs.asciidoc

Added documentation for syslog_host and syslog_port as suggested.

* Update pipeline.yml

Removing payload_raw - this and json are, essentially, the same field and no longer needed after parsing.

* Update pipeline.yml

Changed checks if values are != null to use the filebeat specific ignore_empty_value: true instead.

* Remove set of event.module

Remove the set param for event.module.  Filebeat should add this automatically.

* Apply suggestions from code review

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

* Update test.log

* Use convert instead of set for some fields

Changed ECS sets for IP addresses and ports to converts of type ip and
integer respectively.

* Updates for geoip and autonomous system

* add pensando dfw fields

* fixes from make -C filebeat update

* fixes for filebeat check

* make update changes

* Update filebeat/module/pensando/dfw/config/dfw.yml

Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com>

* Update filebeat/module/pensando/dfw/ingest/pipeline.yml

Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com>

* Update filebeat/module/pensando/dfw/ingest/pipeline.yml

Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com>

* Update filebeat/module/pensando/dfw/ingest/pipeline.yml

Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com>

* Update filebeat/module/pensando/dfw/ingest/pipeline.yml

Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com>

* remove old json file

* ran tests

* Update filebeat/module/pensando/dfw/ingest/pipeline.yml

Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com>

* gen after run of 'mage -v pythonIntegTest'

* Update fields.yml

* mage fmt update request

Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com>
Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

CHANGELOG-developer.next.asciidoc

@@ -104,6 +104,7 @@ The list below covers the major changes between 7.0.0-rc2 and master only.
 - Update Go version to 1.14.7. {pull}20508[20508]
 - Add packaging for docker image based on UBI minimal 8. {pull}20576[20576]
 - Make the mage binary used by the build process in the docker container to be statically compiled. {pull}20827[20827]
+- Add Pensando distributed firewall module. {pull}21063[21063]
 - Update ecszap to v0.3.0 for using ECS 1.6.0 in logs {pull}22267[22267]
 - Add support for customized monitoring API. {pull}22605[22605]
 - Update Go version to 1.15.7. {pull}22495[22495]

filebeat/docs/fields.asciidoc

@@ -69,6 +69,7 @@ grouped in the following categories:
 * <<exported-fields-oracle>>
 * <<exported-fields-osquery>>
 * <<exported-fields-panw>>
+* <<exported-fields-pensando>>
 * <<exported-fields-postgresql>>
 * <<exported-fields-process>>
 * <<exported-fields-proofpoint>>
@@ -105827,6 +105828,147 @@ Specifies the sub type of the log
 
 --
 
+[[exported-fields-pensando]]
+== Pensando fields
+
+pensando Module
+
+
+
+[float]
+=== pensando
+
+Fields from Pensando logs.
+
+
+
+[float]
+=== dfw
+
+Fields for Pensando DFW
+
+
+
+*`pensando.dfw.action`*::
++
+--
+Action on the flow. 
+
+
+type: keyword
+
+--
+
+*`pensando.dfw.app_id`*::
++
+--
+Application ID 
+
+
+type: integer
+
+--
+
+*`pensando.dfw.destination_address`*::
++
+--
+Address of destination. 
+
+
+type: keyword
+
+--
+
+*`pensando.dfw.destination_port`*::
++
+--
+Port of destination. 
+
+
+type: integer
+
+--
+
+*`pensando.dfw.direction`*::
++
+--
+Direction of the flow 
+
+
+type: keyword
+
+--
+
+*`pensando.dfw.protocol`*::
++
+--
+Protocol of the flow 
+
+
+type: keyword
+
+--
+
+*`pensando.dfw.rule_id`*::
++
+--
+Rule ID that was matched. 
+
+
+type: keyword
+
+--
+
+*`pensando.dfw.session_id`*::
++
+--
+Session ID of the flow 
+
+
+type: integer
+
+--
+
+*`pensando.dfw.session_state`*::
++
+--
+Session state of the flow. 
+
+
+type: keyword
+
+--
+
+*`pensando.dfw.source_address`*::
++
+--
+Source address of the flow. 
+
+
+type: keyword
+
+--
+
+*`pensando.dfw.source_port`*::
++
+--
+Source port of the flow. 
+
+
+type: integer
+
+--
+
+*`pensando.dfw.timestamp`*::
++
+--
+Timestamp of the log. 
+
+
+type: date
+
+--
+
 [[exported-fields-postgresql]]
 == PostgreSQL fields
 

filebeat/docs/modules/pensando.asciidoc

@@ -0,0 +1,69 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[filebeat-module-pensando]]
+:modulename: pensando
+:has-dashboards: true
+
+== pensando module
+
+The +{modulename}+ module parses distributed firewall logs created by the
+http://pensando.io/[Pensando] distributed services card (DSC).
+
+
+include::../include/what-happens.asciidoc[]
+
+include::../include/gs-link.asciidoc[]
+
+[float]
+=== Compatibility
+
+The Pensando module has been tested with 1.12.0-E-54 and later.
+
+include::../include/configuring-intro.asciidoc[]
+The following example shows how to set parameters in the +modules.d/{modulename}.yml+
+file to listen for firewall logs sent from the Pensando DSC(s) on port 5514 (default is 9001):
+
+["source","yaml",subs="attributes"]
+-----
+- module: pensando
+  access:
+    enabled: true
+    var.syslog_host: 0.0.0.0
+    var.syslog_port: [9001]
+-----
+:fileset_ex: dfw
+
+include::../include/config-option-intro.asciidoc[]
+
+TODO: document the variables from each fileset. If you're describing a variable
+that's common to other modules, you can reuse shared descriptions by including
+the relevant file. For example:
+
+[float]
+==== `dfw` log fileset settings
+
+include::../include/var-paths.asciidoc[]
+
+[float]
+=== Example dashboard
+
+This module comes with a sample dashboard. For example:
+
+[role="screenshot"]
+image::./images/filebeat-pensando-dfw.png[]
+
+:has-dashboards!:
+
+:fileset_ex!:
+
+:modulename!:
+
+
+[float]
+=== Fields
+
+For a description of each field in the module, see the
+<<exported-fields-pensando,exported fields>> section.
+

filebeat/docs/modules_list.asciidoc

@@ -50,6 +50,7 @@ This file is generated! See scripts/docs_collector.py
   * <<filebeat-module-oracle>>
   * <<filebeat-module-osquery>>
   * <<filebeat-module-panw>>
+  * <<filebeat-module-pensando>>
   * <<filebeat-module-postgresql>>
   * <<filebeat-module-proofpoint>>
   * <<filebeat-module-rabbitmq>>
@@ -121,6 +122,7 @@ include::modules/okta.asciidoc[]
 include::modules/oracle.asciidoc[]
 include::modules/osquery.asciidoc[]
 include::modules/panw.asciidoc[]
+include::modules/pensando.asciidoc[]
 include::modules/postgresql.asciidoc[]
 include::modules/proofpoint.asciidoc[]
 include::modules/rabbitmq.asciidoc[]

filebeat/filebeat.reference.yml

@@ -335,6 +335,18 @@ filebeat.modules:
     # of the document. The default is true.
     #var.use_namespace: true
 
+#------------------------------- Pensando Module -------------------------------
+- module: pensando
+# Firewall logs
+  dfw:
+    enabled: true
+    var.syslog_host: 0.0.0.0
+    var.syslog_port: 9001
+
+    # Set custom paths for the log files. If left empty,
+    # Filebeat will choose the paths depending on your OS.
+    # var.paths:
+
 #------------------------------ PostgreSQL Module ------------------------------
 #- module: postgresql
   # Logs

filebeat/module/pensando/_meta/config.yml

@@ -0,0 +1,10 @@
+- module: pensando
+# Firewall logs
+  dfw:
+    enabled: true
+    var.syslog_host: 0.0.0.0
+    var.syslog_port: 9001
+
+    # Set custom paths for the log files. If left empty,
+    # Filebeat will choose the paths depending on your OS.
+    # var.paths:

filebeat/module/pensando/_meta/docs.asciidoc

@@ -0,0 +1,56 @@
+:modulename: pensando
+:has-dashboards: true
+
+== pensando module
+
+The +{modulename}+ module parses distributed firewall logs created by the
+http://pensando.io/[Pensando] distributed services card (DSC).
+
+
+include::../include/what-happens.asciidoc[]
+
+include::../include/gs-link.asciidoc[]
+
+[float]
+=== Compatibility
+
+The Pensando module has been tested with 1.12.0-E-54 and later.
+
+include::../include/configuring-intro.asciidoc[]
+The following example shows how to set parameters in the +modules.d/{modulename}.yml+
+file to listen for firewall logs sent from the Pensando DSC(s) on port 5514 (default is 9001):
+
+["source","yaml",subs="attributes"]
+-----
+- module: pensando
+  access:
+    enabled: true
+    var.syslog_host: 0.0.0.0
+    var.syslog_port: [9001]
+-----
+:fileset_ex: dfw
+
+include::../include/config-option-intro.asciidoc[]
+
+TODO: document the variables from each fileset. If you're describing a variable
+that's common to other modules, you can reuse shared descriptions by including
+the relevant file. For example:
+
+[float]
+==== `dfw` log fileset settings
+
+include::../include/var-paths.asciidoc[]
+
+[float]
+=== Example dashboard
+
+This module comes with a sample dashboard. For example:
+
+[role="screenshot"]
+image::./images/filebeat-pensando-dfw.png[]
+
+:has-dashboards!:
+
+:fileset_ex!:
+
+:modulename!:

filebeat/module/pensando/_meta/fields.yml

@@ -0,0 +1,10 @@
+- key: pensando
+  title: Pensando 
+  description: >
+    pensando Module
+  fields:
+    - name: pensando
+      type: group
+      description: >
+        Fields from Pensando logs.
+      fields: