Cisco asa/ftd: Remove _temp_ fields on failure

[adriansr]

What does this PR do?

Updates the shared cisco asa/ftd ingest pipeline to remove the fields under _temp_ in the case of failure.

Why is it important?

Not removing the temporary fields can cause the index mapping to grow too big.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Unfortunately I couldn't add a test because Filebeat module's tests don't allow documents that cause errors.

Tested it locally by adding a test file that would cause an error in the pipeline:

cat module/cisco/asa/test/failure.log

%ASA-5-734001: This message will cause a parsing failure

And confirmed that with the fix, _temp_ fields are no longer part of the output document:

{
  "agent": {
    "hostname": "Adrian-Elastic.local",
    "name": "Adrian-Elastic.local",
    "id": "006114b6-35f7-4c9d-8b9b-99e4dcc9dc50",
    "ephemeral_id": "16b9a465-ad71-4cdc-af6b-0030fdec1984",
    "type": "filebeat",
    "version": "8.0.0"
  },
  "log": {
    "file": {
      "path": "/Users/adrian/go/src/github.com/elastic/beats/x-pack/filebeat/module/cisco/asa/test/failure.log"
    },
    "original": "%ASA-5-734001: This message will cause a parsing failure",
    "offset": 0,
    "level": "notification"
  },
  "message": "This message will cause a parsing failure",
  "fileset": {
    "name": "asa"
  },
  "error": {
    "message": [
      "field [raw_date] not present as part of path [_temp_.raw_date]",
      "Unable to find match for dissect pattern: DAP: User %{user.email}, Addr %{source.address}, Connection %{_temp_.cisco.connection_type}: The following DAP records were selected for this connection: %{_temp_.cisco.dap_records->} against source: This message will cause a parsing failure"
    ]
  },
  "tags": [
    "cisco-asa"
  ],
  "input": {
    "type": "log"
  },
  "@timestamp": "2020-05-08T16:48:31.544Z",
  "ecs": {
    "version": "1.5.0"
  },
  "service": {
    "type": "cisco"
  },
  "host": {
    "name": "Adrian-Elastic.local"
  },
  "event": {
    "severity": 5,
    "timezone": "-02:00",
    "module": "cisco",
    "action": "firewall-rule",
    "dataset": "cisco.asa"
  },
  "cisco": {
    "asa": {
      "message_id": "734001"
    }
  }
}

while before this patch one would get some of them:

{
  "agent": {
    "hostname": "Adrian-Elastic.local",
    "name": "Adrian-Elastic.local",
    "id": "006114b6-35f7-4c9d-8b9b-99e4dcc9dc50",
    "type": "filebeat",
    "ephemeral_id": "ab823767-aa1d-449b-8954-f143b1f03a02",
    "version": "8.0.0"
  },
  "_temp_": {
    "cisco": {
      "message_id": "734001"
    }
  },
  "log": {
    "file": {
      "path": "/Users/adrian/go/src/github.com/elastic/beats/x-pack/filebeat/module/cisco/asa/test/failure.log"
    },
    "original": "%ASA-5-734001: This message will cause a parsing failure",
    "offset": 0,
    "level": "notification"
  },
  "fileset": {
    "name": "asa"
  },
  "message": "This message will cause a parsing failure",
  "error": {
    "message": [
      "field [raw_date] not present as part of path [_temp_.raw_date]",
      "Unable to find match for dissect pattern: DAP: User %{user.email}, Addr %{source.address}, Connection %{_temp_.cisco.connection_type}: The following DAP records were selected for this connection: %{_temp_.cisco.dap_records->} against source: This message will cause a parsing failure"
    ]
  },
  "tags": [
    "cisco-asa"
  ],
  "input": {
    "type": "log"
  },
  "@timestamp": "2020-05-08T16:46:14.593Z",
  "ecs": {
    "version": "1.5.0"
  },
  "service": {
    "type": "cisco"
  },
  "host": {
    "name": "Adrian-Elastic.local"
  },
  "event": {
    "severity": 5,
    "timezone": "-02:00",
    "module": "cisco",
    "action": "firewall-rule",
    "dataset": "cisco.asa"
  }
}

Related issues

• Closes [Filebeat] Cisco module asa-ftd pipeline pollutes the mapping when it fails #18391

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request Cisco asa/ftd: Remove _temp_ fields on failure #18392 updated]

• Start Time: 2020-05-08T18:05:53.209+0000

• Duration: 47 min 35 sec (2795164)

• Commit: 9dd1394

Test stats 🧪

Test	Results
Failed	0
Passed	1180
Skipped	128
Total	1308

[adriansr]

jenkins, test this

[andrewkroh]

FYI @alakahakai since you've been migrating this module to the package manager.

[alakahakai]

FYI @alakahakai since you've been migrating this module to the package manager.

@andrewkroh ACK. Will update the integration