Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for array parsing in azure-eventhub input #18585

Merged
merged 4 commits into from
May 20, 2020

Conversation

narph
Copy link
Contributor

@narph narph commented May 15, 2020

What does this PR do?

Adds support to parse array of eventhub messages into separate events.
Expected type was an object containing a list of messages.

Why is it important?

List of eventhub messages were published in single Elasticsearch event.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Ex before:

          "message" : """[{"data":"{\"Id\":\"https://testnamekey.vault.azure.net/keys/weewewew/....\",\"VaultName\":\"testnamekey\",\"ObjectType\":\"Key\",\"ObjectName\":\"weewewew\",\"Version\":\"....\",\"NBF\":null,\"EXP\":null}","dataschema":"#1","id":"tt589454itekjter94oi","source":"/subscriptions/.../resourceGroups/obs-infrastructure/providers/Microsoft.KeyVault/vaults/testnamekey","specversion":"1.0","subject":"weewewew","time":"2020-05-15T12:08:09.616451Z","type":"Microsoft.KeyVault.KeyNewVersionCreated"}]""",
          "azure" : {
            "offset" : 2560,
            "sequence_number" : 3,
            "enqueued_time" : "2020-05-15T12:08:11.232Z",
            "eventhub" : "keyvalut",
            "consumer_group" : "test"
          },
          "input" : {
            "type" : "azure-eventhub"
          }
        },

to:

          "message" : """{"data":"{\"Id\":\"https://testnamekey.vault.azure.net/keys/weewewew/....\",\"VaultName\":\"testnamekey\",\"ObjectType\":\"Key\",\"ObjectName\":\"weewewew\",\"Version\":\"tt589454itekjter94oi\",\"NBF\":null,\"EXP\":null}","dataschema":"#1","id":"....","source":"/subscriptions/...../resourceGroups/obs-infrastructure/providers/Microsoft.KeyVault/vaults/testnamekey","specversion":"1.0","subject":"weewewew","time":"2020-05-15T12:08:09.616451Z","type":"Microsoft.KeyVault.KeyNewVersionCreated"}""",
          "azure" : {
            "offset" : 2560,
            "sequence_number" : 3,
            "enqueued_time" : "2020-05-15T12:08:11.232Z",
            "eventhub" : "keyvalut",
            "consumer_group" : "test"
          },
          "input" : {
            "type" : "azure-eventhub"
          }
        },

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 15, 2020
@narph narph self-assigned this May 15, 2020
@narph narph added [zube]: In Review Filebeat Filebeat Team:Integrations Label for the Integrations team labels May 15, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 15, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented May 15, 2020

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Started by user Victor Martinez, Replayed Random Crashes #6]

  • Start Time: 2020-05-19T16:28:24.074+0000

  • Duration: 46 min 39 sec (2738638)

Test stats 🧪

Test Results
Failed 0
Passed 2182
Skipped 382
Total 2564

// in some cases the message is an array
a.log.Debugw(fmt.Sprintf("deserializing multiple messages using the group object `records`"), "warning", err)
var arrayObject []interface{}
err = json.Unmarshal(bMessage, &arrayObject)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No error checking in this line?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, somehow I missed it, also updated the tests

@narph narph requested a review from sayden May 19, 2020 13:41
@narph narph merged commit c742662 into elastic:master May 20, 2020
@narph narph deleted the azure-add-array branch May 20, 2020 07:37
@narph narph added [zube]: In Review needs_backport PR is waiting to be backported to other branches. v7.8.0 and removed [zube]: Done labels May 20, 2020
narph added a commit to narph/beats that referenced this pull request May 20, 2020
* add array as parser

* changelog

* update test

(cherry picked from commit c742662)
narph added a commit to narph/beats that referenced this pull request May 20, 2020
* add array as parser

* changelog

* update test

(cherry picked from commit c742662)
narph added a commit that referenced this pull request May 20, 2020
…nthub input (#18651)

* Add support for array parsing in azure-eventhub input (#18585)

* add array as parser

* changelog

* update test

(cherry picked from commit c742662)

* fix changelog
narph added a commit that referenced this pull request May 20, 2020
…nthub input (#18650)

* Add support for array parsing in azure-eventhub input (#18585)

* add array as parser

* changelog

* update test

(cherry picked from commit c742662)

* fix changelog
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Filebeat Filebeat needs_backport PR is waiting to be backported to other branches. Team:Integrations Label for the Integrations team v7.8.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants