Fix httpjson split issue on empty root array #32001

kcreddy · 2022-06-21T05:04:38Z

What does this PR do?

Fix httpjson split issue on empty root array. This fix publishes the event with key in case of empty root array

Why is it important?

This is a bug fix on filebeat httpjson input where no event is being published when splitting empty root array. The desired outcome is to publish event with array.

Checklist

My code follows the style guidelines of this project
~~[ ] I have commented my code, particularly in hard-to-understand areas~~
~~[ ] I have made corresponding change to the default configuration files~~
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

cd x-pack/filebeat/input/httpjson; go test -v -run TestSplit

=== RUN   TestSplit
=== RUN   TestSplit/Two_nested_Split_Arrays_with_keep_parent
=== RUN   TestSplit/A_nested_array_with_a_nested_map
=== RUN   TestSplit/A_nested_array_with_a_nested_map_with_transforms
=== RUN   TestSplit/A_nested_array_with_a_nested_array_in_an_object
=== RUN   TestSplit/A_nested_array_with_an_empty_nested_array_in_an_object_publishes_without_the_key
=== RUN   TestSplit/First_level_split_publishes_with_key_if_no_events
=== RUN   TestSplit/Changes_must_be_local_to_parent_when_nested_splits
=== RUN   TestSplit/Split_string
=== RUN   TestSplit/An_empty_array_in_an_object
=== RUN   TestSplit/A_missing_array_in_an_object
=== RUN   TestSplit/An_empty_map_in_an_object
=== RUN   TestSplit/A_missing_map_in_an_object
=== RUN   TestSplit/An_empty_string
=== RUN   TestSplit/A_missing_string
--- PASS: TestSplit (0.00s)
    --- PASS: TestSplit/Two_nested_Split_Arrays_with_keep_parent (0.00s)
    --- PASS: TestSplit/A_nested_array_with_a_nested_map (0.00s)
    --- PASS: TestSplit/A_nested_array_with_a_nested_map_with_transforms (0.00s)
    --- PASS: TestSplit/A_nested_array_with_a_nested_array_in_an_object (0.00s)
    --- PASS: TestSplit/A_nested_array_with_an_empty_nested_array_in_an_object_publishes_without_the_key (0.00s)
    --- PASS: TestSplit/First_level_split_publishes_with_key_if_no_events (0.00s)
    --- PASS: TestSplit/Changes_must_be_local_to_parent_when_nested_splits (0.00s)
    --- PASS: TestSplit/Split_string (0.00s)
    --- PASS: TestSplit/An_empty_array_in_an_object (0.00s)
    --- PASS: TestSplit/A_missing_array_in_an_object (0.00s)
    --- PASS: TestSplit/An_empty_map_in_an_object (0.00s)
    --- PASS: TestSplit/A_missing_map_in_an_object (0.00s)
    --- PASS: TestSplit/An_empty_string (0.00s)
    --- PASS: TestSplit/A_missing_string (0.00s)
PASS
ok      github.com/elastic/beats/v7/x-pack/filebeat/input/httpjson      0.289s

Related issues

Closes Filebeat httpjson cursor not saved when splitting on empty array. #28221

elasticmachine · 2022-06-21T05:04:40Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2022-06-21T06:21:12Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-09-15T08:57:39.478+0000
Duration: 72 min 37 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2237
Skipped	166
Total	2403

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

efd6

This should have a line in CHANGELOG.next.asciidoc and probably wants a backport to the relevant versions.

P1llus · 2022-06-22T11:19:51Z

@marc-gr does this only need to be applied for split on Array, or also on map/string?

mergify · 2022-06-23T06:15:41Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b httpjson_split_empty_root_arr upstream/httpjson_split_empty_root_arr
git merge upstream/main
git push upstream httpjson_split_empty_root_arr

…y_root_arr

mergify · 2022-06-28T16:49:36Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b httpjson_split_empty_root_arr upstream/httpjson_split_empty_root_arr
git merge upstream/main
git push upstream httpjson_split_empty_root_arr

…y_root_arr

mergify · 2022-07-05T13:35:47Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b httpjson_split_empty_root_arr upstream/httpjson_split_empty_root_arr
git merge upstream/main
git push upstream httpjson_split_empty_root_arr

…y_root_arr

mergify · 2022-07-11T12:48:52Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b httpjson_split_empty_root_arr upstream/httpjson_split_empty_root_arr
git merge upstream/main
git push upstream httpjson_split_empty_root_arr

ShourieG · 2022-07-25T12:49:11Z

@kcreddy , please resolve the conflict , otherwise LGTM

efd6 · 2022-08-22T07:25:58Z

CHANGELOG.next.asciidoc

+- Do not emit error log when filestream reader reaches EOF and `close.reader.on_eof` is enabled. {pull}31109[31109]
+- sophos.xg: Update module to handle new log fields. {issue}31038[31038] {pull}31388[31388]
+- Fix MISP documentation for `var.filters` config option. {pull}31434[31434]
+- Fix type mapping of client.as.number in okta module. {pull}31676[31676]
+- Fix handling of empty array in httpjson input. {pull}32001[32001]
+- Fix last write pagination commit checkpoint on `aws-s3` input for s3 direct polling when using the same bucket and different list prefixes. {pull}31776[31776]
+- If a file is ignored by `filestream` because of ignore_older settings, when it is updated, only the new lines are shipped to the output. {issue}31924[31924] {pull}31972[31972]
+- Adding a fix for threatintel module where MISP was paginating forever. {pull}31784[31784]xs
+- Fix deduplication in Google workspace module by changing fingerprint processor target field from `@metadata.id` to `@metadata._id`. {pull}31898[31898]
+- Fix handling and mapping of syslog priority, facility and severity values in Cisco module. {pull}32025[32025]
+- Fix http_endpoint input TLS handshake failures. {pull}32105[32105]


Suggested change

- Do not emit error log when filestream reader reaches EOF and `close.reader.on_eof` is enabled. {pull}31109[31109]

- sophos.xg: Update module to handle new log fields. {issue}31038[31038] {pull}31388[31388]

- Fix MISP documentation for `var.filters` config option. {pull}31434[31434]

- Fix type mapping of client.as.number in okta module. {pull}31676[31676]

- Fix handling of empty array in httpjson input. {pull}32001[32001]

- Fix last write pagination commit checkpoint on `aws-s3` input for s3 direct polling when using the same bucket and different list prefixes. {pull}31776[31776]

- If a file is ignored by `filestream` because of ignore_older settings, when it is updated, only the new lines are shipped to the output. {issue}31924[31924] {pull}31972[31972]

- Adding a fix for threatintel module where MISP was paginating forever. {pull}31784[31784]xs

- Fix deduplication in Google workspace module by changing fingerprint processor target field from `@metadata.id` to `@metadata._id`. {pull}31898[31898]

- Fix handling and mapping of syslog priority, facility and severity values in Cisco module. {pull}32025[32025]

- Fix http_endpoint input TLS handshake failures. {pull}32105[32105]

- Fix handling of empty array in httpjson input. {pull}32001[32001]

Added the line at the end

andrewkroh · 2022-09-13T03:33:20Z

Other than the changelog conflicts, does this need anything else before merging? @kcreddy

…y_root_arr

kcreddy · 2022-09-13T08:01:35Z

@andrewkroh Sorry my bad, I thought this was merged long back. Resolved merge conflicts. This can be merged once approved

ShourieG

LGTM

efd6

LGTM after changelog is fixed.

CHANGELOG.next.asciidoc

* Fix httpjson split issue on empty root arrayPublish events * update asciidoc * update changelog * remove old lines (cherry picked from commit 0334634) # Conflicts: # x-pack/filebeat/input/httpjson/split.go # x-pack/filebeat/input/httpjson/split_test.go

* Fix httpjson split issue on empty root arrayPublish events * update asciidoc * update changelog * remove old lines (cherry picked from commit 0334634)

* Fix httpjson split issue on empty root arrayPublish events * update asciidoc * update changelog * remove old lines (cherry picked from commit 0334634) Co-authored-by: Krishna Chaitanya Reddy Burri <krish.reddy91@gmail.com>

…33102) * Fix httpjson split issue on empty root array (#32001) * Fix httpjson split issue on empty root arrayPublish events * update asciidoc * update changelog * remove old lines (cherry picked from commit 0334634) # Conflicts: # x-pack/filebeat/input/httpjson/split.go # x-pack/filebeat/input/httpjson/split_test.go * move to internal/v2 * remove as duplicated in internal/v2 * linting error never wrapped Co-authored-by: Krishna Chaitanya Reddy Burri <krish.reddy91@gmail.com>

* Fix httpjson split issue on empty root arrayPublish events * update asciidoc * update changelog * remove old lines

Fix httpjson split issue on empty root arrayPublish events

dd86811

kcreddy added bug Team:Security-External Integrations labels Jun 21, 2022

kcreddy requested a review from a team as a code owner June 21, 2022 05:04

kcreddy assigned marc-gr Jun 21, 2022

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 21, 2022

kcreddy unassigned marc-gr Jun 21, 2022

kcreddy requested a review from marc-gr June 21, 2022 05:36

mergify bot assigned kcreddy Jun 21, 2022

efd6 reviewed Jun 21, 2022

View reviewed changes

update asciidoc

f2eb715

Merge remote-tracking branch 'upstream/main' into httpjson_split_empt…

a23e904

…y_root_arr

kcreddy added backport-v8.3.0 Automated backport with mergify backport-7.17 Automated backport to the 7.17 branch with mergify labels Jul 4, 2022

Merge remote-tracking branch 'upstream/main' into httpjson_split_empt…

38bbb79

…y_root_arr

Merge remote-tracking branch 'upstream/main' into httpjson_split_empt…

0bd65e0

…y_root_arr

efd6 reviewed Aug 22, 2022

View reviewed changes

andrewkroh added backport-v8.4.0 Automated backport with mergify and removed backport-v8.3.0 Automated backport with mergify labels Sep 13, 2022

Merge remote-tracking branch 'upstream/main' into httpjson_split_empt…

0cbdb07

…y_root_arr

update changelog

4c42061

kcreddy requested a review from efd6 September 14, 2022 09:41

ShourieG approved these changes Sep 14, 2022

View reviewed changes

efd6 reviewed Sep 14, 2022

View reviewed changes

CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved

remove old lines

6ad71f7

efd6 approved these changes Sep 15, 2022

View reviewed changes

kcreddy merged commit 0334634 into elastic:main Sep 15, 2022

mergify bot mentioned this pull request Sep 15, 2022

[7.17](backport #32001) Fix httpjson split issue on empty root array #33102

Merged

mergify bot mentioned this pull request Sep 15, 2022

[8.4](backport #32001) Fix httpjson split issue on empty root array #33103

Merged

mergify bot pushed a commit that referenced this pull request Sep 15, 2022

Fix httpjson split issue on empty root array (#32001)

bc1059d

* Fix httpjson split issue on empty root arrayPublish events * update asciidoc * update changelog * remove old lines (cherry picked from commit 0334634)

chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023

Fix httpjson split issue on empty root array (#32001)

433c3a2

* Fix httpjson split issue on empty root arrayPublish events * update asciidoc * update changelog * remove old lines

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix httpjson split issue on empty root array #32001

Fix httpjson split issue on empty root array #32001

kcreddy commented Jun 21, 2022 •

edited

Loading

elasticmachine commented Jun 21, 2022

elasticmachine commented Jun 21, 2022 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

efd6 left a comment

P1llus commented Jun 22, 2022

mergify bot commented Jun 23, 2022

mergify bot commented Jun 28, 2022

mergify bot commented Jul 5, 2022

mergify bot commented Jul 11, 2022

ShourieG commented Jul 25, 2022

efd6 Aug 22, 2022

kcreddy Sep 14, 2022

andrewkroh commented Sep 13, 2022

kcreddy commented Sep 13, 2022 •

edited

Loading

ShourieG left a comment

efd6 left a comment

Fix httpjson split issue on empty root array #32001

Fix httpjson split issue on empty root array #32001

Conversation

kcreddy commented Jun 21, 2022 • edited Loading

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

elasticmachine commented Jun 21, 2022

elasticmachine commented Jun 21, 2022 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

efd6 left a comment

Choose a reason for hiding this comment

P1llus commented Jun 22, 2022

mergify bot commented Jun 23, 2022

mergify bot commented Jun 28, 2022

mergify bot commented Jul 5, 2022

mergify bot commented Jul 11, 2022

ShourieG commented Jul 25, 2022

efd6 Aug 22, 2022

Choose a reason for hiding this comment

kcreddy Sep 14, 2022

Choose a reason for hiding this comment

andrewkroh commented Sep 13, 2022

kcreddy commented Sep 13, 2022 • edited Loading

ShourieG left a comment

Choose a reason for hiding this comment

efd6 left a comment

Choose a reason for hiding this comment

kcreddy commented Jun 21, 2022 •

edited

Loading

elasticmachine commented Jun 21, 2022 •

edited by jenkins-beats-ci bot

Loading

kcreddy commented Sep 13, 2022 •

edited

Loading