Cherry-pick #23678 to 7.x: [Libbeat][New Processor] XML Decode #24049
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* stashing before initial commit * Initial commit * updating go.sum * updating it again * adding feedback from PR comments and removing expandkeys config entry * Updating changelog * removing expanded_keys from allowed fields * adding new changes based on PR comments, a few more changes remains * moving the xml decoder to its own subpackage based on PR comments * reverting back to Target being a string pointer, to be able to differentiate between null and empty string * Updating certain tests to fit the new ignore_failure and ignore_missing options * Updating unit test to test with missing field * updating license headers * adding benchmark test * benchmark, now also with allocation results * updating changelog entry * removing duplicate Changelog entry * changing changelog entry name to new name * Simplify error handling and fix race $ benchcmp old.txt new.txt benchmark old ns/op new ns/op delta BenchmarkProcessor_Run/single_object-12 15691 15686 -0.03% BenchmarkProcessor_Run/nested_and_array_object-12 39673 39098 -1.45% benchmark old allocs new allocs delta BenchmarkProcessor_Run/single_object-12 158 158 +0.00% BenchmarkProcessor_Run/nested_and_array_object-12 376 374 -0.53% benchmark old bytes new bytes delta BenchmarkProcessor_Run/single_object-12 8597 8597 +0.00% BenchmarkProcessor_Run/nested_and_array_object-12 20310 19798 -2.52% * internal xml to json implementation * Use internal xml to json decoder benchmark old ns/op new ns/op delta BenchmarkProcessor_Run/single_object-12 15686 8051 -48.67% BenchmarkProcessor_Run/nested_and_array_object-12 39098 20540 -47.47% benchmark old allocs new allocs delta BenchmarkProcessor_Run/single_object-12 158 75 -52.53% BenchmarkProcessor_Run/nested_and_array_object-12 374 184 -50.80% benchmark old bytes new bytes delta BenchmarkProcessor_Run/single_object-12 8597 3520 -59.06% BenchmarkProcessor_Run/nested_and_array_object-12 19798 7824 -60.48% benchmark old ns/op new ns/op delta BenchmarkProcessor_Run/single_object-12 15686 8051 -48.67% BenchmarkProcessor_Run/nested_and_array_object-12 39098 20540 -47.47% benchmark old allocs new allocs delta BenchmarkProcessor_Run/single_object-12 158 75 -52.53% BenchmarkProcessor_Run/nested_and_array_object-12 374 184 -50.80% benchmark old bytes new bytes delta BenchmarkProcessor_Run/single_object-12 8597 3520 -59.06% BenchmarkProcessor_Run/nested_and_array_object-12 19798 7824 -60.48% * changelog fix * Update docs * Add godoc example of xml to json * updating test name to fit Example naming convention Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> (cherry picked from commit 6839307)
P1llus
added
[zube]: In Review
backport
Team:Elastic-Agent
|
Pinging @elastic/integrations (Team:Integrations) |
|
Pinging @elastic/agent (Team:Agent) |
|
Pinging @elastic/ingest-management (Team:Ingest Management) |
botelastic
bot
added
the
needs_team
|
Pinging @elastic/security-external-integrations (Team:Security-External Integrations) |
botelastic
bot
removed
the
needs_team
kaiyan-sheng
approved these changes
Feb 15, 2021
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
Trends 🧪💚 Flaky test reportTests succeeded. Expand to view the summary
Test stats 🧪
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cherry-pick of PR #23678 to 7.x branch. Original message:
What does this PR do?
This PR adds 2 components to
Libbeat.It adds a small XML Unmarshal helper function to the
commonfolder inLibbeat, the reason this is added is so that not only processors can utilize it if/when needed, but also inputs. For example I plan to reuse this inhttp_endpointto add XML support there as well. Other beats/inputs that would benefit from this isWinlogbeat.The helper function expects a
[]bytewith valid XML object(s), and will return it as a struct reusing the names of the XML object(s).It supports lists, arrays, nested fields, object identifiers (for example
<book seq="1">).If the XML is not valid it will return a proper error message describing why it failed.
For more information on supported formats, please see the included unit test files.
Second component is a
decode_xmlprocessor, this is the first processor I made, so please review it carefully. I have usedurldecode,mime_typeand thedecode_json_fieldsprocessor as a guideline for this one.For the supported objects and more information on supported formats, please see the included unit test files.
Why is it important?
This adds the possibility to read XML files for file/log input, or decoding XML strings in existing message fields, for example XML strings in existing JSON fields.
Implements an easy helper function for other parts of beats to reuse if they want to Unmarshal XML into a struct.
Checklist
CHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.Related issues