[Libbeat] Respect the `parameters` option defined in the ES output. #18326

ph · 2020-05-06T19:19:07Z

This make sure that any values defined in the parameters option of the
ES output is correctly added to every bulk request query string.

Fixes: #18325

What does this PR do?

Why is it important?

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

This make sure that any values defined in the `parameters` option of the ES output is correctly added to every bulk request query string. Fixes: elastic#18325

elasticmachine · 2020-05-06T19:19:30Z

Pinging @elastic/ingest-management (Team:Ingest Management)

ph · 2020-05-06T19:23:21Z

Adding the prefer_v2_parameters and this PR will allow ES to correctly use the right template.

 │ info [o.e.c.m.MetadataCreateIndexService] [winterfell] applying create index request using v2 template [bar]
   │ info [o.e.c.m.MetadataCreateIndexService] [winterfell] [test-ok] creating index, cause [auto(bulk api)], templates [bar], shards [1]/[2], mappings [_doc]

elasticmachine · 2020-05-06T19:33:43Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Started by user Michal Pristas, Replayed The "default config" fallback doesn't work properly #3]
Start Time: 2020-05-07T07:20:45.307+0000
Duration: 87 min 3 sec (5223452)
Commit: 8fdb9c7

Test stats 🧪

Test	Results
Failed	0
Passed	7759
Skipped	1236
Total	8995

ph · 2020-05-06T21:46:20Z

7.8 backport is ready as part of #18318
going to make a 7.x backport after.

libbeat/esleg/eslegclient/bulkapi_mock_test.go

…lastic#18326) (cherry picked from commit 0575e15)

ruflin · 2020-05-07T11:02:21Z

Backport for 7.8: #18318
Backport for 7.x: #18348

This PR add additionnal test over the usager of the `parameters` options in the Elasticsearch output: - When preconfigured params are set without local params - When preconfigured params are set with local params - When no preconfigured params are configured but local are. - When no preconfigured or local params are set. The merge is also done close to the actual calls and will not be executed if anything fails before the bulk request. The test assertion is now more solid and take into consideration any errors. See discussion in elastic#18318 and elastic#18326

…18326) (#18348) (cherry picked from commit 0575e15) Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>

…8393) * [Libbeat] Add more complete tests for opt parameters in ES output This PR add additionnal test over the usager of the `parameters` options in the Elasticsearch output: - When preconfigured params are set without local params - When preconfigured params are set with local params - When no preconfigured params are configured but local are. - When no preconfigured or local params are set. The merge is also done close to the actual calls and will not be executed if anything fails before the bulk request. The test assertion is now more solid and take into consideration any errors. See discussion in #18318 and #18326

…astic#18393) * [Libbeat] Add more complete tests for opt parameters in ES output This PR add additionnal test over the usager of the `parameters` options in the Elasticsearch output: - When preconfigured params are set without local params - When preconfigured params are set with local params - When no preconfigured params are configured but local are. - When no preconfigured or local params are set. The merge is also done close to the actual calls and will not be executed if anything fails before the bulk request. The test assertion is now more solid and take into consideration any errors. See discussion in elastic#18318 and elastic#18326 (cherry picked from commit 9c8bbaa)

…8393) (#18491) * [Libbeat] Add more complete tests for opt parameters in ES output This PR add additionnal test over the usager of the `parameters` options in the Elasticsearch output: - When preconfigured params are set without local params - When preconfigured params are set with local params - When no preconfigured params are configured but local are. - When no preconfigured or local params are set. The merge is also done close to the actual calls and will not be executed if anything fails before the bulk request. The test assertion is now more solid and take into consideration any errors. See discussion in #18318 and #18326 (cherry picked from commit 9c8bbaa)

…8393) (#18492) * [Libbeat] Add more complete tests for opt parameters in ES output This PR add additionnal test over the usager of the `parameters` options in the Elasticsearch output: - When preconfigured params are set without local params - When preconfigured params are set with local params - When no preconfigured params are configured but local are. - When no preconfigured or local params are set. The merge is also done close to the actual calls and will not be executed if anything fails before the bulk request. The test assertion is now more solid and take into consideration any errors. See discussion in #18318 and #18326 (cherry picked from commit 9c8bbaa)

[Libbeat] Respect the parameters option defined in the ES output.

9fbfeb7

This make sure that any values defined in the `parameters` option of the ES output is correctly added to every bulk request query string. Fixes: elastic#18325

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 6, 2020

ph requested a review from urso May 6, 2020 19:19

ph self-assigned this May 6, 2020

ph added Ingest Management:alpha1 Group issues for ingest management alpha1 Team:Ingest Management labels May 6, 2020

ph added [zube]: In Review blocker review and removed needs_team Indicates that the issue/PR needs a Team:* label labels May 6, 2020

changelog

0ca284f

ph added the needs_backport PR is waiting to be backported to other branches. label May 6, 2020

ph mentioned this pull request May 6, 2020

[Elastic Agent] Add params prefer_v2_templates=true on the ES output #18318

Merged

6 tasks

Adding tests

8fdb9c7

urso reviewed May 7, 2020

View reviewed changes

libbeat/esleg/eslegclient/bulkapi_mock_test.go Show resolved Hide resolved

urso reviewed May 7, 2020

View reviewed changes

libbeat/esleg/eslegclient/bulkapi_mock_test.go Show resolved Hide resolved

urso approved these changes May 7, 2020

View reviewed changes

urso merged commit 0575e15 into elastic:master May 7, 2020

ruflin pushed a commit to ruflin/beats that referenced this pull request May 7, 2020

[Libbeat] Respect the parameters option defined in the ES output. (e…

a676502

…lastic#18326) (cherry picked from commit 0575e15)

ph added v7.8.0 and removed needs_backport PR is waiting to be backported to other branches. labels May 8, 2020

ph mentioned this pull request May 8, 2020

[Libbeat] Add more complete tests for opt parameters in ES output #18393

Merged

6 tasks

ph added a commit that referenced this pull request May 11, 2020

[Libbeat] Respect the parameters option defined in the ES output. (#…

c8e8875

…18326) (#18348) (cherry picked from commit 0575e15) Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>

ph mentioned this pull request May 13, 2020

Cherry-pick #18393 to 7.x: [Libbeat] Add more complete tests for opt parameters in ES output #18491

Merged

6 tasks

ph mentioned this pull request May 13, 2020

Cherry-pick #18393 to 7.8: [Libbeat] Add more complete tests for opt parameters in ES output #18492

Merged

6 tasks

ycombinator mentioned this pull request May 27, 2020

Monitoring: only include X-Pack API params when appropriate #18787

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Libbeat] Respect the `parameters` option defined in the ES output. #18326

[Libbeat] Respect the `parameters` option defined in the ES output. #18326

ph commented May 6, 2020 •

edited

Loading

elasticmachine commented May 6, 2020

ph commented May 6, 2020

elasticmachine commented May 6, 2020 •

edited

Loading

Build stats

Test stats 🧪

ph commented May 6, 2020

ruflin commented May 7, 2020

[Libbeat] Respect the parameters option defined in the ES output. #18326

[Libbeat] Respect the parameters option defined in the ES output. #18326

Conversation

ph commented May 6, 2020 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine commented May 6, 2020

ph commented May 6, 2020

elasticmachine commented May 6, 2020 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

ph commented May 6, 2020

ruflin commented May 7, 2020

[Libbeat] Respect the `parameters` option defined in the ES output. #18326

[Libbeat] Respect the `parameters` option defined in the ES output. #18326

ph commented May 6, 2020 •

edited

Loading

elasticmachine commented May 6, 2020 •

edited

Loading