Feat: data-stream support + logging improvements #988
Conversation
kares
changed the title
kares
changed the title
kares
changed the title
resolves running bare bone against LS 8.0 SNAPSHOT: ``` NoMethodError: undefined method `stoppable_sleep' for Stud:Module ```
before output would only wait for template to be installed
these will be checked as data-stream compatible
also re-arranged pool code for better readability
to reduce ERROR logging noise in specs
and also extract after successful connection parts
we've seen some weird behavior on ES 7.10.2 where a _bulk request to index 69 documents returned 135 entries this leads to an ugly `NoMethodError: undefined method ``[]' for nil:NilClass` resolves logstash-plugins#989
due failures when spec was run individually
this will play fine with LS' existing overrides e.g.
```
es = LogStash::Outputs::ElasticSearch.new(@es_options)
es.instance_eval { @logger = logger }
es.build_client
```
| @@ -198,12 +198,16 @@ | |||
| let (:settings) { super().merge!({ 'ilm_enabled' => true }) } | |||
|
|
|||
| it 'should raise a configuration error' do | |||
| # TODO should be refactored not to rely on plugin internals | |||
There was a problem hiding this comment.
integration test was relying on a raise from the other thread to propagate, which is problematic.
the new logic to rescue everything from the async thread and propagate exceptions back as thread.value - making it a bit hard to test -> spec example should be rewritten or migrated to a unit test.
There was a problem hiding this comment.
Covers the basics and builds cleanly. I recommend that we merge this huge body of work when it's ready, and immediately follow up with a docs-only PR. Suggestions for enhancements are here: #1006. These kinds of suggestions/enhancements are hard to notate and implement in the github workflow. LMKWYT about this approach.
UPDATED: Please add a data streams section (similar to the ILM section), and include the two configs from the issue. We can handle and track the other doc improvements as part of the follow up issue: #1006
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Data stream support, specification elastic/logstash#12178
pending issue: can not check ES version when hooking up plugin #1001
use at leastrequire_alias=truewith_bulkwhen using DS?!?DS configuration diverged from the proposed implementation plan:
Simply because it seemed fine to just detect the configuration directly, given:
LogStash::PluginMixins::ElasticSearch::APIConfigsmixinthere's a
APIConfigs::CONFIG_PARAMSconvention to make sure introducing a new param will not break DS detectiondata_stream_prefix convention for DS specific configuration paramsInitially I extracted a base-line for the later
include DataStreamSupportbut I've run into (existing) spec problems due mocking. I decided to fix these along the way and also bring back logging which was disabled previously due tests.Splitting this work into separate PR would require extra work as spec improvements rely on some of the DS related refactorinfs for the updated mocking, thus its all here with minor changes besides DS support, namely:
happening after successful_connection(from install template etc.).or!logical ends(might be just me but I never appreciate logs 'screaming' at me with
!)private(with respect to LS re-use in monitoring)