feat(sinks elasticsearch): Add compatibility between elasticsearch cl… #13881
Conversation
|
|
✅ Deploy Preview for vector-project canceled.
|
…ient in 7.x and server in 7.x or 8.x Signed-off-by: guillaume.micouin-jorda <guillaume.micouin-jorda@renault.com>
gmicouin
force-pushed
the
dev/gmic/compatibility_es_v8
branch
from
74529ab
to
13ecb12
Compare
|
@gmicouin Do you mind providing references to the documentation used to show why this should be valid? |
tobz
added
the
ci-condition: integration tests enable
Soak Test ResultsBaseline: 4611307 ExplanationA soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core. The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed. Changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
|
Soak Test ResultsBaseline: bdff6b8 ExplanationA soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core. The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed. No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%: Fine details of change detection per experiment.
|
Soak Test ResultsBaseline: 4611307 ExplanationA soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core. The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed. No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%: Fine details of change detection per experiment.
|
|
Hi @tobz ! And here the code of official python client which replaces the Content-Type and Accept header by "application/vnd.elasticsearch+json;compatible-with=7" I'm newbie in Rust, perhaps a better way exists to do that ;) |
|
@gmicouin No worries, your change is totally valid. My main question would be: it sounds like, from the PR description, you tested this against Elasticsearch 7.12, but did you also test it against 8.x? Did it not work with 8.x without this change? From reading the REST API compatibility documentation, it appears that there also needs to be an |
|
@tobz Yes I test this code with Elastic 8.x, and without this change the Elastic server 8.0 does not accept the request. Yes you're right, I have forgotten the Accept header, because during my tests it was possible to configure it with requests.header. Do you think it would be possible to configure the Content-Type in the vector.toml ? It would be more generic, what do you think ? |
@gmicouin OK, great. I'd say that this PR should also be setting the
It's technically possible, but it's not a change we would make. Elasticsearch only supports a given set of encodings, and we need to ensure the codec we use to generate bulk requests matches what we send in I think what we'd want to do, if anything, is expose a configuration setting that allows setting the version value that gets used in compatibility mode i.e. But like I said above, let's do the change to also update the |
|
Hi @tobz ! But I know this code doesn't compile, and I don't know how do that :s |
|
If we're considering adding a Going even further we should be able to query the cluster on startup and identify the version from that request and self-configure the sink at runtime, rather than having the user configure ahead of time. I haven't fully thought through the implications of this, and it's certainly a larger change. |
|
Hi @spencergilbert, |
|
We are closing this for now in favour of coming up with a better way to handling behaviour appropriately according to a specified or detected version of ElasticSearch. Ref #10342 |
Add compatibility between the elasticsearch client and elasticsearch server in 8.x and after 7.12 (tested from 7.12)