feat(http sink): Add configuration options `payload_prefix` and `payload_suffix`. #15696

jdiebold · 2022-12-22T11:10:37Z

resolves: #15651

bits-bot · 2022-12-22T11:10:41Z

All committers have signed the CLA.

netlify · 2022-12-22T11:10:41Z

✅ Deploy Preview for vrl-playground canceled.

Name	Link
🔨 Latest commit	`dc06a47`
🔍 Latest deploy log	https://app.netlify.com/sites/vrl-playground/deploys/63bf19b7720be60008f0e4f1

netlify · 2022-12-22T11:10:41Z

✅ Deploy Preview for vector-project canceled.

Name	Link
🔨 Latest commit	`dc06a47`
🔍 Latest deploy log	https://app.netlify.com/sites/vector-project/deploys/63bf19b7c60c2b0008fc5fc3

neuronull

Thanks for the contribution @jdiebold !
I think the additional functionality is a good enhancement.

Left a comment below regarding validation of input.

src/sinks/http.rs

neuronull

Looking good! Thanks for adding the validation and tests.

I am being particular about the doc comments for the new config options because we are currently undergoing an effort to make the external documentation for Vector at vector.dev , be generated automatically from these very comments in the configuration structures.
So these comments will be user facing.

On that subject- for now (we aren't done with that automated documentation generation yet) , this change should include these new options to the cue documentation under the /website directory for thus sink.

See https://github.com/vectordotdev/vector/blob/master/docs/DOCUMENTING.md for more information. make check-docs can be used to validate the cue file correctness.

One final note- I haven't approved CI to run on this yet, but I believe that clippy might complain about a thing or two. If you haven't already, do run make check and make check-clippy.

src/sinks/http.rs

jdiebold · 2023-01-10T12:38:45Z

Looking good! Thanks for adding the validation and tests.

I am being particular about the doc comments for the new config options because we are currently undergoing an effort to make the external documentation for Vector at vector.dev , be generated automatically from these very comments in the configuration structures. So these comments will be user facing.

On that subject- for now (we aren't done with that automated documentation generation yet) , this change should include these new options to the cue documentation under the /website directory for thus sink.

See https://github.com/vectordotdev/vector/blob/master/docs/DOCUMENTING.md for more information. make check-docs can be used to validate the cue file correctness.

One final note- I haven't approved CI to run on this yet, but I believe that clippy might complain about a thing or two. If you haven't already, do run make check and make check-clippy.

make check and make check-clippy ran without remark. I still get a lot of errors for make check-docs, but I think they are all from other files (they are also there when i remove roll my changes to http.cue back).
they are all in the form of the following for a lot of components:

components.sources.stdin.configuration.multiline: operand required of '!' not concrete (was bool):
    ./website/cue/reference.cue:331:6

changes made

github-actions · 2023-01-10T18:51:11Z

Regression Test Results

Run ID: fc82d1a1-bc79-4117-bb04-8886ce65161d
Baseline: d1f9d17
Comparison: 9f31aa3
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

Changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%:

experiment	Δ mean	Δ mean %	confidence
syslog_log2metric_humio_metrics	-359.14KiB/CPU-s	-5.64	100.00%

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
splunk_hec_route_s3	121.65KiB/CPU-s	1.03	100.00%	11.54MiB/CPU-s	621.07KiB/CPU-s	8.01KiB/CPU-s	0.052559	11.66MiB/CPU-s	578.5KiB/CPU-s	7.47KiB/CPU-s	0.048458	False	False
syslog_regex_logs2metric_ddmetrics	35.87KiB/CPU-s	0.97	100.00%	3.62MiB/CPU-s	407.24KiB/CPU-s	5.26KiB/CPU-s	0.109788	3.66MiB/CPU-s	436.01KiB/CPU-s	5.63KiB/CPU-s	0.116417	True	False
syslog_log2metric_splunk_hec_metrics	54.29KiB/CPU-s	0.57	100.00%	9.29MiB/CPU-s	325.48KiB/CPU-s	4.2KiB/CPU-s	0.034193	9.35MiB/CPU-s	459.73KiB/CPU-s	5.94KiB/CPU-s	0.048024	False	False
http_to_http_json	59.06KiB/CPU-s	0.43	100.00%	13.41MiB/CPU-s	463.37KiB/CPU-s	5.98KiB/CPU-s	0.03373	13.47MiB/CPU-s	406.0KiB/CPU-s	5.24KiB/CPU-s	0.029428	False	False
datadog_agent_remap_datadog_logs_acks	78.81KiB/CPU-s	0.23	99.85%	33.9MiB/CPU-s	1.41MiB/CPU-s	18.63KiB/CPU-s	0.041611	33.97MiB/CPU-s	1.24MiB/CPU-s	16.41KiB/CPU-s	0.036541	False	False
http_to_http_acks	9.26KiB/CPU-s	0.17	14.34%	5.24MiB/CPU-s	2.77MiB/CPU-s	36.65KiB/CPU-s	0.52923	5.25MiB/CPU-s	2.71MiB/CPU-s	35.8KiB/CPU-s	0.515901	True	False
socket_to_socket_blackhole	18.61KiB/CPU-s	0.13	99.86%	13.61MiB/CPU-s	348.12KiB/CPU-s	4.49KiB/CPU-s	0.02497	13.63MiB/CPU-s	285.27KiB/CPU-s	3.68KiB/CPU-s	0.020435	False	False
enterprise_http_to_http	14.63KiB/CPU-s	0.1	99.30%	13.61MiB/CPU-s	373.03KiB/CPU-s	4.81KiB/CPU-s	0.026768	13.62MiB/CPU-s	193.12KiB/CPU-s	2.49KiB/CPU-s	0.013844	False	False
splunk_hec_indexer_ack_blackhole	2.78KiB/CPU-s	0.02	46.13%	13.62MiB/CPU-s	255.11KiB/CPU-s	3.29KiB/CPU-s	0.018295	13.62MiB/CPU-s	240.62KiB/CPU-s	3.1KiB/CPU-s	0.017252	False	False
splunk_hec_to_splunk_hec_logs_noack	2.9KiB/CPU-s	0.02	56.35%	13.62MiB/CPU-s	216.31KiB/CPU-s	2.79KiB/CPU-s	0.015506	13.63MiB/CPU-s	191.72KiB/CPU-s	2.48KiB/CPU-s	0.01374	False	False
syslog_splunk_hec_logs	259.33B/CPU-s	0.0	3.80%	8.96MiB/CPU-s	260.61KiB/CPU-s	3.36KiB/CPU-s	0.028405	8.96MiB/CPU-s	319.27KiB/CPU-s	4.12KiB/CPU-s	0.034798	False	False
http_to_http_noack	53.52B/CPU-s	0.0	0.75%	13.61MiB/CPU-s	306.93KiB/CPU-s	3.96KiB/CPU-s	0.022015	13.61MiB/CPU-s	305.13KiB/CPU-s	3.94KiB/CPU-s	0.021886	False	False
fluent_elasticsearch	120.96B/CPU-s	0.0	17.23%	45.41MiB/CPU-s	30.18KiB/CPU-s	394.52B/CPU-s	0.000649	45.41MiB/CPU-s	29.96KiB/CPU-s	391.45B/CPU-s	0.000644	False	False
file_to_blackhole	561.55B/CPU-s	0.0	2.00%	54.5MiB/CPU-s	1.16MiB/CPU-s	15.33KiB/CPU-s	0.021293	54.5MiB/CPU-s	1.18MiB/CPU-s	15.61KiB/CPU-s	0.021708	False	False
splunk_hec_to_splunk_hec_logs_acks	-2.22KiB/CPU-s	-0.02	26.49%	13.62MiB/CPU-s	350.44KiB/CPU-s	4.52KiB/CPU-s	0.025131	13.61MiB/CPU-s	369.5KiB/CPU-s	4.77KiB/CPU-s	0.026502	False	False
syslog_loki	-30.47KiB/CPU-s	-0.33	100.00%	8.94MiB/CPU-s	164.19KiB/CPU-s	2.12KiB/CPU-s	0.017931	8.91MiB/CPU-s	140.58KiB/CPU-s	1.81KiB/CPU-s	0.015404	False	False
syslog_humio_logs	-40.06KiB/CPU-s	-0.43	100.00%	9.17MiB/CPU-s	174.75KiB/CPU-s	2.26KiB/CPU-s	0.018612	9.13MiB/CPU-s	451.46KiB/CPU-s	5.83KiB/CPU-s	0.04829	False	False
otlp_http_to_blackhole	-8.49KiB/CPU-s	-0.54	99.99%	1.55MiB/CPU-s	111.96KiB/CPU-s	1.45KiB/CPU-s	0.070675	1.54MiB/CPU-s	118.84KiB/CPU-s	1.53KiB/CPU-s	0.075419	False	False
datadog_agent_remap_blackhole_acks	-191.22KiB/CPU-s	-0.6	100.00%	30.9MiB/CPU-s	1.38MiB/CPU-s	18.3KiB/CPU-s	0.04478	30.71MiB/CPU-s	1.45MiB/CPU-s	19.22KiB/CPU-s	0.047346	False	False
otlp_grpc_to_blackhole	-9.69KiB/CPU-s	-0.9	100.00%	1.05MiB/CPU-s	38.11KiB/CPU-s	503.73B/CPU-s	0.03557	1.04MiB/CPU-s	47.07KiB/CPU-s	622.15B/CPU-s	0.04434	False	False
datadog_agent_remap_datadog_logs	-316.22KiB/CPU-s	-0.92	100.00%	33.57MiB/CPU-s	1.48MiB/CPU-s	19.59KiB/CPU-s	0.044174	33.27MiB/CPU-s	1.71MiB/CPU-s	22.54KiB/CPU-s	0.051284	False	False
http_text_to_http_json	-388.76KiB/CPU-s	-1.48	100.00%	25.59MiB/CPU-s	598.55KiB/CPU-s	7.73KiB/CPU-s	0.022836	25.22MiB/CPU-s	619.07KiB/CPU-s	7.99KiB/CPU-s	0.023974	False	False
datadog_agent_remap_blackhole	-872.46KiB/CPU-s	-2.72	100.00%	31.32MiB/CPU-s	1.11MiB/CPU-s	14.63KiB/CPU-s	0.035337	30.47MiB/CPU-s	1.38MiB/CPU-s	18.2KiB/CPU-s	0.045177	False	False
syslog_log2metric_humio_metrics	-359.14KiB/CPU-s	-5.64	100.00%	6.21MiB/CPU-s	139.45KiB/CPU-s	1.8KiB/CPU-s	0.021915	5.86MiB/CPU-s	352.34KiB/CPU-s	4.55KiB/CPU-s	0.058682	False	False

neuronull · 2023-01-10T20:35:21Z

make check and make check-clippy ran without remark. I still get a lot of errors for make check-docs, but I think they are all from other files (they are also there when i remove roll my changes to http.cue back). they are all in the form of the following for a lot of components:
components.sources.stdin.configuration.multiline: operand required of '!' not concrete (was bool):
    ./website/cue/reference.cue:331:6

Are you by chance developing on macOS ?

neuronull · 2023-01-10T23:00:20Z

make check and make check-clippy ran without remark. I still get a lot of errors for make check-docs, but I think they are all from other files (they are also there when i remove roll my changes to http.cue back). they are all in the form of the following for a lot of components:
components.sources.stdin.configuration.multiline: operand required of '!' not concrete (was bool):
    ./website/cue/reference.cue:331:6
Are you by chance developing on macOS ?

Actually, it's more specifically likely a versioning issue with cue. We use v0.4.2 in CI so if you are not using that, I'd suggest changing to use that version.

neuronull

I updated the cue and just have a final hygiene related comment that I didn't notice earlier.

src/sinks/http.rs

Co-authored-by: neuronull <kyle.criddle@datadoghq.com>

jdiebold · 2023-01-11T07:27:53Z

make check and make check-clippy ran without remark. I still get a lot of errors for make check-docs, but I think they are all from other files (they are also there when i remove roll my changes to http.cue back). they are all in the form of the following for a lot of components:
components.sources.stdin.configuration.multiline: operand required of '!' not concrete (was bool):
    ./website/cue/reference.cue:331:6
Are you by chance developing on macOS ?
Actually, it's more specifically likely a versioning issue with cue. We use v0.4.2 in CI so if you are not using that, I'd suggest changing to use that version.

Yes, i did and the validation was successful

jdiebold · 2023-01-11T09:49:24Z

maybe it's worth adding the cue version to the DOCUMENTING.md to avoid the problem next time.

neuronull · 2023-01-11T16:28:36Z

maybe it's worth adding the cue version to the DOCUMENTING.md to avoid the problem next time.

Yes, looking into the best ways to communicate/validate that 👍

neuronull · 2023-01-11T16:47:45Z

looks like code formatting is off, try running make check-fmt

github-actions · 2023-01-11T17:14:45Z

Regression Test Results

Run ID: cf166cef-ae40-4514-be69-998d5ad300c9
Baseline: 135fe23
Comparison: 229cceb
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%.

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
splunk_hec_route_s3	449.2KiB/CPU-s	3.9	100.00%	11.25MiB/CPU-s	809.42KiB/CPU-s	10.44KiB/CPU-s	0.070252	11.69MiB/CPU-s	581.31KiB/CPU-s	7.5KiB/CPU-s	0.04856	False	False
http_text_to_http_json	871.93KiB/CPU-s	3.44	100.00%	24.77MiB/CPU-s	776.77KiB/CPU-s	10.03KiB/CPU-s	0.030626	25.62MiB/CPU-s	751.46KiB/CPU-s	9.69KiB/CPU-s	0.028643	False	False
syslog_splunk_hec_logs	270.46KiB/CPU-s	2.97	100.00%	8.89MiB/CPU-s	246.73KiB/CPU-s	3.19KiB/CPU-s	0.027092	9.16MiB/CPU-s	264.86KiB/CPU-s	3.42KiB/CPU-s	0.028243	False	False
syslog_log2metric_splunk_hec_metrics	236.19KiB/CPU-s	2.52	100.00%	9.16MiB/CPU-s	391.17KiB/CPU-s	5.05KiB/CPU-s	0.041709	9.39MiB/CPU-s	343.13KiB/CPU-s	4.43KiB/CPU-s	0.035687	False	False
otlp_http_to_blackhole	28.68KiB/CPU-s	1.86	100.00%	1.5MiB/CPU-s	120.66KiB/CPU-s	1.56KiB/CPU-s	0.078405	1.53MiB/CPU-s	125.97KiB/CPU-s	1.63KiB/CPU-s	0.080361	False	False
otlp_grpc_to_blackhole	17.67KiB/CPU-s	1.68	100.00%	1.03MiB/CPU-s	39.56KiB/CPU-s	522.95B/CPU-s	0.037628	1.04MiB/CPU-s	55.0KiB/CPU-s	726.78B/CPU-s	0.051454	False	False
datadog_agent_remap_blackhole	483.51KiB/CPU-s	1.56	100.00%	30.21MiB/CPU-s	1.8MiB/CPU-s	23.85KiB/CPU-s	0.05973	30.68MiB/CPU-s	1.63MiB/CPU-s	21.51KiB/CPU-s	0.053038	False	False
datadog_agent_remap_blackhole_acks	465.66KiB/CPU-s	1.46	100.00%	31.14MiB/CPU-s	1.46MiB/CPU-s	19.24KiB/CPU-s	0.046719	31.6MiB/CPU-s	953.79KiB/CPU-s	12.31KiB/CPU-s	0.029476	False	False
socket_to_socket_blackhole	143.26KiB/CPU-s	1.04	100.00%	13.41MiB/CPU-s	250.57KiB/CPU-s	3.23KiB/CPU-s	0.018248	13.55MiB/CPU-s	395.83KiB/CPU-s	5.11KiB/CPU-s	0.028529	False	False
syslog_loki	88.0KiB/CPU-s	0.97	100.00%	8.83MiB/CPU-s	156.06KiB/CPU-s	2.01KiB/CPU-s	0.01725	8.92MiB/CPU-s	199.83KiB/CPU-s	2.58KiB/CPU-s	0.021876	False	False
syslog_humio_logs	24.22KiB/CPU-s	0.26	100.00%	9.1MiB/CPU-s	163.25KiB/CPU-s	2.11KiB/CPU-s	0.017525	9.12MiB/CPU-s	349.37KiB/CPU-s	4.51KiB/CPU-s	0.037407	False	False
splunk_hec_to_splunk_hec_logs_noack	9.0KiB/CPU-s	0.06	97.04%	13.62MiB/CPU-s	256.44KiB/CPU-s	3.31KiB/CPU-s	0.01839	13.63MiB/CPU-s	192.05KiB/CPU-s	2.48KiB/CPU-s	0.013764	False	False
enterprise_http_to_http	4.67KiB/CPU-s	0.03	82.31%	13.62MiB/CPU-s	224.08KiB/CPU-s	2.89KiB/CPU-s	0.016066	13.62MiB/CPU-s	147.28KiB/CPU-s	1.9KiB/CPU-s	0.010556	False	False
http_to_http_noack	4.02KiB/CPU-s	0.03	56.73%	13.61MiB/CPU-s	305.34KiB/CPU-s	3.94KiB/CPU-s	0.021901	13.62MiB/CPU-s	254.41KiB/CPU-s	3.28KiB/CPU-s	0.018243	False	False
splunk_hec_to_splunk_hec_logs_acks	2.02KiB/CPU-s	0.01	25.83%	13.61MiB/CPU-s	340.34KiB/CPU-s	4.39KiB/CPU-s	0.02441	13.62MiB/CPU-s	332.2KiB/CPU-s	4.29KiB/CPU-s	0.023823	False	False
splunk_hec_indexer_ack_blackhole	1.34KiB/CPU-s	0.01	23.35%	13.62MiB/CPU-s	250.77KiB/CPU-s	3.23KiB/CPU-s	0.017983	13.62MiB/CPU-s	245.38KiB/CPU-s	3.17KiB/CPU-s	0.017595	False	False
fluent_elasticsearch	273.47B/CPU-s	0.0	37.62%	45.41MiB/CPU-s	30.31KiB/CPU-s	396.27B/CPU-s	0.000652	45.41MiB/CPU-s	30.01KiB/CPU-s	392.2B/CPU-s	0.000645	False	False
file_to_blackhole	2.74KiB/CPU-s	0.0	10.22%	54.5MiB/CPU-s	1.18MiB/CPU-s	15.59KiB/CPU-s	0.021659	54.5MiB/CPU-s	1.11MiB/CPU-s	14.62KiB/CPU-s	0.020319	False	False
datadog_agent_remap_datadog_logs_acks	-157.62KiB/CPU-s	-0.46	100.00%	33.47MiB/CPU-s	1.34MiB/CPU-s	17.72KiB/CPU-s	0.040069	33.31MiB/CPU-s	1.54MiB/CPU-s	20.35KiB/CPU-s	0.046224	False	False
datadog_agent_remap_datadog_logs	-274.8KiB/CPU-s	-0.79	100.00%	33.86MiB/CPU-s	1.29MiB/CPU-s	17.06KiB/CPU-s	0.038144	33.59MiB/CPU-s	1.96MiB/CPU-s	25.89KiB/CPU-s	0.058335	False	False
http_to_http_json	-114.87KiB/CPU-s	-0.83	100.00%	13.56MiB/CPU-s	308.18KiB/CPU-s	3.98KiB/CPU-s	0.022189	13.45MiB/CPU-s	406.76KiB/CPU-s	5.25KiB/CPU-s	0.029531	False	False
http_to_http_acks	-111.1KiB/CPU-s	-2.03	97.07%	5.34MiB/CPU-s	2.76MiB/CPU-s	36.5KiB/CPU-s	0.51704	5.23MiB/CPU-s	2.69MiB/CPU-s	35.59KiB/CPU-s	0.514524	True	False
syslog_regex_logs2metric_ddmetrics	-86.84KiB/CPU-s	-2.23	100.00%	3.8MiB/CPU-s	464.08KiB/CPU-s	5.99KiB/CPU-s	0.119202	3.72MiB/CPU-s	465.52KiB/CPU-s	6.01KiB/CPU-s	0.122298	True	False
syslog_log2metric_humio_metrics	-216.64KiB/CPU-s	-3.42	100.00%	6.19MiB/CPU-s	144.01KiB/CPU-s	1.86KiB/CPU-s	0.022719	5.98MiB/CPU-s	333.82KiB/CPU-s	4.31KiB/CPU-s	0.054528	False	False

neuronull · 2023-01-11T18:52:24Z

maybe it's worth adding the cue version to the DOCUMENTING.md to avoid the problem next time.

Yes, looking into the best ways to communicate/validate that 👍

See https://github.com/vectordotdev/vector/blob/master/docs/DOCUMENTING.md#installing-cue

changes made

github-actions · 2023-01-11T21:12:10Z

Regression Test Results

Run ID: 6a81a309-0f5a-4376-ad65-3ae9cd74c398
Baseline: 19e9102
Comparison: dc06a47
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

Changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%:

experiment	Δ mean	Δ mean %	confidence
http_text_to_http_json	1.65MiB/CPU-s	6.87	100.00%

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
http_text_to_http_json	1.65MiB/CPU-s	6.87	100.00%	24.08MiB/CPU-s	585.8KiB/CPU-s	7.56KiB/CPU-s	0.023751	25.74MiB/CPU-s	677.98KiB/CPU-s	8.75KiB/CPU-s	0.025721	False	False
syslog_splunk_hec_logs	409.91KiB/CPU-s	4.55	100.00%	8.8MiB/CPU-s	239.88KiB/CPU-s	3.1KiB/CPU-s	0.026625	9.2MiB/CPU-s	194.88KiB/CPU-s	2.51KiB/CPU-s	0.020689	False	False
socket_to_socket_blackhole	374.45KiB/CPU-s	2.75	100.00%	13.28MiB/CPU-s	309.61KiB/CPU-s	4.0KiB/CPU-s	0.02276	13.65MiB/CPU-s	317.42KiB/CPU-s	4.1KiB/CPU-s	0.022708	False	False
otlp_http_to_blackhole	34.01KiB/CPU-s	2.2	100.00%	1.51MiB/CPU-s	121.21KiB/CPU-s	1.56KiB/CPU-s	0.078278	1.55MiB/CPU-s	109.81KiB/CPU-s	1.42KiB/CPU-s	0.069393	False	False
datadog_agent_remap_blackhole_acks	632.98KiB/CPU-s	2.01	100.00%	30.73MiB/CPU-s	1.4MiB/CPU-s	18.51KiB/CPU-s	0.045564	31.35MiB/CPU-s	1.3MiB/CPU-s	17.19KiB/CPU-s	0.041476	False	False
syslog_log2metric_splunk_hec_metrics	186.94KiB/CPU-s	1.99	100.00%	9.19MiB/CPU-s	290.63KiB/CPU-s	3.75KiB/CPU-s	0.030891	9.37MiB/CPU-s	306.85KiB/CPU-s	3.96KiB/CPU-s	0.031979	False	False
syslog_loki	140.73KiB/CPU-s	1.56	100.00%	8.83MiB/CPU-s	111.78KiB/CPU-s	1.44KiB/CPU-s	0.012364	8.96MiB/CPU-s	131.67KiB/CPU-s	1.7KiB/CPU-s	0.014342	False	False
syslog_humio_logs	141.88KiB/CPU-s	1.56	100.00%	8.89MiB/CPU-s	282.0KiB/CPU-s	3.64KiB/CPU-s	0.030991	9.02MiB/CPU-s	453.31KiB/CPU-s	5.85KiB/CPU-s	0.049054	False	False
syslog_regex_logs2metric_ddmetrics	53.17KiB/CPU-s	1.5	100.00%	3.46MiB/CPU-s	550.65KiB/CPU-s	7.11KiB/CPU-s	0.155233	3.52MiB/CPU-s	521.98KiB/CPU-s	6.74KiB/CPU-s	0.144976	True	False
syslog_log2metric_humio_metrics	86.01KiB/CPU-s	1.48	100.00%	5.69MiB/CPU-s	388.34KiB/CPU-s	5.01KiB/CPU-s	0.066671	5.77MiB/CPU-s	343.99KiB/CPU-s	4.44KiB/CPU-s	0.058198	False	False
splunk_hec_route_s3	171.05KiB/CPU-s	1.43	100.00%	11.65MiB/CPU-s	671.85KiB/CPU-s	8.67KiB/CPU-s	0.05633	11.81MiB/CPU-s	543.68KiB/CPU-s	7.02KiB/CPU-s	0.044939	False	False
otlp_grpc_to_blackhole	11.3KiB/CPU-s	1.07	100.00%	1.03MiB/CPU-s	41.01KiB/CPU-s	542.07B/CPU-s	0.038733	1.04MiB/CPU-s	52.6KiB/CPU-s	694.98B/CPU-s	0.04916	False	False
datadog_agent_remap_datadog_logs_acks	281.0KiB/CPU-s	0.82	100.00%	33.61MiB/CPU-s	1.22MiB/CPU-s	16.06KiB/CPU-s	0.036155	33.89MiB/CPU-s	1.13MiB/CPU-s	14.9KiB/CPU-s	0.033282	False	False
enterprise_http_to_http	5.98KiB/CPU-s	0.04	82.88%	13.62MiB/CPU-s	278.67KiB/CPU-s	3.6KiB/CPU-s	0.019985	13.62MiB/CPU-s	191.99KiB/CPU-s	2.48KiB/CPU-s	0.013763	False	False
splunk_hec_to_splunk_hec_logs_noack	5.86KiB/CPU-s	0.04	86.78%	13.62MiB/CPU-s	234.21KiB/CPU-s	3.02KiB/CPU-s	0.016792	13.63MiB/CPU-s	189.88KiB/CPU-s	2.45KiB/CPU-s	0.013608	False	False
splunk_hec_to_splunk_hec_logs_acks	789.01B/CPU-s	0.01	10.23%	13.62MiB/CPU-s	331.77KiB/CPU-s	4.28KiB/CPU-s	0.02379	13.62MiB/CPU-s	324.84KiB/CPU-s	4.19KiB/CPU-s	0.023292	False	False
splunk_hec_indexer_ack_blackhole	1.05KiB/CPU-s	0.01	18.00%	13.62MiB/CPU-s	254.53KiB/CPU-s	3.28KiB/CPU-s	0.018254	13.62MiB/CPU-s	251.03KiB/CPU-s	3.24KiB/CPU-s	0.018002	False	False
fluent_elasticsearch	253.02B/CPU-s	0.0	35.14%	45.41MiB/CPU-s	29.9KiB/CPU-s	390.9B/CPU-s	0.000643	45.41MiB/CPU-s	30.15KiB/CPU-s	394.18B/CPU-s	0.000648	False	False
file_to_blackhole	-6.92KiB/CPU-s	-0.01	24.60%	54.5MiB/CPU-s	1.12MiB/CPU-s	14.78KiB/CPU-s	0.020527	54.49MiB/CPU-s	1.24MiB/CPU-s	16.39KiB/CPU-s	0.022792	False	False
http_to_http_noack	-1.92KiB/CPU-s	-0.01	24.20%	13.61MiB/CPU-s	332.36KiB/CPU-s	4.29KiB/CPU-s	0.023843	13.61MiB/CPU-s	350.85KiB/CPU-s	4.53KiB/CPU-s	0.025173	False	False
http_to_http_json	-26.33KiB/CPU-s	-0.19	100.00%	13.62MiB/CPU-s	211.64KiB/CPU-s	2.73KiB/CPU-s	0.015171	13.6MiB/CPU-s	253.27KiB/CPU-s	3.27KiB/CPU-s	0.01819	False	False
datadog_agent_remap_datadog_logs	-212.65KiB/CPU-s	-0.61	100.00%	34.13MiB/CPU-s	1.17MiB/CPU-s	15.43KiB/CPU-s	0.034227	33.92MiB/CPU-s	1.44MiB/CPU-s	19.06KiB/CPU-s	0.042524	False	False
http_to_http_acks	-34.91KiB/CPU-s	-0.65	49.92%	5.28MiB/CPU-s	2.8MiB/CPU-s	36.97KiB/CPU-s	0.530134	5.24MiB/CPU-s	2.75MiB/CPU-s	36.37KiB/CPU-s	0.524932	True	False
datadog_agent_remap_blackhole	-303.0KiB/CPU-s	-0.96	100.00%	30.79MiB/CPU-s	1.31MiB/CPU-s	17.29KiB/CPU-s	0.042454	30.5MiB/CPU-s	1.43MiB/CPU-s	18.9KiB/CPU-s	0.046891	False	False

neuronull

Thanks for the contribution @jdiebold !

jdiebold · 2023-01-11T21:58:17Z

Thanks, @neuronull! It was fun working on it!

feat(sinks): add configurable payload_prefix and _suffix for http sink

efe50ac

github-actions bot added the domain: sinks Anything related to the Vector's sinks label Dec 22, 2022

jszwedko requested a review from neuronull December 22, 2022 17:04

jszwedko assigned neuronull Dec 22, 2022

Merge branch 'master' into 15651-prefix-suffix-http-sink-payload

921dadf

neuronull reviewed Jan 4, 2023

View reviewed changes

src/sinks/http.rs Outdated Show resolved Hide resolved

src/sinks/http.rs Show resolved Hide resolved

jdiebold added 3 commits January 6, 2023 10:09

add validation to produce valid json payload

462a10d

only validate payload wrapper if character delimited json encoding

8a3f166

add comments for config about validation

2c31dc6

neuronull self-requested a review January 6, 2023 18:19

neuronull previously requested changes Jan 9, 2023

View reviewed changes

src/sinks/http.rs Outdated Show resolved Hide resolved

src/sinks/http.rs Outdated Show resolved Hide resolved

src/sinks/http.rs Outdated Show resolved Hide resolved

src/sinks/http.rs Outdated Show resolved Hide resolved

src/sinks/http.rs Outdated Show resolved Hide resolved

jdiebold added 2 commits January 10, 2023 11:03

refactor validation

415cd8f

docs: add docs for new http sink configs

1091a97

github-actions bot added the domain: external docs Anything related to Vector's external, public documentation label Jan 10, 2023

docs: add suggestions to config doc comment

a399b0b

neuronull added 2 commits January 10, 2023 09:44

Merge branch 'master' into 15651-prefix-suffix-http-sink-payload

736c58e

update docs

48eb240

neuronull changed the title ~~feat(sinks): Add configurable payload_prefix and _suffix for HTTP sink~~ feat(http sink): Add configuration options payload_prefix and payload_suffix. Jan 10, 2023

neuronull self-requested a review January 10, 2023 17:54

revert host_metrics cue

9f31aa3

neuronull previously requested changes Jan 11, 2023

View reviewed changes

src/sinks/http.rs Outdated Show resolved Hide resolved

Update src/sinks/http.rs

229cceb

Co-authored-by: neuronull <kyle.criddle@datadoghq.com>

neuronull self-requested a review January 11, 2023 16:29

neuronull mentioned this pull request Jan 11, 2023

Properly versioning external tool dependencies #15909

Open

fix formatting

dc06a47

neuronull approved these changes Jan 11, 2023

View reviewed changes

neuronull merged commit 1da9769 into vectordotdev:master Jan 11, 2023

jdiebold deleted the 15651-prefix-suffix-http-sink-payload branch January 12, 2023 08:21

Anonymodesu mentioned this pull request Jan 30, 2023

Custom request body for HTTP sink #15963

Closed

jszwedko mentioned this pull request Feb 28, 2023

Add the ability to wrap batches in an envelope #16637

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(http sink): Add configuration options `payload_prefix` and `payload_suffix`. #15696

feat(http sink): Add configuration options `payload_prefix` and `payload_suffix`. #15696

jdiebold commented Dec 22, 2022 •

edited by neuronull

Loading

bits-bot commented Dec 22, 2022 •

edited

Loading

netlify bot commented Dec 22, 2022 •

edited

Loading

netlify bot commented Dec 22, 2022 •

edited

Loading

neuronull left a comment

neuronull left a comment

jdiebold commented Jan 10, 2023

github-actions bot commented Jan 10, 2023

neuronull commented Jan 10, 2023 •

edited

Loading

neuronull commented Jan 10, 2023

neuronull left a comment

jdiebold commented Jan 11, 2023

jdiebold commented Jan 11, 2023

neuronull commented Jan 11, 2023

neuronull commented Jan 11, 2023

github-actions bot commented Jan 11, 2023

neuronull commented Jan 11, 2023

github-actions bot commented Jan 11, 2023

neuronull left a comment

jdiebold commented Jan 11, 2023

feat(http sink): Add configuration options payload_prefix and payload_suffix. #15696

feat(http sink): Add configuration options payload_prefix and payload_suffix. #15696

Conversation

jdiebold commented Dec 22, 2022 • edited by neuronull Loading

bits-bot commented Dec 22, 2022 • edited Loading

netlify bot commented Dec 22, 2022 • edited Loading

✅ Deploy Preview for vrl-playground canceled.

netlify bot commented Dec 22, 2022 • edited Loading

✅ Deploy Preview for vector-project canceled.

neuronull left a comment

Choose a reason for hiding this comment

neuronull left a comment

Choose a reason for hiding this comment

jdiebold commented Jan 10, 2023

github-actions bot commented Jan 10, 2023

Regression Test Results

neuronull commented Jan 10, 2023 • edited Loading

neuronull commented Jan 10, 2023

neuronull left a comment

Choose a reason for hiding this comment

jdiebold commented Jan 11, 2023

jdiebold commented Jan 11, 2023

neuronull commented Jan 11, 2023

neuronull commented Jan 11, 2023

github-actions bot commented Jan 11, 2023

Regression Test Results

neuronull commented Jan 11, 2023

github-actions bot commented Jan 11, 2023

Regression Test Results

neuronull left a comment

Choose a reason for hiding this comment

jdiebold commented Jan 11, 2023

feat(http sink): Add configuration options `payload_prefix` and `payload_suffix`. #15696

feat(http sink): Add configuration options `payload_prefix` and `payload_suffix`. #15696

jdiebold commented Dec 22, 2022 •

edited by neuronull

Loading

bits-bot commented Dec 22, 2022 •

edited

Loading

netlify bot commented Dec 22, 2022 •

edited

Loading

netlify bot commented Dec 22, 2022 •

edited

Loading

neuronull commented Jan 10, 2023 •

edited

Loading