Allow templating headers in http sink

[binarylogic]

For certain services the ability to dynamically set HTTP headers based on log content is important. From a user on a related project:

I am sending Docker container logs to Sumo Logic, and would like to dynamically set log metadata based on the incoming log records. Sumo Logic's HTTP receivers accept custom source names, hosts, and categories by setting HTTP headers named X-Sumo-Name, X-Sumo-Host, X-Sumo-Category.

Describe the solution you'd like

I would like a way to set dynamically header values in the http output plugin. For example, let's say there is a record key named category. There should be a way to use this as the header value.

[1154103724, {"key1"=>12345, "category"=>"abc"}]

[sinks.sumologic]
    Name           http
    Match          *
    Host           127.0.0.1
    Port           9000
    Header         X-Key-A Value_A
    Header_record  X-Key-B category
    URI            /something

In the above configuration, the header X-Key-A would have the value Value_A, but X-Key-B would have the value abc from the record.

Additional context

Without the ability to customize the log metadata via headers, I cannot effectively sort and manage logs in Sumo Logic.

[binarylogic]

Thinking about this more, how do we handle multiple different values? Does it create multiple buffers / requests? In a way, I'm thinking this should be addressed with a higher-level feature like a "group_by" transform.

[jalaziz]

Still new to Rust, but could this be solved with the relatively new PartitionHttpSink?

[binarylogic]

cc @bruceg on that one.

[bruceg]

Yes, PartitionHttpSink looks to be the ideal target for this kind of modification, and would end up being ideal sample code for other HTTP sinks to adapt when they need this kind of enhancement. The key to the partition types would be the HTTP headers so that the sink batches events with like headers together.

[jszwedko]

Closed by #7189

[jalaziz]

@jszwedko Has this actually been fixed? #7189 was an issue that was closed in favor of this one. I can't seem to find a PR related to either.

[jszwedko]

Hi @jalaziz !

Apologies, you are right, I misread this as just allowing headers to be set, but I see this is about allowing templated HTTP headers. I'll reopen.

[goedelsoup-clari]

The original request is specifically to support the use of dynamic source categories within SumoLogic. Is templating support through the use of PartitionHttpSink still preferred or would a SumoLogic logs sink be another option at this point?

We are beginning to launch a Vector-first implementation of our observability topology on EKS and while we can use the current HTTP sink in conjunction with Field Extraction Rules to make things palatable, it becomes hard to manage lifecycles of logs and it doesn't integrate well with enabling development teams to manage pipelines and data stores/retention independently.

Having the full control of a dedicated sink would be great, but templating via headers does directly solve the problem. Would be happy to try to get company approval to tackle this if the Vector team is in agreement on the approach.

[jszwedko]

Hi @goedelsoup-clari !

Regarding having http headers be templatable we are definitely open to this. It's a fairly common request even outside of the context of sending logs to SumoLogic. It would mean partitioning the batches using the header values.

We'd very likely be open to a dedicated SumoLogic sink too. If you want to pursue that route, would you want to open a separate issue? We have this checklist that we like to run through for new component proposals: https://github.com/vectordotdev/vector/blob/master/CONTRIBUTING.md#new-sources-sinks-and-transforms

[DGolubets]

This feature would be nice to have.

[sreisich]

I need exactly the same for Parseable. In order for a stream to be created in Parseable, I need to set the header X-P-Stream. And I need the streams to be the kubernetes namespaces:

request:
    headers:
        X-P-Stream: "{{ .kubernetes_pod_namespace }}"

Any chance this will be implemented soon?