Setting host.* in Beats that forward data

[andrewkroh]

There are several use cases in Beats where the data reported by a Beat did not originate on that Beat host. Some examples are syslog, windows forwarded events, router netflow data, and cloud watch logs. In these cases it would be appropriate to set the host.* field to information about the originating machine.

From ECS:

ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken.

Some issues related to this:

• host.name behavior inconsistent across the Elastic stack #13777
• [winlogbeat] Use the original host for host.name in Windows Event Logs #13706
• [Filebeat] The host.name sent from Filebeat doesn't match the same field from Metricbeat #13589
• Not always send host.name and have host metadata processor enabled #10698

I think we need way for inputs and modules to be able to "designate" that host.* should not be set by default. The output pipeline and also the add_host_metadata processor will need to honor this "designation".

[andrewkroh]

@webmat Would it be appropriate to populate observer.* in any of these above use cases? If so which host's data would go there?

[webmat]

Here are cases that should populate observer:

• A Beat is actively monitoring a different host. E.g. Metricbeat collecting MySQL metrics from another host, like an AWS RDS instance.
  • In this case, the host running Metricbeat should go to observer.hostname & so on with other observer fields
  • In this case, the host being monitored should go into host.hostname & so on with other host fields.
• A Beat is collecting logs locally from another agent acting as an observer (e.g. Zeek monitoring a network tap, or a Beat installed on a network appliance).
  • In this case, since both the Beat and the software doing the monitoring are on the same host, observer.hostname & other fields should be populated with that host's detail
  • If the data being collected contains information about the monitored hosts, then this goes to host.*
  • If the data being collected does not contain information about the monitored hosts (e.g. network flow stats), then host.* cannot be populated.

Here are cases that should not populate observer:

• A Beat is receiving Syslog events and passing them along.
• A Beat is receiving Windows Event Logs and passing them along.

In both of these cases, if the data stream contains the source host's detail, it should to go to host.*.

[webmat]

The case of monitoring containers may require a chat, I see a few cases, there may be more:

• An agent can be installed on the host running Docker directly (not using Docker)
  • It can then monitor containers
  • It can monitor the host itself
• An agent can be installed as a sidecar container, with the purpose of monitoring other containers local to the host
• An agent can be installed in a container, with the purpose of monitoring the host

I'm not sure the current semantics as defined by ECS are clear nor sufficient to fully capture this. Maybe I'm wrong. But I'd be more than happy to have a chat with folks specialized with monitoring Docker, and hash out ideas here. Let me know if that would be useful.

[missnebun]

I just installed filebeat 7.4 and I am using the netflow module. I have the same problem. agent.hostname is the agent.hostname for the hostname of netflow input server instead of the sender. I had to update some visualization for Kibana and replace agent.hostname with observer.ip.

[urso]

It is not only the processors, but also libbeat directly adding some fields. See: https://github.com/elastic/beats/blob/master/libbeat/publisher/processing/default.go#L78

I'm in favor of not automatically modifying any event once it hits libbeat. All modification should be opt-in via processors.

[exekias]

I think we need way for inputs and modules to be able to "designate" that host.* should not be set by default. The output pipeline and also the add_host_metadata processor will need to honor this "designation".

I totally agree, we have this same problem with cloud metadata and cloud monitoring modules (AWS, Azure, etc). Something we have with add_cloud_metadata is that it won't override any info sent by the input/module itself:

beats/libbeat/processors/add_cloud_metadata/add_cloud_metadata.go

Lines 112 to 117 in 1fb41ea

	if !p.initData.overwrite {
	cloudValue, _ := event.GetValue("cloud")
	if cloudValue != nil {
	return event, nil
	}
	}

In this case, cloud metadata for the agent is not sent, which may not be ideal (it could make sense to have it under observer.cloud?)

Something like this could make sense for add_host_metadata, where it could decide to put the metadata somewhere else (as you said, maybe observer)

The case of monitoring containers may require a chat, I see a few cases, there may be more:

• An agent can be installed on the host running Docker directly (not using Docker)

  • It can then monitor containers
  • It can monitor the host itself

• An agent can be installed as a sidecar container, with the purpose of monitoring other containers local to the host

• An agent can be installed in a container, with the purpose of monitoring the host

I'm not sure the current semantics as defined by ECS are clear nor sufficient to fully capture this. Maybe I'm wrong. But I'd be more than happy to have a chat with folks specialized with monitoring Docker, and hash out ideas here. Let me know if that would be useful.

Happy to participate in this conversation @webmat!