Making clear about to what device is host object related to

[vbohata]

A lot if issues here (including mine) needs to somehow implement following:

• What is the network peer host name I received logs from?
• What device originated the event?
• What device transmitted it?

I propose to make some unification about it by prefixing host field with "device.SOURCENAME", where SOURCENAME is:

• originator - the generator of the event (Filebeat)
• collector - who received it (Logstash)
• relay - who relayed it to another relay or collector ... so maybe relay0, relay1, ...

Also there should be peer object so device.collector.peer.host.ip - means IP of the TCP session peer from which for example the Logstash received the event (can be different from device.relay.host.ip or even device.originator.host.ip).

[ruflin]

Can you share a bit of background on how you plan to query and use this data? Especially interested in the collector and relay in the above.

I definitively see value on knowing through which hops an event went at the same time this can contain more then just 3 hops as you pointed out in the above with relay0, relay1 and don't know yet how we should cover that. Would order matter? What I was thinking so far is that originator = agent.

[vbohata]

One use case: currently we collect logs from our switches this way:

1. switch sends syslog events to rsyslog server [switch is the originator]
2. filebeat forwards logs from rsyslog logged events to Logstash [filebeat is agent, but not originator, it is just relay this time]
3. logstash receives logs from filebeat and forwards it to kafka [logstash is relay]
4. another logstash reads logs from kafka and stores it in elasticsearch [this logstash is collector]
   But lets say, 3+4 is collector. For us it is important to know the originator IP and hostname, the filebeat (here relay) IP and hostname and IP address of peer from which the collector received logs from relay (can be different from relay management IP because of NAT for example).

Would order matter?
Probably.

[ruflin]

So in your case you are interested in the switch and filebeat info but not the other hops? Or you would add all of them?

[vbohata]

Yes, in this case I am interested about the switch and filebeat info. Btw. this use case also shows I need to distinguish log data from metadata. For switch log it is not clear if something like device.originator is related to the event generator or to some event content (for example 802.1x auth info where for the first time device.originator can look like the device authenticating to the switch).

[webmat]

I'm hesitant on the value of adding too much metadata surrounding how events are gathered or transported to this common schema, at least this early.

If we clarified how to use ECS while also adding other fields that aren't part of ECS (or not yet), would that help? I've seen internal discussions about this, but it hasn't really been communicated in this repo yet.

[ruflin]

++ on the comment from @webmat

One thing that we started for such proposal is that it can be provided as a use case (see notes there): https://github.com/elastic/ecs#use-cases

[ebeahan]

observer.* has been introduced, which I think addresses some of the original concerns here, in part.

There's a meta issue in #940 tracking past issues/discussions around pipeline support and guidance in ECS. I've noted this issue in the list captured there.

If there are any current thoughts or concerns on the topic, let's take the discussion there.