Compatible? Compliant? How to discuss ECS-ness of an integration

[MikePaquette]

This is a discussion regarding how to describe integrations that are designed to work with ECS, and how to ensure that we realize value of ECS in re-usable analysis content. Would love to get some feedback and thoughts.

We've discussed the idea of defining a certain set of ECS fields that are common, generalized fields that are likely to be used by analysis content (searches, visualizations, dashboards, alerts, machine learning jobs, reports) across use cases. Ideally, any analysis content designed to operate on these fields should work properly on data from any relevant source. For now, let us call them "ECS-Core" fields.

What I've seen so far from folks that have tried to map their fields to ECS is that they start with their existing set of event fields, and they try to find the appropriate ECS field name for each. This is the basic and necessary step in mapping to ECS, but I think it is insufficient. Call it Step 1.

I think we also need to implement a new step 2, where we start by walking through the list of all ECS-Core fields, and consider "how can I best populate these fields with the data I have available (even if I've mapped all my fields in step 1)? Call this Step 2.

The reason for requiring Step 2 is to ensure reusability of ECS analysis content. If I build a dashboard that uses ECS-core fields, I expect my dashboard to be useful on any source implementation that contains relevant data. For example, two proposed ECS-Core fields are inbound_bytes and inbound packets. If I build an enterprise visualization that uses those ECS-Core fields, I would expect that an ECS bro ingestion config would populate them. Same for an ECS netflow implementation. You can see that if the bro config followed only Step 1 above, my ECS dashboard would not work with the bro implementation even though both were built with ECS in mind. (Because bro conn.log does not have default fields that would map to inbound_bytes and inbound packets. Only by taking step 2 and figuring out how to populate inbound_bytes and inbound packets would the integration work with my dashboard.

We can then define two levels of ECS-ness:

1. All existing event fields are mapped to ECS-Core fields. (Perhaps ECS-Compatible)
2. In addition, as many ECS-Core fields as practical are populated, even if that means duplication, or adding metadata, or deriving values via manipulation and math. (Perhaps ECS-Compliant)

[webmat]

I agree we should define some sort of minimum set of fields that should be present most of the time. I think we should even start documenting the thinking process you're describe in the README right away as well. The idea that after doing the basic mapping of what you already happened to have (step 1), you should also try to fill out as many of the fields as possible (step 2) is a very good idea. It may seem obvious after you've heard it, but it was eye opening for me the first time I heard it :-)

Part of this explanation would benefit from having the Core vs Extended fields identified (#93) as well, however. A user should work more towards making sure they fill up as many of the Core fields than Extended ones.

On the other hand, we haven't fully mapped anything to ECS, yet. We also haven't built any significant downstream artifacts based on such ECS streams either (be it dashboards, Kibana apps, ML & so on). So from that point of view, I think it may be a bit early to define a precise approach like the one you describe above, with named standards (Compatible vs Compliant).

WDYT?

[ruflin]

If we look at Metricbeat, most events will fulfill a subset of ECS Core and adding network.inbound.bytes and network.outbound.bytes does not make too much sense from my perspective in most cases. There are other cases in Filebeat modules where we have similar issues. Both Beats already started using ECS today and will use even more fields in the near future but are unlikely to have all fields we are thinking of core for now.

There are three solutions I can see:

1. Lots of Beats events will just not be what we call compatible
2. We reduce the scope of Core to be really minimal
3. We find an other solution

I don't like 1 as I think Beats is one of our main ingestion tool and they should definitively be compatible. If we reduce the scope of Core to a minimum that all will have the fields and I think we come down to just @timestamp but that is not useful for dashboards. So we end up with option 3.

The problem from my perspective is that every area is very different: web server logs vs metrics vs audit security vs apm. There will not be a dashboard for all this data and I think there shouldn't be. Still ECS is a foundation for all these use cases.

What I would propose instead is defining areas which we know have common fields. If someone wants to use his data with "auditing" dashboards these x fields are required. The fields can be in Core, Extended or what ever other levels we come up with. The separation into Core and Extended is to make it easier for users to get started with ECS but I think for all use cases we come up they will contain fields in Core and Extended.

What I describe above could probably described as a data source type. It will fulfill certain queries in Elasticsearch.

I realised I mentioned above use cases, perhaps we should reuse what we call use cases at the moment for this purpose.

[ebeahan]

The concept of ECS Core vs. Extended fields is now documented here: https://www.elastic.co/guide/en/ecs/current/ecs-guidelines.html#_ecs_field_levels

Defining the concepts of ECS-compliant and ECS-compatible remains and comes up from time-to-time, but let's open a new discussion when we're ready to move those discussions forward actively.