New field "time"

[vbohata]

For at least IIS, Apache2, IIS DHCP logs I need a time field. Just event.duration is not good for use because it do not describe what has taken what time.

I propose to create "time" (or some other named) fields like this:

• time.taken_seconds
• time.taken_milliseconds
• time.duration_seconds
• time.duration_milliseconds

So for IIS, Apache2 logs I could you http.response.time.taken_milliseconds, for dhcp I could use dhcp.lease.time.duration_seconds, ...

[ruflin]

Thanks for opening this request. As usual I'm a bit hesitant to add metrics which are not very generic to ECS as there are just too many different metrics out there.

Can you describe a bit more in detail what taken and duration are in the above?

I'm hoping we would not need seconds and milliseconds but only unit: elastic/elasticsearch#31244

[vbohata]

Taken would describe number of time unit the action took to process, duration is more generic as for example lease duration - for how much time for example lease has been assigned.

In case one unit will be used, this general "time" field will not be needed ... probably. From the linked issue I did not get if there will be new duration field in Elasticsearch (as mentioned in this issue) or there will be just ingest filter to convert everything to ms for example.

[webmat]

We should not add multiple fields for different scales of a numeric value (millis vs sec). Rather, we should store the most precise information we have in a numeric field, then have our application reading from it convert the value to whatever makes most sense (human readable, seconds, hours) as needed.

For example, in Kibana index patterns, you can specify a field formatter that would specify "these values are milliseconds, and I want to display them human readable". If you have another custom application reading from ES, it should also incorporate some knowledge of how to interpret the data.

Your issue brings up a good point, however. For a general schema like ECS, we can't know for a given duration field, what kind of resolution the source will have. If we arbitrarily decide that "all duration fields" should be stored as milliseconds, we will force systems with finer resolutions to discard the trailing information, whether it's micro or nano seconds, which is not great.

[ruflin]

For now we should define the unit that "duration" should be. Long term I'm hoping for the duration type so it can be ingested in different units.

[vbohata]

My duration ranges for different data sources are very large. For example for http response time it is OK to be in milliseconds. For ttl values in DNS etc. the long data type is not enough for some kind of unified value in micro/nano seconds.
Maybe it will worth try something like this one: host.timezone.offset.sec (already in ECS).
So there will be http.request.response_time.ms, dns.ttl.sec, extreme_precision_field.ns, ... Kibana could handle this automatically ...

[ruflin]

What are the largest values you expect and in which data source (as an example)?

[vbohata]

My mistake, seems to fit even for max possible TTL in DNS records converted from secs to nsecs. So it is supposed to store everything in ns? Most of logged time values are limited to 32bit number, so 64bit long is wasting of space.

[vbohata]

Btw. what is current usage of event.duration? Could I use it for extracting "execution time" strings from messages or is it dedicated for time taken to generate (how measured?) the event?

[ruflin]

In ES it does not make much a difference if you use integer vs long as long as the values are small.

I think event.duration can be used for various use cases. And you describe one above.

[vbohata]

Currently found some cases for which I need to store duration times also as keyword with multi-field). I need it for splitting in ML for DNS TTL, DHCP lease time ... values which are usually 1800, 3600, ... seconds. Here it does not make sense to store it in nanoseconds. What about for each time/duration value use suffix .sec, .msec, .usec, .nsec (user defined, for precision the user needs) ... and once the ES will be able to handle the time duration, just copy this field value to some general .period or .interval suffix?

So for now for example:

• event.duration.nsec
• dhcp.lease_duration.sec

After interval support in ES copy it (for backward compatibility):

• event.duration.nsec -> event.duration.period
• dhcp.lease_duration.sec -> dhcp.lease_duration.period

Pros: does not need to set format for each time value in Kibana, easy to understand for users, easy to write in search bar in Kibana, better visibility during searching (seeing or typing 3 seconds in secods is much better and typos then value in nanoseconds) ...

[ruflin]

I worry a lot about field explosion in ECS here. ECS should have 1 unit until it's supported differently by Elasticsearch.

[vbohata]

I understand the reason, but this makes much worse to filter these values in Kibana. I just tested it can be nicely translated to human units automatically in discover or in visualisations. Timelions does not support it so I had to manual add divide here. Also for each filter in discover I have to put the original value, so nanoseconds. For the "clicked" filters the Kibana will show it as wanted, but still editing in nanoseconds - so editing in nanoseconds, filter displays the value as defined in Kibana (very confusing for big numbers). Also to "lucene search bar" users have to put original values and Kibana shows in this bar original values ... which is logical but even more confusing - users will see the same value in different scales at the same time.

[vbohata]

For some fields it does not make sense to store it in other then seconds like host.timezone.offset.sec which is part of ECS. So maybe other fields which are always in seconds like dhcp.lease_duration.sec (for example, dont know if will be part of ECS) should have ".sec" suffix. For other values like event.duration it make sense to store it in nanoseconds but for users it should be clear in which unit the fields is stored in (before applying Kibana display modifications). So what about adding .sec to all fields for which other unit is useless and for other fields use nanoseconds and add .nsec suffix for them ... so event.duration.nsec?

[ruflin]

I actually have second thoughts about host.timezone.offset.sec and we should potentially rename it to host.timezone.offset. It would be part of the ECS docs to state which unit they are in. And I have some ideas around how we could make Elasticsearch and Kibana aware of it but it's not in there yet and ECS should also work with older version.

Units is definitively something we need to still put more thoughts into.