Data Stream fields: Move to stage 3

[ruflin]

Criteria for consideration for this stage:

• Opened pull request for this candidate revising the existing candidate
• Included multiple real world example source documents
• Existing or newly raised questions and concerns are addressed
• No objections from sponsor, ECS team, or subject matter experts

[ph]

@ruflin Looping back our other docs, should we document "logs" to be the default values of data_stream.type?

[ruflin]

@ph I don't think there should be a default in the spec. I rather thing certain application should define their defaults. In the case of logstash it is probably logs, in the case of metricbeat, metrics etc.

[ebeahan]

We've recently removed Stage 4 from the RFC process.

A decision to make is whether we feel this RFC is stable enough to target stage 3 still or update the target stage to the updated stage 2 (equivalent to legacy stage 3).

Updated proposal stages and their requirements: https://elastic.github.io/ecs/stages.html

[ebeahan]

Now that the data stream naming scheme was added to the Elasticsearch docs, I think a link to it would make a good addition as a reference:

https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-a-data-stream.html#elastic-data-stream-naming-scheme.

[ruflin]

Happy to add the link. What is not clear what else I need to do on this PR to move it forward?

[ebeahan]

We first need to make a decision about if this proposal is considered finished or not.

The options to move forward:

1. We advance this proposal as a Candidate. The data_stream fields will be added to the ECS docs and schema but marked as beta fields.
2. Or we advance it to Finished. data_stream fields are added to ECS as GA.

I'm in favor of advancing the proposal to Finished. The fields are already used by data stream naming schema, so the likelihood of any changes to these fields happening seems very unlikely. I'm also not seeing any outstanding concerns from the proposal that need further discussion.

I updated the PR's description with criteria. If we agree on advancing as Finished, I believe we just need final reviews.

@ruflin You're listed as the sponsor here. Is that still accurate? Is there anyone else from observability we should have weigh-in?

[ruflin]

++ on going to Final. I'm indeed filling both roles at the moment, but would be good to have a second person having a look at this. I'm thinking of @jpountz .

[jpountz]

I think we will want to retain the data_stream fields in the future given their tight relationship with how we now think about event data in Observability+Elasticsearch. I can think of two reasons why we might want to wait a bit before making this final:

• It's not fully clear (to me at least) how to deal with datasets that really are sources of data rather than datasets like Prometheus.
• Currently the data_stream fields are explicit in every document. Maybe at some point we will want to consider dynamically extracting them from the name of the data_stream. Said otherwise, querying data_stream.type would actually look at the part of the name of the data stream that is before the first -. Even if these fields compress well, they contribute to a perceived heavy weight of using Elastic for Observability by making documents larger.

[ruflin]

@jpountz As we already heavily use these fields today and we can't easily change it, I assume the basic structure will stay as it is. There are some open questions around prometheus for example but in the end, we must make it fit into the current naming scheme I think.

For your second comment: I wonder how the _source part would look like in this case. As it is a constant_keyword, it would work well for the querying I assume as in case it is set in the mapping, it just includes all docs even if they don't contain the field itself? But a user looking at the doc would miss some of the information.

My current suggestion would be to move this forward as it is already in use today. My assumption is that future changes will be additions and not breaking changes.

[ebeahan]

@felixbarny has a suggested addition to the naming restrictions section over in #1304. Can we incorporate that change here?

[ebeahan]

My current suggestion would be to move this forward as it is already in use today. My assumption is that future changes will be additions and not breaking changes.

I suggest we leave this open for a few more days for any additional comments or concerns, but I agree with moving forward and advancing this proposal to Finished.

[ebeahan]

Correcting the advancement date in #1306

[ruflin]

@ebeahan Thanks for the cleanup and sorry for the early merge.