-
Notifications
You must be signed in to change notification settings - Fork 444
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add datastream fields to all datasets #213
Conversation
919ca47
to
9107ee6
Compare
Change for the agent can be found here: elastic/beats#20402 |
@mtojek I will probably need your help to get this over the line. After we get this in, what is the best way to release a new version of the packages to snapshot and then promote it through? |
currently it's the command |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we have any plans for Kibana dashboards?
If so, there are few places in https://github.com/elastic/integrations/blob/master/dev/import-beats/kibana.go to replace dataset.name
.
BTW Does it mean we don't have anymore datasets and just datastreams? Should we update all places in integrations? code, READMEs. Code changes can be postponed, but what about READMEs?
I had plan to promote packages today. I can push changes in this PR too. |
@mtojek At the moment the Agent ships both fields so we need both mappings. But the plan is to follow up on removing How tricky do you think it's going to be to adjust all dashboards and README's? |
@mtojek I'll do a follow up PR to adjust all the scripts if that is ok for you. |
Elastic Agent and Elasticsearch are switching over from using dataset.* to datastream.*. This adjust the aggregation on the dataset page to get the datastreams. For this to work properly, the most recent version of Elasticsearch 7.9 must be used and is pending updates on all the packages to ship also the datastream fields, see elastic/integrations#213
Type: "constant_keyword", | ||
Description: "Datastream namespace.", | ||
}, | ||
// TODO: This should be removed as soon as it is not a requirement anymore by the validation |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm.. shouldn't be validation in the package-registry already adjusted?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we adjust it now, all old packages are invalid.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PR for validation: elastic/package-registry#618
once this one is pushed, you can update the reference to the commit
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can update the reference in docs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
You need to bump up versions in all manifest.yml files. |
I cherry-picked the changes from #216 into this one. I initially expected the script changes to be larger and the version bumps went into the wrong branch. Please have a look again. |
description: Datastream type. | ||
- name: datastream.dataset | ||
type: constant_keyword | ||
description: Datastream dataset name. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Datastream dataset
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
adjusted in all packages.
Side note: we have some packages in the package storage: barracuda and endpoint, and there is bunch of new waiting for merge: elastic/package-storage#181 any plan for these ones? |
My thinking for the pending packages in snapshot was:
|
OK, I'll push first all integrations in the package-storage to the production and leave the ones that we (integrations) do not own. |
It is planned to move from dataset.* to datastream.*. To make the path easier to 7.9 Elasticsearch (elastic/elasticsearch#60592) and the Elastic Agent already ship with the datastream.* fields. Because of this, these should also be added to the mappings of the packages.
The agent will ship datastream.* fields. Because of this, the import scripts can be adjusted to only use these fields. This PR doese not rename any Golang variables. This should be done in a follow up and in sync with potential changes to the registry.
d5c82fc
to
9f6e769
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Elastic Agent and Elasticsearch are switching over from using dataset.* to datastream.*. This adjust the aggregation on the dataset page to get the datastreams. For this to work properly, the most recent version of Elasticsearch 7.9 must be used and is pending updates on all the packages to ship also the datastream fields, see elastic/integrations#213
…#74342) * [Ingest Manager] Adjust dataset aggs to use datastream fields instead Elastic Agent and Elasticsearch are switching over from using dataset.* to datastream.*. This adjust the aggregation on the dataset page to get the datastreams. For this to work properly, the most recent version of Elasticsearch 7.9 must be used and is pending updates on all the packages to ship also the datastream fields, see elastic/integrations#213 * Update datastream to data_stream * Update data stream name generation * Fix typo * Temporarily use datastream instead of data_stream * updating to use `data_stream` instead of `datastream` Co-authored-by: ruflin <spam@ruflin.com> Co-authored-by: Jen Huang <its.jenetic@gmail.com>
…elastic#74342) * [Ingest Manager] Adjust dataset aggs to use datastream fields instead Elastic Agent and Elasticsearch are switching over from using dataset.* to datastream.*. This adjust the aggregation on the dataset page to get the datastreams. For this to work properly, the most recent version of Elasticsearch 7.9 must be used and is pending updates on all the packages to ship also the datastream fields, see elastic/integrations#213 * Update datastream to data_stream * Update data stream name generation * Fix typo * Temporarily use datastream instead of data_stream * updating to use `data_stream` instead of `datastream` Co-authored-by: ruflin <spam@ruflin.com> Co-authored-by: Jen Huang <its.jenetic@gmail.com>
…elastic#74342) * [Ingest Manager] Adjust dataset aggs to use datastream fields instead Elastic Agent and Elasticsearch are switching over from using dataset.* to datastream.*. This adjust the aggregation on the dataset page to get the datastreams. For this to work properly, the most recent version of Elasticsearch 7.9 must be used and is pending updates on all the packages to ship also the datastream fields, see elastic/integrations#213 * Update datastream to data_stream * Update data stream name generation * Fix typo * Temporarily use datastream instead of data_stream * updating to use `data_stream` instead of `datastream` Co-authored-by: ruflin <spam@ruflin.com> Co-authored-by: Jen Huang <its.jenetic@gmail.com>
…#74342) (#74355) * [Ingest Manager] Adjust dataset aggs to use datastream fields instead Elastic Agent and Elasticsearch are switching over from using dataset.* to datastream.*. This adjust the aggregation on the dataset page to get the datastreams. For this to work properly, the most recent version of Elasticsearch 7.9 must be used and is pending updates on all the packages to ship also the datastream fields, see elastic/integrations#213 * Update datastream to data_stream * Update data stream name generation * Fix typo * Temporarily use datastream instead of data_stream * updating to use `data_stream` instead of `datastream` Co-authored-by: ruflin <spam@ruflin.com> Co-authored-by: Jen Huang <its.jenetic@gmail.com> Co-authored-by: ruflin <spam@ruflin.com> Co-authored-by: Jen Huang <its.jenetic@gmail.com>
…#74342) (#74356) Elastic Agent and Elasticsearch are switching over from using dataset.* to datastream.*. This adjust the aggregation on the dataset page to get the datastreams. For this to work properly, the most recent version of Elasticsearch 7.9 must be used and is pending updates on all the packages to ship also the datastream fields, see elastic/integrations#213
It is planned to move from dataset.* to datastream.*. To make the path easier to 7.9 Elasticsearch (elastic/elasticsearch#60592) and the Elastic Agent already ship with the datastream.* fields. Because of this, these should also be added to the mappings of the packages. The agent will ship datastream.* fields. Because of this, the import scripts can be adjusted to only use these fields. This PR doese not rename any Golang variables. This should be done in a follow up and in sync with potential changes to the registry.
It is planned to move from dataset.* to datastream.. To make the path easier to 7.9 Elasticsearch (elastic/elasticsearch#60592) and the Elastic Agent already ship with the datastream. fields. Because of this, these should also be added to the mappings of the packages.