Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add datastream fields to all datasets #213

Merged
merged 9 commits into from
Aug 4, 2020

Conversation

ruflin
Copy link
Member

@ruflin ruflin commented Aug 3, 2020

It is planned to move from dataset.* to datastream.. To make the path easier to 7.9 Elasticsearch (elastic/elasticsearch#60592) and the Elastic Agent already ship with the datastream. fields. Because of this, these should also be added to the mappings of the packages.

@ruflin ruflin self-assigned this Aug 3, 2020
@elasticmachine
Copy link

elasticmachine commented Aug 3, 2020

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #213 updated]

  • Start Time: 2020-08-04T11:31:19.078+0000

  • Duration: 3 min 1 sec

@ruflin
Copy link
Member Author

ruflin commented Aug 3, 2020

Change for the agent can be found here: elastic/beats#20402

@ruflin ruflin marked this pull request as ready for review August 3, 2020 14:57
@ruflin
Copy link
Member Author

ruflin commented Aug 3, 2020

@mtojek I will probably need your help to get this over the line. After we get this in, what is the best way to release a new version of the packages to snapshot and then promote it through?

@mtojek
Copy link
Contributor

mtojek commented Aug 4, 2020

what is the best way to release a new version of the packages to snapshot and then promote it through?

currently it's the command mage UpdatePackageStorage and later on juggling packages through branches of the package-storage

Copy link
Contributor

@mtojek mtojek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have any plans for Kibana dashboards?

If so, there are few places in https://github.com/elastic/integrations/blob/master/dev/import-beats/kibana.go to replace dataset.name.

BTW Does it mean we don't have anymore datasets and just datastreams? Should we update all places in integrations? code, READMEs. Code changes can be postponed, but what about READMEs?

@mtojek
Copy link
Contributor

mtojek commented Aug 4, 2020

I had plan to promote packages today. I can push changes in this PR too.

@ruflin
Copy link
Member Author

ruflin commented Aug 4, 2020

@mtojek At the moment the Agent ships both fields so we need both mappings. But the plan is to follow up on removing dataset.* completely. The first step is getting rid of it in queries (dashboards), second step adjust the package spec but we plan to do this later on in small steps.

How tricky do you think it's going to be to adjust all dashboards and README's?

@ruflin
Copy link
Member Author

ruflin commented Aug 4, 2020

@mtojek I'll do a follow up PR to adjust all the scripts if that is ok for you.

@ruflin
Copy link
Member Author

ruflin commented Aug 4, 2020

@mtojek Here is the script change PR: #216

ruflin added a commit to ruflin/kibana that referenced this pull request Aug 4, 2020
Elastic Agent and Elasticsearch are switching over from using dataset.* to datastream.*. This adjust the aggregation on the dataset page to get the datastreams.

For this to work properly, the most recent version of Elasticsearch 7.9 must be used and is pending updates on all the packages to ship also the datastream fields, see elastic/integrations#213
Type: "constant_keyword",
Description: "Datastream namespace.",
},
// TODO: This should be removed as soon as it is not a requirement anymore by the validation
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm.. shouldn't be validation in the package-registry already adjusted?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we adjust it now, all old packages are invalid.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR for validation: elastic/package-registry#618

once this one is pushed, you can update the reference to the commit

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can update the reference in docs.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@mtojek
Copy link
Contributor

mtojek commented Aug 4, 2020

You need to bump up versions in all manifest.yml files.

@ruflin
Copy link
Member Author

ruflin commented Aug 4, 2020

I cherry-picked the changes from #216 into this one. I initially expected the script changes to be larger and the version bumps went into the wrong branch. Please have a look again.

description: Datastream type.
- name: datastream.dataset
type: constant_keyword
description: Datastream dataset name.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Datastream dataset

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

adjusted in all packages.

@mtojek
Copy link
Contributor

mtojek commented Aug 4, 2020

Side note:

we have some packages in the package storage: barracuda and endpoint, and there is bunch of new waiting for merge: elastic/package-storage#181

any plan for these ones?

@ruflin
Copy link
Member Author

ruflin commented Aug 4, 2020

My thinking for the pending packages in snapshot was:

  • Update registry first for production, so validation passes
  • Later updated it for snapshot and with it also all the packages

@mtojek
Copy link
Contributor

mtojek commented Aug 4, 2020

OK, I'll push first all integrations in the package-storage to the production and leave the ones that we (integrations) do not own.

go.mod Outdated Show resolved Hide resolved
It is planned to move from dataset.* to datastream.*. To make the path easier to 7.9 Elasticsearch (elastic/elasticsearch#60592) and the Elastic Agent already ship with the datastream.* fields. Because of this, these should also be added to the mappings of the packages.
The agent will ship datastream.* fields. Because of this, the import scripts can be adjusted to only use these fields.

This PR doese not rename any Golang variables. This should be done in a follow up and in sync with potential changes to the registry.
@mtojek mtojek self-requested a review August 4, 2020 11:41
Copy link
Contributor

@mtojek mtojek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ruflin ruflin merged commit da4873a into elastic:master Aug 4, 2020
@ruflin ruflin deleted the add-datastream-fields branch August 4, 2020 11:57
skh pushed a commit to skh/kibana that referenced this pull request Aug 5, 2020
Elastic Agent and Elasticsearch are switching over from using dataset.* to datastream.*. This adjust the aggregation on the dataset page to get the datastreams.

For this to work properly, the most recent version of Elasticsearch 7.9 must be used and is pending updates on all the packages to ship also the datastream fields, see elastic/integrations#213
skh added a commit to elastic/kibana that referenced this pull request Aug 5, 2020
…#74342)

* [Ingest Manager] Adjust dataset aggs to use datastream fields instead

Elastic Agent and Elasticsearch are switching over from using dataset.* to datastream.*. This adjust the aggregation on the dataset page to get the datastreams.

For this to work properly, the most recent version of Elasticsearch 7.9 must be used and is pending updates on all the packages to ship also the datastream fields, see elastic/integrations#213

* Update datastream to data_stream

* Update data stream name generation

* Fix typo

* Temporarily use datastream instead of data_stream

* updating to use `data_stream` instead of `datastream`

Co-authored-by: ruflin <spam@ruflin.com>
Co-authored-by: Jen Huang <its.jenetic@gmail.com>
skh added a commit to skh/kibana that referenced this pull request Aug 5, 2020
…elastic#74342)

* [Ingest Manager] Adjust dataset aggs to use datastream fields instead

Elastic Agent and Elasticsearch are switching over from using dataset.* to datastream.*. This adjust the aggregation on the dataset page to get the datastreams.

For this to work properly, the most recent version of Elasticsearch 7.9 must be used and is pending updates on all the packages to ship also the datastream fields, see elastic/integrations#213

* Update datastream to data_stream

* Update data stream name generation

* Fix typo

* Temporarily use datastream instead of data_stream

* updating to use `data_stream` instead of `datastream`

Co-authored-by: ruflin <spam@ruflin.com>
Co-authored-by: Jen Huang <its.jenetic@gmail.com>
skh added a commit to skh/kibana that referenced this pull request Aug 5, 2020
…elastic#74342)

* [Ingest Manager] Adjust dataset aggs to use datastream fields instead

Elastic Agent and Elasticsearch are switching over from using dataset.* to datastream.*. This adjust the aggregation on the dataset page to get the datastreams.

For this to work properly, the most recent version of Elasticsearch 7.9 must be used and is pending updates on all the packages to ship also the datastream fields, see elastic/integrations#213

* Update datastream to data_stream

* Update data stream name generation

* Fix typo

* Temporarily use datastream instead of data_stream

* updating to use `data_stream` instead of `datastream`

Co-authored-by: ruflin <spam@ruflin.com>
Co-authored-by: Jen Huang <its.jenetic@gmail.com>
skh added a commit to elastic/kibana that referenced this pull request Aug 5, 2020
…#74342) (#74355)

* [Ingest Manager] Adjust dataset aggs to use datastream fields instead

Elastic Agent and Elasticsearch are switching over from using dataset.* to datastream.*. This adjust the aggregation on the dataset page to get the datastreams.

For this to work properly, the most recent version of Elasticsearch 7.9 must be used and is pending updates on all the packages to ship also the datastream fields, see elastic/integrations#213

* Update datastream to data_stream

* Update data stream name generation

* Fix typo

* Temporarily use datastream instead of data_stream

* updating to use `data_stream` instead of `datastream`

Co-authored-by: ruflin <spam@ruflin.com>
Co-authored-by: Jen Huang <its.jenetic@gmail.com>

Co-authored-by: ruflin <spam@ruflin.com>
Co-authored-by: Jen Huang <its.jenetic@gmail.com>
ruflin pushed a commit to elastic/kibana that referenced this pull request Aug 5, 2020
…#74342) (#74356)

Elastic Agent and Elasticsearch are switching over from using dataset.* to datastream.*. This adjust the aggregation on the dataset page to get the datastreams.

For this to work properly, the most recent version of Elasticsearch 7.9 must be used and is pending updates on all the packages to ship also the datastream fields, see elastic/integrations#213
eyalkraft pushed a commit to build-security/integrations that referenced this pull request Mar 30, 2022
It is planned to move from dataset.* to datastream.*. To make the path easier to 7.9 Elasticsearch (elastic/elasticsearch#60592) and the Elastic Agent already ship with the datastream.* fields. Because of this, these should also be added to the mappings of the packages.

The agent will ship datastream.* fields. Because of this, the import scripts can be adjusted to only use these fields.

This PR doese not rename any Golang variables. This should be done in a follow up and in sync with potential changes to the registry.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants