Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hidden data streams #63987

Merged
merged 33 commits into from
Oct 30, 2020
Merged

Hidden data streams #63987

merged 33 commits into from
Oct 30, 2020

Conversation

probakowski
Copy link
Contributor

This change adds possibility to mark data stream as hidden. To use it you set usual "index.hidden":true setting in data stream template.
This change also relaxes naming convention for data streams - data stream can start with . but can't start with backing index prefix (.ds- currently) to avoid clashes.

Closes #63889

martijnvg
martijnvg reviewed Oct 21, 2020
View reviewed changes
Copy link
Member

@martijnvg martijnvg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The implementation looks good at first glance. Can you also add a rest test for hiding a data stream? iirc some changes are also needed in security.

danhermann
danhermann reviewed Oct 21, 2020
View reviewed changes
Copy link
Contributor

@danhermann danhermann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this change the behavior of the GetDataStream API? I think we'll need to add the expand_wildcards option to that API for consistency in the handling of hidden data streams.

@probakowski probakowski marked this pull request as ready for review October 22, 2020 21:48
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (:Core/Features/Data streams)

@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label Oct 22, 2020
@probakowski
Copy link
Contributor Author

probakowski commented Oct 22, 2020 •
edited
Loading

Thanks for review, could I ask for another round?
@martijnvg do you remember what it was about security? I've tested it with security on locally and used RBAC as well, it all worked correctly. I've added REST tests.
@danhermann for now I've updated GetDataStream API to show all data streams, the same goes for DeleteDataStream API. I think adding expand_wildcards param is good idea but I think it can be done in separate issue, WDYT?

@probakowski
Copy link
Contributor Author

@elasticmachine run elasticsearch-ci/1

@probakowski
Copy link
Contributor Author

@elasticmachine run elasticsearch-ci/1 (known unrelated failure)

martijnvg
martijnvg approved these changes Oct 30, 2020
View reviewed changes
Copy link
Member

@martijnvg martijnvg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@probakowski probakowski merged commit 0e81fc6 into elastic:master Oct 30, 2020
@probakowski probakowski deleted the hidden-data-streams branch October 30, 2020 10:42
@probakowski probakowski mentioned this pull request Oct 30, 2020
Merged
probakowski added a commit to probakowski/elasticsearch that referenced this pull request Oct 30, 2020
@probakowski
Hidden data streams (elastic#63987)
eaac4da 
* Hidden data streams

* whitespace reverted

* stricter ds name

* Revert "stricter ds name"

This reverts commit 100dba5.

* String.format removed

* fix test

* fix GetDataStream action

* fix test

* fix test

* rest test

* rest test

* spotless

* tests

* Delete a.json

* added expand_wildcards for GetDataStream and DeleteDataStream

* unused imports

* add hidden setting to data stream template

* fix expand_wildcards

* spotless

* fix compilation

* unused import

* yaml test

* fix test

* fix cleanup

* review

* compilation fix

* fix javadoc

* fix javadoc
@webmat webmat mentioned this pull request Oct 30, 2020
Merged
probakowski added a commit that referenced this pull request Nov 2, 2020
@probakowski
[7.x] Hidden data streams (#63987) (#64402)
a06b65c 
Backports the following commits to 7.x:

Hidden data streams (#63987)
@imotov imotov mentioned this pull request Nov 5, 2020
Closed
@cjcenizal cjcenizal mentioned this pull request Nov 18, 2020
Closed
@cjcenizal
Copy link
Contributor

Opened elastic/kibana#83709 to track the need to reflect these changes in the UI.

@yuliacech yuliacech mentioned this pull request Nov 20, 2020
Closed
@jrodewig jrodewig mentioned this pull request Dec 3, 2020
Merged
This was referenced Dec 7, 2020
Merged
Merged
Merged
Merged
@yuliacech yuliacech mentioned this pull request Dec 10, 2020
Closed
@stevejgordon stevejgordon mentioned this pull request Dec 17, 2020
Closed
@alisonelizabeth alisonelizabeth mentioned this pull request Jan 12, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@martijnvg martijnvg martijnvg approved these changes

@danhermann danhermann Awaiting requested review from danhermann

Assignees
No one assigned
Labels
:Data Management/Data streams Data streams and their lifecycles >enhancement Team:Data Management Meta label for data/management team v7.11.0 v8.0.0-alpha1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support hidden data streams
6 participants
@probakowski @elasticmachine @danhermann @cjcenizal @martijnvg @jakelandis