Description
A goal of the system indices project is to, as much as possible, allow users not to care about system indices (and system data streams). We are not there yet, so we need to figure out how to explain what system indices are and provide enough information to troubleshoot issues that users might have with them, without getting into implementation details.
We have an item about system indices on our 8.x API Conventions page. It reads:
Elasticsearch modules and plugins can store configuration and state information in internal system indices. You should not directly access or modify system indices as they contain data essential to the operation of the system.
Here are some situations where a user might interact with system indices:
- The negative case: "I just upgraded, but where did my
.kibana
and.security
indices go? Did I lose them in the upgrade?" The answer: no, they just became hidden system indices and you can't see them anymore. - The investigation case: "I'm seeing log messages related to an index, but it's not one that I created and not one that I can view in the API." In order to find out what's going on, you will need
superuser
privileges. - The privilege hack: "We keep having trouble with this one system index, so I'd like to give my non-superuser admins permission to view and investigate it." In this case, you would want to create a new security role with an index permission that targets the index in question and has
allow_restricted_indices
set totrue
. (See Defining Roles: Indices Privileges.) - The indirect exposure case: Several diagnostic or informational APIs do nothing to hide the existence of system indices. For example, the
_cat/shards
API will show shards for system indices as well as for normal indices, and system indices are listed in snapshot data.
Finally, we may want to clarify that for the superuser, system indices are like hidden indices, but they have certain additional protection around their settings and mappings.
We may also want to warn about the case where legacy templates that target *
will be applied to system indices, which can sometimes break a cluster in ways that are hard to diagnose.
I'm not exactly sure how to put all this together in the documents, but, basically, we need to give users enough information that, when they run into system indices somehow, they can figure out what's going on.