Description
Elasticsearch Version
8.7
Installed Plugins
No response
Java Version
bundled
OS Version
any
Problem Description
As of 8.7 the default _source behaviour linked to index.mode: time_series was changed. Prior to 8.7, _source was explicitly stored, from this version onward it became synthetics.
This change cannot be easily visualised by users, as the corresponding setting ("_source": {"mode": "synthetic"}) doesn't appear in the resulting indices.
Moreover, if a user sets this by hand in a template, indices created don't have the setting. It is as if the setting is ignored by Elasticsearch upon index creation. This is extremely confusing for users.
In general, this behaviour is misleading, since there is no clear indication to the user that the _source isn't fully stored. Some customers also explicitly require their _source not to be tampered with, for legal and auditing reasons.
This behaviour (if intended) needs to be clearly documented in the TSDS or _source documentation page.
We further propose that any mapping or setting explicitly written in a template should be explicitly written in any index that is instantiated from it. Anything else is too misleading.
It is also somewhat of a concern that there now appear to be two separate sources of truth for "mode": "synthetic": one in mappings and another hidden somewhere in our code.
Steps to Reproduce
This can be reproduced by running the following (notice the change of behaviour pre/post 8.7):
PUT /_index_template/test
{
"index_patterns": [
"foobar*"
],
"template": {
"settings": {
"index": {
"mode": "time_series",
"routing_path": [
"host.name"
]
}
},
"mappings": {
"_source": {
"mode": "synthetic"
}
}
},
"composed_of": [
],
"priority": 500,
"data_stream": {
}
}
POST /_index_template/_simulate/test
Gives the following on 8.6.x:
"mappings": {
"_source": {
"mode": "synthetic"
}
},
This very section disappears on 8.7.x, while synthetic source remains activated.