Skip to content

Composable Templates #53101

New issue
New issue
Closed
Closed
Composable Templates#53101

Description

@martijnvg

martijnvg
openedon Mar 4, 2020

The meta issue will track the development of the second version of index templates.

Tasks:

Tasks that may be done after initial release:

  • Mark APIs as stable for 7.9
  • Switch internal use of templates to use v2 templates (like ILM/SLM history indices, ML, etc)
  • Switch all internal tests to use v2 templates instead of v1 templates
  • Enforce only delete or update operations for v1 templates are allowed on 8.0+ (no creation of new v1 templates)
  • Deprecate the entire v1 template API in 8.0+
  • Add a method to migrate a v1 template into a v2 template? (maybe)
  • Add simulate API for a template ie. /_index_template/{template-name}/_simulate, without any body. It will return the final resolved version of the template (merging the component templates)

Background

Index templates as it stands today has a number of shortcomings mainly around index template inheritance:

  • At index creation time, multiple index templates can be applied. If no template order is specified or templates have the same order then how templates are applied to create the final mappings and index setting is undefined. This can result in settings/mappings being applied in an unexpected manner.
  • With template inheritance, multiple index templates can merged together. The end result of that merge can result in invalid settings and mappings.

Version 2 changes

The biggest change is that reuse of template configuration via template inheritance that exists today is going to get replaced by template composition. A new index will no longer be able to match with multiple index templates. If a new index matches with multiple index templates then the index template with the highest priority is selected and used. If multiple index templates match that have the same priority then that results into an error. Either at alias template creation time or when alias templates are evaluated as part of index creation.

New indices may require the same index template configuration be applied to them, but their names may be so different that a single index template's index pattern can't capture them and because with index templates v2 only a single template can match with a new index. To solve this problem, we'll abstract the index template config into reusable ‘component templates', which users can refer to when composing index templates. A component template only contains settings, mappings and aliases. Index templates can be composed of one or more component templates. The order in which the names of component templates are specified in an index template determines the order in which the component templates are applied. The setting, mapping and aliases specified inside an index template are applied last.

Component templates never match at index creation time directly and are only included via a matching index template. Component templates can’t be composed of other component templates.

Index and component templates can only contain settings and mappings that are valid on their own. The settings and mappings of an individual index template should result in a successful index creation.

Component template merging

The order in which the names of component templates are specified in an index template determines the order in which the component templates are applied. The setting, mapping and aliases specified inside an index template are applied last.

Settings and mappings are essentially nested key value pairs. If a component template is applied and a key already exists then it will replace the key and value even if that contains other key value pairs.

For example consider the following snippets:

{
  "my_field": {
      "type": "integer", 
      "ignore_malformed": true
  }
}
{
  "my_field": {
     "type": "keyword", 
     "ignore_above": 1024
  }
}

With the merging of index templates that exist today, the following result would be created:

{
  "my_field": {
      "ignore_malformed": true,
      "type": "keyword", 
      "ignore_above": 1024
  }
}

In order to avoid this, at the time a template with snippet B is applied then that would completely overwrite the ‘my_field’ key in order to avoid creating an illegal mapping. With this logic this would be the result:

{
  "my_field": {
     "type": "keyword", 
     "ignore_above": 1024
  }
}

Merging will happen on a field granular level, and the V2 templates will ensure no doc types are specified.

Also dynamic templates should be merged in the same manner.

Backwards compatibility

We should strive towards ensuring that existing index templates should be able to work like index templates v2. If index templates v1 are not relying on template inheritance then no action should be required.

With index templates v2 configuration reuse is achieved by selective composability whereas with index templates v1 this is achieved via template inheritance. Template inheritance should be allowed throughout the 7.x version timeline, so that users can upgrade their index templates to use component template. In this period of time we should emit deprecation warnings if a new index matches with more than one index template in the v1 structure. After the deprecation period, if multiple v1 templates match we should fail with an error at index creation time. Users will then have to upgrade to use component templates.

For interop with both v1 and v2 templates the following will be used:

If * in v1, allow but warn to put v2
Runtime resolution will favor v2
If PUT v1 (not ‘*’) and overlaps with v2, then error (v2 will take precedence)
If PUT v1 (with ‘*’) and overlaps with v2, then warn (v2 may take precedence)
If PUT v2 with ‘*’ allow the overlap, warn about taking precedence
If PUT v2 that matches a V1 template, warn that v2 will take precedence
In 8.0 - only allow update/delete of v1, not create (provide an escape hatch setting/property)
In 7.x, deprecate the entire /_template endpoint

In 9.0 the deprecated /_template endpoint will be removed.

The structure and field names have changed slightly with index templates v1. The current structure and field names will be supported throughout the 7.x version timeline. After which we can automatically rewrite the older format into the new format upon upgrading to version 8. (we may be able to provide a conversion API, but we cannot automatically upgrade them since v1 templates rely on inheritance)

Config and APIs

Component templates are stored in the cluster state as separate resources. Index templates remain to be stored in the same place inside the cluster state.

Index Templates v2 Config

Config:

{
    "index_patterns": [...],
    "priority": 0-9223372036854775808,
    "template": {
       "settings": {...},
       "mappings": {...},
       "aliases": {...}
    },
    "composed_of": [...],
    "version": 0-9223372036854775808,
    "_meta": { ... }
}

Explanation of the configuration fields:

  • index_patterns (required), a string array. Each string contains a wildcard pattern.
  • template (optional), a json object at least settings or mappings or aliases.
    • settings (optional), a json object representing index settings.
    • mappings (optional), a json object representing mappings.
    • aliases (optional), a json object representing aliases.
  • priority (optional), a numeric value indicating precedence over other index templates. When indexing into a non existing index/alias, the name may match with patterns of multiple index templates, the alias template with the highest order value will be selected. Other index templates will be ignored. Defaults to 0.
  • composed_of (optional), a string array with the names of components templates that will be applied on a new index together with the current template. The order in the string array determines the order in which the component templates are applied. If composed_of is specified then the index template is applied last. This is to ensure that the settings/mappings for the index template are never overwritten by component templates it references to.
  • version (optional), a numeric value. External systems can use this version number to simplify template management. The version parameter is completely optional and not automatically generated by Elasticsearch. Defaults to null.
  • _meta (optional), a json object with meta information about a template, not used by Elasticsearch, but for third party systems and users.
    Index Template APIs
    Index v2 templates will remain to use the existing index template APIs. Additionally there will be an api to simulate what index template would match with a given index name. Also the final index settings, mappings and aliases would be returned.

Simulate index template api:

GET /_index_template/[index_name]/_simulate

We should also support sending an index template (v2) body with the simulate API, so a UI can simulate adding an index template and seeing what it would look like.

Component Template Config

Config:

{
    "template": {
       "settings": {...},
       "mappings": {...},
       "aliases": {...}
    },
    "version": 0-9223372036854775808,
    "_meta": { ... }
}

Explanation of the configuration fields:

  • template (required), a json object at least settings or mappings or aliases.
    • settings (optional), a json object representing index settings.
    • mappings (optional), a json object representing mappings.
    • aliases (optional), a json object representing aliases.
  • version (optional), a numeric value. External systems can use this version number to simplify template management. The version parameter is completely optional and not automatically generated by Elasticsearch. Defaults to null.
  • _meta (optional), a json object with meta information about a template, not used by Elasticsearch, but for third party systems and users.

Component Template APIs

Put component template api:

PUT /_component_template/[name]
{
   ...
}

Get component template api:

GET /_component_template/[name]

Delete component template api:

DELETE /_component_template/[name]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

Labels

:Data Management/Indices APIsAPIs to create and manage indices and templatesMetaTeam:Data ManagementMeta label for data/management teamTop Ask

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions