Setup data streams and send events to them (#28450)

## What does this PR do?
  
  We are introducing data streams to Beats. It means that all Beats are going to send events to data streams instead of indices regardless of ES version. Do not confuse it with the data stream naming convention we use in integrations. Naming does not change in Beats, only the underlying data storage method in Elasticsearch.
  
  The name of the data stream is going to be `{beatname}-{version}` and the index pattern is `{beatname}-{version}`.
  
  With this change, the option `setup.template.type` no longer makes sense. Hence, it is removed completely from 8.x. If you are loading JSON index templates by specifying a file in `setup.template.json.path`, make sure you move from the legacy format to composable index templates.
  
  Beats no longer load an alias to Elasticsearch, instead all data can be reached through the data stream.
  
  One of the limitations is that only create operations are supported in data streams. Thus, there is no way to use e.g. "index" or "delete" operation types when sending events to ES.
  
  ## Why is it important?
  
  Simplify loading ILM, templates, and use the specialized data streams for output events.

CHANGELOG.next.asciidoc

@@ -25,6 +25,9 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Remove `auto` from the available options of `setup.ilm.enabled` and set the default value to `true`. {pull}28671[28671]
 - add_process_metadata processor: Replace usage of deprecated `process.ppid` field with `process.parent.pid`. {pull}28620[28620]
 - add_docker_metadata processor: Replace usage of deprecated `process.ppid` field with `process.parent.pid`. {pull}28620[28620]
+- Use data streams instead of indices for storing events from Beats. {pull}28450[28450]
+- Remove option `setup.template.type` and always load composable template with data streams. {pull}28450[28450]
+- Remove several ILM options (`rollover_alias` and `pattern`) as data streams does not require index aliases. {pull}28450[28450] 
 - Index template's default_fields setting is only populated with ECS fields. {pull}28596[28596] {issue}28215[28215]
 - Remove deprecated `--template` and `--ilm-policy` flags. Use `--index-management` instead. {pull}28870[28870]
 - Remove options `logging.files.suffix` and default to datetime endings. {pull}28927[28927]

auditbeat/auditbeat.reference.yml

@@ -1145,19 +1145,13 @@ output.elasticsearch:
 # Set to false to disable template loading.
 #setup.template.enabled: true
 
-# Select the kind of index template. From Elasticsearch 7.8, it is possible to
-# use component templates. Available options: legacy, component, index.
-# By default auditbeat uses the legacy index templates.
-#setup.template.type: legacy
-
 # Template name. By default the template name is "auditbeat-%{[agent.version]}"
 # The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
 #setup.template.name: "auditbeat-%{[agent.version]}"
 
-# Template pattern. By default the template pattern is "-%{[agent.version]}-*" to apply to the default index settings.
-# The first part is the version of the beat and then -* is used to match all daily indices.
+# Template pattern. By default the template pattern is "auditbeat-%{[agent.version]}" to apply to the default index settings.
 # The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
-#setup.template.pattern: "auditbeat-%{[agent.version]}-*"
+#setup.template.pattern: "auditbeat-%{[agent.version]}"
 
 # Path to fields.yml file to generate the template
 #setup.template.fields: "${path.config}/fields.yml"
@@ -1204,17 +1198,8 @@ setup.template.settings:
 # output.elasticsearch.index is ignored, and the write alias is used to set the
 # index name.
 
-# Enable ILM support. Valid values are true, false, and auto. When set to auto
-# (the default), the Beat uses index lifecycle management when it connects to a
-# cluster that supports ILM; otherwise, it creates daily indices.
-#setup.ilm.enabled: auto
-
-# Set the prefix used in the index lifecycle write alias name. The default alias
-# name is 'auditbeat-%{[agent.version]}'.
-#setup.ilm.rollover_alias: 'auditbeat'
-
-# Set the rollover index pattern. The default is "%{now/d}-000001".
-#setup.ilm.pattern: "{now/d}-000001"
+# Enable ILM support. Valid values are true, false.
+#setup.ilm.enabled: true
 
 # Set the lifecycle policy name. The default policy name is
 # 'beatname'.

filebeat/filebeat.reference.yml

@@ -2078,19 +2078,13 @@ output.elasticsearch:
 # Set to false to disable template loading.
 #setup.template.enabled: true
 
-# Select the kind of index template. From Elasticsearch 7.8, it is possible to
-# use component templates. Available options: legacy, component, index.
-# By default filebeat uses the legacy index templates.
-#setup.template.type: legacy
-
 # Template name. By default the template name is "filebeat-%{[agent.version]}"
 # The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
 #setup.template.name: "filebeat-%{[agent.version]}"
 
-# Template pattern. By default the template pattern is "-%{[agent.version]}-*" to apply to the default index settings.
-# The first part is the version of the beat and then -* is used to match all daily indices.
+# Template pattern. By default the template pattern is "filebeat-%{[agent.version]}" to apply to the default index settings.
 # The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
-#setup.template.pattern: "filebeat-%{[agent.version]}-*"
+#setup.template.pattern: "filebeat-%{[agent.version]}"
 
 # Path to fields.yml file to generate the template
 #setup.template.fields: "${path.config}/fields.yml"
@@ -2137,17 +2131,8 @@ setup.template.settings:
 # output.elasticsearch.index is ignored, and the write alias is used to set the
 # index name.
 
-# Enable ILM support. Valid values are true, false, and auto. When set to auto
-# (the default), the Beat uses index lifecycle management when it connects to a
-# cluster that supports ILM; otherwise, it creates daily indices.
-#setup.ilm.enabled: auto
-
-# Set the prefix used in the index lifecycle write alias name. The default alias
-# name is 'filebeat-%{[agent.version]}'.
-#setup.ilm.rollover_alias: 'filebeat'
-
-# Set the rollover index pattern. The default is "%{now/d}-000001".
-#setup.ilm.pattern: "{now/d}-000001"
+# Enable ILM support. Valid values are true, false.
+#setup.ilm.enabled: true
 
 # Set the lifecycle policy name. The default policy name is
 # 'beatname'.

filebeat/tests/system/config/filebeat.yml.j2

@@ -120,14 +120,8 @@ filebeat.config.{{ reload_type|default("inputs") }}:
 
 {% if ilm %}
 setup.ilm:
-  enabled: {{ ilm.enabled | default("auto") }}
+  enabled: {{ ilm.enabled | default("true") }}
   policy_name: libbeat-test-default-policy
-  {% if ilm.pattern %}
-  pattern: {{ ilm.pattern }}
-  {% endif %}
-  {% if ilm.rollover_alias %}
-  rollover_alias: {{ ilm.rollover_alias }}
-  {% endif %}
 {% endif %}
 
 

filebeat/tests/system/config/filebeat_inputs.yml.j2

@@ -17,14 +17,8 @@ filebeat.registry:
 
 {% if ilm %}
 setup.ilm:
-  enabled: {{ ilm.enabled | default("auto") }}
+  enabled: {{ ilm.enabled | default("true") }}
   policy_name: libbeat-test-default-policy
-  {% if ilm.pattern %}
-  pattern: {{ ilm.pattern }}
-  {% endif %}
-  {% if ilm.rollover_alias %}
-  rollover_alias: {{ ilm.rollover_alias }}
-  {% endif %}
 {% endif %}
 
 

filebeat/tests/system/test_modules.py

@@ -104,7 +104,7 @@ def run_on_file(self, module, fileset, test_file, cfgfile):
         self.assert_explicit_ecs_version_set(module, fileset)
 
         try:
-            self.es.indices.delete(index=self.index_name)
+            self.es.indices.delete_data_stream(self.index_name)
         except BaseException:
             pass
         self.wait_until(lambda: not self.es.indices.exists(self.index_name))
@@ -242,7 +242,7 @@ def clean_keys(obj):
         host_keys.append("host.name")
 
     # The create timestamps area always new
-    time_keys = ["event.created", "event.ingested"]
+    time_keys = ["event.created", "event.ingested", "@timestamp"]
     # source path and agent.version can be different for each run
     other_keys = ["log.file.path", "agent.version"]
     # ECS versions change for any ECS release, large or small

heartbeat/heartbeat.reference.yml

@@ -1291,19 +1291,13 @@ output.elasticsearch:
 # Set to false to disable template loading.
 #setup.template.enabled: true
 
-# Select the kind of index template. From Elasticsearch 7.8, it is possible to
-# use component templates. Available options: legacy, component, index.
-# By default heartbeat uses the legacy index templates.
-#setup.template.type: legacy
-
 # Template name. By default the template name is "heartbeat-%{[agent.version]}"
 # The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
 #setup.template.name: "heartbeat-%{[agent.version]}"
 
-# Template pattern. By default the template pattern is "-%{[agent.version]}-*" to apply to the default index settings.
-# The first part is the version of the beat and then -* is used to match all daily indices.
+# Template pattern. By default the template pattern is "heartbeat-%{[agent.version]}" to apply to the default index settings.
 # The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
-#setup.template.pattern: "heartbeat-%{[agent.version]}-*"
+#setup.template.pattern: "heartbeat-%{[agent.version]}"
 
 # Path to fields.yml file to generate the template
 #setup.template.fields: "${path.config}/fields.yml"
@@ -1350,17 +1344,8 @@ setup.template.settings:
 # output.elasticsearch.index is ignored, and the write alias is used to set the
 # index name.
 
-# Enable ILM support. Valid values are true, false, and auto. When set to auto
-# (the default), the Beat uses index lifecycle management when it connects to a
-# cluster that supports ILM; otherwise, it creates daily indices.
-#setup.ilm.enabled: auto
-
-# Set the prefix used in the index lifecycle write alias name. The default alias
-# name is 'heartbeat-%{[agent.version]}'.
-#setup.ilm.rollover_alias: 'heartbeat'
-
-# Set the rollover index pattern. The default is "%{now/d}-000001".
-#setup.ilm.pattern: "{now/d}-000001"
+# Enable ILM support. Valid values are true, false.
+#setup.ilm.enabled: true
 
 # Set the lifecycle policy name. The default policy name is
 # 'beatname'.

libbeat/_meta/config/setup.ilm.reference.yml.tmpl

@@ -5,17 +5,8 @@
 # output.elasticsearch.index is ignored, and the write alias is used to set the
 # index name.
 
-# Enable ILM support. Valid values are true, false, and auto. When set to auto
-# (the default), the Beat uses index lifecycle management when it connects to a
-# cluster that supports ILM; otherwise, it creates daily indices.
-#setup.ilm.enabled: auto
-
-# Set the prefix used in the index lifecycle write alias name. The default alias
-# name is '{{.BeatName}}-%{[agent.version]}'.
-#setup.ilm.rollover_alias: '{{.BeatIndexPrefix}}'
-
-# Set the rollover index pattern. The default is "%{now/d}-000001".
-#setup.ilm.pattern: "{now/d}-000001"
+# Enable ILM support. Valid values are true, false.
+#setup.ilm.enabled: true
 
 # Set the lifecycle policy name. The default policy name is
 # 'beatname'.

libbeat/_meta/config/setup.template.reference.yml.tmpl

@@ -7,19 +7,13 @@
 # Set to false to disable template loading.
 #setup.template.enabled: true
 
-# Select the kind of index template. From Elasticsearch 7.8, it is possible to
-# use component templates. Available options: legacy, component, index.
-# By default {{.BeatName}} uses the legacy index templates.
-#setup.template.type: legacy
-
 # Template name. By default the template name is "{{.BeatIndexPrefix}}-%{[agent.version]}"
 # The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
 #setup.template.name: "{{.BeatIndexPrefix}}-%{[agent.version]}"
 
-# Template pattern. By default the template pattern is "-%{[agent.version]}-*" to apply to the default index settings.
-# The first part is the version of the beat and then -* is used to match all daily indices.
+# Template pattern. By default the template pattern is "{{.BeatIndexPrefix}}-%{[agent.version]}" to apply to the default index settings.
 # The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
-#setup.template.pattern: "{{.BeatIndexPrefix}}-%{[agent.version]}-*"
+#setup.template.pattern: "{{.BeatIndexPrefix}}-%{[agent.version]}"
 
 # Path to fields.yml file to generate the template
 #setup.template.fields: "${path.config}/fields.yml"

libbeat/docs/howto/load-index-templates.asciidoc

@@ -224,7 +224,7 @@ PS > Invoke-RestMethod -Method Delete "http://localhost:9200/{beatname_lc}-*"
 ----------------------------------------------------------------------
 endif::win_os[]
 
-This command deletes all indices that match the pattern +{beat_default_index_prefix}-*+.
+This command deletes all indices that match the pattern +{beat_default_index_prefix}+.
 Before running this command, make sure you want to delete all indices that match
 the pattern.
 

libbeat/docs/template-config.asciidoc

@@ -26,13 +26,6 @@ existing one.
 *`setup.template.enabled`*:: Set to false to disable template loading. If this is set to false,
 you must <<load-template-manually,load the template manually>>.
 
-ifndef::apm-server[]
-*`setup.template.type`*:: The type of template to use. Available options: `legacy` (default), index templates
-before Elasticsearch v7.8. Use this to avoid breaking existing deployments. New options are `component`
-and `index`. Selecting `component` loads a component template which can be included in new index templates.
-The option `index` loads the new index template.
-endif::[]
-
 *`setup.template.name`*:: The name of the template. The default is
 +{beatname_lc}+. The {beatname_uc} version is always appended to the given
 name, so the final name is +{beatname_lc}-%{[{beat_version_key}]}+.
@@ -44,17 +37,16 @@ name, so the final name is +{beatname_lc}-%{[{beat_version_key}]}+.
 // the example as expected.
 
 *`setup.template.pattern`*:: The template pattern to apply to the default index
-settings. The default pattern is +{beat_default_index_prefix}-\*+. The {beatname_uc} version is always
+settings. The default pattern is +{beat_default_index_prefix}+. The {beatname_uc} version is always
 included in the pattern, so the final pattern is
-+{beat_default_index_prefix}-%{[{beat_version_key}]}-*+. The wildcard character `-*` is used to
-match all daily indices.
++{beat_default_index_prefix}-%{[{beat_version_key}]}+. 
 +
 Example:
 +
 ["source","yaml",subs="attributes"]
 ----------------------------------------------------------------------
 setup.template.name: "{beatname_lc}"
-setup.template.pattern: "{beat_default_index_prefix}-*"
+setup.template.pattern: "{beat_default_index_prefix}"
 ----------------------------------------------------------------------
 
 *`setup.template.fields`*:: The path to the YAML file describing the fields. The default is +fields.yml+. If a