New indexes created for datastreams after update to `8.15.0` are without lifecycle policies

[lahsivjar]

APM Server version (apm-server version): 8.15.0

Description of the problem including expected versus actual behavior: In 8.15.0 we migrated from ILM to DSL. New indexes created for clusters which migrate to 8.15.0 don't have any lifecycle attached as existing datastream needs to be updated explicitly: https://www.elastic.co/guide/en/elasticsearch/reference/current/tutorial-manage-existing-data-stream.html.

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including server configuration, agent(s) used, etc. The easier you make it
for us to reproduce it, the more likely that somebody will take the time to
look at it.

1. Create a cluster with version < 8.15.0
2. Send data to the cluster (for example: traces) (and continue sending it throughout the steps)
3. Perform a rollover on the datastream getting data
4. Upgrade the cluster to 8.15.0
5. Observe that the latest indices are Unmanaged (for example: by using GET /_data_stream/traces-apm-default for traces datastream)
6. Perform a rollover on the datastream getting data
7. Observe that the newly created index is also Unmanaged (for example: by using GET /_data_stream/traces-apm-default for traces datastream)

Temporary mitigation is to explicitly set the lifecycle using the PUT API. For example, the below operations would set data retention based on APM defaults for all APM datastreams:

PUT _data_stream/traces-apm-*/_lifecycle
{
  "data_retention": "10d" 
}

PUT _data_stream/traces-apm.rum*/_lifecycle
{
  "data_retention": "90d" 
}

PUT _data_stream/traces-apm.sampled*/_lifecycle
{
  "data_retention": "1h" 
}

PUT _data_stream/metrics-apm.*.1m-*/_lifecycle
{
  "data_retention": "90d" 
}

PUT _data_stream/metrics-apm.*.10m-*/_lifecycle
{
  "data_retention": "180d" 
}

PUT _data_stream/metrics-apm.*.60m-*/_lifecycle
{
  "data_retention": "390d" 
}

PUT _data_stream/metrics-apm.internal-*/_lifecycle
{
  "data_retention": "90d" 
}

PUT _data_stream/metrics-apm.app.*/_lifecycle
{
  "data_retention": "90d" 
}

PUT _data_stream/logs-apm.*/_lifecycle
{
  "data_retention": "10d" 
}

Provide logs (if relevant): N/A

[lahsivjar]

The fix for v8.15.1 has been merged via elastic/elasticsearch#112097, however, the same patch cannot be applied to v8.16.0. The details of this are tracked in elastic/elasticsearch#112137

[axw]

I don't love elastic/elasticsearch#112097 as a solution, since it kinda implies we'll keep the old ILM config around in the index templates for perpetuity.

Perhaps we can remove that while also addressing serverless, by dynamically injecting the ILM config into the index template if there are existing indices with ILM config?

[axw]

We need to handle the following scenarios:

• Create a new 8.15.1+ cluster, with defaults: DLM should be used
• Create a new 8.15.1+ cluster, with customised ILM (https://www.elastic.co/guide/en/observability/current/apm-ilm-how-to.html#apm-data-streams-custom-three): customised ILM policy should be used
• Upgrade 8.14.x to 8.15.1+ with defaults: ILM should continue to be used for old indices, DLM should be used for new indices
• Upgrade 8.14.x to 8.15.1+ with customised ILM (same guide as above): ILM should continue to be used for old indices, and also for new indices

[lahsivjar]

With the current changes, the custom ILM policies would break even for versions >= 8.15.1 as index templates cannot be overridden. elastic/elasticsearch#112432 provides a fix for this by moving the fallback to component template which could be overridden as required. NOTE that this would also require us to update our documents for configuring custom ILM policy to include prefer_ilm: true for datastreams created on or after 8.15.x

Testing elastic/elasticsearch#112432 locally

Create a new 8.15.1+ cluster, with defaults: DLM should be used

✅ Tested (new cluster created locally)

{
  "data_streams": [
    {
      "name": "traces-apm-default",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-traces-apm-default-2024.09.02-000001",
          "index_uuid": "4rLT17ZQQIC6mFDlPS1SNA",
          "prefer_ilm": false,
          "ilm_policy": "traces-apm.traces-default_policy",
          "managed_by": "Data stream lifecycle"
        }
      ],
      "generation": 1,
      "_meta": {
        "description": "Index template for traces-apm-*",
        "managed": true
      },
      "status": "YELLOW",
      "template": "traces-apm@template",
      "lifecycle": {
        "enabled": true,
        "data_retention": "10d"
      },
      "ilm_policy": "traces-apm.traces-default_policy",
      "next_generation_managed_by": "Data stream lifecycle",
      "prefer_ilm": false,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "rollover_on_write": false,
      "failure_store": {
        "enabled": false,
        "rollover_on_write": true,
        "indices": []
      }
    }
  ]
}

Create a new 8.15.1+ cluster, with customised ILM (https://www.elastic.co/guide/en/observability/current/apm-ilm-how-to.html#apm-data-streams-custom-three): customised ILM policy should be used

✅ Tested by created traces-apm@custom component template with a custom ILM policy but prefer_ilm as false

Since the prefer_ilm is false, DSL took priority.

{
  "data_streams": [
    {
      "name": "traces-apm-default",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-traces-apm-default-2024.09.02-000001",
          "index_uuid": "4rLT17ZQQIC6mFDlPS1SNA",
          "prefer_ilm": false,
          "ilm_policy": "traces-apm.traces-default_policy",
          "managed_by": "Data stream lifecycle"
        },
        {
          "index_name": ".ds-traces-apm-default-2024.09.02-000002",
          "index_uuid": "FeVicze2S7ex5o7R7yeLaQ",
          "prefer_ilm": false,
          "ilm_policy": "custom-ilm-override-dsl",
          "managed_by": "Data stream lifecycle"
        }
      ],
      "generation": 2,
      "_meta": {
        "description": "Index template for traces-apm-*",
        "managed": true
      },
      "status": "YELLOW",
      "template": "traces-apm@template",
      "lifecycle": {
        "enabled": true,
        "data_retention": "10d"
      },
      "ilm_policy": "custom-ilm-override-dsl",
      "next_generation_managed_by": "Data stream lifecycle",
      "prefer_ilm": false,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "rollover_on_write": false,
      "failure_store": {
        "enabled": false,
        "rollover_on_write": true,
        "indices": []
      }
    }
  ]
}

✅ Tested by updating traces-apm@custom component template to include prefer_ilm as true

{
  "data_streams": [
    {
      "name": "traces-apm-default",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-traces-apm-default-2024.09.02-000001",
          "index_uuid": "4rLT17ZQQIC6mFDlPS1SNA",
          "prefer_ilm": false,
          "ilm_policy": "traces-apm.traces-default_policy",
          "managed_by": "Data stream lifecycle"
        },
        {
          "index_name": ".ds-traces-apm-default-2024.09.02-000002",
          "index_uuid": "FeVicze2S7ex5o7R7yeLaQ",
          "prefer_ilm": false,
          "ilm_policy": "custom-ilm-override-dsl",
          "managed_by": "Data stream lifecycle"
        },
        {
          "index_name": ".ds-traces-apm-default-2024.09.02-000003",
          "index_uuid": "7ufMCy5RTBaxTBJ78dxC-A",
          "prefer_ilm": true,
          "ilm_policy": "custom-ilm-override-dsl",
          "managed_by": "Index Lifecycle Management"
        }
      ],
      "generation": 3,
      "_meta": {
        "description": "Index template for traces-apm-*",
        "managed": true
      },
      "status": "YELLOW",
      "template": "traces-apm@template",
      "lifecycle": {
        "enabled": true,
        "data_retention": "10d"
      },
      "ilm_policy": "custom-ilm-override-dsl",
      "next_generation_managed_by": "Index Lifecycle Management",
      "prefer_ilm": true,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "rollover_on_write": false,
      "failure_store": {
        "enabled": false,
        "rollover_on_write": true,
        "indices": []
      }
    }
  ]
}

Upgrade 8.14.x to 8.15.1+ with defaults: ILM should continue to be used for old indices, DLM should be used for new indices

@axw this case, as expected, did not work. The reason is that Datastream Lifecycle needs to be explicitly configured for ALREADY created datastreams. So, for any old cluster which has used APM and upgraded to 8.15.1+ would continue to use ILM (even for new indices) unless they explicitly configure DSL using the PUT API.

❌ Tested by updating from 8.14.x to 8.15.1

{
  "data_streams": [
    {
      "name": "traces-apm-default",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-traces-apm-default-2024.09.02-000001",
          "index_uuid": "farCK-0lRI-H64IxGfRUuQ",
          "prefer_ilm": true,
          "ilm_policy": "traces-apm.traces-default_policy",
          "managed_by": "Index Lifecycle Management"
        },
        {
          "index_name": ".ds-traces-apm-default-2024.09.02-000002",
          "index_uuid": "C_qI1J2FQ5O-81VuHMl82A",
          "prefer_ilm": true,
          "ilm_policy": "traces-apm.traces-default_policy",
          "managed_by": "Index Lifecycle Management"
        },
        {
          "index_name": ".ds-traces-apm-default-2024.09.02-000003",
          "index_uuid": "HVQRbVrtSEm6UmeyPWj8bA",
          "prefer_ilm": false,
          "ilm_policy": "traces-apm.traces-default_policy",
          "managed_by": "Index Lifecycle Management"
        }
      ],
      "generation": 3,
      "_meta": {
        "package": {
          "name": "apm"
        },
        "managed_by": "fleet",
        "managed": true
      },
      "status": "YELLOW",
      "template": "traces-apm@template",
      "ilm_policy": "traces-apm.traces-default_policy",
      "next_generation_managed_by": "Index Lifecycle Management",
      "prefer_ilm": false,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "rollover_on_write": false,
      "failure_store": {
        "enabled": false,
        "rollover_on_write": false,
        "indices": []
      }
    }
  ]
}

Upgrade 8.14.x to 8.15.1+ with customised ILM (same guide as above): ILM should continue to be used for old indices, and also for new indices

✅ Tested (cluster created locally with 8.14.3 and upgraded to 8.15.1)

{
  "data_streams": [
    {
      "name": "traces-apm-default",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-traces-apm-default-2024.09.02-000001",
          "index_uuid": "I5k3QG4QRGGZLdUnNAWpJQ",
          "prefer_ilm": true,
          "ilm_policy": "traces-apm.traces-default_policy",
          "managed_by": "Index Lifecycle Management"
        },
        {
          "index_name": ".ds-traces-apm-default-2024.09.02-000002",
          "index_uuid": "IoauvZ7pQqC-xsRkDTLBkA",
          "prefer_ilm": true,
          "ilm_policy": "custom-ilmdlm-test",
          "managed_by": "Index Lifecycle Management"
        },
        {
          "index_name": ".ds-traces-apm-default-2024.09.02-000003",
          "index_uuid": "DDJu8FbnTUCapW6Xr13-Kw",
          "prefer_ilm": false,
          "ilm_policy": "custom-ilmdlm-test",
          "managed_by": "Index Lifecycle Management"
        }
      ],
      "generation": 3,
      "_meta": {
        "package": {
          "name": "apm"
        },
        "managed_by": "fleet",
        "managed": true
      },
      "status": "YELLOW",
      "template": "traces-apm@template",
      "ilm_policy": "custom-ilmdlm-test",
      "next_generation_managed_by": "Index Lifecycle Management",
      "prefer_ilm": false,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "rollover_on_write": false,
      "failure_store": {
        "enabled": false,
        "rollover_on_write": false,
        "indices": []
      }
    }
  ]
}

Upgrade 8.14.x to 8.15.0 with custom ILM policy: custom ILM policy should work as expected (no bugs for this case)

✅ Tested by upgrading from 8.14.3 with custom ILM to 8.15.0

{
  "data_streams": [
    {
      "name": "traces-apm-default",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-traces-apm-default-2024.09.02-000001",
          "index_uuid": "dFLoaPf3QZC4t7ttplDSqg",
          "prefer_ilm": true,
          "ilm_policy": "custom-ilmdlm-test",
          "managed_by": "Index Lifecycle Management"
        },
        {
          "index_name": ".ds-traces-apm-default-2024.09.02-000002",
          "index_uuid": "fJ__qQv7Tumu9n_j8hEljA",
          "prefer_ilm": true,
          "ilm_policy": "custom-ilmdlm-test",
          "managed_by": "Index Lifecycle Management"
        },
        {
          "index_name": ".ds-traces-apm-default-2024.09.02-000003",
          "index_uuid": "kNSWO6pMQ06eZo7yHqKQCw",
          "prefer_ilm": true,
          "ilm_policy": "custom-ilmdlm-test",
          "managed_by": "Index Lifecycle Management"
        }
      ],
      "generation": 3,
      "_meta": {
        "package": {
          "name": "apm"
        },
        "managed_by": "fleet",
        "managed": true
      },
      "status": "YELLOW",
      "template": "traces-apm@template",
      "ilm_policy": "custom-ilmdlm-test",
      "next_generation_managed_by": "Index Lifecycle Management",
      "prefer_ilm": true,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "rollover_on_write": false
    }
  ]
}

Upgrade 8.14.x to 8.15.0 with default ILM policy and then upgrade to 8.15.1

This, as expected, creates unmanaged indices. We would need to suggest workaround for this in our changelog/release-notes. The simplest way would be to have users with default ILM settings explicitly configure DSL for all datastreams. Since default ILM and DSL are identical, there would be no impact to the users. In future, if the user wants to move to ILM, they can do so by creating a custom component template. WDYT @axw ?

⚠️ Tested: leads to unmanaged indices for indices that were created in 8.15.0

{
  "data_streams": [
    {
      "name": "traces-apm-default",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-traces-apm-default-2024.09.02-000001",
          "index_uuid": "4MAl7phPSkexDJFda-38RQ",
          "prefer_ilm": true,
          "ilm_policy": "traces-apm.traces-default_policy",
          "managed_by": "Index Lifecycle Management"
        },
        {
          "index_name": ".ds-traces-apm-default-2024.09.02-000002",
          "index_uuid": "hv1XnUBeQmim7VkOgvHAiw",
          "prefer_ilm": true,
          "ilm_policy": "traces-apm.traces-default_policy",
          "managed_by": "Index Lifecycle Management"
        },
        {
          "index_name": ".ds-traces-apm-default-2024.09.02-000003",
          "index_uuid": "jb2UpzXgQ_CmcVPTW2ICwA",
          "prefer_ilm": true,
          "managed_by": "Unmanaged"
        },
        {
          "index_name": ".ds-traces-apm-default-2024.09.02-000004",
          "index_uuid": "4e9MCYGWSqKR8BZ2JYKZeA",
          "prefer_ilm": true,
          "managed_by": "Unmanaged"
        },
        {
          "index_name": ".ds-traces-apm-default-2024.09.02-000005",
          "index_uuid": "OlSp2sXGRA6ozl86UIe3Lw",
          "prefer_ilm": false,
          "ilm_policy": "traces-apm.traces-default_policy",
          "managed_by": "Index Lifecycle Management"
        }
      ],
      "generation": 5,
      "_meta": {
        "package": {
          "name": "apm"
        },
        "managed_by": "fleet",
        "managed": true
      },
      "status": "YELLOW",
      "template": "traces-apm@template",
      "ilm_policy": "traces-apm.traces-default_policy",
      "next_generation_managed_by": "Index Lifecycle Management",
      "prefer_ilm": false,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "rollover_on_write": false,
      "failure_store": {
        "enabled": false,
        "rollover_on_write": false,
        "indices": []
      }
    }
  ]
}

[simitt]

Reading through the test cases, can you clarify if this is true: customers on 8.14.3 with custom ILM policies, who upgrade to 8.15.1 will not have to do any manual interactions for their custom ILM policies to continue to be applied?

Please also provide test scenarios where users already have upgraded to 8.15.0 and then upgrade to 8.15.1.