[DOCS] How to Increase "index.mapping.total_fields.limit" for new indices

[iquirino]

Hey guys, I hope you're well!

Describe the bug
I need to have more than 1000 fields on my otel indices, this is a must have to my solution.
I've tried to add it to the data-prepper template_file but it's no working as expected.

I want to have the following command automatically applied to each index created by dataprepper:

PUT otel-v1-apm-span-*/_settings
{
  "index.mapping.total_fields.limit": 2500
}

References:

• https://opensearch.org/docs/2.7/data-prepper/pipelines/configuration/sinks/opensearch/
• https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch/src/main/resources/otel-v1-apm-span-index-template.json
• https://stackoverflow.com/questions/55372330/what-does-limit-of-total-fields-1000-in-index-has-been-exceeded-means-in

To Reproduce

pipelines.yaml

sink:
  - opensearch:
      hosts: [ "https://node-0.example.com:9200" ]
      #index_type: trace-analytics-raw
      index_type: custom
      template_file: "/usr/share/data-prepper/templates/index-template.json"
      cert: "/usr/share/data-prepper/root-ca.pem"
      username: "admin"
      password: "admin"
      socket_timeout: 0
      connect_timeout: 0

index-template.json

{
    "version": 1,
    "settings": {
      "index": { "mapping": { "total_fields": { "limit": "2500" } } }
    },
    "mappings": {
        // ...
    }
}

GET otel-v1-apm-span-*/_settings

{
  "otel-v1-apm-span-": {
    "settings": {
      "index": {
        "number_of_shards": "1",
        "provided_name": "otel-v1-apm-span-",
        "creation_date": "1685351383685",
        "number_of_replicas": "1",
        "uuid": "P_ss8jsCRj2kreesYZWPbw",
        "version": {
          "created": "136277827"
        }
      }
    }
  }
}

I am required to run the following command for each index created to have my index settings as desired:

PUT otel-v1-apm-span-*/_settings
{
  "index.mapping.total_fields.limit": 2500
}

Result:

{
  "otel-v1-apm-span-": {
    "settings": {
      "index": {
        "mapping": {
          "total_fields": {
            "limit": "2500"
          }
        },
        "number_of_shards": "1",
        "provided_name": "otel-v1-apm-span-",
        "creation_date": "1685351383685",
        "number_of_replicas": "1",
        "uuid": "P_ss8jsCRj2kreesYZWPbw",
        "version": {
          "created": "136277827"
        }
      }
    }
  }
}

Expected behavior
Data prepper should follow my index settings on template_file or have any other way to configure it's index settings.

Screenshots
My setting is not going to be used by data-prepper to create the index:

When i've applied it manually

Environment (please complete the following information):

• OS: Manjaro Latest and Ubuntu Latest

Additional context
No

[iquirino]

Template File doesn't work if you don't specify an ism_policy_file.

I've attached the original ism from source code and it worked well:

data-prepper-plugins/opensearch/src/main/resources/raw-span-policy-with-ism-template.json

{
  "policy": {
    "description": "Managing raw spans for trace analytics",
    "default_state": "current_write_index",
    "states": [
      {
        "name": "current_write_index",
        "actions": [
          {
            "rollover": {
              "min_size": "50gb",
              "min_index_age": "24h"
            }
          }
        ]
      }
    ],
    "ism_template": {
      "index_patterns": ["otel-v1-apm-span-*"]
    }
  }
}

[iquirino]

I am changing the title because it is not a bug, just a lack of documentation.
Please add this details to the documentation page.
An example on documentation should work better ;)

[dlvenable]

@iquirino , Thank you for this feedback. If I understand correctly, the problem you had was that you wanted to change the template file, but was surprised that you also had to change the ISM policy. Correct? We can make a note to update the documentation on this, if this is the problem.

[iquirino]

Yes, I didn't made any changes to ISM Policy, but I am required to attach it to the sink configuration.
You should mention this on your documentation, or remove this requirement.

[JannikBrand]

In general, I think the question should be "How to set index settings for Data Prepper managed indices" (e.g., otel-v1-apm-span-* indices) which come with an index template by default. For example, setting the number of shards or any other index setting seems interesting to me.
@iquirino chose the way to set the index_type to custom and specifying his own template file containing index settings and probably the copied mappings of the original index template. That would be one way how to do it.

Currently, I would solve this by keeping the trace-analytics-raw or trace-analytics-service-map index_type and separately uploading a (legacy) index template which sets the index settings for the respective indices. Both, Data Prepper's templates and the separately uploaded template would be applied to individual indices.

Since version 2.3.0 one can also decide to use the new index template OpenSearch API instead of the legacy API. This would prohibit the previous approach, since only Data Prepper's index template would be applied because it takes preference over any matching legacy template and also only one index template (with the highest priority) can be applied to an index simultaneously.
Here, the only option would be to use the approach from @iquirino described at the top.
I would have an idea for a better solution: Introducing a new property within the opensearch plugin which allows to specify a component template. This component template will be uploaded by Data Prepper before the index template gets uploaded. The component template can contain possible index settings. Then, the default index template references this component template within the composed_of field, allowing index settings to be injected into the template.
Reference: Composable Templates

@dlvenable, What do you think about this approach?

[BarRozner-Salt]

@dlvenable I think easy settings "num_of_shards" for the otel-v1-apm-span-* indices via the sink conf is crucial. I suggest that this configuration will be applied regardless of type, well especially in builtin types..