[BUG] illegal_argument_exception error after upgrading to 3.3.0

[tronboto]

Describe the bug

After performing a rolling upgrade from form 3.2.0 to 3.3.0, we started seeing our vector nodes fail to ingest to Opensearch with the following error:

{
  "errors": true,
  "items": [
    {
      "create": {
        "_index": "catch-all-2025.09.29-000001",
        "status": 400,
        "error": {
          "type": "illegal_argument_exception",
          "reason": "cannot change field \"@timestamp\" from docValuesSkipIndexType=NONE to inconsistent docValuesSkipIndexType=RANGE"
        }
      }
    }
  ]
}

The settings for this index are:

{
  "catch-all-2025.09.29-000001": {
    "settings": {
      "index.codec": "zstd_no_dict",
      "index.creation_date": "1759154069750",
      "index.creation_date_string": "2025-09-29T13:54:29.750Z",
      "index.mapping.ignore_malformed": "true",
      "index.number_of_replicas": "1",
      "index.number_of_shards": "1",
      "index.plugins.index_state_management.rollover_alias": "catch-all",
      "index.priority": "100",
      "index.provided_name": "<catch-all-{now/d}-000001>",
      "index.refresh_interval": "10s",
      "index.replication.type": "SEGMENT",
      "index.routing.allocation.total_shards_per_node": "1",
      "index.unassigned.node_left.delayed_timeout": "5m",
      "index.uuid": "1Edu7vQqQEuSpCMVx6jzrg",
      "index.version.created": "136407827",
      "index.version.created_string": "2.19.0"
    }
  }
}

So it looks like the index was created in 2.19.0 and then we upgraded first to 3.2.0 and then to 3.3.0. We only started seeing this error after upgrading to 3.3.0.

The mapping for @timestamp looks like:

{
  "catch-all-2025.09.29-000001": {
    "mappings": {
      "@timestamp": {
        "full_name": "@timestamp",
        "mapping": {
          "@timestamp": {
            "type": "date"
          }
        }
      }
    }
  }
}

Note that ingest resumes again without error after hitting /catch-all/_rollover.

Related component

Other

To Reproduce

Haven't had time to verify a minimal reproducer just yet

Expected behavior

Ingest shouldn't stop after upgrading

Additional Details

Plugins
n/a

Screenshots
n/a

Host/Environment (please complete the following information):

• OS: Ubuntu 22.04
• Version: opensearch/opensearch:3.3.0 container

Additional context
n/a

[Luzifer]

Same here with FluentBit log ingestion (and other indices). Might be related to #19480

[smortex]

So it looks like the index was created in 2.19.0 and then we upgraded first to 3.2.0 and then to 3.3.0. We only started seeing this error after upgrading to 3.3.0.

We hit that issue, but we have logs and metrics indices created on a daily basis, so the "blocked" ones where created by OpenSearch 3.2.0.

Note that ingest resumes again without error after hitting /catch-all/_rollover.

Workaround confirmed: in our case we had to clone the "live" indices which where not usable, removed the old indices, reindexed the old data in new indices and remove the clones. Data streams would have helped 😁

[getsaurabh02]

@asimmahmood1 is actively looking into the issue and getting a fix in place. We will have a PR out soon.

@peterzhuamazon @andrross we will need to get a patch release out soon with bwc validation.