Add ES8 backwards compatibility for workflow classifiers

[rajeshwar-nu]

Summary

• Add the classifier keyword mapping to existing ES8 workflow indices during startup.
• Treat legacy workflow documents without a classifier field as regular workflows when filtering for classifier=workflow.
• Keep classifier=agent filtering strict.
• Limit the compatibility behavior to the ES8 persistence module.

Why

The workflow classifier field was added after some Elasticsearch workflow indices had already been created. Those indices use dynamic: false, so classifier values can appear in _source without being indexed. The Conductor UI filters workflow searches using classifier=workflow, which causes legacy workflow documents with no indexed classifier to disappear from results.

Updating the index template only fixes newly created indices. Existing indices need an additive mapping update, while documents created before classifiers existed need a compatibility query that preserves their previous semantic meaning as regular workflows.

Implementation

During ES8 index initialization, Conductor now applies an additive keyword mapping for classifier through the workflow alias. ES8 structured queries for the workflow classifier match either an explicit workflow value or documents where the classifier field is absent. Agent classifier queries and all non-ES8 persistence implementations are unchanged.

For a direct upgrade from a pre-classifier release, mapping reconciliation runs before new writes, so legacy documents remain visible without a bulk backfill and subsequent classifier values are indexed correctly.

Upgrade caveat

Elasticsearch does not retroactively index values that were already written to _source while the field was unmapped. A deployment that previously ran a classifier-enabled Conductor build against an older unmapped workflow index may therefore have explicit classifier values stored only in _source. Those documents require reindexing to recover strict classifier semantics; until then Elasticsearch cannot distinguish an unindexed explicit classifier from a genuinely legacy document with no classifier. This change prevents that state for direct upgrades but cannot reconstruct information Elasticsearch never indexed.

When automatic ES8 index management is disabled, operators remain responsible for applying the mapping before classifier-enabled writes, consistent with the existing external index-management contract.

Testing

• Added query parser coverage for legacy documents when filtering for workflows.
• Added coverage confirming agent classifier filtering remains strict.
• Added an ES8 integration test verifying startup mapping reconciliation on a legacy workflow index.

[codeCraft-Ritik]

Solid solution

Regarding the compatibility query structure:

• Assuming the classifier=workflow fallback uses a bool query with filter context (bool.should with minimum_should_match: 1 wrapping the term and must_not: { exists: "classifier" }), this will be fully cacheable by Elasticsearch and won't introduce scoring overhead on hot search paths.
• Preserving strict equality for classifier=agent ensures no false positives for new agentic workloads.

The test cases covering both query generation and startup reconciliation on legacy indices give a lot of confidence. Approved! 🚀

[rajeshwar-nu]

@v1r3n @nthmost-orkes Can I get some review on this please? This fixes a regression brought in #1268 for ES8 indexing backend