Fix missing search aliases after reindex by deleting the concrete index atomically within the alias swap - #28667
Merged
Merged
Conversation
github-actions
Bot
added
backend
safe to test
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens search index promotion during (re)indexing to handle the “fresh install” shape where the canonical name (e.g., *_search_index) is still a concrete index (and therefore cannot simultaneously be an alias). It introduces an atomic alias swap that can also remove the conflicting concrete index in the same request, preventing the canonical name (including all) from becoming orphaned.
Changes:
- Extend
swapAliasesto optionally delete concrete indices atomically within the same aliases update request (ES/OS). - Make promotion/finalization attach aliases derived only from indexMapping.json (deterministic), and abort safely when no aliases resolve (orphan guard).
- Add/adjust unit tests to cover first-install and orphan-prevention scenarios.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| openmetadata-service/src/main/java/org/openmetadata/service/search/DefaultRecreateHandler.java | Adds orphan-guard + canonical concrete removal resolution; uses deterministic aliases from mapping and performs atomic swap with optional remove_index. |
| openmetadata-service/src/main/java/org/openmetadata/service/search/IndexManagementClient.java | Updates swapAliases API to accept indicesToRemove and provides a backward-compatible default overload. |
| openmetadata-service/src/main/java/org/openmetadata/service/search/elasticsearch/ElasticSearchClient.java | Wires through the new swapAliases(..., indicesToRemove) signature. |
| openmetadata-service/src/main/java/org/openmetadata/service/search/elasticsearch/ElasticSearchIndexManager.java | Implements atomic alias swap that can also remove_index (with mustExist(false)) in ES. |
| openmetadata-service/src/main/java/org/openmetadata/service/search/opensearch/OpenSearchClient.java | Wires through the new swapAliases(..., indicesToRemove) signature. |
| openmetadata-service/src/main/java/org/openmetadata/service/search/opensearch/OpenSearchIndexManager.java | Implements atomic alias swap that can also remove_index in OpenSearch. |
| openmetadata-service/src/test/java/org/openmetadata/service/search/DefaultRecreateHandlerTest.java | Updates existing expectations and adds coverage for first-install concrete canonical removal + orphan-guard behavior. |
mohityadav766
changed the title
pmbrull
previously approved these changes
Jun 3, 2026
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mohityadav766
added
the
To release
Contributor
🟡 Playwright Results — all passed (13 flaky)✅ 4260 passed · ❌ 0 failed · 🟡 13 flaky · ⏭️ 88 skipped
🟡 13 flaky test(s) (passed on retry)
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |
Code Review ✅ Approved 1 resolved / 1 findingsImplements atomic index deletion and alias swapping during reindexing to prevent orphan states, while unifying alias derivation via index mappings. The OpenSearch removeIndex operation now correctly handles existence checks, resolving all identified missing alias issues. ✅ 1 resolved✅ Bug: OpenSearch removeIndex omits mustExist(false) unlike ES path
OptionsDisplay: compact → Showing less information. Comment with these commands to change:
Was this helpful? React with 👍 / 👎 | Gitar |
Contributor
|
Failed to cherry-pick changes to the 1.13 branch. |
|
Shreyansh100704
pushed a commit
that referenced
this pull request
Jun 4, 2026
… succeeds (#28700) PR #28667 added an atomic alias swap that folds the canonical concrete-index delete into the _aliases request via a remove_index action. The action was built as removeIndex(index).mustExist(false). OpenSearch's _aliases parser does not accept must_exist on remove_index and rejects the whole request with "[remove_index] unknown field [must_exist]" -> "[aliases] failed to parse field [actions]" (HTTP 400), so the alias add in the same body never applies. Elasticsearch tolerates the field, which is why it passed review. On a fresh install every canonical *_search_index is a concrete index, so the remove_index action fires for all entities and every swap fails -> no canonical aliases are attached -> the canonical name resolves to nothing. This surfaced as the AI Platform CAIP integration test failing with "table_search_index missing embedding field". Drop must_exist from the remove_index action in both OpenSearchIndexManager and ElasticSearchIndexManager. It is unnecessary: resolveCanonicalRemoval only forwards indices already confirmed to exist via indexExists(). Both engine paths are now identical. Add AliasSwapConcreteRemovalIT, which exercises the first-install shape (concrete canonical in indicesToRemove) against the live cluster: it fails before this fix on OpenSearch and passes after. The existing DefaultRecreateHandlerTest mocks swapAliases, so it never exercised the request body where this bug lives. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mohityadav766
added a commit
that referenced
this pull request
Jun 4, 2026
… succeeds (#28700) PR #28667 added an atomic alias swap that folds the canonical concrete-index delete into the _aliases request via a remove_index action. The action was built as removeIndex(index).mustExist(false). OpenSearch's _aliases parser does not accept must_exist on remove_index and rejects the whole request with "[remove_index] unknown field [must_exist]" -> "[aliases] failed to parse field [actions]" (HTTP 400), so the alias add in the same body never applies. Elasticsearch tolerates the field, which is why it passed review. On a fresh install every canonical *_search_index is a concrete index, so the remove_index action fires for all entities and every swap fails -> no canonical aliases are attached -> the canonical name resolves to nothing. This surfaced as the AI Platform CAIP integration test failing with "table_search_index missing embedding field". Drop must_exist from the remove_index action in both OpenSearchIndexManager and ElasticSearchIndexManager. It is unnecessary: resolveCanonicalRemoval only forwards indices already confirmed to exist via indexExists(). Both engine paths are now identical. Add AliasSwapConcreteRemovalIT, which exercises the first-install shape (concrete canonical in indicesToRemove) against the live cluster: it fails before this fix on OpenSearch and passes after. The existing DefaultRecreateHandlerTest mocks swapAliases, so it never exercised the request body where this bug lives. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit d800ba4)
mohityadav766
added a commit
that referenced
this pull request
Jul 3, 2026
… succeeds (#28700) (#29723) PR #28667 added an atomic alias swap that folds the canonical concrete-index delete into the _aliases request via a remove_index action. The action was built as removeIndex(index).mustExist(false). OpenSearch's _aliases parser does not accept must_exist on remove_index and rejects the whole request with "[remove_index] unknown field [must_exist]" -> "[aliases] failed to parse field [actions]" (HTTP 400), so the alias add in the same body never applies. Elasticsearch tolerates the field, which is why it passed review. On a fresh install every canonical *_search_index is a concrete index, so the remove_index action fires for all entities and every swap fails -> no canonical aliases are attached -> the canonical name resolves to nothing. This surfaced as the AI Platform CAIP integration test failing with "table_search_index missing embedding field". Drop must_exist from the remove_index action in both OpenSearchIndexManager and ElasticSearchIndexManager. It is unnecessary: resolveCanonicalRemoval only forwards indices already confirmed to exist via indexExists(). Both engine paths are now identical. Add AliasSwapConcreteRemovalIT, which exercises the first-install shape (concrete canonical in indicesToRemove) against the live cluster: it fails before this fix on OpenSearch and passes after. The existing DefaultRecreateHandlerTest mocks swapAliases, so it never exercised the request body where this bug lives. (cherry picked from commit d800ba4) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes:
Fixes the recurring "Failed to find index
openmetadata_*_search_index" error caused by canonical search aliases going missing or pointing at nothing after a "Recreate Indexes" reindex.Symptom
Users see, typically after the daily reindex:
The Elasticsearch/OpenSearch cluster is healthy and the data is intact — a
*_search_index_rebuild_<ts>index exists with all documents, but the canonical alias (*_search_index) is attached to nothing. It is intermittent: re-running "Recreate Indexes" sometimes fixes it, and the only consistent manual workaround is to repoint the alias at the live_rebuild_*index by hand.Root cause
The zero-downtime recreate flow builds
*_rebuild_<ts>and then swaps the canonical alias onto it. On a fresh install (and after any prior orphaning), the canonical nametable_search_indexexists as a concrete index, not an alias — and OS/ES forbid an alias sharing a name with an existing index. So the handler did two separate cluster operations:If step 2 fails or is interrupted between the two — e.g. the index delete hasn't fully propagated and ES/OS still rejects the alias-add with "an index exists with the same name as the alias" — the concrete index is gone, the alias is not attached, and the canonical name resolves to nothing → orphan. Because it's a propagation race it's intermittent, and
createMissingIndexes()(run on every server boot) recreates the canonical as a concrete index again, re-arming the same window each cycle.Two secondary issues made this worse / harder to reason about:
finalizeReindexread the alias set off the live cluster (getAliases(activeIndexName)), which is non-deterministic (propagates stray aliases), adds a round-trip, and diverged from the distributed promotion path that already derived aliases fromindexMapping.json.Changes
Atomic concrete-index removal during the swap. Added an overload
swapAliases(oldIndices, newIndex, aliases, indicesToRemove)that emits aremove_indexaction (ES withmustExist(false)) in the sameupdateAliasesrequest as the alias add. The concrete canonical index is now deleted and the alias attached in one atomic operation — so a failure is a no-op (the concrete index and its live aliases survive and the reindex simply retries) instead of an orphan. The old 3-arg signature is kept as a default delegating with an empty removal set.DefaultRecreateHandlerno longer callsdeleteIndexWithBackoff(canonicalIndex)before the swap; it usesresolveCanonicalRemoval(...)to decide what to hand to the atomic swap.Never delete the old index when no aliases resolve.
finalizeReindex/promoteEntityIndexnow abort (abortPromotionWithoutAliases) and record a promotion failure if the alias set comes up empty — the old serving index is left intact for a retry instead of being deleted into a void.Aliases come solely from
indexMapping.json. Removed thegetAliases(activeIndexName)cluster read. Both the recreate and finalize paths now derive the set via the singlegetAliasesFromMappinghelper ({ parent aliases, short alias, raw index name }), matching whatpromoteEntityIndexalready did. The set is deterministic and the two promotion paths are unified.Type of change:
High-level design:
IndexManagementClient: new atomicswapAliases(..., indicesToRemove); old 3-arg is now adefaultdelegating withSet.of().ElasticSearchIndexManager/OpenSearchIndexManager: build a singleUpdateAliasesRequestcontainingremove(alias from old indices) +remove_index(concrete) +add(alias to staged) actions.ElasticSearchClient/OpenSearchClient: delegate the 4-arg overload to their managers.DefaultRecreateHandler:resolveCanonicalRemoval(...)— returns the concrete canonical index (if any) to remove atomically and prunes the alias name from the delete set.abortPromotionWithoutAliases(...)— guards the empty-alias case without deleting the old index.recreateIndexFromMapping/finalizeReindexderive aliases viagetAliasesFromMapping; no cluster read.Test plan:
Added/updated unit tests in
DefaultRecreateHandlerTest(all 33 pass):testFinalizeReindexRemovesConcreteCanonicalAtomically— the concrete canonical is removed inside the atomic swap, never via a separatedeleteIndexWithBackoff.testFinalizeReindexFailedSwapDoesNotOrphanConcreteCanonical— on a failed atomic swap the concrete index and its aliases survive (no orphan). This is the regression guard for the reported bug.testPromoteEntityIndexDoesNotOrphanAliasWhenMappingHasNoAliases/testFinalizeReindexDoesNotOrphanAliasWhenNoAliasesResolved— empty alias set never deletes the old serving index.testRecreateIndexFromMappingUsesAliasTargetAsActiveIndex— assertsverify(never()).getAliases(...); aliases are derived from the mapping only.The orphan was reproduced deterministically against the pre-fix code (concrete canonical + forced swap failure → concrete deleted, alias resolves to nothing) and confirmed gone after the fix.
UI changes:
Not applicable.
Checklist: