Tags: open-metadata/OpenMetadata
Tags
fix(search): drop must_exist on remove_index so OpenSearch alias swap… … succeeds (#28700) (#29723) PR #28667 added an atomic alias swap that folds the canonical concrete-index delete into the _aliases request via a remove_index action. The action was built as removeIndex(index).mustExist(false). OpenSearch's _aliases parser does not accept must_exist on remove_index and rejects the whole request with "[remove_index] unknown field [must_exist]" -> "[aliases] failed to parse field [actions]" (HTTP 400), so the alias add in the same body never applies. Elasticsearch tolerates the field, which is why it passed review. On a fresh install every canonical *_search_index is a concrete index, so the remove_index action fires for all entities and every swap fails -> no canonical aliases are attached -> the canonical name resolves to nothing. This surfaced as the AI Platform CAIP integration test failing with "table_search_index missing embedding field". Drop must_exist from the remove_index action in both OpenSearchIndexManager and ElasticSearchIndexManager. It is unnecessary: resolveCanonicalRemoval only forwards indices already confirmed to exist via indexExists(). Both engine paths are now identical. Add AliasSwapConcreteRemovalIT, which exercises the first-install shape (concrete canonical in indicesToRemove) against the live cluster: it fails before this fix on OpenSearch and passes after. The existing DefaultRecreateHandlerTest mocks swapAliases, so it never exercised the request body where this bug lives. (cherry picked from commit d800ba4) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fix(jdbi3): chunk IN-list batch queries to stay under DB parameter li… …mit (#28752) (#28777) * fix(jdbi3): chunk IN-list batch queries to stay under DB parameter limit Bulk subtree delete/restore fans an entire tree level's ids into a single IN (...) query. JDBI @BindList emits one bind parameter per element, so a level with ~100k entities (e.g. a service with 100k tables) exceeds PostgreSQL's 65,535 parameter ceiling ("Given query has 100,002 parameters") and the operation fails. The reported findToBatchAllTypes crash was only the first uncaught offender: getExtensionsBatch hits the same limit earlier but is swallowed by populateRelationFields' per-entity fallback, and ~13 other findToBatch*/findFromBatch* and tag batches were vulnerable in other paths. Add EntityDAO.queryInChunks / updateInChunks helpers (chunk at MAX_IN_LIST_CHUNK_SIZE, pass straight through for small lists so single-query behavior is preserved) and route every @BindList batch method through them: findToBatch*/findFromBatch*, entity_extension deleteAllBatch / getExtensionsBatch / getExtensionBatch, and tag_usage getTagsInternalBatch / getCertTagsInternalBatch. Also extract the duplicated deletedCondition(include) clause builder. Adds InListChunkingTest covering the chunk-size cap, exact input coverage (no dropped/duplicated ids), and per-chunk result aggregation. * fix(jdbi3): dedup ids before chunking IN-list batches queryInChunks / updateInChunks split a large id list into chunks, so a duplicate value landing in two different chunks would be queried (or deleted) twice — duplicating result rows / issuing a redundant statement, unlike a single IN (...) which ignores duplicate values. De-duplicate (encounter order preserved) before chunking, matching the existing findEntitiesByIds pattern. Adds InListChunkingTest cases covering duplicates split across a chunk boundary for both the query and update helpers. * fix(jdbi3): chunk three more unbounded IN-list paths Audit of the remaining @BindList paths surfaced three more methods that fan a caller-controlled, catalog-scaled list into one IN (...) and can hit the same 65,535 parameter ceiling. All routed through the queryInChunks/updateInChunks helpers: - EntityRelationshipDAO.bulkUpdateFromId — reparenting a glossary term updates every nested descendant's relationship row in one UPDATE; a subtree with >65k terms blew the cap. Now chunks toIds via updateInChunks. - TagUsageDAO.deleteTagsByTargets — clearing column tags on a bulk table update/import passes every column FQN across the batch (wide/nested tables x ~100 per batch). Now chunks via updateInChunks. - EntityTimeSeriesDAO.getLatestExtensionBatch — listing ingestion pipelines with fields=pipelineStatuses and a large limit (capped at 1,000,000) binds one FQN-hash per row. Now chunks via queryInChunks. Adds InListChunkingTest cases for all three. * test(jdbi3): cover IN-list chunking pass-through with duplicate ids The existing tests all feed an over-limit list, leaving queryInChunks / updateInChunks' else branch (size <= MAX_IN_LIST_CHUNK_SIZE) untested. Add two cases proving a within-limit list with duplicates issues exactly one query/statement and is passed through verbatim (no dedup) — the database collapses the duplicates via IN(...) set semantics. Complements the existing over-limit dedup tests, which assert against the distinct set. * fix(jdbi3): chunk five more catalog-scaled IN-list paths Follow-up to the IN-list chunking work — five more @BindList paths fan a catalog-scaled list into one IN (...) and can still hit the 65,535 parameter ceiling. All routed through the existing queryInChunks/updateInChunks helpers. - EntityRelationshipDAO.bulkRemoveTo / bulkRemoveFrom: the 'remove' siblings of bulkUpdateFromId (already chunked). Reached via the bulkRemove{To,From}Relationship wrappers (their only callers) from DataProductRepository.batchMigrateAssetDomains (all assets of a type under one data product), TeamRepository, DomainRepository. - UsageDAO.getLatestUsageBatch: GET /v1/tables?fields=usageSummary (and dashboards/pipelines/topics/mlmodels/...) resolves the field over the whole page via setFieldsInBulk; list limit is @max(1000000). - DataQualityDataTimeSeriesDAO.getLatestRecordBatch: GET /v1/dataQuality/testCases?fields=testCaseResult, same whole-page resolution. - TestCaseResultTimeSeriesDAO.listResultSummariesForTestSuites: GET /v1/dataQuality/testSuites?fields=summary, binds one id per executable test suite. Adds InListChunkingTest cases for all five (16 tests pass). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com> [1.13 backport] Dropped the findToBatchAllTypes(List<Integer> relations, Include) chunking hunk and its findToBatchAllTypesWithRelationsCondition delegate: that overload originates in temporal lineage (#28426), which is not on the 1.13 branch. The two queryInChunks dedup / pass-through tests that exercised it were retargeted to the existing int-relation findToBatchAllTypes overload (same queryInChunks path). All 16 InListChunkingTest cases pass. (cherry picked from commit 5af1ba5) Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
PreviousNext