[refactor](paimon) Per-catalog Paimon metadata cache with two-level table+snapshot structure#60478
Merged
morningman merged 12 commits intoapache:masterfrom Feb 11, 2026
Merged
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run external |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
Author
|
run external |
Contributor
FE Regression Coverage ReportIncrement line coverage |
morningman
reviewed
Feb 10, 2026
fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
Outdated
Show resolved
Hide resolved
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 29989 ms |
ClickBench: Total hot run time: 28.18 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
morningman
approved these changes
Feb 11, 2026
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
CalvinKirs
approved these changes
Feb 11, 2026
suxiaogang223
added a commit
to suxiaogang223/doris
that referenced
this pull request
Feb 13, 2026
…able+snapshot structure (apache#60478) Refactor Paimon metadata cache from a single global instance to per-catalog instances, introduce a two-level Table+Snapshot cache structure, and unify TTL resolution logic across Iceberg/Paimon/Schema caches. The previous design shared a single `PaimonMetadataCache` instance and a single global `snapshotCache` across all Paimon catalogs. This caused: - Different catalogs could not independently configure cache TTL. - Cache keys had to carry `catalogId` for isolation; invalidation required scanning all keys and filtering. - `PaimonExternalTable` eagerly fetched the `Table` object at construction time, incurring remote calls even when the table was never subsequently accessed. (cherry picked from commit 2c85148)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Summary
Refactor Paimon metadata cache from a single global instance to per-catalog instances,
introduce a two-level Table+Snapshot cache structure, and unify TTL resolution logic
across Iceberg/Paimon/Schema caches.
Motivation
The previous design shared a single
PaimonMetadataCacheinstance and a single globalsnapshotCacheacross all Paimon catalogs. This caused:catalogIdfor isolation; invalidation required scanning allkeys and filtering.
PaimonExternalTableeagerly fetched theTableobject at construction time, incurringremote calls even when the table was never subsequently accessed.
Changes
Per-catalog cache instantiation
CatalogScopedCacheMgr<T>, a generic catalog-keyed cache manager backed byConcurrentHashMap.computeIfAbsent.PaimonMetadataCacheMgrnow extendsCatalogScopedCacheMgr<PaimonMetadataCache>;each catalog owns an independent
PaimonMetadataCache.icebergCacheMap(previously hand-rolled double-checked locking)in
ExternalMetaCacheMgrto the sameCatalogScopedCacheMgr.Two-level table + snapshot cache
snapshotCachewithtableCache(key:
PaimonTableCacheKey, value:PaimonTableCacheValue).PaimonTableCacheValueholds the PaimonTableobject and lazy-loadsPaimonSnapshotCacheValuevia double-checked locking.Tableobject is managed by CaffeineLoadingCache, subject to TTL/maxSize.When TTL expires, Caffeine creates a new
PaimonTableCacheValue, and the snapshotis re-lazily-loaded on next access.
tableCachedirectly; MTMV queries go through the explicitsnapshot path; branch queries call the Paimon catalog directly (branches have
independent schemas, not suitable for the main table cache).
Unified TTL resolution
ExternalCatalog.resolveCacheTtlSpec()to centralize TTL property parsing:null→ use global default (external_cache_expire_time_seconds_after_access)-1→ no expiry (Caffeine does not setexpireAfterAccess)0→ disable cache (maxSize=0, Caffeine evicts immediately)>0→ use asexpireAfterAccessIcebergMetadataCache,PaimonMetadataCache, andExternalSchemaCache.paimon.table.meta.cache.ttl-secondcatalog property with validation incheckProperties().ALTER CATALOG SETtriggers cache reinitialization vianotifyPropertiesUpdated(), consistent with Iceberg's existing pattern.Lazy table access in PaimonExternalTable
paimonTablefield from the constructor.Tableobject access now goes throughPaimonUtils→PaimonMetadataCache.tableCache, making it lazy and cache-aware.PaimonUtilsas the centralized static accessor for Paimon cacheoperations, simplifying call sites.
Iceberg invalidation fast path
IcebergMetadataCache.invalidateTableCache()now attempts a direct key lookup(
getIfPresent) first. On hit, invalidate immediately; on miss, fall back to fullcache scan matching by local name. Avoids unnecessary iteration on the common path.
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)