[python] Backtick quoting for identifiers, exists_batch optimization by tub · Pull Request #7347 · apache/paimon

tub · 2026-03-05T15:14:27Z

Summary

Add backtick quoting to Identifier for SQL-safe formatting
Add exists_batch() for bulk file existence checks
Expose read_changelog in manifest list manager for future streaming reads

Stacked PR series

This is PR 1a/5 in the Python streaming read series:

[python] Backtick quoting for identifiers, exists_batch optimization #7347 — caching infrastructure + utilities
[python] Add scanners, sharding, and row kind support #7348 — scanners, sharding, row kind
[python] Add consumer management for streaming progress #7349 — consumer management
[python] Add StreamReadBuilder and AsyncStreamingTableScan #7350 — StreamReadBuilder + AsyncStreamingTableScan
[python] Add paimon tail CLI for streaming table reads #7351 — paimon tail CLI

Test plan

flake8 passes on all changed files
python -m pytest passes (630/630, 9 pre-existing lance skips)
New tests: identifier_test.py, manifest_cache_test.py

- Add backtick quoting to Identifier for SQL-safe formatting - Add ChangelogProducer enum to core_options - Add exists_batch() for bulk file existence checks - Add LRU caching to ManifestFileManager and ManifestListManager - Add snapshot caching and traversal helpers to SnapshotManager - Add cachetools dependency Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…tEquals Extract shared base class for ManifestFileCacheTest and ManifestListCacheTest, add _make_snapshot() helper, and fix deprecated assertEquals (removed in 3.12). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

paimon-python/pypaimon/common/options/core_options.py

paimon-python/pypaimon/common/identifier.py

paimon-python/pypaimon/filesystem/pyarrow_file_io.py

paimon-python/pypaimon/manifest/manifest_file_manager.py

paimon-python/pypaimon/manifest/manifest_list_manager.py

paimon-python/pypaimon/snapshot/snapshot_manager.py

…rim docs, remove ChangelogProducer - Upgrade cachetools to >=7,<8 for cachedmethod(info=True) support - Remove ChangelogProducer enum (belongs in apache#7348 scanners branch) - Replace manual cache hit/miss counters with @cachedmethod(info=True) decorator on ManifestFileManager, ManifestListManager, SnapshotManager - Trim verbose docstrings across identifier, file_io, pyarrow_file_io, manifest_list_manager, and snapshot_manager - Update cache tests to use cache_info() instead of manual counters Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…efault-size tests - Move shared cache-behaviour tests (second_read, disabled_when_zero) into _CacheBehaviourMixin so they run for both manager types without duplication - Extract _EMPTY_ROW / _EMPTY_STATS module constants to reduce DataFileMeta boilerplate - Remove test_default_cache_size tests (just assert constructor defaults) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…rim docs, remove ChangelogProducer - Upgrade cachetools to >=7,<8 for cachedmethod(info=True) support - Remove ChangelogProducer enum (belongs in apache#7348 scanners branch) - Replace manual cache hit/miss counters with @cachedmethod(info=True) decorator on ManifestFileManager, ManifestListManager, SnapshotManager - Trim verbose docstrings across identifier, file_io, pyarrow_file_io, manifest_list_manager, and snapshot_manager - Update cache tests to use cache_info() instead of manual counters Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

cachetools 7.x requires Python >=3.10 but the project supports 3.6+. Drop info=True and explicit key= from @cachedmethod (both 7.x-only features) while keeping the decorator itself (available since 4.x). Replace cache_info()-based test assertions with unittest.mock spies on file_io.new_input_stream, testing the actual caching effect without any production code counters. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

JingsongLi · 2026-03-07T03:37:02Z

paimon-python/pypaimon/snapshot/snapshot_manager.py

+
+        return results
+
+    def find_next_scannable(


Do we really need it? Just reading the snapshot file to determine, does this really need to be optimized?

JingsongLi · 2026-03-07T03:38:30Z

Can you explain the specific function of Cache? It seems that streaming reading does not repeat reading files?

JingsongLi · 2026-03-07T04:12:47Z

paimon-python/dev/requirements.txt

 pyarrow>=6,<7; python_version < "3.8"
 pyarrow>=16,<20; python_version >= "3.8"
-pylance>=0.20,<1; python_version>="3.9"
-pylance>=0.10,<1; python_version>="3.8" and python_version<"3.9"


I did a small commit to remove these deps. Please rebase master.

Remove LRU caches from ManifestFileManager, ManifestListManager, and SnapshotManager — they have near-zero hit rates in practice (batch reads create new manager instances; streaming reads see unique manifest names per snapshot). Caching will be re-added in PR apache#7350 where streaming actually benefits. Remove find_next_scannable and get_snapshots_batch from SnapshotManager as they have zero callers on this branch. They will be added where needed in downstream PRs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Revert manifest_file_manager.py to upstream/master (caching split no longer needed) - Restore original docstring for get_snapshot_by_id - Revert assertEquals -> assertEqual drive-by change Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Dynamic bucket options were accidentally removed during rebase. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

tub · 2026-03-09T15:41:46Z

Can you explain the specific function of Cache? It seems that streaming reading does not repeat reading files?

I introduced the cache when I was working on the catch-up streaming scenario - and only one of them ended up being used. I've removed them from here and will add them back if/when they're needed.

tub · 2026-03-09T15:43:25Z

@JingsongLi thanks a lot for the review, sorry about some of the useless changes - they crept in from a bunch of performance testing I was doing at the end of the CLI work.

…rim docs, remove ChangelogProducer - Upgrade cachetools to >=7,<8 for cachedmethod(info=True) support - Remove ChangelogProducer enum (belongs in apache#7348 scanners branch) - Replace manual cache hit/miss counters with @cachedmethod(info=True) decorator on ManifestFileManager, ManifestListManager, SnapshotManager - Trim verbose docstrings across identifier, file_io, pyarrow_file_io, manifest_list_manager, and snapshot_manager - Update cache tests to use cache_info() instead of manual counters Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

JingsongLi

+1 Thanks @tub

tub mentioned this pull request Mar 5, 2026

[Feature] [python] Streaming Read Support in paimon-python #7152

Open

2 tasks

tub marked this pull request as ready for review March 5, 2026 16:30

This was referenced Mar 5, 2026

[python] Add streaming infrastructure: scanners, consumers, caching #7342

Closed

[python] Add StreamReadBuilder and AsyncStreamingTableScan #7343

Closed

[python] Add paimon tail CLI for streaming table reads #7344

Closed

tub and others added 2 commits March 5, 2026 18:07

Making examples more generic

faee9aa

tub marked this pull request as draft March 6, 2026 11:25

tub commented Mar 6, 2026

View reviewed changes

tub and others added 2 commits March 6, 2026 12:21

tub marked this pull request as ready for review March 6, 2026 12:44

tub and others added 2 commits March 6, 2026 14:00

[python] Fix trailing blank line in snapshot_manager.py (flake8 W391)

43c00b1

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

JingsongLi reviewed Mar 7, 2026

View reviewed changes

Merge branch 'master' into python-streaming-1a-caching

6af3313

tub force-pushed the python-streaming-1a-caching branch from dfb750d to 6af3313 Compare March 9, 2026 10:06

tub and others added 3 commits March 9, 2026 12:27

Reverting ordering-only changes

626241b

tub marked this pull request as draft March 9, 2026 13:59

Restore core_options.py to upstream/master

5942d40

Dynamic bucket options were accidentally removed during rebase. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

tub marked this pull request as ready for review March 9, 2026 15:42

tub marked this pull request as draft March 9, 2026 15:44

tub changed the title ~~[python] Add caching infrastructure and utilities~~ [python] Backtick quoting for identifiers, exists_batch optimization Mar 9, 2026

tub marked this pull request as ready for review March 9, 2026 16:19

JingsongLi approved these changes Mar 10, 2026

View reviewed changes

JingsongLi merged commit 87a4c91 into apache:master Mar 10, 2026
5 checks passed

Conversation

tub commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Stacked PR series

Test plan

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JingsongLi Mar 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tub Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

JingsongLi commented Mar 7, 2026

Uh oh!

JingsongLi Mar 7, 2026

Choose a reason for hiding this comment

Uh oh!

tub Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

tub commented Mar 9, 2026

Uh oh!

tub commented Mar 9, 2026

Uh oh!

JingsongLi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tub commented Mar 5, 2026 •

edited

Loading

JingsongLi Mar 7, 2026 •

edited

Loading