Remove redundant storage calls and timeline-server round trips found by an instrumented FS-call audit #20257

Task Description

What needs to be done:

On object storage every file-system call (HEAD, GET, LIST, PUT) is a round trip, and every executor request to the
embedded timeline server is a driver round trip served under a lock. Small redundant costs per file, file group, task,
partition or instant add up to the dominant latency on large tables, and they are invisible in tests that use a
handful of files on one thread.

An audit of master instrumented every storage call (and every executor to timeline-server request) made by Spark
writes, table services, queries, SQL DMLs and HoodieStreamer, across COW and MOR, table versions 6, 8 and current, the
metadata table on and off, the timeline server on and off, the index types and the write handles. It found redundant
calls such as: metadata-table HFiles opened and downloaded three times per read, base-file footers read twice and
status-checked twice per file, column stats re-reading every new file's footer, executor tasks rebuilding metadata
readers and file-system views, per-task timeline-server lookups the driver could answer once, and a per-request
lock plus timeline hash in the timeline server. It also surfaced several correctness bugs.

This issue tracks the fixes. Each sub-issue describes one problem with its evidence; each PR carries a test that
counts the calls (or reproduces the bug) so the regression class is guarded.

Why this task is needed:

On object storage each redundant call is a 10-100 ms round trip, and the costs found grow with files, file groups, tasks, partitions or instants, so they dominate latency on large tables.

Task Type

Performance optimization

Related Issues

Parent feature issue: (none)
Related issues:

• Read each metadata table native HFile log file once instead of three times #20258 (PR perf(metadata): read each metadata table HFile once per reader #20259): metadata table HFile log files read once instead of three times

• MOR upserts silently drop records when hoodie.allow.operation.metadata.field is enabled #20260 (PR fix(writer): do not drop MOR records when the operation field is enabled #20261): MOR records silently dropped when the operation metadata field is enabled (bug)

• Bloom index key check asks the timeline server for each candidate base file twice per task #20262 (PR perf(index): reuse driver-resolved base files in the bloom key check #20263): bloom index key check reuses driver-resolved base files instead of timeline-server lookups

• Timeline server serializes every file system view request on the view lock while hashing the timeline #20264 (PR perf(timeline-service): check view freshness without the view lock #20265): timeline server checks view freshness without the per-table view lock

• Data skipping returns too few rows on a METADATA_ONLY bootstrapped table once column stats are enabled #20266 (PR fix(spark): keep bootstrap skeleton files and partitions as data skipping candidates #20267): data skipping returned too few rows on METADATA_ONLY bootstrap tables with column stats (bug)