perf(ingest/bigquery): partition-ID sentinel constants + prebuilt system-tables deny pattern [2/9]

[acrylJonny]

Summary

PR 2 of 9 splitting #12825. Stacked on #19478.

Small, behaviour-preserving cleanup that the partition-discovery layer later in the stack depends on:

• Hoist BigQuery's virtual/sentinel partition IDs (__NULL__, __UNPARTITIONED__, __STREAMING_UNPARTITIONED__) into named constants + a BQ_SPECIAL_PARTITION_IDS frozenset in common.py, and interpolate them into the legacy-stats dataset query in queries.py rather than repeating the magic strings inline.
• Build the immutable system-tables AllowDenyPattern once at module scope instead of reconstructing it on every BigQueryFilter.is_allowed() call (invoked per table during ingestion).

Stack

Base ← #19478 (1/9). Next: PartitionInfo multi-column model (3/9).

Test plan

• pytest tests/unit/bigquery/test_bigquery_source.py — 135 passed
• Verified the f-string conversion in queries.py keeps the .format(project_id=…, dataset_name=…, table_filter=…) placeholders intact for the (unchanged) callers
• ruff + mypy clean

---

Summary by cubic

Hoists BigQuery's sentinel partition IDs into named constants and prebuilds the system-tables deny pattern once, avoiding per-call reconstruction during ingestion. Both the legacy-stats and partition-stats queries in queries.py now interpolate the constants instead of hardcoding the magic strings, so the two paths can't drift. Behavior is unchanged; the constants are consumed by the upcoming partition-discovery work.

^{Written for commit 02953d2. Summary will update on new commits.}

[cursor]

PR Summary

Overview
Behavior-preserving refactor for the BigQuery v2 ingestion source ahead of partition-discovery work.

Adds shared constants in common.py for BigQuery virtual partition IDs (__NULL__, __UNPARTITIONED__, __STREAMING_UNPARTITIONED__) plus a BQ_SPECIAL_PARTITION_IDS frozenset, and wires the legacy-stats and partition-stats dataset SQL in queries.py to interpolate those values instead of duplicating literal strings when excluding sentinels from num_partitions / max_partition_id.

Performance: BigQueryFilter.is_allowed() now reuses a module-level _SYSTEM_TABLES_ALLOW_DENY AllowDenyPattern instead of constructing one on every table check during ingestion.

^{Reviewed by Cursor Bugbot for commit 02953d2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cubic-dev-ai]

All reported issues were addressed across 2 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[codecov]

Codecov Report

❌ Patch coverage is 85.71429% with 1 line in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...src/datahub/ingestion/source/bigquery_v2/common.py	83.33%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[datahub-connector-tests]

Connector Tests Results

All connector tests passed for commit 02953d2

View full test logs →

To skip connector tests, add the skip-connector-tests label (org members only).

Autogenerated by the connector-tests CI pipeline.