Skip to content

feat(cli): emit per_node.db in recce init --cloud [PR 2/3]#1335

Merged
even-wei merged 2 commits into
feature/drc-3295-pr1-per-node-db-emitterfrom
feature/drc-3295-pr2-cli-integration
Apr 23, 2026
Merged

feat(cli): emit per_node.db in recce init --cloud [PR 2/3]#1335
even-wei merged 2 commits into
feature/drc-3295-pr1-per-node-db-emitterfrom
feature/drc-3295-pr2-cli-integration

Conversation

@even-wei
Copy link
Copy Markdown
Contributor

PR checklist

  • Ensure you have added or ran the appropriate tests for your PR.
  • DCO signed

What type of PR is this?

feat — wires the emitter into recce init --cloud and adds the cache.db handling.

What this PR does / why we need it:

Stacked on top of #1334 (PR 1/3 — the pure emitter module). Please review #1334 first; this PR shows the merge diff against that branch.

Wires PerNodeDbWriter from #1334 into recce init --cloud:

  • After build_full_cll_map() succeeds, emits per_node.db for both base and current envs from the loaded manifests + catalogs
  • Uploads per_node.db via a new per_node_db_url presigned-URL key (graceful-degraded: missing key logs a warning, doesn't fail the command)
  • Cloud mode's cll_cache.db now lives in tempfile.mkdtemp(prefix="recce-cll-") instead of ~/.recce/cll_cache.db. Purpose: keep the warm-cache perf (build_full_cll_map re-uses previous CLL slices) while guaranteeing the cache DB never ends up on S3 or on the ECS task host after shutdown
  • Removed the cloud-mode cll_cache_upload_url upload path entirely (cache.db is not a cloud artifact)
  • shutil.rmtree cleanup of the tempdir on success via try/finally

Local (non-cloud) recce init behavior is unchanged: still writes ~/.recce/cll_cache.db (or whatever --cache-db points to), still doesn't emit per_node.db.

Which issue(s) this PR fixes:

Part of DRC-3295 (epic DRC-3294, project "Lineage API Data Transfer Optimization" Phase III).

Special notes for your reviewer:

  • Cloud-side prerequisite: the API server needs to add per_node_db_url to the response of get_upload_urls_by_session_id (same presigned-URL pattern as cll_map_url, content type application/octet-stream). CLI handles missing key gracefully, so this PR can land independently of that work.
  • 7 new tests in tests/test_cli_per_node_db.py:
    • Cloud happy path (PUTs verified)
    • Missing-key graceful degradation
    • Scratch-dir cleanup after success
    • Local-mode non-regression
    • Contract parity vs. DbtAdapter.get_model() on the tests/manifest.json + catalog.json jaffle_shop fixture
    • Mode-switching safety
  • One test in tests/test_cli_cache.py retargeted (test_init_cloud_upload_partial_failure_shows_warning) since the cll_cache_url upload branch is now deleted
  • Full suite: 1106 passed, 5 pre-existing SPA failures (unchanged from feat: precompute per_node.db in recce init --cloud [DRC-3295] #1334test_spa_route_* requires a built frontend that's not in the worktree)
  • Review focus: correctness of the cloud / local branch split around cache_db path; try/finally cleanup semantics; contract-test coverage

Does this PR introduce a user-facing change?:

Cloud users (implicit): running recce init --cloud now uploads an extra per_node.db artifact and no longer uploads cll_cache.db. Consumers in Recce Cloud will begin reading per_node.db once the Cloud-side URL key is added. Nothing breaks in the meantime — the cll_cache.db upload path was already only read via the cll_cache_url key, which Cloud will continue to provide transparently during migration (CLI just ignores it).

Local users (no change): local recce init behavior is unchanged.

NONE (user-visible), but the cloud artifact set does change — covered by the cross-repo coordination note above.

PR 2/3 of DRC-3295. Wires PR 1's per-node SQLite emitter into the
`recce init --cloud` command so Cloud can stream lineage rows without
proxying to an ephemeral Recce instance.

Key changes:
- In cloud mode, write `cll_cache.db` to a `tempfile.mkdtemp(prefix=
  "recce-cll-")` scratch dir so build_full_cll_map keeps its warm-cache
  perf, but the cache.db never leaves the container (ECS task GC on
  exit). A user-provided --cache-db is intentionally ignored in cloud
  mode.
- After build_full_cll_map(), emit per_node.db (manifest + catalog rows,
  not_null/unique tests, primary_key, edges) to the same scratch dir
  and upload it via the new `per_node_db_url` upload key.
- Graceful degradation: if Cloud has not yet added per_node_db_url, log
  a warning and continue — old CLI + new Cloud and vice versa stay
  compatible.
- Remove the cll_cache.db cloud upload block; cache.db is now local-only
  in cloud mode.
- Clean up the scratch dir on successful upload; leave it on failure for
  debugging (ECS reclaims it anyway).

Tests:
- tests/test_cli_per_node_db.py: integration tests for per_node.db
  upload, local-mode non-regression, tempdir cleanup, mode-switching
  safety, and a contract test vs. DbtAdapter.get_model() using the
  jaffle_shop fixture (manifest + catalog round-trip → reconstructed
  get_model() payload matches live adapter output).
- Update test_cli_cache.py::test_init_cloud_upload_partial_failure to
  use per_node_db_url instead of the removed cll_cache_url upload path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: even-wei <evenwei@infuseai.io>
@even-wei
Copy link
Copy Markdown
Contributor Author

Heads-up — pushing a revert commit shortly.

I had this PR suppress the cll_cache.db upload in cloud mode based on a misread of the design. cll_cache.db is actually load-bearing: it enables cross-session warm-cache reuse for OSS users and cloud sessions alike (downloaded at session init, uploaded at completion, reused by build_full_cll_map to skip already-computed CLL slices).

Incoming fix:

  • cache_db default path restored to ~/.recce/cll_cache.db (or --cache-db override)
  • cll_cache_upload_url upload block restored
  • per_node.db stays tempdir-scoped (it IS genuinely a throwaway per-task artifact)
  • Tests updated to assert cll_cache.db IS uploaded in cloud mode

Fix commit inbound. Please hold review until it lands — or review just the per_node.db emission path if it helps.

Original PR 2 removed the cll_cache.db cloud upload path
under the mistaken assumption that cache.db was unwanted
in cloud mode. cache.db is load-bearing for cross-session
warm-cache reuse (OSS local users and cloud sessions alike
— downloaded at init, uploaded at completion, reused by
build_full_cll_map). Restore the cll_cache_upload_url path
and return cache_db to its pre-PR-2 default path.

per_node.db remains tempdir-scoped (genuinely throwaway
per-task artifact) and continues to upload via
per_node_db_url.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: even-wei <evenwei@infuseai.io>
@even-wei even-wei merged commit eff86d6 into feature/drc-3295-pr1-per-node-db-emitter Apr 23, 2026
1 check passed
@even-wei even-wei deleted the feature/drc-3295-pr2-cli-integration branch April 23, 2026 02:56
@even-wei
Copy link
Copy Markdown
Contributor Author

Consolidated into #1334 for commit-by-commit review. PR 2's two commits (cb8d74f1 CLI wire-in + eff86d6e cache.db fix) are in #1334 now via fast-forward. GitHub auto-marked this as merged when the source branch was absorbed — that's accurate from its perspective. Deleting the source branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant