fix: honor in_memory everywhere, plugging a throwaway SQLite catalog by shcheklein · Pull Request #1863 · datachain-ai/datachain

shcheklein · 2026-07-17T02:42:55Z

Problem

read_storage(..., in_memory=True) is silently ignored in two independent places:

Session.get only honors in_memory when it is the call that creates the global session — in any script that already touched datachain, the flag does nothing.
The catalog loader returns the env-configured metastore/warehouse (DATACHAIN__METASTORE / DATACHAIN__WAREHOUSE serialized objects) before ever looking at in_memory, and raises for import-path style config.

The practical consequence in platform jobs: using read_storage('<local dir>').to_storage('s3://...') to push job results to S3 persists a listing of the worker's ephemeral working directory into the shared metastore — every nightly run mints new meaningless lst__file:///tmp/<workdir>/... datasets that outlive the worker filesystem they describe. There was no way to opt out: the one knob that should cover this (in_memory=True) was dropped on the floor at both layers.

Keeping persistent listings as the default is intentional and unchanged — read_storage(update=True) to refresh a listing for the file browser, NFS-style stable local storages, lineage deps, etc. all still work exactly as before. This PR only makes the existing explicit opt-out actually work.

What changed

loader: an explicit in_memory=True now always returns in-memory SQLiteMetastore/SQLiteWarehouse, winning over environment-provided config instead of being silently ignored (serialized style) or raising (import-path style).
Session.get: when an in-memory catalog is requested while the resolved session is backed by a persistent one, route to a dedicated process-wide cached in-memory session (_get_in_memory_session, inherits client_config from the session it shadows). A single cached session is reused so all in_memory=True chains share one shared-cache database and can be combined. The existing client_config-mismatch branch composes with it unchanged.
Jobs: sessions with an in-memory catalog use a session-local job. DATACHAIN_JOB_ID is deliberately ignored there — that job lives in the configured metastore, not in the temporary one (looking it up would raise JobNotFoundError) — and the process-wide _CURRENT_JOB cache is never shared with in-memory sessions in either direction. Job creation is one shared _create_job() helper for both the process-wide and session-local paths, so in-memory jobs carry normal script/query metadata (UDF checkpoints require a job).
Mixed-catalog guard: combining chains across an in-memory and a persistent catalog previously failed deep inside SQL execution with a confusing missing-table error. DatasetQuery.union/join/subtract now raise an explicit ValueError up front. Two in-memory chains (one shared-cache database) and two persistent chains remain combinable as before.
read_dataset(..., in_memory=True): added for symmetry with datasets(), delete_dataset(), move_dataset() — without it, a dataset saved through an in-memory chain could only be read back by holding on to the session object.
docs: documented in_memory on read_storage and read_dataset.

Already in place and unchanged: distributed dispatch bypass for in-memory catalogs, explicit errors for workers/processes with in-memory, single-file read_storage creating no listing.

Resulting semantics

# inside a platform job (env-configured Postgres/ClickHouse catalog):
dc.read_storage("results_dir", in_memory=True).to_storage("s3://bucket/out")

lists into a throwaway SQLite catalog; nothing is written to the shared metastore, and the listing disappears with the process. Default behavior (no flag) is byte-for-byte unchanged.

Scenario matrix

Scenario	Behavior
Fresh process, first call has `in_memory=True`	identical to any later call: resolves to the process in-memory session; the global slot is never occupied by a throwaway catalog, in any environment (the retry example and tests now pass the flag explicitly on every call)
Persistent session exists, call has `in_memory=True`	routed to the cached dedicated in-memory session
Ambient session (context/global) already in-memory	reused as-is
Explicit persistent `session=` / `catalog=` + `in_memory=True`	explicit `ValueError` — conflicting safety arguments are never silently ignored
Explicit in-memory session/catalog + `in_memory=True`	works
Repeated `in_memory=True` with a different effective `client_config` (explicit or inherited from the ambient session)	explicit `ValueError` — the process-wide throwaway database is bound to a single storage identity, so silent credential/endpoint rebinding is refused. Implicit resolution never mutates the context stack
Persistent `client_config` override without a session	owned, cached, non-context session scoped to such calls; later unflagged calls resolve to the global session again (previously the override entered an implicit context and changed later defaults)
Caller mutates the (possibly nested) config dict after the call	session config unaffected — configs are structurally copied when frozen
Explicit in-memory `catalog=`	honored by object identity (previously validated, then silently replaced by the process cache when a global session existed)
`in_memory=True` inside an explicit in-memory context	context reused when config matches; conflict raises; `Session.__exit__` now removes only itself from the context stack, so out-of-order exits can no longer corrupt it
`.save("name")` via in-memory chain	lands in throwaway catalog; readable via `read_dataset(name, in_memory=True)`
combine in-memory × persistent (`union`/`merge`/`subtract`/`join`)	explicit `ValueError` (was: confusing SQL failure at execution)
combine in-memory × in-memory	works (shared-cache database)
`workers=` / `processes=` / distributed with in-memory	pre-existing explicit errors / bypass, unchanged
`DATACHAIN_JOB_ID` set (Studio job)	ignored by in-memory sessions; session-local job; no leakage into/out of `_CURRENT_JOB`
`DATACHAIN_PROJECT`/`DATACHAIN_NAMESPACE` set (Studio job)	applies (env is process-global): project auto-created in the throwaway metastore (`is_studio()` ⇒ `create=True`); save and `read_dataset(in_memory=True)` resolve symmetrically
`read_dataset(in_memory=True)` of a dataset not in the throwaway catalog	in Studio: clean `DatasetNotFoundError` (no remote fallback, pre-existing `is_studio()` gate); locally: documented pull-through from Studio into the throwaway catalog

Cross-database validation beyond the mode check is tracked in #1864.

Tests

loader: in_memory=True wins over serialized-env and import-path config; the env branch is provably never consulted.
session: dedicated in-memory session routing + caching; cleanup_for_tests reset; job isolation from DATACHAIN_JOB_ID in both directions.
end-to-end: read_storage(local_dir, in_memory=True) alongside a persistent session — listing dataset and saved datasets exist only in the throwaway catalog; read_dataset(..., in_memory=True) reads them back; unflagged read_dataset does not see them.
mixed-combine: union/merge/subtract across in-memory and persistent chains raise ValueError in both directions.
local pull-through: read_dataset(..., in_memory=True) pulls a Studio dataset into the throwaway catalog, not the persistent one.
Studio-job simulation (DATACHAIN_IS_STUDIO + DATACHAIN_JOB_ID + DATACHAIN_PROJECT): in-memory save auto-creates the env project in the throwaway metastore with the session-local job attributed; symmetric read-back (short and fully-qualified names); nothing leaks into the persistent catalog; unknown datasets fail cleanly.

🤖 Generated with Claude Code

Previously read_storage(..., in_memory=True) was silently ignored in two places: Session.get() dropped it whenever any session already existed, and the catalog loader dropped it whenever the metastore/warehouse came from environment config (serialized objects), or raised for import-path config. In a platform job this meant listings of ephemeral local directories (e.g. read_storage of a job's working dir to push results to S3) were persisted to the shared metastore as meaningless lst__file:///tmp/... datasets that outlive the worker. Now in_memory=True reliably means: run this chain against a temporary in-memory SQLite catalog. - loader: an explicit in-memory request wins over env-provided metastore/warehouse config instead of being ignored (serialized) or raising (import path). - Session.get: when an in-memory catalog is requested while the resolved session is backed by a persistent one, route to a dedicated cached in-memory session (inheriting client_config) instead of silently ignoring the request. - Jobs: sessions with an in-memory catalog use a session-local job created in the throwaway metastore. DATACHAIN_JOB_ID is deliberately ignored there - that job lives in the configured metastore, not in the temporary one - and the process-wide job cache is never shared with in-memory sessions in either direction. Execution guards for in-memory catalogs (distributed dispatch bypass, workers/processes errors) already existed and are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

cloudflare-workers-and-pages · 2026-07-17T02:45:06Z

Deploying datachain with Cloudflare Pages

Latest commit:	`f45b587`
Status:	✅ Deploy successful!
Preview URL:	https://5f2f7530.datachain-2g6.pages.dev
Branch Preview URL:	https://fix-in-memory-honored-everyw.datachain-2g6.pages.dev

View logs

codecov · 2026-07-17T02:50:56Z

Codecov Report

❌ Patch coverage is 97.27273% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/datachain/query/session.py	96.90%	1 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

Address review feedback on the initial implementation: - Extract _create_job() and reuse it for both the process-wide job and the session-local job of in-memory catalogs, instead of a duplicated ad-hoc create_job call with a synthetic name. In-memory jobs now carry the same script name/query metadata as regular local jobs. - Move the dedicated in-memory session creation out of Session.get into _get_in_memory_session() and let the existing client_config-mismatch branch compose with it. - Drop the redundant IN_MEMORY_SESSION_CTX block from _global_cleanup - the _ALL_SESSIONS loop already closes it. - Combining chains across in-memory and persistent catalogs previously failed deep inside SQL execution with a confusing missing-table error; union/join/subtract now raise an explicit ValueError up front. Two in-memory chains (shared-cache database) and two persistent chains remain combinable as before. - Add in_memory to read_dataset() for symmetry with datasets(), delete_dataset() and move_dataset(): without it, a dataset saved through an in-memory chain could only be read back by holding on to the session object. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…o jobs End-to-end simulation of a Studio job (DATACHAIN_IS_STUDIO, DATACHAIN_JOB_ID pointing at the persistent metastore, DATACHAIN_PROJECT routing): - save through an in-memory chain resolves the env project and auto-creates it in the throwaway metastore (is_studio => create=True), attributing the session-local job, not the env job - read_dataset(..., in_memory=True) resolves through the same env path, both short and fully-qualified names - nothing leaks into the persistent catalog - no Studio remote fallback: unknown datasets fail with a clean DatasetNotFoundError Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The unflagged read_dataset() assertion relied on the local default namespace suppressing Studio remote fallback. In the studio CI environment the persistent catalog has a non-local default namespace and a Studio config, so the lookup went into pull_dataset and failed with a connection error instead of DatasetNotFoundError. Check the persistent catalog directly — stronger and environment-independent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

This PR fixes cases where in_memory=True was being ignored, ensuring ephemeral reads/writes (like job-local read_storage(...).to_storage(...)) don’t persist listings/datasets into the configured shared metastore/warehouse. It introduces consistent “throwaway SQLite catalog” behavior across loader/session resolution, job handling, and dataset combination operations.

Changes:

Ensure catalog loader always honors explicit in_memory=True, overriding env-configured serialized/import-path catalogs.
Route Session.get(in_memory=True) to a dedicated cached in-memory session when a persistent ambient/global session exists, with job isolation for in-memory catalogs.
Add an explicit mixed-catalog guard for combining queries (e.g., union/join/subtract) and add read_dataset(..., in_memory=True) for symmetry and round-tripping.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/unit/test_session.py	Adds coverage for in-memory session routing/caching, job isolation, storage listing isolation, studio env simulation, and mixed-catalog combine errors.
tests/unit/test_catalog_loader.py	Updates/adds tests to ensure `in_memory=True` bypasses env-configured metastore/warehouse (serialized + import-path).
src/datachain/query/session.py	Adds process-wide cached in-memory session routing, session-local jobs for in-memory catalogs, and shared job creation helper.
src/datachain/query/dataset.py	Adds upfront validation to prevent combining chains across persistent vs in-memory catalogs.
src/datachain/lib/dc/storage.py	Documents `in_memory` behavior for `read_storage`.
src/datachain/lib/dc/datasets.py	Adds `in_memory` option to `read_dataset` and routes session resolution accordingly.
src/datachain/catalog/loader.py	Makes explicit `in_memory=True` win over environment-provided metastore/warehouse configuration.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Address Copilot review: the __init__ comment pointed at a helper removed in the refactor, and the get_or_create_job Behavior bullets described job name/query in a way that never matched _create_job. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

…ript Address Copilot review round 2: - get_or_create_job docstring no longer implies finalize hooks apply to the session-local in-memory job; rewritten as a short resolution-order paragraph. - _create_job also catches UnicodeDecodeError when reading the script for the job query (a non-UTF-8 script must not fail job creation). Also trim verbose comments across the in-memory changes (net -30 lines). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

- Explicit persistent session= or catalog= combined with in_memory=True now raises ValueError instead of silently ignoring the flag - the same persistence hazard this PR eliminates elsewhere. Tests that passed the flag redundantly alongside an explicit session are cleaned up. - An in-memory request with a different client_config gets its own session (cached by config, backed by the same shared-cache database, so chains stay combinable) instead of falling into the generic mismatch branch, which entered a context and could hijack later unflagged calls. - In Studio, a first-call in_memory=True no longer makes the throwaway catalog the process default; locally the legacy behavior is kept for backward compatibility and now documented and regression-tested. - Add local Studio pull-through coverage: read_dataset(in_memory=True) pulls into the throwaway catalog, not the persistent one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Address review: the two-tier cache (special first session + keyed extras) made reuse order-dependent and could hand back a session whose config no longer matched the ambient one (e.g. empty ambient config vs a non-empty cached one). Now there is a single cache keyed by effective client_config (explicit, else inherited from the ambient session), so resolution is deterministic regardless of call order. IN_MEMORY_SESSION_CTX stays as the first-created session pointer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… stack Address re-review round 2 by removing machinery rather than adding it: - One implicit in-memory session per process, client_config frozen at creation (explicit, else inherited). A different explicit config now raises with a pointer to Session(in_memory=True, client_config=...): the shared throwaway database cannot isolate data per config, so a per-config session pool implied isolation that did not exist (same URI would alias across endpoints, File rows rebind to the reader's credentials - which is also the pre-existing semantics of persistent catalogs). The config-keyed pool is gone, and with it the mutable dict / stringified cache key concern. - An accepted explicit in-memory catalog= is now actually used (cached by object identity), instead of being validated and then silently replaced by the process cache when a global session exists. - in_memory=True requests resolve through the in-memory resolver even when the ambient session is already in-memory: a config conflict raises instead of falling into the generic mismatch branch, which entered an implicit context. - Session.__exit__ removes only itself from SESSION_CONTEXTS: exiting a non-context session or exiting out of order no longer pops (and thereby corrupts) someone else's context. - Rename _ensure_combinable to _ensure_same_catalog_mode - it checks catalog mode, not database identity, and should not claim more. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Third review round: - The implicit in-memory session now enforces its single-config contract against the *effective* config of every call - explicit or inherited from the ambient session - instead of only explicit ones. Reusing the session under a different storage identity (endpoint/credentials) raised silent-misrouting concerns; now it raises. The error message no longer suggests explicit sessions as an isolated escape hatch, since all in-memory catalogs share one process database. - client_config is now structurally copied (nested mappings/sequences) when freezing it, so caller-side mutation of nested client_kwargs etc. cannot change a session's configuration after creation. Leaf objects (credential providers, SSL contexts) stay by reference. - A persistent client_config override no longer enters an implicit context (action at a distance: one override changed all later unflagged calls). It returns an owned, cached, non-context session; later unflagged calls resolve to the global session again. This aligns persistent and in-memory override semantics. - IN_MEMORY_SESSIONS renamed to SIDE_SESSIONS: it now owns both explicit-catalog wrappers and persistent config overrides. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… docs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Simplification pass over the accumulated session changes: - Session.get is now a five-rule precedence resolver (explicit session, explicit in-memory catalog, in_memory, ambient, config override) with one small helper per rule, instead of accreted special cases. - The legacy fork is gone: in_memory=True never occupies the global slot, in any environment. First-call and later-call semantics are identical, and the is_studio() special case is deleted. The retry example and tests that relied on 'later unflagged calls become in-memory automatically' are updated to pass the flag explicitly. - That legacy turned out to be an accident: internal code (catalog version updates, delta processing) constructs DatasetQuery / read_dataset without threading the session, which used to capture the global slot with the in-memory catalog. Those gaps are fixed properly now - delta helpers thread the chain's session, and an explicitly provided in-memory catalog resolves to a cached wrapper session instead of becoming the global session. - Session.__exit__ is idempotent and skips temp-dataset cleanup when the database is already closed: exiting a session after its catalog owner closed the database used to silently reconnect and leak the new connection (this was the source of a long-standing class of ResourceWarning flakes in tests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

diff funnels through merge/join, so the primitive-level guard covers it; assert that so a future diff reimplementation cannot silently lose it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Parametrized over union, merge, subtract, diff, compare_and_split and file_diff, with the in-memory chain on either side: all must reject a mixed in-memory x persistent pair up front. Pins the invariant at the public API surface instead of relying on knowledge of which primitives each operation composes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…og rule - Two test_session-based tests accidentally received in_memory=True in a bulk edit; the shared-cache test database masked it on SQLite while the Studio backend failed with DatasetNotFoundError. They now thread session=test_session explicitly, deterministic on every backend. - Session.__exit__ removes this session's most recent context entry and does so outside the idempotence guard, so re-entrant 'with session:' blocks unwind the stack correctly. - A client_config override rebuilds its catalog from default/env configuration; when the ambient session wraps an explicitly provided catalog object that cannot be reconstructed, raise instead of silently switching the metastore. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

shcheklein and others added 3 commits July 16, 2026 20:24

shcheklein requested a review from Copilot July 17, 2026 03:57

shcheklein self-assigned this Jul 17, 2026

Copilot started reviewing on behalf of shcheklein July 17, 2026 03:57 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Comment thread src/datachain/query/session.py Outdated

Comment thread src/datachain/query/session.py Outdated

shcheklein requested a review from Copilot July 17, 2026 04:15

Copilot started reviewing on behalf of shcheklein July 17, 2026 04:15 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Comment thread src/datachain/query/session.py Outdated

Comment thread src/datachain/query/session.py

shcheklein requested a review from Copilot July 17, 2026 04:50

Copilot started reviewing on behalf of shcheklein July 17, 2026 04:50 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Comment thread src/datachain/query/session.py Outdated

shcheklein and others added 3 commits July 16, 2026 22:05

shcheklein requested a review from Copilot July 17, 2026 05:59

Copilot started reviewing on behalf of shcheklein July 17, 2026 05:59 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Comment thread src/datachain/query/session.py

shcheklein mentioned this pull request Jul 17, 2026

Catalog needs a database-identity contract for early cross-catalog validation #1864

Open

shcheklein requested a review from Copilot July 17, 2026 14:57

Copilot started reviewing on behalf of shcheklein July 17, 2026 14:57 View session

shcheklein and others added 2 commits July 17, 2026 07:59

docs: simplify in-memory comments, drop Studio references from public…

58bf1d9

… docs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

docs: drop SQLite implementation detail from in_memory docs

921b3fc

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Comment thread src/datachain/query/session.py Outdated

Comment thread src/datachain/query/session.py Outdated

shcheklein and others added 4 commits July 17, 2026 08:36

test: pin transitive mixed-catalog guard through diff

ec41d3d

diff funnels through merge/join, so the primitive-level guard covers it; assert that so a future diff reimplementation cannot silently lose it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Navigation Menu

Uh oh!

fix: honor in_memory everywhere, plugging a throwaway SQLite catalog - #1863

fix: honor in_memory everywhere, plugging a throwaway SQLite catalog#1863
shcheklein wants to merge 16 commits into
mainfrom
fix/in-memory-honored-everywhere

shcheklein commented Jul 17, 2026 •

edited

Loading

Uh oh!

cloudflare-workers-and-pages Bot commented Jul 17, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jul 17, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

shcheklein commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

What changed

Resulting semantics

Scenario matrix

Tests

Uh oh!

cloudflare-workers-and-pages Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying datachain with Cloudflare Pages

Uh oh!

codecov Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

shcheklein commented Jul 17, 2026 •

edited

Loading

cloudflare-workers-and-pages Bot commented Jul 17, 2026 •

edited

Loading

codecov Bot commented Jul 17, 2026 •

edited

Loading