Skip to content

/load_expr re-runs the full stat pipeline on every POST even when the session already exists #899

Description

@paddymul

Problem

`LoadExprHandler.post` (`handlers.py:398`) runs the full expression load and stat pipeline unconditionally on every request, before consulting the session cache. The cache check — `sessions.get_or_create(session_id, build_dir)` at `handlers.py:463` — happens after `load_expr_build_dir` and `XorqServerDataflow.init` have already completed. A repeat POST with the same `session_id` and `build_dir` triggers the same ~3–4 s pipeline as the first call and discards the result.

The handler has no early-exit path for an already-loaded session.

Evidence

Instrumenting `xo.connect` and timing each phase of a `/load_expr` POST for a warm stat cache:

load_expr_build_dir:  25ms   (1 xo.connect, guarded by default_backend check)
XorqServerDataflow: 1015ms   (stat pipeline — reads 17 cached parquet files)
/load_expr total:   ~3800ms  (remainder is execution inside the dataflow)

Calling `/load_expr` three consecutive times with the same `build_dir` and distinct `session_id`s (to avoid the dict lookup):

call 1:  3829ms
call 2:  3766ms
call 3:  3771ms

All three pay the full cost. The session dict provides no relief because the expensive work finishes before it is consulted.

Impact

A companion server that wraps Buckaroo (e.g. tallyman) must maintain its own session map and skip `/load_expr` entirely on repeat views to avoid the cost. After a server restart, the first navigation to each catalog entry blocks for ~4 s while the session is re-established — one per entry, serialised, with no way to pre-warm without paying the full cost.

Suggested fix

At the top of `LoadExprHandler.post`, after parsing `session_id` and `build_dir`, check whether the session already exists with a matching build_dir:

```python
sessions = self.application.settings["sessions"]
existing = sessions.get(session_id)
if existing and getattr(existing, "build_dir", None) == build_dir:
self.write({"session": session_id, ...existing metadata...})
return
```

This short-circuits the load for repeat POSTs. A companion can then call `/load_expr` on every page view rather than maintaining its own session map, and a startup pre-warm pass becomes cheap.

Context

Identified while profiling the tallyman companion server. Related: #896 (connection accumulation — fixing that issue reduces per-call memory cost but does not eliminate the redundant stat-pipeline execution this issue tracks).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions