Skip to content

latest symlink is environment-blind in multi-environment workflows #389

@stack72

Description

@stack72

Problem

When the same model runs across multiple environments, all data versions are stored under a single path with one latest symlink:

.swamp/data/command/shell/{model-id}/result/
├── 1/       (dev run)
├── 2/       (prod run)
├── 3/       (dev run)
└── latest -> 3   # always points to most recent, regardless of environment

The latest symlink points to whichever version was written most recently — in this case, version 3 from a dev run. There is no concept of "latest for prod" or "latest for staging" at the filesystem level.

Why This Matters

The latest symlink serves two purposes:

  1. Quick access — users browsing the data directory can find the most recent output
  2. CEL resolution — expressions like model.deploy-app.resource.result.latest.attributes.* resolve through this symlink

In a single-environment setup, latest always points to the most recent run, which is correct. But in a multi-environment setup, latest is misleading:

  • A user looking at latest after a dev run might think it's the current prod state
  • A CEL expression referencing latest in a prod workflow might resolve to dev data if dev ran more recently
  • There is no way to browse to "latest prod result" without searching through version metadata

This creates a correctness hazard: the most convenient and obvious access path (latest) returns environment-incorrect data in multi-environment scenarios.

Current Workaround

Users can use swamp data search --tag environment=prod to find the right version, but this requires knowing to avoid the latest symlink and using tag-based queries instead. CEL expressions have no equivalent workaround — data.latest() and the implicit latest path always resolve to the globally most recent version.

Use Case

A deployment workflow runs multiple times per day across dev, staging, and prod. A downstream monitoring model references the deployment model's output via CEL to check the last deployed version. Without environment-aware latest resolution, the monitoring model reports whichever environment deployed most recently, not the specific environment being monitored.

This issue is related to the need for tag-filtered data access in CEL (see related issue), which would solve the CEL side. This issue specifically tracks the filesystem-level latest symlink behavior for users browsing data directly.

Metadata

Metadata

Assignees

Labels

in-discussionA feature or issue that is in active discussion

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions