Proposal: Introduce Snapshot as the Foundation for Time Travel and Metadata Table

[majin1102]

Motivation

Case 1: No Time-Based Version Access

Currently, there is no convenient way to access historical versions by timestamp. Users must first list all versions, find the desired timestamp, then explicitly checkout that version. This workflow is cumbersome for time travel use cases.

Case 2: Full manifests scanning for version listing

The existing implementation (Dataset.versions()) reads manifests sequentially when listing versions. For large tables with many versions, this causes significant memory and time overhead. Additionally, there is no support for range-based queries, forcing a full manifest load each time we only need a page of version history. (as illustrated at the bottom)

Case 3: No unified metadata entry for metadata table

Version and Transaction are separate types with no unified interface for accessing snapshot-level metadata. This forces upper layers to handle multiple types and understand their relationships.

Proposed Solution

Introduce a new Snapshot concept as a unified, lightweight entry point for all snapshot-level metadata.

Key Goals

1. Lazy loading: Only load manifest when needed. For example, for a 20-record page, we load at most 20 manifests, which callers can parallelize
2. Range queries: Version number and timestamp-based filtering, eliminating unnecessary manifest loading
3. Unified abstraction: Entry point for snapshot metadata covering version and transaction to directly support cases like metadata table.

Interface

A prototype for this: #5886

Snapshot

pub struct Snapshot {
    dataset: Arc<Dataset>,
    manifest_location: ManifestLocation,
}

Snapshots API

// Get snapshots handler
let snapshots = dataset.snapshots();

// List all snapshots (lazy, returns metadata only)
let all = snapshots.list().await?;

// Get latest snapshot
let latest = snapshots.latest().await?;

// Version range query [start, end)
let range = snapshots.within(1, 100).await?;  // versions 1-99

// Timestamp range query [start, end)
let by_time = snapshots.within_timestamp(ts1, ts2).await?;

// Directional queries with limit
let earlier = snapshots.earlier_than(5, 10).await?;  // versions < 5, limit 10
let later = snapshots.later_than(100, 20).await?;     // versions >= 100, limit 20

Snapshot API

// Lightweight access without loading manifest
let ts = snapshot.timestamp();
let version = snapshot.version();

// Lazy manifest loading
let manifest = snapshot.manifest().await?;

// Checkout as new Dataset (reuses loaded manifest)
let historical = snapshot.checkout().await?;

// Get transaction if available
let tx = snapshot.transaction().await?;

Benefits Scenarios

Time Travel

// Travel by timestamp (e.g., yesterday)
let historical = dataset.snapshots()
    .earlier_than_timestamp(Utc::now() - chrono::Duration::days(1), 1)
    .await?
    .remove(0)
    .checkout()
    .await?;

Metadata Table

This is a real use case from integrating Lance with Apache Amoro:

Current implementation:

let all_versions = dataset.versions(); // reads all manifests sequentially
for version in all_versions {
    let current_dataset = dataset.checkout(version.version) // re-read this manifest
    let transaction = current_dataset.read_transaction()
    let snapshot = map_snapshot(version, transaction) // construct a snapshot like object
}

This approach loads all manifests upfront and cannot be paginated server-side.

With Snapshot, we can implement efficient pagination:

// Efficiently list versions with metadata, loading only what's needed
let page_snapshots = dataset.snapshots().within(last_version, last_version + 20).await?;

[jackye1995]

Do we need to introduce a new concept Snapshot here? I think for systems like Iceberg, snapshot and table version are 2 different things, and are tracked separately, thus 2 concepts. But here the are the same thing. I feel it would be nice to keep the number of concepts minimum if possible. Curious what others think, cc @wjones127

Also I feel there is a mix of bad user experience and performance concerns.

• For case 1, I agree we should expose more friendly API for time travel
• For case 2, could we do some binary search to find the right time if there are many versions?
• For case 3, there is a dedicated thread for metadata table, maybe we should discuss more there: Expose lance metadata as tables #5856

[majin1102]

Do we need to introduce a new concept Snapshot here?

Totally agree. I considered to extend Version instead. But not sure if Version is the right place to do this lazy manifest loading and we must consider API compatibility. I think we could make things clear in this discussion.

For case 2, could we do some binary search to find the right time if there are many versions?

My first step is to decouple list_manifests and read_manifest in dataset.versions(), since they represent distinct storage operations. This would allow us to skip unnecessary read_manifest calls when only a subset of versions is needed. Furthermore, by exposing lazy loading through the API (e.g., snapshot.version()), callers could parallelize manifest reads as needed.

For binary searching, I'm not sure if you are meaning an optimization for list_manifests?

For case 3, there is a dedicated thread for metadata table, maybe we should discuss more there: #5856

I see this as foundational work for that discussion (not a point of conflict), and I'd love to share my thoughts in that thread.

[jackye1995]

For binary searching, I'm not sure if you are meaning an optimization for list_manifests?

I mean for deciding what manifest to read. Here is what I was imagining, drafted by calude:

  use chrono::{DateTime, Utc};

  /// Find the oldest version created at or after the given timestamp.
  ///
  /// Uses binary search to minimize manifest reads - O(log N) instead of O(N).
  ///
  /// # Arguments
  /// * `since` - The minimum timestamp (inclusive)
  ///
  /// # Returns
  /// * `Ok(Some(version))` - The oldest version at or after the timestamp
  /// * `Ok(None)` - No version exists at or after the timestamp
  pub async fn find_oldest_version_since(&self, since: DateTime<Utc>) -> Result<Option<Version>> {
      let mut locations: Vec<ManifestLocation> = self
          .commit_handler
          .list_manifest_locations(&self.base, &self.object_store, false)
          .try_collect()
          .await?;

      if locations.is_empty() {
          return Ok(None);
      }

      locations.sort_by_key(|l| l.version);

      let target_nanos = timestamp_to_nanos(since);

      // Binary search for first version with timestamp >= target
      let mut lo = 0;
      let mut hi = locations.len();

      while lo < hi {
          let mid = lo + (hi - lo) / 2;
          let manifest = read_manifest(
              &self.object_store,
              &locations[mid].path,
              locations[mid].size,
          )
          .await?;

          if manifest.timestamp_nanos >= target_nanos {
              hi = mid;
          } else {
              lo = mid + 1;
          }
      }

      if lo >= locations.len() {
          return Ok(None);
      }

      let manifest = read_manifest(
          &self.object_store,
          &locations[lo].path,
          locations[lo].size,
      )
      .await?;

      Ok(Some(Version::from(&manifest)))
  }

[majin1102]

Yeah, I see.

I was just using ObjectMeta.last_modified as the timestamp of each version in my prototype. However, this seems like a shortcut because manifests are immutable once created, so the timestamp could be obtained earlier — prior to reading the manifests. I’m wondering whether this approach carries any risks

[jackye1995]

I think we do want to use the timestamp in the manifest as the source of truth for a few reasons:

1. there is a slight difference between the time recorded in manifest and the actual time recorded in the storage system, and we want the manfiest one to be the source of truth.
2. using ObjectMeta.last_modified would add additional dependency on the file system supporting last_modified. It might not be a big requirement, but it could still be unreliable depending on the storage (especially a distributed one).
3. for external manifest store, this gets more complicated as the external manifest store needs to provide a timestamp if we go with using storage provided timestamp, and also we (best effort) commit a manifest in storage with the right version. That means the manifest could be committed at a much later time than the actual creation time recorded in manifest.

Delta also moved from using storage timestamp to recording a timestamp in log: https://github.com/delta-io/delta/blob/master/PROTOCOL.md#in-commit-timestamps, so I think we are doing the right thing here.

[westonpace]

There seems to be two parts to this proposal (but I may be misunderstanding):

• We need some API equivalent of "manifest location (and time)" so we can return something cheaply from a range query without materializing all the data
• We need to combine version and transaction

I think I understand (and agree) with the first point. I'm not sure about the second. Let's say you issue a range query and get back 20 matching locations. Is the goal here to reduce 40 reads (manifest + transaction) into 20 reads (combined manifest/transaction)?

[majin1102]

Yes. Your summary is spot on.

The first point is really the core of this proposal. Building on that foundation, we could introduce time-based APIs to better support time travel use cases.

As for the second point, it's not important but could consider to pick up along the way. There are use cases that benefit from accessing manifest and transaction metadata together. But I agree the existing approach is sufficient