[Vote] Sub-Project Proposal: lance-context – The Multimodal Native Persistence Layer for Agentic AI

[beinan]

Project Name: lance-context
Dependencies: lance, lancedb,lance-namespace,lance-graph

1. Abstract

lance-context is a specialized library designed to manage the lifecycle of Multimodal Agentic Context. Unlike standard vector stores that treat data as flat lists of text embeddings, lance-context treats agent history as a versioned, branching, multimodal dataset. It leverages Lance's native columnar storage to unify text, images, PDF documents, and structured DataFrames into a single, queryable "Context Stream."

2. Motivation

Building production-grade autonomous agents requires complex state management beyond simple chat logs. Modern agents are multimodal; they view screens, read reports, and analyze data.

The Problem:

Developers currently fragment context storage:

• Text goes to Vector DBs.
• Images/PDFs go to S3 (with loose links in the DB).
• DataFrames are pickled or lost after the session ends.

The Gap:

There is no unified "Memory" that can hold a user's instruction alongside the image they uploaded and the dataframe the agent generated in response.

The Solution:

A unified persistence layer where an agent can "rollback" to a state that includes both the conversation history and the specific version of the dataframe it was working on.

3. Why Lance?

Lance is uniquely positioned to solve this problem where other formats fail:

• Native Multimodal Storage: Lance is a columnar format that efficiently stores mixed types (Scalars, Vectors, Binary Blobs, Nested Structs) in the same file. We do not need a separate object store for small-to-medium media assets.

• Zero-Copy Versioning: Lance's dataset versioning allows us to "branch" the context (e.g., for "Tree of Thoughts" exploration) without duplicating the underlying data storage.

• Fast Random Access: lance allows fast retrieval of full context windows by ID (row_id single row seek or chat_id multi row scan), which is critical for rebuilding context for the LLM.

API Design (Draft)

The API aims to be "framework agnostic" (usable by LangChain, CrewAI, or vanilla OpenAI scripts).

import lance_context as lc
import pandas as pd
from PIL import Image

# Initialize persistent context on disk or S3
ctx = lc.Context.create("s3://agent-memory/session_multimodal_01")

# --- 1. Multimodal Ingestion ---

# Add Text (Standard)
ctx.add(role="user", content="Analyze this invoice and this dataset.")

# Add Image (Multimodal)
# Automatically handles image serialization and optional CLIP embedding
img = Image.open("invoice.png")
ctx.add(role="user", content=img, data_type="image/png")

# Add DataFrame (Structured Data)
# Stores the dataframe natively so the agent can retrieve it later
df = pd.read_csv("financials.csv")
ctx.add(role="assistant", content=df, data_type="dataframe")


# --- 2. Advanced State Management ---

# Snapshot: Save state before executing risky code
safe_point = ctx.snapshot(label="pre_code_execution")

# Branching: Create a fork to try an approach
branch_a = ctx.fork(branch_name="attempt_pandas")
branch_a.add(role="assistant", content="Trying Pandas approach...")

# Rollback: If Branch A fails, revert main context to safe point
ctx.checkout(safe_point)

4 Why lance-format? (Project Positioning)

This project belongs as a sub-project within lance-format (alongside lance-graph or lance-spark) rather than an external application library for three key reasons:

1. It is a Connector, not an Application lance-context does not define how an agent thinks or prompts; it strictly defines how agent state is serialized. It acts as the standard "Connector" between the Lance format and high-level AI frameworks. By standardizing this at the Lance level, we enable interoperability across the entire ecosystem.

2. It is Infrastructure, not Logic. The core challenges here are storage challenges: zero-copy branching, efficient blob storage, and manifest management. These are database concerns, not AI agent concerns. This project builds on low-level Lance primitives to solve infrastructure problems, making it distinct from application-layer memory tools.

3. It Requires Deep Lance Integration To make features like "Context Forking" (Tree of Thoughts) performant, the library needs close proximity to Lance's internal versioning and manifest APIs. The API must be transparent to the AI application—the user simply calls .snapshot(), and lance-context handles the low-level storage mechanics. This level of optimization is best achieved within the Lance engineering domain.

[wjones127]

Could you elaborate more on why it makes sense for this to be a sub project within Lance, rather than a separate project that depends on Lance?

[beinan]

Great question, as we discussed in the meeting this morning, I just added a new section to the proposal to explain why this to be a sub project

[prrao87]

For all those who have seen this, please add a +1 to upvote whether this should be added as a sub-project within Lance. Thanks!

[jackye1995]

Note that this is a subproject vote, so the vote will last for 3 days

[jackye1995]

Based on discussion in community sync, looks like there is value in trying this out, so +1 (binding)

[prrao87]

I am interested in seeing how the community benefits from this, +1 from me (binding).

[yanghua]

+1 binding

[wjones127]

+1

[jackye1995]

The vote time has passed and we have enough vote, I have created lance-context repo, let's go from there!