Skip to content

Add metadata to Passages and Sources tables to track embedding model/dimentions #719

Closed
@sarahwooders

Description

We should add metadata fields to the Passages table that include:

  • embedding_model: Model used to generate the embedding (also in Sources table)
  • embedding_dim: Dimensions of embedding field (also in Sources table)
  • num_tokens: Number of tokens contained in passage
  • num_chars: Number of characters contained in passages

This will allow us to add additional checks:

  • When querying archival memory, filter by embedding_model field to avoid
  • Warn users when they change their embedding model, which loaded sources will not longer be accessible to their agent

Deign Decisions

  • When a user change their embedding model, should we remove previously imported embeddings from the agent (as they will still be saved in the data sources table)?

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions