Add metadata to Passages
and Sources
tables to track embedding model/dimentions #719
Closed
Description
We should add metadata fields to the Passages
table that include:
embedding_model
: Model used to generate the embedding (also inSources
table)embedding_dim
: Dimensions of embedding field (also inSources
table)num_tokens
: Number of tokens contained in passagenum_chars
: Number of characters contained in passages
This will allow us to add additional checks:
- When querying archival memory, filter by
embedding_model
field to avoid - Warn users when they change their embedding model, which loaded sources will not longer be accessible to their agent
Deign Decisions
- When a user change their embedding model, should we remove previously imported embeddings from the agent (as they will still be saved in the data sources table)?