Skip to content

[DRAFT] ESQL: Add TEXT_EMBEDDING function for dense vector embeddings #131131

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

afoucret
Copy link
Contributor

@afoucret afoucret commented Jul 11, 2025

Summary

Implements the TEXT_EMBEDDING function for ES|QL to generate dense vector embeddings from text using inference model.

Function Signature: TEXT_EMBEDDING(text: string, inference_id: string) -> dense_vector

Example Usage:

FROM documents 
| WHERE KNN(embedding_field, TEXT_EMBEDDING(content, "my-embedding-deployment"), 10)

Implementation Status

Completed in this PR:

  • TEXT_EMBEDDING_FUNCTION capability with snapshot build gating

  • Core Function Infrastructure

    • TextEmbedding function class with proper type validation and serialization
    • InferenceFunction interface for inference-based functions
    • Function registration in EsqlFunctionRegistry
  • Analysis of the inference function (validate existence and type of the inference endpoint)

    • Refactored pre-analysis, so it is able to collect inference ids form both Inference plans and inference function
    • Added validation for inference function in the analysis
  • Add a pre-optimizer async phase to the ES|QL query execution

  • Documentation generated from the annotations

🚧 TODO (Before Merge):

  • Execute the inference in the pre-optimizer
  • Integration tests and end-to-end validation

Notes

The function is enabled only in snapshot builds.
TEXT_EMBEDDING function is tracked into #131022

@elasticsearchmachine elasticsearchmachine added needs:triage Requires assignment of a team area label v9.2.0 labels Jul 11, 2025
@afoucret afoucret marked this pull request as draft July 11, 2025 21:27
Copy link
Contributor

github-actions bot commented Jul 11, 2025

🔍 Preview links for changed docs

@afoucret afoucret changed the title [DRAFT] ESQL: Add EMBED_TEXT function for dense vector embeddings [DRAFT] ESQL: Add TEXT_EMBEDDING function for dense vector embeddings Jul 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs:triage Requires assignment of a team area label v9.2.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants