Semantic_text match_all with Highlighter #128702

Samiul-TheSoccerFan · 2025-05-30T21:10:37Z

This PR addresses the missing highlighting for semantic_text fields when a match_all query is used. We guide users to use semantic highlighting to view their chunks, but highlighting does not currently return any results with match_all. Highlighting is usually unnecessary for non-inference fields with match_all, but it's essential for semantic_text fields to return the generated chunks. This change ensures those highlighting results are returned as expected.

Test cases:

Only text fields

PUT my-index/
{
  "mappings": {
    "properties": {
      "semantic1": {
        "type": "text"
      },
      "semantic2": {
        "type": "text"
      }
    }
  }
}

POST my-index/_doc/1
{
  "semantic1": "Puggles are pugs and beagles",
  "semantic2": "Chiweenies are chihuahuas and dachshunds"
}


GET my-index/_search
{
  "query": {
    "match_all": {}
  }
}

only inference fields

PUT my-semantic-index/
{
  "mappings": {
    "properties": {
      "semantic1": {
        "type": "semantic_text"
      },
      "semantic2": {
        "type": "semantic_text"
      }
    }
  }
}

POST my-semantic-index/_doc/1
{
  "semantic1": "Puggles are pugs and beagles",
  "semantic2": "Chiweenies are chihuahuas and dachshunds"
}

GET my-semantic-index/_search
{
  "query": {
    "match_all": {}
  },
  "highlight": {
    "fields": {
      "semantic1": {
        "type": "semantic",
        "number_of_fragments": 1
      },
      "semantic2": {
        "type": "semantic",
        "number_of_fragments": 1
      }
    }
  }
}

Both infer and non-inference fields

PUT test-index
{
  "mappings": {
    "properties": {
      "test_field": {
        "type": "semantic_text"
      },
      "non_infer_field": {
        "type": "text"
      }
    }
  }
}

PUT test-index/_doc/doc1
{
  "test_field": "these are not the droids you're looking for. He's free to go around",
  "non_infer_field": "this is a non-inference field"
}

GET test-index/_search
{
  "query": {
    "match_all": {}
  },
  "highlight": {
    "fields": {
      "test_field": {
        "type": "semantic",
        "number_of_fragments": 1
      }
    }
  }
}

Both dense vector fields

PUT _inference/text_embedding/my-e5-model
{
  "service": "elasticsearch",
  "service_settings": {
    "num_allocations": 1,
    "num_threads": 1,
    "model_id": ".multilingual-e5-small"
  }
}

PUT my-dense-semantic-index/
{
  "mappings": {
    "properties": {
      "semantic1": {
        "type": "semantic_text",
        "inference_id": "my-e5-model"
      },
      "semantic2": {
        "type": "semantic_text",
        "inference_id": "my-e5-model"
      }
    }
  }
}

POST my-dense-semantic-index/_doc/1
{
  "semantic1": ["Puggles are pugs and beagles", "    ", "now with chunks"],
  "semantic2": ["Chiweenies are chihuahuas and dachshunds"  , "    ", "now with chunks"]
}

GET my-dense-semantic-index/_search
{
  "query": {
    "match_all": {}
  },
  "highlight": {
    "fields": {
      "semantic1": {
        "type": "semantic",
        "number_of_fragments": 3
      },
      "semantic2": {
        "type": "semantic",
        "number_of_fragments": 3
      }
    }
  }
}

Some additional cases

GET my-*/_search
{
  "query": {
    "match_all": {}
  },
  "highlight": {
    "fields": {
      "semantic1": {
        "number_of_fragments": 1
      },
      "semantic2": {
        "number_of_fragments": 1
      }
    }
  }
}

GET my-*/_search
{
  "query": {
    "match_all": {}
  },
  "highlight": {
    "fields": {
      "semantic1": {
        "type": "semantic",
        "number_of_fragments": 1
      },
      "semantic2": {
        "type": "semantic",
        "number_of_fragments": 1
      }
    }
  }
}

// throws error as fields are semantic_text field
GET my-*/_search
{
  "query": {
    "match_all": {}
  },
  "highlight": {
    "fields": {
      "semantic1": {
        "type": "score",
        "number_of_fragments": 1
      },
      "semantic2": {
        "type": "score",
        "number_of_fragments": 1
      }
    }
  }
}

kderusso

Nice start, I've added some high level comments on overall organization. We can do a deeper review when it's closer to ready.

.../java/org/elasticsearch/xpack/inference/queries/SemanticMatchAllQueryRewriteInterceptor.java

...src/main/java/org/elasticsearch/xpack/inference/queries/SemanticQueryRewriteInterceptor.java

Mikep86

There's a much simpler way to fix this by updating the semantic highlighter. Take a look at how you can change QueryVisitor in extractDenseVectorQueries and extractSparseVectorQueries: https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/highlight/SemanticTextHighlighter.java#L250

Samiul-TheSoccerFan · 2025-06-03T13:12:51Z

@elasticmachine update branch

elasticsearchmachine · 2025-06-03T13:40:37Z

Pinging @elastic/search-eng (Team:SearchOrg)

elasticsearchmachine · 2025-06-03T13:40:37Z

Pinging @elastic/search-relevance (Team:Search - Relevance)

elasticsearchmachine · 2025-06-03T13:43:21Z

Hi @Samiul-TheSoccerFan, I've created a changelog YAML for you.

kderusso

Thanks for these changes, looks great @Samiul-TheSoccerFan !

kderusso · 2025-06-03T14:53:37Z

docs/changelog/128702.yaml

@@ -0,0 +1,5 @@
+pr: 128702
+summary: Highlighting support added to semantic_text with `match_all`


Nitpick: Since this is a bug, perhaps we call the changelog summary something like `Fix issue where highlighting was not returned with match_all queries"

kderusso · 2025-06-03T14:54:05Z

...rence/src/main/java/org/elasticsearch/xpack/inference/highlight/SemanticTextHighlighter.java

@@ -267,6 +268,8 @@ public void visitLeaf(Query query) {
                    queries.add(fieldType.createExactKnnQuery(VectorData.fromFloats(knnQuery.getTargetCopy()), null));
                } else if (query instanceof KnnByteVectorQuery knnQuery) {
                    queries.add(fieldType.createExactKnnQuery(VectorData.fromBytes(knnQuery.getTargetCopy()), null));
+                } else if (query instanceof MatchAllDocsQuery) {


😍 Nice optimization from the first solution!

Samiul-TheSoccerFan · 2025-06-03T16:30:03Z

@elasticmachine update branch

Mikep86

Approach looks good! Caught a copy/paste error in the tests that we should fix though.

...nce/src/yamlRestTest/resources/rest-api-spec/test/inference/90_semantic_text_highlighter.yml

...src/yamlRestTest/resources/rest-api-spec/test/inference/90_semantic_text_highlighter_bwc.yml

...nce/src/yamlRestTest/resources/rest-api-spec/test/inference/90_semantic_text_highlighter.yml

...src/yamlRestTest/resources/rest-api-spec/test/inference/90_semantic_text_highlighter_bwc.yml

Mikep86

LGTM! Good clean solution and tests :)

Samiul-TheSoccerFan · 2025-06-03T20:57:36Z

@elasticmachine update branch

Samiul-TheSoccerFan · 2025-06-04T12:22:00Z

@elasticmachine update branch

elasticsearchmachine · 2025-06-04T13:34:39Z

💔 Backport failed

Status	Branch	Result
❌	8.19	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 128702

Samiul-TheSoccerFan · 2025-06-04T14:22:09Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Questions ?

Please refer to the Backport tool documentation

* initial implementation for match_All * reformat * [CI] Auto commit changes from spotless * Excluding matchAllintercepter * Adding matchAllDocs support for vector fields * [CI] Auto commit changes from spotless * Remove previous implementation * Adding yaml tests for match_all * fixed yaml tests * Update docs/changelog/128702.yaml * Update changelog * changelog - update summary * Fix wrong inference names for the yaml tests --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> (cherry picked from commit d1b5532) # Conflicts: # x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/InferenceFeatures.java

Samiul-TheSoccerFan added 2 commits May 30, 2025 16:41

initial implementation for match_All

faa2689

reformat

ddd7112

elasticsearchmachine added the v9.1.0 label May 30, 2025

[CI] Auto commit changes from spotless

d9bcbe3

kderusso reviewed Jun 2, 2025

View reviewed changes

Mikep86 reviewed Jun 2, 2025

View reviewed changes

Samiul-TheSoccerFan and others added 5 commits June 2, 2025 16:00

Excluding matchAllintercepter

60da6f4

Adding matchAllDocs support for vector fields

1139842

[CI] Auto commit changes from spotless

12c1547

Remove previous implementation

e788bfb

Adding yaml tests for match_all

9248b7f

elasticmachine and others added 2 commits June 3, 2025 16:12

Merge branch 'main' into match_all_with_highlighter

4a36076

fixed yaml tests

c53cf5b

Samiul-TheSoccerFan marked this pull request as ready for review June 3, 2025 13:39

elasticsearchmachine added the needs:triage Requires assignment of a team area label label Jun 3, 2025

Samiul-TheSoccerFan added auto-backport Automatically create backport pull requests when merged :SearchOrg/Relevance Label for the Search (solution/org) Relevance team v8.19.0 and removed needs:triage Requires assignment of a team area label labels Jun 3, 2025

elasticsearchmachine added Team:SearchOrg Meta label for the Search Org (Enterprise Search) Team:Search - Relevance The Search organization Search Relevance team labels Jun 3, 2025

Samiul-TheSoccerFan added the >bug label Jun 3, 2025

Samiul-TheSoccerFan requested review from kderusso and Mikep86 June 3, 2025 13:43

Samiul-TheSoccerFan and others added 2 commits June 3, 2025 09:43

Update docs/changelog/128702.yaml

b70dc11

Update changelog

51e21c9

kderusso approved these changes Jun 3, 2025

View reviewed changes

changelog - update summary

ed44ca1

Merge branch 'main' into match_all_with_highlighter

9c86416

Mikep86 reviewed Jun 3, 2025

View reviewed changes

Fix wrong inference names for the yaml tests

9e42c98

Mikep86 approved these changes Jun 3, 2025

View reviewed changes

Merge branch 'main' into match_all_with_highlighter

0ed1798

Merge branch 'main' into match_all_with_highlighter

dc54ae7

Samiul-TheSoccerFan merged commit d1b5532 into elastic:main Jun 4, 2025
18 checks passed

elasticsearchmachine added the backport pending label Jun 4, 2025

Samiul-TheSoccerFan mentioned this pull request Jun 4, 2025

[8.19] Semantic_text match_all with Highlighter (#128702) #128903

Merged

		@@ -0,0 +1,5 @@
		pr: 128702
		summary: Highlighting support added to semantic_text with `match_all`

Semantic_text match_all with Highlighter #128702

Semantic_text match_all with Highlighter #128702

Conversation

Samiul-TheSoccerFan commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test cases:

Only text fields

only inference fields

Both infer and non-inference fields

Both dense vector fields

Some additional cases

Uh oh!

kderusso left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Mikep86 left a comment

Choose a reason for hiding this comment

Uh oh!

Samiul-TheSoccerFan commented Jun 3, 2025

Uh oh!

elasticsearchmachine commented Jun 3, 2025

Uh oh!

elasticsearchmachine commented Jun 3, 2025

Uh oh!

elasticsearchmachine commented Jun 3, 2025

Uh oh!

kderusso left a comment

Choose a reason for hiding this comment

Uh oh!

kderusso Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

kderusso Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

Samiul-TheSoccerFan commented Jun 3, 2025

Uh oh!

Mikep86 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Mikep86 left a comment

Choose a reason for hiding this comment

Uh oh!

Samiul-TheSoccerFan commented Jun 3, 2025

Uh oh!

Samiul-TheSoccerFan commented Jun 4, 2025

Uh oh!

Uh oh!

elasticsearchmachine commented Jun 4, 2025

💔 Backport failed

Uh oh!

Samiul-TheSoccerFan commented Jun 4, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Uh oh!

Samiul-TheSoccerFan commented May 30, 2025 •

edited

Loading