Skip to content

Commit 324c3cd

Browse files
meili-bors[bot]Strift
meili-bors[bot]
and
Strift
authored
Merge #827
827: Add AI-powered search features r=brunoocasali a=Strift # Pull Request ## Related issue Fixes #817 ## What does this PR do? ### Update settings to handle embedders Docs: https://www.meilisearch.com/docs/reference/api/settings#embedders Update the methods `getEmbedders`, `updateEmbedders`, `resetEmbedders`. Also, the method `updateSettings` should be able to accept the new `embedders` parameter. Here is the list of fields in the `embedders` object: - [x] `source` sub field is available and accepts: `ollama`, `rest`, `openAI`, `huggingFace` and `userProvided` - [x] `apiKey` sub field is available (string) - optional because not compatible with all sources. Only for `openAi`, `ollama`, `rest`. - [x] `model` sub field is available (string) - optional because not compatible with all sources. Only for `ollama`, `openAI`, `huggingFace` - [x] `documentTemplate` sub field is available (string) - optional - [x] `dimensions` - optional because not compatible with all sources. Only for `openAi`, `huggingFace`, `ollama`, and `rest` - [x] `distribution` - optional - [x] `request` - mandatory only if using `rest` embedder - [x] `response` - mandatory only if using `rest` embedder - [x] `documentTemplateMaxBytes` - optional - [x] `revision` - optional, only for `huggingFace` - [x] `headers` - optional, only for `rest` - [x] `binaryQuantized` - optional ### Update search to handle vector search and hybrid search Docs: https://www.meilisearch.com/docs/reference/api/search Update the `search` method:: - [x] `hybrid` search parameter, with sub fields `semanticRatio` and `embedder`. `embedder` is mandatory if `hybrid` is set. - [x] `vector` parameter is available - [x] `retrieveVectors` parameter available - [ ] ~~`semanticHitCount` in search response~~ - [ ] ~~Accept `_semanticScore` in the search response (optional)~~ - [ ] ~~`vector` should be returned in the search response, but optional (because depends on search parameters)~~ - [x] `_vectors` present in the search response, but optional ### Add similar documents endpoint Docs: https://www.meilisearch.com/docs/reference/api/similar - [x] Implement `searchSimilarDocuments` associated with the `POST /indexes/:uid/similar`. Do NOT implement with `GET`. ## PR checklist Please check if your PR fulfills the following requirements: - [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)? - [x] Have you read the contributing guidelines? - [x] Have you made sure that the title is accurate and descriptive of the changes? Thank you so much for contributing to Meilisearch! <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added support for managing embedders settings, including retrieving, updating, and resetting embedders on indexes. - Introduced semantic similarity search for documents. - Added hybrid search configuration and vector-based search with options to retrieve vector data in search results. - Introduced new models for embedder configuration, embedder distribution, and hybrid search. - **Bug Fixes** - Improved JSON serialization to omit optional fields with null values in search requests and similar document requests. - **Tests** - Added comprehensive tests for embedders settings, semantic similarity search, hybrid search, vector search, and serialization of new features. - **Chores** - Enhanced test workflow logging for better visibility during CI runs. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Strift <lau.cazanove@gmail.com> Co-authored-by: Laurent Cazanove <lau.cazanove@gmail.com>
2 parents 5e0caf7 + b91b92b commit 324c3cd

17 files changed

+629
-52
lines changed

‎.github/workflows/tests.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ jobs:
2929
- name: Meilisearch (latest) setup with Docker
3030
run: docker run -d -p 7700:7700 getmeili/meilisearch:latest meilisearch --master-key=masterKey --no-analytics
3131
- name: Build and run unit and integration tests
32-
run: ./gradlew build integrationTest
32+
run: ./gradlew build integrationTest --info
3333
- name: Archive test report
3434
uses: actions/upload-artifact@v4
3535
if: failure()

‎src/main/java/com/meilisearch/sdk/Index.java

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1242,11 +1242,57 @@ public TaskInfo resetSearchCutoffMsSettings() throws MeilisearchException {
12421242
return this.settingsHandler.resetSearchCutoffMsSettings(this.uid);
12431243
}
12441244

1245+
/**
1246+
* Retrieves documents that are semantically similar to a given document
1247+
*
1248+
* @param query SimilarDocumentRequest containing parameters for the similar documents search
1249+
* @return SimilarDocumentsResults containing the search results
1250+
* @throws MeilisearchException if an error occurs
1251+
* @see <a href="https://www.meilisearch.com/docs/reference/api/similar">API specification</a>
1252+
*/
12451253
public SimilarDocumentsResults searchSimilarDocuments(SimilarDocumentRequest query)
12461254
throws MeilisearchException {
12471255
return this.config.httpClient.post(
12481256
new URLBuilder("/indexes").addSubroute(this.uid).addSubroute("/similar").getURL(),
12491257
query,
12501258
SimilarDocumentsResults.class);
12511259
}
1260+
1261+
/**
1262+
* Gets the embedders settings of the index
1263+
*
1264+
* @return a Map that contains all embedders settings
1265+
* @throws MeilisearchException if an error occurs
1266+
* @see <a href="https://www.meilisearch.com/docs/reference/api/settings#get-embedders">API
1267+
* specification</a>
1268+
*/
1269+
public Map<String, Embedder> getEmbeddersSettings() throws MeilisearchException {
1270+
return this.settingsHandler.getEmbedders(this.uid);
1271+
}
1272+
1273+
/**
1274+
* Updates the embedders settings of the index
1275+
*
1276+
* @param embedders a Map that contains the new embedders settings
1277+
* @return TaskInfo instance
1278+
* @throws MeilisearchException if an error occurs
1279+
* @see <a href="https://www.meilisearch.com/docs/reference/api/settings#update-embedders">API
1280+
* specification</a>
1281+
*/
1282+
public TaskInfo updateEmbeddersSettings(Map<String, Embedder> embedders)
1283+
throws MeilisearchException {
1284+
return this.settingsHandler.updateEmbedders(this.uid, embedders);
1285+
}
1286+
1287+
/**
1288+
* Resets the embedders settings of the index
1289+
*
1290+
* @return TaskInfo instance
1291+
* @throws MeilisearchException if an error occurs
1292+
* @see <a href="https://www.meilisearch.com/docs/reference/api/settings#reset-embedders">API
1293+
* specification</a>
1294+
*/
1295+
public TaskInfo resetEmbeddersSettings() throws MeilisearchException {
1296+
return this.settingsHandler.resetEmbedders(this.uid);
1297+
}
12521298
}

‎src/main/java/com/meilisearch/sdk/IndexSearchRequest.java

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,13 +38,14 @@ public class IndexSearchRequest {
3838
private FederationOptions federationOptions;
3939
protected String[] locales;
4040
protected String distinct;
41+
protected Boolean retrieveVectors;
4142

4243
/**
4344
* Constructor for MultiSearchRequest for building search queries with the default values:
4445
* offset: 0, limit: 20, attributesToRetrieve: ["*"], attributesToCrop: null, cropLength: 200,
4546
* attributesToHighlight: null, filter: null, showMatchesPosition: false, facets: null, sort:
4647
* null, showRankingScore: false, showRankingScoreDetails: false, rankingScoreThreshold: null
47-
* distinct: null
48+
* distinct: null, retrieveVectors: false
4849
*
4950
* @param indexUid uid of the requested index String
5051
*/
@@ -106,7 +107,8 @@ public String toString() {
106107
.putOpt("rankingScoreThreshold", this.rankingScoreThreshold)
107108
.putOpt("attributesToSearchOn", this.attributesToSearchOn)
108109
.putOpt("locales", this.locales)
109-
.putOpt("distinct", this.distinct);
110+
.putOpt("distinct", this.distinct)
111+
.putOpt("retrieveVectors", this.retrieveVectors);
110112

111113
return jsonObject.toString();
112114
}

‎src/main/java/com/meilisearch/sdk/SearchRequest.java

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
package com.meilisearch.sdk;
22

3+
import com.meilisearch.sdk.model.Hybrid;
34
import com.meilisearch.sdk.model.MatchingStrategy;
45
import lombok.AccessLevel;
56
import lombok.AllArgsConstructor;
@@ -42,12 +43,15 @@ public class SearchRequest {
4243
protected Double rankingScoreThreshold;
4344
protected String[] locales;
4445
protected String distinct;
45-
46+
protected Hybrid hybrid;
47+
protected Double[] vector;
48+
protected Boolean retrieveVectors;
4649
/**
4750
* Constructor for SearchRequest for building search queries with the default values: offset: 0,
4851
* limit: 20, attributesToRetrieve: ["*"], attributesToCrop: null, cropLength: 200,
4952
* attributesToHighlight: null, filter: null, showMatchesPosition: false, facets: null, sort:
50-
* null, showRankingScore: false, showRankingScoreDetails: false, rankingScoreThreshold: null
53+
* null, showRankingScore: false, showRankingScoreDetails: false, rankingScoreThreshold: null,
54+
* retrieveVectors: false
5155
*
5256
* @param q Query String
5357
*/
@@ -104,7 +108,13 @@ public String toString() {
104108
.putOpt("showRankingScoreDetails", this.showRankingScoreDetails)
105109
.putOpt("rankingScoreThreshold", this.rankingScoreThreshold)
106110
.putOpt("locales", this.locales)
107-
.putOpt("distinct", this.distinct);
111+
.putOpt("distinct", this.distinct)
112+
.putOpt("vector", this.vector)
113+
.putOpt("retrieveVectors", this.retrieveVectors);
114+
115+
if (this.hybrid != null) {
116+
jsonObject.put("hybrid", this.hybrid.toJSONObject());
117+
}
108118

109119
return jsonObject.toString();
110120
}

‎src/main/java/com/meilisearch/sdk/SettingsHandler.java

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
import com.meilisearch.sdk.exceptions.MeilisearchException;
44
import com.meilisearch.sdk.http.URLBuilder;
5+
import com.meilisearch.sdk.model.Embedder;
56
import com.meilisearch.sdk.model.Faceting;
67
import com.meilisearch.sdk.model.LocalizedAttribute;
78
import com.meilisearch.sdk.model.Pagination;
@@ -769,4 +770,47 @@ public TaskInfo resetNonSeparatorTokensSettings(String uid) {
769770
return httpClient.delete(
770771
settingsPath(uid).addSubroute("non-separator-tokens").getURL(), TaskInfo.class);
771772
}
773+
774+
/**
775+
* Gets the embedders settings of the index
776+
*
777+
* @param uid Index identifier
778+
* @return a Map that contains all embedders settings
779+
* @throws MeilisearchException if an error occurs
780+
*/
781+
Map<String, Embedder> getEmbedders(String uid) throws MeilisearchException {
782+
return httpClient.get(
783+
settingsPath(uid).addSubroute("embedders").getURL(),
784+
Map.class,
785+
String.class,
786+
Embedder.class);
787+
}
788+
789+
/**
790+
* Updates the embedders settings of the index
791+
*
792+
* @param uid Index identifier
793+
* @param embedders a Map that contains the new embedders settings
794+
* @return TaskInfo instance
795+
* @throws MeilisearchException if an error occurs
796+
*/
797+
TaskInfo updateEmbedders(String uid, Map<String, Embedder> embedders)
798+
throws MeilisearchException {
799+
return httpClient.patch(
800+
settingsPath(uid).addSubroute("embedders").getURL(),
801+
embedders == null ? null : httpClient.jsonHandler.encode(embedders),
802+
TaskInfo.class);
803+
}
804+
805+
/**
806+
* Resets the embedders settings of the index
807+
*
808+
* @param uid Index identifier
809+
* @return TaskInfo instance
810+
* @throws MeilisearchException if an error occurs
811+
*/
812+
TaskInfo resetEmbedders(String uid) throws MeilisearchException {
813+
return httpClient.delete(
814+
settingsPath(uid).addSubroute("embedders").getURL(), TaskInfo.class);
815+
}
772816
}

‎src/main/java/com/meilisearch/sdk/SimilarDocumentRequest.java

Lines changed: 12 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -21,28 +21,22 @@ public class SimilarDocumentRequest {
2121
private Double rankingScoreThreshold;
2222
private Boolean retrieveVectors;
2323

24-
/**
25-
* Constructor for SimilarDocumentsRequest for building search request for similar documents
26-
* with the default values: id null, embedder "default", attributesToRetrieve ["*"], offset 0,
27-
* limit 20, filter null, showRankingScore false, showRankingScoreDetails false,
28-
* rankingScoreThreshold null, retrieveVectors false
29-
*/
24+
/** Constructor for SimilarDocumentsRequest for building search request for similar documents */
3025
public SimilarDocumentRequest() {}
3126

3227
@Override
3328
public String toString() {
34-
JSONObject jsonObject =
35-
new JSONObject()
36-
.put("id", this.id)
37-
.put("embedder", this.embedder)
38-
.put("attributesToRetrieve", this.attributesToRetrieve)
39-
.put("offset", this.offset)
40-
.put("limit", this.limit)
41-
.put("filter", this.filter)
42-
.put("showRankingScore", this.showRankingScore)
43-
.put("showRankingScoreDetails", this.showRankingScoreDetails)
44-
.put("rankingScoreThreshold", this.rankingScoreThreshold)
45-
.put("retrieveVectors", this.retrieveVectors);
29+
JSONObject jsonObject = new JSONObject();
30+
jsonObject.put("id", this.id);
31+
jsonObject.put("embedder", this.embedder);
32+
jsonObject.putOpt("attributesToRetrieve", this.attributesToRetrieve);
33+
jsonObject.putOpt("offset", this.offset);
34+
jsonObject.putOpt("limit", this.limit);
35+
jsonObject.putOpt("filter", this.filter);
36+
jsonObject.putOpt("showRankingScore", this.showRankingScore);
37+
jsonObject.putOpt("showRankingScoreDetails", this.showRankingScoreDetails);
38+
jsonObject.putOpt("rankingScoreThreshold", this.rankingScoreThreshold);
39+
jsonObject.putOpt("retrieveVectors", this.retrieveVectors);
4640

4741
return jsonObject.toString();
4842
}

‎src/main/java/com/meilisearch/sdk/model/Embedder.java

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
package com.meilisearch.sdk.model;
2+
3+
import com.fasterxml.jackson.annotation.JsonInclude;
4+
import java.util.Map;
5+
import lombok.*;
6+
import lombok.experimental.Accessors;
7+
8+
@Builder
9+
@AllArgsConstructor(access = AccessLevel.PACKAGE)
10+
@Getter
11+
@Setter
12+
@Accessors(chain = true)
13+
@JsonInclude(JsonInclude.Include.NON_NULL)
14+
public class Embedder {
15+
/** Source of the embedder. Accepts: ollama, rest, openAI, huggingFace and userProvided */
16+
protected EmbedderSource source;
17+
18+
/**
19+
* API key for authentication with the embedder service. Optional: Only applicable for openAi,
20+
* ollama, and rest sources.
21+
*/
22+
protected String apiKey;
23+
24+
/**
25+
* Model to use for generating embeddings. Optional: Only applicable for ollama, openAI, and
26+
* huggingFace sources.
27+
*/
28+
protected String model;
29+
30+
/** Template for document embedding. Optional. */
31+
protected String documentTemplate;
32+
33+
/**
34+
* Dimensions of the embedding vectors. Optional: Only applicable for openAi, huggingFace,
35+
* ollama, and rest sources.
36+
*/
37+
protected Integer dimensions;
38+
39+
/** Distribution configuration. Optional. */
40+
protected EmbedderDistribution distribution;
41+
42+
/** Request configuration. Mandatory only when using rest embedder, optional otherwise. */
43+
protected Map<String, Object> request;
44+
45+
/** Response configuration. Mandatory only when using rest embedder, optional otherwise. */
46+
protected Map<String, Object> response;
47+
48+
/** Maximum bytes for document template. Optional. */
49+
protected Integer documentTemplateMaxBytes;
50+
51+
/** Revision identifier. Optional: Only applicable for huggingFace. */
52+
protected String revision;
53+
54+
/** HTTP headers. Optional: Only applicable for rest. */
55+
protected Map<String, String> headers;
56+
57+
/** Whether to use binary quantization. Optional. */
58+
protected Boolean binaryQuantized;
59+
60+
/** URL for the embedder service. Optional. */
61+
protected String url;
62+
63+
/** Input fields for the embedder. Optional. */
64+
protected String[] inputField;
65+
66+
/** Type of input for the embedder. Optional. */
67+
protected EmbedderInputType inputType;
68+
69+
/** Query for the embedder. Optional. */
70+
protected String query;
71+
72+
public Embedder() {}
73+
}

‎src/main/java/com/meilisearch/sdk/model/EmbedderDistribution.java

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
package com.meilisearch.sdk.model;
2+
3+
import com.fasterxml.jackson.annotation.JsonInclude;
4+
import lombok.AccessLevel;
5+
import lombok.AllArgsConstructor;
6+
import lombok.Builder;
7+
import lombok.Getter;
8+
import lombok.NoArgsConstructor;
9+
import lombok.Setter;
10+
import lombok.experimental.Accessors;
11+
12+
/**
13+
* Describes the natural distribution of search results for embedders. Contains mean and sigma
14+
* values, each between 0 and 1.
15+
*/
16+
@Builder
17+
@AllArgsConstructor(access = AccessLevel.PACKAGE)
18+
@NoArgsConstructor(access = AccessLevel.PUBLIC)
19+
@Getter
20+
@Setter
21+
@Accessors(chain = true)
22+
@JsonInclude(JsonInclude.Include.NON_NULL)
23+
public class EmbedderDistribution {
24+
/** Mean value of the distribution, between 0 and 1 */
25+
private Double mean;
26+
27+
/** Sigma (standard deviation) value of the distribution, between 0 and 1 */
28+
private Double sigma;
29+
30+
/**
31+
* Creates a uniform distribution with default values
32+
*
33+
* @return An EmbedderDistribution instance with mean=0.5 and sigma=0.5
34+
*/
35+
public static EmbedderDistribution uniform() {
36+
return new EmbedderDistribution().setMean(0.5).setSigma(0.5);
37+
}
38+
39+
/**
40+
* Creates a custom distribution with specified mean and sigma values
41+
*
42+
* @param mean Mean value between 0 and 1
43+
* @param sigma Sigma value between 0 and 1
44+
* @return An EmbedderDistribution instance with the specified values
45+
* @throws IllegalArgumentException if mean or sigma are outside the valid range
46+
*/
47+
public static EmbedderDistribution custom(double mean, double sigma) {
48+
if (mean < 0 || mean > 1) {
49+
throw new IllegalArgumentException("Mean must be between 0 and 1");
50+
}
51+
if (sigma < 0 || sigma > 1) {
52+
throw new IllegalArgumentException("Sigma must be between 0 and 1");
53+
}
54+
return new EmbedderDistribution().setMean(mean).setSigma(sigma);
55+
}
56+
}

‎src/main/java/com/meilisearch/sdk/model/Embedders.java

Lines changed: 0 additions & 24 deletions
This file was deleted.

0 commit comments

Comments
 (0)