Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.Net: Added implementation of SQLite connector for new memory design #9164

Merged
merged 128 commits into from
Oct 17, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
128 commits
Select commit Hold shift + click to select a range
d7287bb
.Net: ADR for Text Search Abstractions (#8307)
markwallace-microsoft Aug 22, 2024
590745f
.Net: Add TextSearchExtension methods to create KernelPlugins and Ker…
markwallace-microsoft Aug 27, 2024
3c09d91
.Net: Update Bing Search to new Text Search Design (#8343)
markwallace-microsoft Aug 29, 2024
187afca
Merge branch 'main' into feature-vector-search
markwallace-microsoft Aug 29, 2024
16f4fa1
.Net: Update Google Search to new Text Search Design (#8394)
markwallace-microsoft Aug 29, 2024
5af6540
.Net: Add text search concepts demonstrating RAG and Function Calling…
markwallace-microsoft Aug 30, 2024
6d13254
Merge branch 'main' into feature-vector-search
markwallace-microsoft Aug 30, 2024
f806fc2
.Net: Simplify how additional text search filter parameters are speci…
markwallace-microsoft Sep 2, 2024
890c0f8
Merge branch 'main' into feature-vector-search
markwallace-microsoft Sep 2, 2024
cfbac04
Merge branch 'main' into feature-vector-search
markwallace-microsoft Sep 3, 2024
b96d607
Merge branch 'main' into feature-vector-search
markwallace-microsoft Sep 3, 2024
4b8406e
.Net: Add Vector Search ADR document and abstractions (#8494)
westey-m Sep 4, 2024
d02005a
.Net: Add support for advanced search keywords (#8489)
markwallace-microsoft Sep 4, 2024
ac540db
.Net: Use interfaces for mapper abstractions (#8424)
markwallace-microsoft Sep 4, 2024
d745d57
.Net: Add vector search implementation for azure ai search (#8507)
westey-m Sep 5, 2024
7836721
.Net: Add qdrant vector search implementation. (#8508)
westey-m Sep 5, 2024
ec10424
.Net: Add Redis vector search implementation (#8510)
westey-m Sep 5, 2024
d43c5ff
.Net: Unify text and vector search and move all to data namespace. (#…
westey-m Sep 6, 2024
70a98ee
.Net: Add volatile vector search and enforcing collection type. (#8546)
westey-m Sep 6, 2024
ffaa93f
Update adr document with filter changes. (#8590)
westey-m Sep 9, 2024
639694b
Resolve merge conflicts
markwallace-microsoft Sep 11, 2024
c1cd203
Fix formatting
markwallace-microsoft Sep 11, 2024
7bdf519
Merge branch 'main' into feature-vector-search
westey-m Sep 12, 2024
6b92a18
Merge branch 'main' into feature-vector-search
markwallace-microsoft Sep 13, 2024
1fc7e91
.Net: Switch to using interfaces for search instead of query objects.…
westey-m Sep 13, 2024
9c8e438
Merge branch 'main' into feature-vector-search
markwallace-microsoft Sep 13, 2024
031a653
Merge branch 'main' into feature-vector-search
westey-m Sep 17, 2024
971afd6
.Net: Suppress moq warning after taking package updates. (#8838)
westey-m Sep 17, 2024
bf34a51
.Net: Basic implementation of ITextSearch for an IVectorStore impleme…
markwallace-microsoft Sep 17, 2024
434498a
.Net: Add float64 support for redis vector search. (#8847)
westey-m Sep 17, 2024
092a9ac
Merge branch 'main' into feature-vector-search
dmytrostruk Sep 18, 2024
f35d051
.Net: Enhance volatile memory connector to allow collection to be ser…
markwallace-microsoft Sep 18, 2024
8dc7dc3
.Net: Added vector search implementation for Azure CosmosDB for Mongo…
dmytrostruk Sep 18, 2024
fa461ce
.Net: Vector Search Bug fixes (#8890)
westey-m Sep 18, 2024
be531dd
.Net: [Feature branch] Added support for Filter and Offset parameters…
dmytrostruk Sep 19, 2024
36aa3f2
.Net: Vector search sample: Multi Vector and Paging (#8920)
westey-m Sep 20, 2024
8fa1e8e
.Net: Adding a multi-store vector search sample. (#8909)
westey-m Sep 20, 2024
b5c649f
.Net: [Feature Branch] Added vector search implementation for Azure C…
dmytrostruk Sep 20, 2024
952d679
Merge branch 'main' into feature-vector-search
markwallace-microsoft Sep 20, 2024
6c4aa5d
.Net: Add VectorStoreTextSearch concepts and unit tests (#8891)
markwallace-microsoft Sep 23, 2024
cb00332
Merge branch 'main' into feature-vector-search
westey-m Sep 23, 2024
21f8a27
.Net: [Feature Branch] Added vector search implementation for Weaviat…
dmytrostruk Sep 23, 2024
d6b6019
.Net: Add filtering support for VectoreStoreTextSearch (#8947)
markwallace-microsoft Sep 24, 2024
1c35c01
Merge branch 'main' into feature-vector-search
markwallace-microsoft Sep 24, 2024
6d041b3
.Net: Adding generic data type support to the volatile vector store. …
westey-m Sep 24, 2024
16f787e
.Net: Rename a number of properties as agreed and remove dangerous de…
westey-m Sep 25, 2024
12a85d5
Merge branch 'main' into feature-vector-search
dmytrostruk Sep 26, 2024
66d608c
.Net: Add delegates for the string and result mappers (#9000)
markwallace-microsoft Sep 26, 2024
d08eef3
.Net: Integration tests for Bing and Google text search (#8991)
markwallace-microsoft Sep 27, 2024
21e37a0
.Net: Add DI registration helpers for collections and search (#9007)
westey-m Sep 27, 2024
a6dc6b3
Merge branch 'main' into feature-vector-search
dmytrostruk Sep 30, 2024
2758ff7
Fix after merge
dmytrostruk Sep 30, 2024
da7291b
Implemented database manipulation methods
dmytrostruk Oct 1, 2024
667533e
Merge branch 'main' into feature-vector-search
markwallace-microsoft Oct 1, 2024
7975fed
Merge branch 'main' into feature-vector-search
markwallace-microsoft Oct 1, 2024
6931b79
Merge branch 'main' into feature-vector-search
markwallace-microsoft Oct 1, 2024
5bce499
.Net: Fix issues after vector split merge. (#9050)
westey-m Oct 1, 2024
1046bd3
Merge branch 'feature-vector-search' into sqlite-connector
dmytrostruk Oct 1, 2024
467191e
.Net: Support DI for Text Search Services (#9026)
markwallace-microsoft Oct 1, 2024
a608659
Fixed compilation errors after merge
dmytrostruk Oct 1, 2024
136fc0f
Added creation of virtual table for vectors
dmytrostruk Oct 1, 2024
d6ec1ee
Added deletion of virtual table for vectors
dmytrostruk Oct 1, 2024
2b94b08
.Net: Adapt In-Memory Connector to new Text Search Design (#9046)
markwallace-microsoft Oct 2, 2024
7710e36
Merge branch 'main' into feature-vector-search
markwallace-microsoft Oct 2, 2024
e2e22b6
.Net: Vector search recordreader manual merge (#9059)
westey-m Oct 2, 2024
6b05639
Implemented default mapper
dmytrostruk Oct 2, 2024
c4f3c66
Merge branch 'feature-vector-search' into sqlite-connector
dmytrostruk Oct 2, 2024
3b02888
Implemented Get and Upsert operations
dmytrostruk Oct 3, 2024
d8eb4b5
.Net: Adapt Azure AI Search Connector to new Text Search Design (#9061)
markwallace-microsoft Oct 3, 2024
d98e218
Implemented Delete record operations. Fixed batch operations.
dmytrostruk Oct 3, 2024
098ba75
Exposed virtual table name configuration
dmytrostruk Oct 3, 2024
e668013
Added test for string keys
dmytrostruk Oct 3, 2024
0179733
Added distance function mapping
dmytrostruk Oct 3, 2024
cc8fe45
.Net: Add response container type for vector search results. (#9082)
westey-m Oct 4, 2024
fba0aba
.Net: Use AsynEnumerable search interface for azure ai search. (#9090)
westey-m Oct 4, 2024
c198b09
Vector search and code refactoring
dmytrostruk Oct 4, 2024
d09e70f
Added offset logic
dmytrostruk Oct 5, 2024
03b89fe
Added filter usage
dmytrostruk Oct 5, 2024
69b9fc4
.Net: Update Qdrant Memory Connector to new Text Search Design (#9076)
markwallace-microsoft Oct 7, 2024
6d32d34
.Net: Add attributes to add to a model which can be converted to a Te…
markwallace-microsoft Oct 7, 2024
805febe
.Net: Change signature of the TextSearchResult constructor (#9153)
markwallace-microsoft Oct 8, 2024
add4461
Merge branch 'feature-vector-search' into sqlite-connector
dmytrostruk Oct 8, 2024
ea335bc
Merge from feature branch
dmytrostruk Oct 8, 2024
22e24a5
Merge branch 'main' into feature-vector-search
dmytrostruk Oct 8, 2024
db95435
Merge branch 'feature-vector-search' into sqlite-connector
dmytrostruk Oct 8, 2024
2d88cfb
Added support for generic data model
dmytrostruk Oct 8, 2024
bb2b457
Added vector store class
dmytrostruk Oct 9, 2024
806829d
Added extension methods
dmytrostruk Oct 9, 2024
4625afb
Small fix
dmytrostruk Oct 9, 2024
46ec215
Small fix
dmytrostruk Oct 9, 2024
b85b0e7
Fixed warnings
dmytrostruk Oct 9, 2024
a8af704
Fixed warning
dmytrostruk Oct 9, 2024
f29e110
Merge branch 'main' into sqlite-connector
dmytrostruk Oct 9, 2024
0dec618
Fix after merge
dmytrostruk Oct 9, 2024
59e9bba
Fixed usings
dmytrostruk Oct 9, 2024
21cf1ac
Added unit tests
dmytrostruk Oct 9, 2024
4687d06
More unit tests
dmytrostruk Oct 9, 2024
52c8a4c
Merge branch 'main' into sqlite-connector
dmytrostruk Oct 10, 2024
7da8523
Merge branch 'main' into sqlite-connector
dmytrostruk Oct 11, 2024
22c9491
Fixes after merge
dmytrostruk Oct 11, 2024
270e9e1
Moved Sqlite unit test project
dmytrostruk Oct 11, 2024
f7cbaf2
Addressed PR feedback
dmytrostruk Oct 11, 2024
9094eaa
Added more comments
dmytrostruk Oct 11, 2024
aa4b4b6
More improvements and fixes
dmytrostruk Oct 11, 2024
79dac79
Merge branch 'main' into sqlite-connector
dmytrostruk Oct 11, 2024
77fe3c2
Updated mapping and fixed tests
dmytrostruk Oct 14, 2024
4ecf78c
Fixed usings
dmytrostruk Oct 14, 2024
a34fc23
Added unit tests for command builder
dmytrostruk Oct 14, 2024
ecf6de9
Added unit tests for record collection class
dmytrostruk Oct 14, 2024
43900fd
Added integration test to get existing record
dmytrostruk Oct 14, 2024
7453e21
Fixed warnings
dmytrostruk Oct 14, 2024
3a797fb
Updated extension methods
dmytrostruk Oct 14, 2024
1b75ccd
Merge branch 'main' into sqlite-connector
dmytrostruk Oct 15, 2024
44e366f
Fixes based on merge from main
dmytrostruk Oct 15, 2024
cbd2902
Added more tests
dmytrostruk Oct 16, 2024
5048f9f
Small fix
dmytrostruk Oct 16, 2024
4458478
Small refactoring
dmytrostruk Oct 16, 2024
c4df0c6
Addressed PR feedback
dmytrostruk Oct 16, 2024
c55e350
Added more tests for collection class
dmytrostruk Oct 16, 2024
d8e72b3
Added unit tests for default record mapper
dmytrostruk Oct 16, 2024
53e3c5a
Added more unit tests
dmytrostruk Oct 16, 2024
9884082
Merge branch 'main' into sqlite-connector
dmytrostruk Oct 16, 2024
e10ccf7
Merge branch 'main' into sqlite-connector
dmytrostruk Oct 17, 2024
a335a61
Addressed PR feedback
dmytrostruk Oct 17, 2024
6510a66
Updated SQLite connector package version suffix
dmytrostruk Oct 17, 2024
c79b8c6
Merge branch 'main' into sqlite-connector
dmytrostruk Oct 17, 2024
5b5399e
Removed kernel builder extensions
dmytrostruk Oct 17, 2024
033ca59
Merge branch 'main' into sqlite-connector
dmytrostruk Oct 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
.Net: Use interfaces for mapper abstractions (#8424)
### Motivation and Context

Closes #8390 

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
  • Loading branch information
markwallace-microsoft authored Sep 4, 2024
commit ac540dba1e56335e9f19ce8245feb5481f0ee957
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ public async Task FunctionCallingWithBingTextSearchUsingSiteArgumentAsync()
Console.WriteLine(await kernel.InvokePromptAsync("What is the Semantic Kernel? Only include results from techcommunity.microsoft.com. Include citations to the relevant information where it is referenced in the response.", arguments));
}

private static KernelFunction CreateSearchBySite(BingTextSearch textSearch, BasicFilterOptions? basicFilter = null, MapSearchResultToString? mapToString = null)
private static KernelFunction CreateSearchBySite(BingTextSearch textSearch, BasicFilterOptions? basicFilter = null)
{
var options = new KernelFunctionFromMethodOptions()
{
Expand Down
16 changes: 14 additions & 2 deletions dotnet/samples/Concepts/Search/Bing_TextSearch.cs
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ public async Task UsingBingTextSearchWithACustomMapperAsync()
var textSearch = new BingTextSearch(apiKey: TestConfiguration.Bing.ApiKey, options: new()
{
HttpClient = httpClient,
MapToString = webPage => JsonSerializer.Serialize(webPage),
StringMapper = new TestTextSearchStringMapper(),
});

var query = "What is the Semantic Kernel?";
Expand Down Expand Up @@ -103,7 +103,7 @@ public async Task UsingBingTextSearchWithASiteFilterAsync()
var textSearch = new BingTextSearch(apiKey: TestConfiguration.Bing.ApiKey, options: new()
{
HttpClient = httpClient,
MapToString = webPage => JsonSerializer.Serialize(webPage),
StringMapper = new TestTextSearchStringMapper(),
});

var query = "What is the Semantic Kernel?";
Expand All @@ -121,5 +121,17 @@ public async Task UsingBingTextSearchWithASiteFilterAsync()

#region private
private const int HorizontalRuleLength = 80;

/// <summary>
/// Test mapper which converts an arbitrary search result to a string using JSON serialization.
/// </summary>
private sealed class TestTextSearchStringMapper : ITextSearchStringMapper
{
/// <inheritdoc />
public string MapFromResultToString(object result)
{
return JsonSerializer.Serialize(result);
}
}
#endregion
}
14 changes: 13 additions & 1 deletion dotnet/samples/Concepts/Search/Google_TextSearch.cs
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ public async Task UsingGoogleTextSearchWithACustomMapperAsync()
var textSearch = new GoogleTextSearch(
searchEngineId: TestConfiguration.Google.SearchEngineId,
apiKey: TestConfiguration.Google.ApiKey,
options: new() { MapToString = webPage => JsonSerializer.Serialize(webPage) });
options: new() { StringMapper = new TestTextSearchStringMapper() });

var query = "What is the Semantic Kernel?";

Expand Down Expand Up @@ -110,6 +110,18 @@ public async Task UsingGoogleTextSearchWithASiteSearchFilterAsync()
#region private
private const int HorizontalRuleLength = 80;

/// <summary>
/// Test mapper which converts an arbitrary search result to a string using JSON serialization.
/// </summary>
private sealed class TestTextSearchStringMapper : ITextSearchStringMapper
{
/// <inheritdoc />
public string MapFromResultToString(object result)
{
return JsonSerializer.Serialize(result);
}
}

/// <summary>
/// Implementation of <see cref="ConfigurableMessageHandler"/> which logs HTTP responses.
/// </summary>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -104,13 +104,13 @@ public async Task GetSearchResultsReturnsSuccessfullyAsync()
}

[Fact]
public async Task SearchWithCustomMapperReturnsSuccessfullyAsync()
public async Task SearchWithCustomStringMapperReturnsSuccessfullyAsync()
{
// Arrange
this._messageHandlerStub.AddJsonResponse(File.ReadAllText(WhatIsTheSKResponseJson));

// Create an ITextSearch instance using Bing search
var textSearch = new BingTextSearch(apiKey: "ApiKey", options: new() { HttpClient = this._httpClient, MapToString = webPage => JsonSerializer.Serialize(webPage) });
var textSearch = new BingTextSearch(apiKey: "ApiKey", options: new() { HttpClient = this._httpClient, StringMapper = new TestTextSearchStringMapper() });

// Act
KernelSearchResults<string> result = await textSearch.SearchAsync("What is the Semantic Kernel?", new() { Count = 10, Offset = 0 });
Expand All @@ -129,6 +129,33 @@ public async Task SearchWithCustomMapperReturnsSuccessfullyAsync()
}
}

[Fact]
public async Task GetTextSearchResultsWithCustomResultMapperReturnsSuccessfullyAsync()
{
// Arrange
this._messageHandlerStub.AddJsonResponse(File.ReadAllText(WhatIsTheSKResponseJson));

// Create an ITextSearch instance using Bing search
var textSearch = new BingTextSearch(apiKey: "ApiKey", options: new() { HttpClient = this._httpClient, ResultMapper = new TestTextSearchResultMapper() });

// Act
KernelSearchResults<TextSearchResult> result = await textSearch.GetTextSearchResultsAsync("What is the Semantic Kernel?", new() { Count = 10, Offset = 0 });

// Assert
Assert.NotNull(result);
Assert.NotNull(result.Results);
var resultList = await result.Results.ToListAsync();
Assert.NotNull(resultList);
Assert.Equal(10, resultList.Count);
foreach (var textSearchResult in resultList)
{
Assert.NotNull(textSearchResult);
Assert.Equal(textSearchResult.Name, textSearchResult.Name?.ToUpperInvariant());
Assert.Equal(textSearchResult.Value, textSearchResult.Value?.ToUpperInvariant());
Assert.Equal(textSearchResult.Link, textSearchResult.Link?.ToUpperInvariant());
}
}

[Theory]
[InlineData("answerCount", 5, "https://api.bing.microsoft.com/v7.0/search?q=What%20is%20the%20Semantic%20Kernel%3F&count=4&offset=0&answerCount=5")]
[InlineData("cc", "AR", "https://api.bing.microsoft.com/v7.0/search?q=What%20is%20the%20Semantic%20Kernel%3F&count=4&offset=0&cc=AR")]
Expand Down Expand Up @@ -205,5 +232,39 @@ public void Dispose()
private readonly MultipleHttpMessageHandlerStub _messageHandlerStub;
private readonly HttpClient _httpClient;
private readonly Kernel _kernel;

/// <summary>
/// Test mapper which converts a BingWebPage search result to a string using JSON serialization.
/// </summary>
private sealed class TestTextSearchStringMapper : ITextSearchStringMapper
{
/// <inheritdoc />
public string MapFromResultToString(object result)
{
return JsonSerializer.Serialize(result);
}
}

/// <summary>
/// Test mapper which converts a BingWebPage search result to a string using JSON serialization.
/// </summary>
private sealed class TestTextSearchResultMapper : ITextSearchResultMapper
{
/// <inheritdoc />
public TextSearchResult MapFromResultToTextSearchResult(object result)
{
if (result is not BingWebPage webPage)
{
throw new ArgumentException("Result must be a BingWebPage", nameof(result));
}

return new TextSearchResult
{
Name = webPage.Name?.ToUpperInvariant(),
Value = webPage.Snippet?.ToUpperInvariant(),
Link = webPage.DisplayUrl?.ToUpperInvariant(),
};
}
}
#endregion
}
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
using System;
using System.IO;
using System.Linq;
using System.Text.Json;
using System.Threading.Tasks;
using Google.Apis.CustomSearchAPI.v1.Data;
using Google.Apis.Http;
Expand Down Expand Up @@ -109,6 +110,65 @@ public async Task GetSearchResultsReturnsSuccessfullyAsync()
}
}

[Fact]
public async Task SearchWithCustomStringMapperReturnsSuccessfullyAsync()
{
// Arrange
this._messageHandlerStub.AddJsonResponse(File.ReadAllText(WhatIsTheSKResponseJson));

// Create an ITextSearch instance using Google search
using var textSearch = new GoogleTextSearch(
initializer: new() { ApiKey = "ApiKey", HttpClientFactory = this._clientFactory },
searchEngineId: "SearchEngineId",
options: new() { StringMapper = new TestTextSearchStringMapper() });

// Act
KernelSearchResults<string> result = await textSearch.SearchAsync("What is the Semantic Kernel?", new() { Count = 4, Offset = 0 });

// Assert
Assert.NotNull(result);
Assert.NotNull(result.Results);
var resultList = await result.Results.ToListAsync();
Assert.NotNull(resultList);
Assert.Equal(4, resultList.Count);
foreach (var stringResult in resultList)
{
Assert.NotEmpty(stringResult);
var googleResult = JsonSerializer.Deserialize<global::Google.Apis.CustomSearchAPI.v1.Data.Result>(stringResult);
Assert.NotNull(googleResult);
}
}

[Fact]
public async Task GetTextSearchResultsWithCustomResultMapperReturnsSuccessfullyAsync()
{
// Arrange
this._messageHandlerStub.AddJsonResponse(File.ReadAllText(WhatIsTheSKResponseJson));

// Create an ITextSearch instance using Google search
using var textSearch = new GoogleTextSearch(
initializer: new() { ApiKey = "ApiKey", HttpClientFactory = this._clientFactory },
searchEngineId: "SearchEngineId",
options: new() { ResultMapper = new TestTextSearchResultMapper() });

// Act
KernelSearchResults<TextSearchResult> result = await textSearch.GetTextSearchResultsAsync("What is the Semantic Kernel?", new() { Count = 4, Offset = 0 });

// Assert
Assert.NotNull(result);
Assert.NotNull(result.Results);
var resultList = await result.Results.ToListAsync();
Assert.NotNull(resultList);
Assert.Equal(4, resultList.Count);
foreach (var textSearchResult in resultList)
{
Assert.NotNull(textSearchResult);
Assert.Equal(textSearchResult.Name, textSearchResult.Name?.ToUpperInvariant());
Assert.Equal(textSearchResult.Value, textSearchResult.Value?.ToUpperInvariant());
Assert.Equal(textSearchResult.Link, textSearchResult.Link?.ToUpperInvariant());
}
}

[Theory]
[InlineData("cr", "countryAF", "https://customsearch.googleapis.com/customsearch/v1?key=ApiKey&cr=countryAF&cx=SearchEngineId&num=4&q=What%20is%20the%20Semantic%20Kernel%3F&start=0")]
[InlineData("dateRestrict", "d[5]", "https://customsearch.googleapis.com/customsearch/v1?key=ApiKey&cx=SearchEngineId&dateRestrict=d%5B5%5D&num=4&q=What%20is%20the%20Semantic%20Kernel%3F&start=0")]
Expand Down Expand Up @@ -197,5 +257,39 @@ public void Dispose()
GC.SuppressFinalize(this);
}
}

/// <summary>
/// Test mapper which converts a global::Google.Apis.CustomSearchAPI.v1.Data.Result search result to a string using JSON serialization.
/// </summary>
private sealed class TestTextSearchStringMapper : ITextSearchStringMapper
{
/// <inheritdoc />
public string MapFromResultToString(object result)
{
return JsonSerializer.Serialize(result);
}
}

/// <summary>
/// Test mapper which converts a global::Google.Apis.CustomSearchAPI.v1.Data.Result search result to a string using JSON serialization.
/// </summary>
private sealed class TestTextSearchResultMapper : ITextSearchResultMapper
{
/// <inheritdoc />
public TextSearchResult MapFromResultToTextSearchResult(object result)
{
if (result is not global::Google.Apis.CustomSearchAPI.v1.Data.Result googleResult)
{
throw new ArgumentException("Result must be a Google Result", nameof(result));
}

return new TextSearchResult
{
Name = googleResult.Title?.ToUpperInvariant(),
Value = googleResult.Snippet?.ToUpperInvariant(),
Link = googleResult.Link?.ToUpperInvariant(),
};
}
}
#endregion
}
47 changes: 34 additions & 13 deletions dotnet/src/Plugins/Plugins.Web/Bing/BingTextSearch.cs
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,12 @@ public BingTextSearch(string apiKey, BingTextSearchOptions? options = null)
this._httpClient = options?.HttpClient ?? HttpClientProvider.GetHttpClient();
this._httpClient.DefaultRequestHeaders.Add("User-Agent", HttpHeaderConstant.Values.UserAgent);
this._httpClient.DefaultRequestHeaders.Add(HttpHeaderConstant.Names.SemanticKernelVersion, HttpHeaderConstant.Values.GetAssemblyVersion(typeof(BingTextSearch)));
this._mapToString = options?.MapToString ?? DefaultMapToString;
this._mapToTextSearchResult = options?.MapToTextSearchResult ?? DefaultMapToTextSearchResult;
this._stringMapper = options?.StringMapper ?? s_defaultStringMapper;
this._resultMapper = options?.ResultMapper ?? s_defaultResultMapper;
}

/// <inheritdoc/>
async public Task<KernelSearchResults<string>> SearchAsync(string query, TextSearchOptions? searchOptions = null, CancellationToken cancellationToken = default)
public async Task<KernelSearchResults<string>> SearchAsync(string query, TextSearchOptions? searchOptions = null, CancellationToken cancellationToken = default)
{
searchOptions ??= new TextSearchOptions();
BingSearchResponse<BingWebPage>? searchResponse = await this.ExecuteSearchAsync(query, searchOptions, cancellationToken).ConfigureAwait(false);
Expand All @@ -52,7 +52,7 @@ async public Task<KernelSearchResults<string>> SearchAsync(string query, TextSea
}

/// <inheritdoc/>
async public Task<KernelSearchResults<TextSearchResult>> GetTextSearchResultsAsync(string query, TextSearchOptions? searchOptions = null, CancellationToken cancellationToken = default)
public async Task<KernelSearchResults<TextSearchResult>> GetTextSearchResultsAsync(string query, TextSearchOptions? searchOptions = null, CancellationToken cancellationToken = default)
{
searchOptions ??= new TextSearchOptions();
BingSearchResponse<BingWebPage>? searchResponse = await this.ExecuteSearchAsync(query, searchOptions, cancellationToken).ConfigureAwait(false);
Expand All @@ -63,7 +63,7 @@ async public Task<KernelSearchResults<TextSearchResult>> GetTextSearchResultsAsy
}

/// <inheritdoc/>
async public Task<KernelSearchResults<object>> GetSearchResultsAsync(string query, TextSearchOptions? searchOptions = null, CancellationToken cancellationToken = default)
public async Task<KernelSearchResults<object>> GetSearchResultsAsync(string query, TextSearchOptions? searchOptions = null, CancellationToken cancellationToken = default)
{
searchOptions ??= new TextSearchOptions();
BingSearchResponse<BingWebPage>? searchResponse = await this.ExecuteSearchAsync(query, searchOptions, cancellationToken).ConfigureAwait(false);
Expand All @@ -79,8 +79,11 @@ async public Task<KernelSearchResults<object>> GetSearchResultsAsync(string quer
private readonly HttpClient _httpClient;
private readonly string? _apiKey;
private readonly Uri? _uri = null;
private readonly MapBingWebPageToString _mapToString;
private readonly MapBingWebPageToTextSearchResult _mapToTextSearchResult;
private readonly ITextSearchStringMapper _stringMapper;
private readonly ITextSearchResultMapper _resultMapper;

private static readonly ITextSearchStringMapper s_defaultStringMapper = new DefaultTextSearchStringMapper();
private static readonly ITextSearchResultMapper s_defaultResultMapper = new DefaultTextSearchResultMapper();

// See https://learn.microsoft.com/en-us/bing/search-apis/bing-web-search/reference/query-parameters
private static readonly string[] s_queryParameters = ["answerCount", "cc", "freshness", "mkt", "promote", "responseFilter", "safeSearch", "setLang", "textDecorations", "textFormat"];
Expand Down Expand Up @@ -170,7 +173,7 @@ private async IAsyncEnumerable<TextSearchResult> GetResultsAsTextSearchResultAsy

foreach (var webPage in searchResponse.WebPages.Value)
{
yield return this._mapToTextSearchResult(webPage);
yield return this._resultMapper.MapFromResultToTextSearchResult(webPage);
await Task.Yield();
}
}
Expand All @@ -189,7 +192,7 @@ private async IAsyncEnumerable<string> GetResultsAsStringAsync(BingSearchRespons

foreach (var webPage in searchResponse.WebPages.Value)
{
yield return this._mapToString(webPage);
yield return this._stringMapper.MapFromResultToString(webPage);
await Task.Yield();
}
}
Expand All @@ -211,17 +214,35 @@ private async IAsyncEnumerable<string> GetResultsAsStringAsync(BingSearchRespons
/// <summary>
/// Default implementation which maps from a <see cref="BingWebPage"/> to a <see cref="string"/>
/// </summary>
private static string DefaultMapToString(BingWebPage webPage)
private sealed class DefaultTextSearchStringMapper : ITextSearchStringMapper
{
return webPage.Snippet ?? string.Empty;
/// <inheritdoc />
public string MapFromResultToString(object result)
{
if (result is not BingWebPage webPage)
{
throw new ArgumentException("Result must be a BingWebPage", nameof(result));
}

return webPage.Snippet ?? string.Empty;
}
}

/// <summary>
/// Default implementation which maps from a <see cref="BingWebPage"/> to a <see cref="TextSearchResult"/>
/// </summary>
private static TextSearchResult DefaultMapToTextSearchResult(BingWebPage webPage)
private sealed class DefaultTextSearchResultMapper : ITextSearchResultMapper
{
return new TextSearchResult(webPage.Name, webPage.Snippet, webPage.Url);
/// <inheritdoc />
public TextSearchResult MapFromResultToTextSearchResult(object result)
{
if (result is not BingWebPage webPage)
{
throw new ArgumentException("Result must be a BingWebPage", nameof(result));
}

return new TextSearchResult(webPage.Name, webPage.Snippet, webPage.Url);
}
}

/// <summary>
Expand Down
Loading
Loading