Skip to content

Add semantic memory retrieval test#781

Merged
Bryan-Roe merged 3 commits intomainfrom
3jhfjp-codex/validate-semantic-memory-retrieval-accuracy
Aug 3, 2025
Merged

Add semantic memory retrieval test#781
Bryan-Roe merged 3 commits intomainfrom
3jhfjp-codex/validate-semantic-memory-retrieval-accuracy

Conversation

@Bryan-Roe
Copy link
Collaborator

@Bryan-Roe Bryan-Roe commented Jul 27, 2025

Summary

  • add unit test to verify semantic memory retrieval

Testing

  • python 01-core-implementations/python/tests/unit/memory/test_semantic_memory_retrieval.py (fails: ModuleNotFoundError: No module named 'numpy')

https://chatgpt.com/codex/tasks/task_e_6886a79400d083228045f0d4e0ce5e79

Summary by Sourcery

Tests:

  • Introduce test_semantic_memory_retrieval.py with FakeEmbeddingGenerator and create_memory helper to test SemanticTextMemory.search accuracy for various queries.

Copilot AI review requested due to automatic review settings July 27, 2025 22:45
@sourcery-ai
Copy link

sourcery-ai bot commented Jul 27, 2025

Reviewer's Guide

Introduce a new pytest unit test for SemanticTextMemory that uses a FakeEmbeddingGenerator and VolatileMemoryStore to verify accurate retrieval based on simple synthetic embeddings.

File-Level Changes

Change Details Files
Add semantic memory retrieval unit test
  • Implement FakeEmbeddingGenerator to produce deterministic embeddings from text
  • Add create_memory helper to initialize SemanticTextMemory with volatile storage
  • Define async pytest test saving entries and asserting search results for 'hiking', 'tour', and 'Iceland'
01-core-implementations/python/tests/unit/memory/test_semantic_memory_retrieval.py

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds a unit test to verify semantic memory retrieval functionality. The test validates that the semantic memory system can accurately retrieve stored information based on search queries.

Key Changes

  • Creates a new test file for semantic memory retrieval functionality
  • Implements a fake embedding generator for testing purposes
  • Adds test cases to verify accurate retrieval of stored text information
Comments suppressed due to low confidence (1)

01-core-implementations/python/tests/unit/memory/test_semantic_memory_retrieval.py:28

  • [nitpick] The test only verifies exact matches but doesn't test semantic similarity or ranking when multiple results are returned. Consider adding test cases for partial matches and verifying the ranking order of search results.
async def test_retrieval_accuracy():

Copy link

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @Bryan-Roe - I've reviewed your changes - here's some feedback:

  • Add numpy to the project’s test or dev dependencies so that import numpy works when running tests.
  • Consider parameterizing the query/text assertions in test_retrieval_accuracy to reduce repetition and improve readability.
  • Add a test case with limit > 1 (and multiple expected results) to verify the search result ordering beyond single-item lookups.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- Add numpy to the project’s test or dev dependencies so that `import numpy` works when running tests.
- Consider parameterizing the query/text assertions in `test_retrieval_accuracy` to reduce repetition and improve readability.
- Add a test case with limit > 1 (and multiple expected results) to verify the search result ordering beyond single-item lookups.

## Individual Comments

### Comment 1
<location> `01-core-implementations/python/tests/unit/memory/test_semantic_memory_retrieval.py:2` </location>
<code_context>
+import asyncio
+import numpy as np
+import pytest
+
</code_context>

<issue_to_address>
Test requires numpy but does not skip or handle missing dependency.

Add a check or use pytest.skipif to skip the test if numpy is not installed, ensuring the test suite runs smoothly without numpy.

Suggested implementation:

```python
import asyncio
import pytest

try:
    import numpy as np
    HAS_NUMPY = True
except ImportError:
    HAS_NUMPY = False

from semantic_kernel.memory.volatile_memory_store import VolatileMemoryStore

```

```python
@pytest.mark.skipif(not HAS_NUMPY, reason="numpy is not installed")
class FakeEmbeddingGenerator(EmbeddingGeneratorBase):

```
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Bryan-Roe and others added 2 commits August 3, 2025 01:35
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Bryan <74067792+Bryan-Roe@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Bryan <74067792+Bryan-Roe@users.noreply.github.com>
@Bryan-Roe Bryan-Roe merged commit 07da280 into main Aug 3, 2025
7 of 11 checks passed
@Bryan-Roe Bryan-Roe deleted the 3jhfjp-codex/validate-semantic-memory-retrieval-accuracy branch August 3, 2025 08:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant