Add post-filter support for VSIM vector search results by hailangx · Pull Request #1570 · microsoft/garnet

hailangx · 2026-02-20T00:53:49Z

Adds post-filter support for VSIM vector search results by introducing a JSON-attribute filter expression engine and integrating it into VectorManager after similarity search.

Changes:

Introduces a tokenizer/parser/evaluator for filter expressions over JSON attributes (and/or/not, comparisons, arithmetic, in, grouping).
Integrates post-filtering into VectorManager for both value-based and element-based similarity search paths.
Adds unit tests for the filter engine and RESP integration tests for VSIM ... FILTER ....

Supported syntax documented at Vector Filter Expressions (VSIM ... FILTER)
website/docs/dev/vector-sets.md

VSIM supports FILTER <expression> for attribute-based post filtering.

VSIM query source can be either ELE <element-id> or VALUES <dimensions> <v1> ... <vN>

Examples

VSIM movies ELE dune FILTER '.year >= 1980 and .rating > 7'
VSIM movies ELE dune FILTER '.genre == "action" && .rating > 8.0'
VSIM movies ELE dune FILTER '"classic" in .tags'
VSIM movies ELE dune FILTER '(.year - 2000) ** 2 < 100 and .rating / 2 > 4'
VSIM movies VALUES 3 0.12 0.34 0.56 FILTER '.year >= 1980 and .rating > 7'

Expression syntax

Arithmetic: +, -, *, /, %, **
Comparison: ==, !=, >, <, >=, <=
Logical: and, or, not (also &&, ||, !)
Containment: in
Grouping: parentheses ()

Field access uses dot notation (for example, .year, .rating, .genre).

Supported values

Numbers
Strings
Booleans (true / false, evaluated as 1 / 0)
Arrays (for in when the right side is an attribute array)

Operator precedence (high to low)

primary / parentheses
unary (not, !, unary -)
power (**, right-associative)
multiplicative (*, /, %)
additive (+, -)
containment (in)
comparison (>, <, >=, <=)
equality (==, !=)
logical and (and, &&)
logical or (or, ||)

Notes

Keywords are lowercase (and, or, not, in, true, false)
Missing attributes are treated as non-matching (null/falsy)
Array literals inside expressions (for example, .director in ["a","b"]) are not currently supported

Implement JSON-path-based filter expressions that are evaluated against vector element attributes after similarity search. The filter engine includes a tokenizer, expression parser, and evaluator supporting comparison operators, logical operators (and/or/not), arithmetic, string equality, containment (in), and parenthesized grouping. Integrate post-filtering into VectorManager for both VSIM code paths, rejecting requests that specify a filter without WITHATTRIBS.

Copilot

Pull request overview

Adds post-filter support for VSIM vector search results by introducing a JSON-attribute filter expression engine and integrating it into VectorManager after similarity search.

Changes:

Introduces a tokenizer/parser/evaluator for filter expressions over JSON attributes (and/or/not, comparisons, arithmetic, in, grouping).
Integrates post-filtering into VectorManager for both value-based and element-based similarity search paths.
Adds unit tests for the filter engine and RESP integration tests for VSIM ... FILTER ....

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`libs/server/Resp/Vector/VectorManager.cs`	Applies post-filtering to VSIM results and evaluates expressions against per-element attributes.
`libs/server/Resp/Vector/Filter/VectorFilterTokenizer.cs`	Tokenizes filter expressions into numbers/strings/identifiers/operators/keywords.
`libs/server/Resp/Vector/Filter/VectorFilterParser.cs`	Parses tokens into an expression AST with operator precedence.
`libs/server/Resp/Vector/Filter/VectorFilterExpression.cs`	Defines AST node types for literals, member access, unary, and binary ops.
`libs/server/Resp/Vector/Filter/VectorFilterEvaluator.cs`	Evaluates the AST against `JsonElement` attribute data.
`test/Garnet.test/VectorFilterTests.cs`	Unit tests for tokenizer/parser/evaluator behavior.
`test/Garnet.test/RespVectorSetTests.cs`	Adds RESP-level tests verifying VSIM filtering behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

libs/server/Resp/Vector/Filter/VectorFilterEvaluator.cs

libs/server/Resp/Vector/VectorManager.cs

libs/server/Resp/Vector/Filter/VectorFilterEvaluator.cs

test/Garnet.test/RespVectorSetTests.cs

Copilot · 2026-02-20T01:04:19Z

@hailangx I've opened a new pull request, #1571, to work on those changes. Once the pull request is ready, I'll request review from you.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot · 2026-02-20T01:05:50Z

@hailangx I've opened a new pull request, #1572, to work on those changes. Once the pull request is ready, I'll request review from you.

* Initial plan * Avoid per-result allocation in EvaluateFilter by using Utf8JsonReader with ParseValue Co-authored-by: hailangx <3389245+hailangx@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: hailangx <3389245+hailangx@users.noreply.github.com>

…ly (#1572) * Initial plan * Fetch attributes internally for filtering when not returning them Co-authored-by: hailangx <3389245+hailangx@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: hailangx <3389245+hailangx@users.noreply.github.com>

harsha-simhadri · 2026-02-24T17:35:50Z

CAn you link the specification of expression syntax you are implementing here

libs/server/Resp/Vector/Filter/VectorFilterTokenizer.cs

hailangx · 2026-02-24T21:41:22Z

@microsoft-github-policy-service agree company="Microsoft"

hailangx · 2026-02-24T21:42:16Z

CAn you link the specification of expression syntax you are implementing here
added into the vector set document

kevin-montrose

I think some fundamental reworking is needed here, exceptions and allocations need to go - I've left a bunch of guideline comments to get us pointed in the right direction. It's not quite an exhaustive review - there are minor optimizations and style things we can revisit latter when we're closer to mergeable.

kevin-montrose · 2026-02-25T16:04:14Z

libs/server/Resp/Vector/VectorManager.cs

+                return numResults;
+            }
+
+            var filterStr = Encoding.UTF8.GetString(filter);


We definitely do not want to work in terms of strings here, that's some expensive validation plus an allocation in a hot path.

kevin-montrose · 2026-02-25T16:13:15Z

libs/server/Resp/Vector/VectorManager.cs

+                var filteredCount = 0;
+
+                // Parse the filter expression once, then evaluate per result
+                var tokens = VectorFilterTokenizer.Tokenize(filterStr);


Ideally we would be able to do this tokenize and parse in one pass - failing that the definitely shouldn't be allocating a new List on each run. Tokenize into offsets and lengths, try and keep it on stack if we can and promote to a heap allocated array only when we exceed some reasonable maximum (512 bytes is probably fine there).

libs/server/Resp/Vector/VectorManager.cs

kevin-montrose · 2026-02-25T16:18:51Z

libs/server/Resp/Vector/VectorManager.cs

+                var distWritePos = 0;
+                var attrWritePos = 0;
+
+                for (var i = 0; i < numResults; i++)


This will copy many id, attribute, and distance value if filters exclude anything - which we'd expect to be common. For large results sets (or large attributes) that a bunch of work.

A better approach would be to have the ValueSimilarity and ElementSimilarity calls indicate a match count and a set of passing elements (probably actually a bitmap) and then let NetworkVSIM handle skipping elements. Then we have no extra copying.

kevin-montrose · 2026-02-25T16:39:30Z

libs/server/Resp/Vector/Filter/VectorFilterExpression.cs

+    /// Discriminated union value type to eliminate boxing of doubles/strings
+    /// throughout the filter evaluation pipeline.
+    /// </summary>
+    [StructLayout(LayoutKind.Auto)]


nit: this is unnecessary

libs/server/Resp/Vector/Filter/VectorFilterParser.cs

kevin-montrose · 2026-02-25T16:50:01Z

libs/server/Resp/Vector/Filter/VectorFilterParser.cs

+    /// </summary>
+    internal static class VectorFilterParser
+    {
+        public static Expr ParseExpression(List<Token> tokens, int start, out int end)


If I'm reading this correctly, it's extremely recursive - which is very scary. Is a stack overflow possible with reasonable filters here? And how complex a filter can we test with before something breaks?

One complication is that Windows and Linux tend to use different default stack sizes - we mostly dev on Windows, but deployments are more commonly Linux.

refactor it to one pass postfix (reverse-Polish) design which redis is using, so only a bound stack. will be used.

libs/server/Resp/Vector/Filter/VectorFilterEvaluator.cs

kevin-montrose · 2026-02-25T16:57:09Z

libs/server/Resp/Vector/Filter/VectorFilterEvaluator.cs

+    /// Evaluates parsed expression trees against JSON attribute data.
+    /// Returns FilterValue (a struct) to avoid boxing allocations on every evaluation.
+    /// </summary>
+    internal static class VectorFilterEvaluator


As a general note, this feels overly complicated.

We're taking a filter that we've parsed and a whole JSON document. But the filter only applies over top level elements of that document... so really, this is operating over the filter and a dictionary (a dictionary that contains no other dictionaries at that).

I've noted elsewhere we should remove most of these allocations - a natural-ish approach would be to a filter, the attribute span, and a collection of (length, offset) pairs to top level attributes.

hailangx marked this pull request as ready for review February 20, 2026 00:54

Copilot AI review requested due to automatic review settings February 20, 2026 00:54

Copilot started reviewing on behalf of hailangx February 20, 2026 00:55 View session

Copilot AI reviewed Feb 20, 2026

View reviewed changes

fix format

a57c6d5

Copilot AI mentioned this pull request Feb 20, 2026

Avoid per-result byte array allocation in EvaluateFilter #1571

Merged

Update libs/server/Resp/Vector/VectorManager.cs

9c883f7

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI mentioned this pull request Feb 20, 2026

VSIM FILTER works without WITHATTRIBS by fetching attributes internally #1572

Merged

Copilot AI and others added 2 commits February 19, 2026 17:09

hailangx requested review from kevin-montrose and tiagonapoli February 23, 2026 20:26

tiagonapoli reviewed Feb 24, 2026

View reviewed changes

libs/server/Resp/Vector/Filter/VectorFilterTokenizer.cs Outdated Show resolved Hide resolved

tiagonapoli reviewed Feb 24, 2026

View reviewed changes

libs/server/Resp/Vector/Filter/VectorFilterTokenizer.cs Outdated Show resolved Hide resolved

Haiyang Xu added 4 commits February 24, 2026 11:30

optimize code

97010ee

add Supported vector filter syntax

7aa8b13

update doc with syntac

95ba208

fix build

54064a0

Haiyang Xu added 5 commits February 24, 2026 14:58

update test with ELE style syntax

aa07eb0

split the filter engine tests

1e5cd34

remove object value type

c939609

remove object-returning property

54dfc42

fix format error

b65dc7c

kevin-montrose requested changes Feb 25, 2026

View reviewed changes

CI Fix and others added 3 commits February 26, 2026 13:48

resove comments

513be50

refactor to stack-based postfix

6770b05

Merge branch 'main' into haixu/vector-filter-postprocessing

6dd6501

hailangx requested a review from kevin-montrose February 26, 2026 23:18

remove hot path allocate

e09603d

Conversation

hailangx commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Examples

Expression syntax

Supported values

Operator precedence (high to low)

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI commented Feb 20, 2026

Uh oh!

Copilot AI commented Feb 20, 2026

Uh oh!

harsha-simhadri commented Feb 24, 2026

Uh oh!

Uh oh!

Uh oh!

hailangx commented Feb 24, 2026

Uh oh!

hailangx commented Feb 24, 2026

Uh oh!

kevin-montrose left a comment

Choose a reason for hiding this comment

Uh oh!

kevin-montrose Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

kevin-montrose Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kevin-montrose Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

kevin-montrose Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kevin-montrose Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

hailangx Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kevin-montrose Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

hailangx commented Feb 20, 2026 •

edited

Loading