Skip to content

Python: Vector store filters raise NotImplementedError for negative number literals such as x.price > -5 #14571

Description

What happens

A filter lambda that uses a negative number literal, such as lambda x: x.temperature > -5, raises NotImplementedError in nine of the eleven Python vector store connectors. Azure AI Search and Azure Cosmos DB NoSQL accept it.

Python parses -5 as UnaryOp(USub, Constant(5)), not as a constant. Each connector's _lambda_parser sends every UnaryOp other than not to a raise NotImplementedError(...), so the literal can never be translated. A negative bound is a normal thing to filter on (temperatures, scores, offsets, balances).

Where

python/semantic_kernel/connectors/ on current main (cc8a15f), the ast.UnaryOp branch of _lambda_parser:

  • chroma.py line 409
  • qdrant.py line 419
  • mongodb.py line 494
  • pinecone.py line 558
  • weaviate.py line 456
  • redis.py line 469
  • postgres.py line 916
  • sql_server.py line 644
  • oracle.py line 1095

azure_ai_search.py and azure_cosmos_db.py handle USub by prefixing -, which is why x.price > -5 works there. The shared base class builds the AST in VectorSearch._build_filter (python/semantic_kernel/data/vector.py), so the literal can be normalized once for all connectors instead of in each parser.

Steps to reproduce

Real search against in-process Chroma:

import asyncio
from dataclasses import dataclass
from typing import Annotated

import chromadb

from semantic_kernel.connectors.chroma import ChromaCollection
from semantic_kernel.data.vector import VectorStoreField, vectorstoremodel


@vectorstoremodel
@dataclass
class Record:
    id: Annotated[str, VectorStoreField("key")]
    temp: Annotated[int, VectorStoreField("data", is_indexed=True)]
    vec: Annotated[list[float] | None, VectorStoreField("vector", dimensions=2)] = None


async def main():
    collection = ChromaCollection(record_type=Record, collection_name="temps", client=chromadb.EphemeralClient())
    await collection.ensure_collection_exists()
    await collection.upsert([Record("1", -10, [1.0, 0.0]), Record("2", -3, [0.0, 1.0]), Record("3", 4, [1.0, 1.0])])
    for f in ("lambda x: x.temp > -5", "lambda x: x.temp > 0"):
        try:
            results = await collection.search(vector=[1.0, 1.0], filter=f)
            print(f, sorted([r.record.id async for r in results.results]))
        except Exception as ex:
            print(f, type(ex).__name__, ex)


asyncio.run(main())

All connectors, calling the parser directly (collection._build_filter("lambda x: x.price > -5"), collection created with Cls.__new__(Cls) and only definition set):

Actual

chroma:    NotImplementedError: Unary +, -, ~ and ! are not supported in Chroma filters.
qdrant:    NotImplementedError: Unary +, -, ~ are not supported in Qdrant filters.
mongodb:   NotImplementedError: Unary +, -, ~ are not supported in MongoDB filters.
pinecone:  NotImplementedError: Unary +, -, ~ are not supported in Pinecone filters.
weaviate:  NotImplementedError: Unary +, -, ~, ! are not supported in Weaviate filters.
redis:     NotImplementedError: Unary +, -, ~ are not supported in RedisVL filters.
postgres:  NotImplementedError: Unary +, -, ~ are not supported in PostgreSQL filters.
sql_server:NotImplementedError: Unary +, -, ~ are not supported in SQL filters.
oracle:    NotImplementedError: Unsupported UnaryOp: <class 'ast.USub'>
azure_ai:  'price gt -5'
cosmos:    ('c.price > -5', [])

The Chroma script prints lambda x: x.temp > -5 VectorSearchExecutionException An error occurred during the search: Unary +, -, ~ and ! are not supported in Chroma filters. and then lambda x: x.temp > 0 ['3'].

Expected

x.temp > -5 is translated like any other number: {"temp": {"$gt": -5}} for Chroma, MongoDB and Pinecone, a Range(gt=-5) for Qdrant, "temp" > %s with [-5] for Postgres, and so on. The Chroma script returns ['2', '3'] for the first filter.

Suggested fix: in VectorSearch._build_filter, fold UnaryOp(USub, Constant(<int or float>)) into Constant(-value) before the parsers run. Other unary expressions (-x.field, -True, ~5, +5) are left to the parsers as they are today.

I checked open and closed issues and PRs for negative numbers, unary operators and filter parsing and found nothing covering this. I am opening a PR right after filing this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    pythonPull requests for the Python Semantic Kerneltriage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions