FLEX_SEARCH_TOO_MANY_OBJECTS - FlexSearch error when try to use postFilter

[alexisrcausa]

Hello guys! We are currently encountering an issue when utilizing the flexSearch query with the postFilter parameter in our system. Allow me to provide some context.

We have an entity named “Finding” which contains several fields. We employ flexSearch to filter findings by their title and description, and this works perfectly. However, when we attempt to use postFilter in the query to apply additional filters on some related entities, we encounter the following error: FLEX_SEARCH_TOO_MANY_OBJECTS.

We are seeking guidance on whether there are any modifications or updates we can implement to avoid this error. Additionally, we would appreciate an explanation of the potential causes behind this error.

Thank you for your assistance.

[mfusser]

Hi,
let me try to explain the concept behind this error, it seems the documentation for this is missing.

Our goal with flexSearch was to provide an API that always guarantees to have good performance and quick response time.
When using flexSearchFilter you will always use the ArangoSearch Index which means the query should be quite fast.

For the cases where using the flexSearchFilter does not work, we introduced the postFilter which has the capabilities of normal filtering. But because of the way Arango works, it is applied in memory after the flexSearchFilter is applied and can never use any indices (the same applies to any sorting that does not match the flexSearchOrder).

To prevent queries with bad performance we throw this error when a postFilter or sorting is applied to more than 10 000 Objects (the exact number is configurable via the flexSearchMaxFilterableAndSortableAmount ExecutionOption) we throw this FLEX_SEARCH_TOO_MANY_OBJECTS error.

The idea here is the following:

If you have a table with 1 million objects and you use a postFilter your query would be way too slow (and probably time out).
But if you apply a flexSearchFilter first that filters it down to just 1000 objects (e.g. filtering by customer), using a postFilter would be fine.

I hope this helps understanding this error. I will add a ticket to add this to the documentation as well.

If you have any more questions about this, let me know.