Problem
Evaluator::partial_run() documents that only rows selected by its validity bitmap should be
evaluated. and_filters and or_filters update that bitmap after each predicate and pass it to
later expressions through EvalContext.validity.
However, the generic helpers in
src/query/expression/src/function/register_vectorize.rs iterate over every row in a column without
checking ctx.validity. Several custom vectorized functions do the same.
Confirmed examples include:
- string
ILIKE and ILIKE ANY;
- Variant
LIKE and LIKE ANY;
REGEXP / RLIKE;
GLOB through vectorize_with_builder_2_arg;
- other functions built with
vectorize_* and vectorize_with_builder_* helpers.
Inactive results are normally masked by the enclosing expression, and EvalContext::set_error()
ignores errors on inactive rows. This usually preserves query results, but expensive functions can
still perform unnecessary work over complete columns.
Not every function will benefit from checking validity. For cheap branchless arithmetic, a dense
bitmap, or predicates that leave most rows active, a bitmap lookup and branch per row may cost as
much as or more than evaluating the function. Scalar arguments and constant expressions also have
little or no row-level work to skip. The optimization should therefore distinguish sparse,
high-cost evaluation from dense or inexpensive evaluation rather than unconditionally adding a
validity branch to every function.
For example, a selective predicate may leave only a small fraction of a batch active:
SELECT event_time
FROM benchmark_events
WHERE worker = 'worker-7'
AND payload RLIKE 'expensive-pattern'
ORDER BY event_time DESC
LIMIT 50;
The regex currently runs for all rows passed to ordinary vector evaluation, not only rows selected
by the equality predicate. This is particularly visible when FUSE Parquet Prewhere evaluates
multiple predicates through the ordinary expression evaluator rather than Selector.
PR #20211 fixes and tests this behavior specifically for vectorized string LIKE. The same
framework-level gap remains for other functions.
Expected work
- Audit custom vectorized functions that ignore sparse validity.
- Consider adding validity-aware sparse paths to the generic vectorize helpers, or provide explicit
sparse helpers or an opt-in property for expensive functions.
- Keep the current dense path when validity is absent or all true, so common full-column evaluation
does not gain a per-row bitmap branch.
- Emit default placeholders for inactive rows without invoking the scalar operation.
- Preserve nullable, error, scalar/column, nested-type, and non-deterministic function semantics.
- Add call-count and property tests that compare active-row results with full evaluation.
- Benchmark different validity densities and function costs before changing generic helpers used by
many functions. Consider a selectivity threshold if switching paths dynamically is useful.
Relevant code
src/query/expression/src/evaluator.rs: partial_run, eval_and_filters, eval_filters
src/query/expression/src/function/register_vectorize.rs: generic vectorize helpers
src/query/functions/src/scalars/comparison.rs: ILIKE, Variant LIKE, REGEXP, and GLOB
Problem
Evaluator::partial_run()documents that only rows selected by its validity bitmap should beevaluated.
and_filtersandor_filtersupdate that bitmap after each predicate and pass it tolater expressions through
EvalContext.validity.However, the generic helpers in
src/query/expression/src/function/register_vectorize.rsiterate over every row in a column withoutchecking
ctx.validity. Several custom vectorized functions do the same.Confirmed examples include:
ILIKEandILIKE ANY;LIKEandLIKE ANY;REGEXP/RLIKE;GLOBthroughvectorize_with_builder_2_arg;vectorize_*andvectorize_with_builder_*helpers.Inactive results are normally masked by the enclosing expression, and
EvalContext::set_error()ignores errors on inactive rows. This usually preserves query results, but expensive functions can
still perform unnecessary work over complete columns.
Not every function will benefit from checking validity. For cheap branchless arithmetic, a dense
bitmap, or predicates that leave most rows active, a bitmap lookup and branch per row may cost as
much as or more than evaluating the function. Scalar arguments and constant expressions also have
little or no row-level work to skip. The optimization should therefore distinguish sparse,
high-cost evaluation from dense or inexpensive evaluation rather than unconditionally adding a
validity branch to every function.
For example, a selective predicate may leave only a small fraction of a batch active:
The regex currently runs for all rows passed to ordinary vector evaluation, not only rows selected
by the equality predicate. This is particularly visible when FUSE Parquet Prewhere evaluates
multiple predicates through the ordinary expression evaluator rather than
Selector.PR #20211 fixes and tests this behavior specifically for vectorized string
LIKE. The sameframework-level gap remains for other functions.
Expected work
sparse helpers or an opt-in property for expensive functions.
does not gain a per-row bitmap branch.
many functions. Consider a selectivity threshold if switching paths dynamically is useful.
Relevant code
src/query/expression/src/evaluator.rs:partial_run,eval_and_filters,eval_filterssrc/query/expression/src/function/register_vectorize.rs: generic vectorize helperssrc/query/functions/src/scalars/comparison.rs: ILIKE, Variant LIKE, REGEXP, and GLOB