Skip to content

Conversation

@clintropolis
Copy link
Member

Description

The equality filter allows automatic coercion of scalar typed columns to array columns to help assist schema migration between mvds and arrays, as well as scalars to arrays, where the scalar inputs are treated as single element arrays for the purposes of matching.

However, this had an unintended side-effect of allowing the reverse that impacts only native queries, allowing using an equality filter with a scalar match value to match single element arrays of array columns, which is pretty strange (and json_value using SQL).

This PR modifies the equality filter (and array indexes) to no longer allow matching single element arrays in columns with scalar values, while preserving the opposite, where scalar columns can match single element arrays.

This has a side-effect that strictly impacts native query equality filter on mixed type 'auto' columns which contain arrays, that they must now be interacted with as their presenting type, so if any rows are arrays, e.g. the segment metadata and information_schema reports the type as some array type, then the native queries must also filter as if they are some array type. This does not impact SQL, which already has this limitation due to how the type presents itself, and again only impacts mixed type 'auto' columns.

Release note

Native query equals filter on mixed type 'auto' columns which contain arrays must now be filtered as their presenting type, so if any rows are arrays (e.g. the segment metadata and information_schema reports the type as some array type), then the native queries must also filter as if they are some array type. This does not impact SQL, which already has this limitation due to how the type presents itself, and again only impacts mixed type 'auto' columns which contain both scalars and arrays.


This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • been tested in a test Druid cluster.

@clintropolis clintropolis merged commit 1eafe98 into apache:master Dec 8, 2023
@clintropolis clintropolis deleted the fix-array-scalar-equality-filtering branch December 8, 2023 09:22
@LakshSingla LakshSingla added this to the 29.0.0 milestone Jan 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants