Skip to content

Fix incorrect partition pruning for not IN/ not has function#96241

Merged
alexey-milovidov merged 7 commits intoClickHouse:masterfrom
nihalzp:fix-in-set-relaxed-pruning
Feb 8, 2026
Merged

Fix incorrect partition pruning for not IN/ not has function#96241
alexey-milovidov merged 7 commits intoClickHouse:masterfrom
nihalzp:fix-in-set-relaxed-pruning

Conversation

@nihalzp
Copy link
Member

@nihalzp nihalzp commented Feb 6, 2026

For not IN, we always had this bug:

CREATE TABLE test ( i Int32 )
ENGINE = MergeTree PARTITION BY i ORDER BY tuple()
SETTINGS index_granularity = 1;

INSERT INTO test SELECT number FROM numbers(3);

The following query gives [0,2] but should be [0,1,2].

SELECT arraySort(groupArray(i)) FROM test WHERE tuple(i, i) NOT IN (tuple(1, 2));

Since 25.12, for this table:

CREATE TABLE test ( dt DateTime )
ENGINE = MergeTree PARTITION BY toDate(dt) ORDER BY tuple()
SETTINGS index_granularity = 1;

INSERT INTO test SELECT toDateTime('2026-01-01 00:00:00') + number * 3600 FROM numbers(24 * 40);

The following query gives 936 but should be 959.

SELECT count()
FROM test
WHERE NOT has([toDateTime('2026-01-10 00:00:00')], dt);

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Fix incorrect partition pruning for not IN and not has function in some cases.

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

@clickhouse-gh
Copy link
Contributor

clickhouse-gh bot commented Feb 6, 2026

Workflow [PR], commit [115776c]

Summary:

job_name test_name status info comment
AST fuzzer (amd_ubsan) failure
Logical error: Unexpected return type from A. Expected B. Got C. Action: (STID: 1611-343c) FAIL cidb
BuzzHouse (amd_msan) failure
Logical error: 'Part 5e7858f3c23213bee131d13d07715a84_22_22_0_74 intersects part 5e7858f3c23213bee131d13d07715a84_22_22_1. It is a bug or a result of manual intervention in the ZooKeeper data.' (STID: 2352-4570) FAIL cidb

@clickhouse-gh clickhouse-gh bot added the pr-bugfix Pull request with bugfix, not backported by default label Feb 6, 2026
@nihalzp nihalzp changed the title Fix incorrect partition pruning for not has function Fix incorrect partition pruning for not IN/ not has` function Feb 7, 2026
@nihalzp nihalzp changed the title Fix incorrect partition pruning for not IN/ not has` function Fix incorrect partition pruning for not IN/ not has function Feb 7, 2026
if (adjusted_indexes_mapping.size() < set_types.size() || is_constant_transformed)
out.relaxed = true;

if (out.set_index->size() > 1 || out.relaxed)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per my understanding, we do not need:

if (out.set_index->size() > 1 || out.relaxed)
    relaxed = true;

at all for correctness. As a side effect, it only disables exact count projection optimization which can be enabled by removing it. For example, with this check, we do not use the optimization for queries like:

CREATE TABLE test ( i UInt32 )
ENGINE = MergeTree ORDER BY i SETTINGS index_granularity = 1;

INSERT INTO test SELECT number % 30 FROM numbers(1000000);

EXPLAIN indexes = 1
SELECT count() FROM test WHERE i IN (1, 2, 3);

But let's keep it for now to preserve old behavior as we will backport the fix.

@alexey-milovidov alexey-milovidov self-assigned this Feb 8, 2026
@alexey-milovidov alexey-milovidov merged commit 059928d into ClickHouse:master Feb 8, 2026
256 of 264 checks passed
@robot-clickhouse-ci-1 robot-clickhouse-ci-1 added the pr-synced-to-cloud The PR is synced to the cloud repo label Feb 8, 2026
@robot-ch-test-poll4 robot-ch-test-poll4 added the pr-must-backport-synced The `*-must-backport` labels are synced into the cloud Sync PR label Feb 8, 2026
robot-clickhouse-ci-1 added a commit that referenced this pull request Feb 8, 2026
Cherry pick #96241 to 25.12: Fix incorrect partition pruning for `not IN`/ `not has` function
robot-clickhouse added a commit that referenced this pull request Feb 8, 2026
robot-clickhouse-ci-1 added a commit that referenced this pull request Feb 8, 2026
Cherry pick #96241 to 26.1: Fix incorrect partition pruning for `not IN`/ `not has` function
robot-clickhouse added a commit that referenced this pull request Feb 8, 2026
@robot-clickhouse-ci-2 robot-clickhouse-ci-2 added the pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore label Feb 8, 2026
clickhouse-gh bot added a commit that referenced this pull request Feb 8, 2026
Backport #96241 to 26.1: Fix incorrect partition pruning for `not IN`/ `not has` function
nihalzp added a commit that referenced this pull request Feb 8, 2026
Backport #96241 to 25.12: Fix incorrect partition pruning for `not IN`/ `not has` function
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore pr-bugfix Pull request with bugfix, not backported by default pr-must-backport-synced The `*-must-backport` labels are synced into the cloud Sync PR pr-synced-to-cloud The PR is synced to the cloud repo v25.12-must-backport v26.1-must-backport

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants