Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[BACKPORT 2024.1][#21141,21036] YSQL: Enable remote filters for Bitma…
…p Scans Summary: Original commit: 773869c / D32651 Pushing down filters prevents us from fetching unnecessary rows. In the bitmap scan case, if we can push down a filter at the Bitmap Index Scan level, it saves us from 1. retrieving the ybctid 2. using that ybctid in any bitmap operations (AND, OR) 3. retrieving the row corresponding to the ybctid. With this diff, both YB Bitmap Table Scans and Bitmap Index Scans are capable of pushing down filters. == YB Bitmap Table Scan pushdown == YB Bitmap Table Scans can push down any clause. The pushdown logic is similar to the recheck logic - it only pushes down a filter if it is necessary. For example, ```lang=sql /*+ BitmapScan(test_limit) */ EXPLAIN SELECT * FROM test_limit WHERE a < 1000 OR b < 1000; QUERY PLAN -------------------------------------------------------------------------------------- YB Bitmap Table Scan on test_limit (cost=6.91..11.21 rows=10 width=12) -> BitmapOr (cost=6.91..6.91 rows=20 width=0) -> Bitmap Index Scan on test_limit_a_idx (cost=0.00..3.45 rows=10 width=0) Index Cond: (a < 1000) -> Bitmap Index Scan on test_limit_b_idx (cost=0.00..3.45 rows=10 width=0) Index Cond: (b < 1000) (6 rows) ``` does not require a remote or local filter because it knows that the index scans will return only correct results. ``` /*+ BitmapScan(test_limit) */ EXPLAIN SELECT * FROM test_limit WHERE a < 1000 OR b < 1000 AND c < 1; QUERY PLAN -------------------------------------------------------------------------------------- YB Bitmap Table Scan on test_limit (cost=7.06..11.41 rows=10 width=12) Remote Filter: ((a < 1000) OR ((b < 1000) AND (c < 1))) -> BitmapOr (cost=7.06..7.06 rows=20 width=0) -> Bitmap Index Scan on test_limit_a_idx (cost=0.00..3.53 rows=10 width=0) Index Cond: (a < 1000) -> Bitmap Index Scan on test_limit_b_idx (cost=0.00..3.53 rows=10 width=0) Index Cond: (b < 1000) (7 rows) ``` This requires a filter, and this filter can be pushed down. == Bitmap Index Scan pushdown == There are two cases where filters can be pushed down for bitmap index scans. === 1. If the filter clause is at the same nesting level as the index qual in the boolean operation tree === During planning, when the OR is broken up into separate indexes, the pushdown conditions are extracted for each branch of the OR. This allows us to get useful filters for pushdown, knowing that a particular index scan is concerned only by a portion of the whole condition. For example, in the index below, the first Bitmap Index Scan has an index condition `unique1 < 5` and a remote filter `unique1 % 2 = 0`. ```lang=sql /*+ BitmapScan(tenk1) */ EXPLAIN (ANALYZE, DIST, COSTS OFF) SELECT * FROM tenk1 WHERE (unique1 < 5 AND unique1 % 2 = 0) OR unique2 < 3; QUERY PLAN ----------------------------------------------------------------------------- YB Bitmap Table Scan on tenk1 (actual rows=6 loops=1) Remote Filter: (((unique1 < 5) AND ((unique1 % 2) = 0)) OR (unique2 < 3)) Storage Table Read Requests: 1 Storage Table Rows Scanned: 6 -> BitmapOr (actual rows=6 loops=1) -> Bitmap Index Scan on tenk1_unique1 (actual rows=3 loops=1) Index Cond: (unique1 < 5) Remote Index Filter: ((unique1 % 2) = 0) Storage Index Read Requests: 1 Storage Index Rows Scanned: 5 -> Bitmap Index Scan on tenk1_unique2 (actual rows=3 loops=1) Index Cond: (unique2 < 3) Storage Index Read Requests: 1 Storage Index Rows Scanned: 3 Storage Read Requests: 3 Storage Rows Scanned: 14 Storage Write Requests: 0 Storage Flush Requests: 0 (18 rows) ``` When creating a bitmap scan plan, the planner breaks the OR into two parts. a: `(unique1 < 5 AND unique1 % 2 = 0)` b: `unique2 < 3` If it can find an index scan for each part, then a bitmap scan is a valid plan. (If there was a third clause on an unindexed field, c: `string1 = 'hi'`, then there would not be an index to answer each part of the OR so we cannot use a bitmap scan here). The default PG behaviour is to extract index clauses from each branch, so PG would identify `unique1 < 5` for a and `unique2 < 3` for b. With this diff, Yugabyte goes a step further. If a condition is not a valid index condition (e.g. `unique1 % 2 = 0`, then we test if it can be pushed down using the index columns. If yes, then we store it as a valid pushdown filter for a bitmap index scan. So we identify: | | index qual | filter qual | | a | `unique1 < 5` | `unique1 % 2 = 0` | | b | `unique2 < 3` | | Since we found an index path for each branch of the OR, we've found a valid bitmap scan path. === 2. All columns referenced by the top-level condition are contained in the index === In this case, we can push down the entire condition (or a portion of the top-level AND clause) to the index with the included columns. We cannot partially pushdown top-level ORs because this index doesn't know enough about the other branches, but that case was already handled above. For example, in the example below, the first Bitmap Index Scan pushes down the entire condition `(((a < 5) AND ((a % 2) = 0)) OR ((c <= 10) AND ((a % 3) = 0)))` on `multi_c_a_idx` because the every column in the condition can be checked by this index. ```lang=sql /*+ BitmapScan(multi) */ EXPLAIN (ANALYZE, DIST, COSTS OFF) SELECT * FROM multi WHERE (a < 50 AND a % 2 = 0) AND (c <= 100 AND c % 3 = 0); QUERY PLAN ------------------------------------------------------------------------------------------- YB Bitmap Table Scan on multi (actual time=3.655..3.666 rows=16 loops=1) Storage Table Read Requests: 1 Storage Table Read Execution Time: 1.014 ms Storage Table Rows Scanned: 16 -> BitmapAnd (actual time=2.492..2.492 rows=16 loops=1) -> Bitmap Index Scan on multi_c_a_idx (actual time=2.550..2.550 rows=16 loops=1) Index Cond: (c <= 100) Storage Index Filter: ((a < 50) AND ((a % 2) = 0) AND ((c % 3) = 0)) Storage Index Read Requests: 1 Storage Index Read Execution Time: 1.778 ms Storage Index Rows Scanned: 33 -> Bitmap Index Scan on multi_pkey (actual time=1.288..1.288 rows=24 loops=1) Index Cond: (a < 50) Storage Index Filter: ((a % 2) = 0) Storage Table Read Requests: 1 Storage Table Read Execution Time: 1.164 ms Storage Table Rows Scanned: 49 Planning Time: 15.273 ms Execution Time: 4.362 ms Storage Read Requests: 2 Storage Read Execution Time: 2.942 ms Storage Rows Scanned: 82 Storage Write Requests: 0 Catalog Read Requests: 19 Catalog Read Execution Time: 31.753 ms Catalog Write Requests: 0 Storage Flush Requests: 0 Storage Execution Time: 34.695 ms Peak Memory Usage: 119 kB (26 rows) ``` === 3. All columns referenced by the one of the top-level clauses (implicitly ANDed) are contained in the index === Consider the `tenk1` table: ```lang=sql \d tenk1 Table "public.tenk1" ... Indexes: "tenk1_hundred" lsm (hundred ASC) "tenk1_thous_tenthous" lsm (thousand ASC, tenthous ASC) "tenk1_unique1" lsm (unique1 ASC) "tenk1_unique2" lsm (unique2 ASC) ``` Then the query can push down the `tenthous % 2 = 0` condition to the `thousand` bitmap index scan. Even though it will still need to be validated at the Table Scan layer (to check the results from the `unique2` bitmap index scan, it is efficient to reduce the number of rows retrieved. ``` explain (analyze, costs off, dist) /*+ bitmapscan(tenk1) */ select * from tenk1 where thousand < 10 and unique2 < 1000 and tenthous % 2 = 0; QUERY PLAN -------------------------------------------------------------------------------------------------- YB Bitmap Table Scan on tenk1 (actual time=17.917..17.929 rows=4 loops=1) Storage Filter: ((tenthous % 2) = 0) Storage Table Read Requests: 1 Storage Table Read Execution Time: 1.170 ms Storage Table Rows Scanned: 4 -> BitmapAnd (actual time=16.422..16.422 rows=4 loops=1) -> Bitmap Index Scan on tenk1_unique2 (actual time=13.315..13.315 rows=1000 loops=1) Index Cond: (unique2 < 1000) Storage Index Read Requests: 1 Storage Index Read Execution Time: 10.981 ms Storage Index Rows Scanned: 1000 -> Bitmap Index Scan on tenk1_thous_tenthous (actual time=2.845..2.845 rows=50 loops=1) Index Cond: (thousand < 10) Storage Index Filter: ((tenthous % 2) = 0) Storage Index Read Requests: 1 Storage Index Read Execution Time: 2.407 ms Storage Index Rows Scanned: 100 Planning Time: 0.894 ms Execution Time: 18.702 ms Storage Read Requests: 3 Storage Read Execution Time: 14.559 ms Storage Rows Scanned: 1104 Storage Write Requests: 0 Catalog Read Requests: 0 Catalog Write Requests: 0 Storage Flush Requests: 0 Storage Execution Time: 14.559 ms Peak Memory Usage: 180 kB (28 rows) ``` Jira: DB-10083 Test Plan: ``` ./yb_build.sh --java-test 'org.yb.pgsql.TestPgRegressYbBitmapScans' ``` Tested with random query generator Reviewers: amartsinchyk, tnayak Reviewed By: tnayak Subscribers: yql Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D34006
- Loading branch information
