disabled block engine for spilling

[lll-phill-lll]

Changelog entry

Block engine doesn't support wide combiner. But spilling for aggregation only works in it WideCombiner. So, this PR disables block engine when spilling in aggregation is enabled
...

Changelog category

• not-for-changelog

Additional information

...

[lll-phill-lll]

Local test

Deployed local ydb and made 2 requests. Current ydb state is: spilling enabled in all nodes by default, FinalizeByKey(WideCombine) is enabled by default.

Query with spilling disabled:

PRAGMA ydb.EnableSpillingNodes="None";
SELECT a.o_comment, count(*) FROM `/local/tpch/string_s1/orders` a group by a.o_comment order by a.o_comment;

AST:

(
(let $1 (DataType 'Uint64))
(let $2 '('"o_comment" (DataType 'String)))
(let $3 '('('"_logical_id" '573) '('"_id" '"8bbf71ce-d58fbd29-e411365f-10c2463b") '('"_wide_channels" (StructType '('_yql_agg_0 $1) $2))))
(let $4 (DqPhyStage '() (lambda '() (block '(
  (let $13 (KqpTable '"/local/tpch/string_s1/orders" '"72075186232723360:7" '"" '1))
  (let $14 '('"o_comment"))
  (let $15 (KqpBlockReadOlapTableRanges $13 (Void) $14 '() '() (lambda '($16) (TKqpOlapAgg $16 '('('_yql_agg_0 '"count" '"*")) $14))))
  (return (FromFlow $15))
))) $3))
(let $5 (DqCnHashShuffle (TDqOutput $4 '0) '('1)))
(let $6 (StructType '('"column1" $1) $2))
(let $7 '('('"_logical_id" '963) '('"_id" '"1d6715cc-5b35ae17-8c4c1ef5-25f86eb2") '('"_wide_channels" $6)))
(let $8 (DqPhyStage '($5) (lambda '($17) (block '(
  (let $18 (lambda '($22 $23 $24) $23 $22 $24))
  (let $19 (AggBlockApplyState '"count_all" $1 $1))
  (let $20 (BlockMergeFinalizeHashed (WideMap (ToFlow $17) $18) '('0) '('($19 '1))))
  (let $21 (lambda '($25 $26 $27) $26 $25 $27))
  (return (FromFlow (WideFromBlocks (WideMap (WideSortBlocks $20 '('('0 (Bool 'true)))) $21))))
))) $7))
(let $9 (DqCnMerge (TDqOutput $8 '0) '('('1 '"Asc"))))
(let $10 (DqPhyStage '($9) (lambda '($28) (FromFlow (NarrowMap (ToFlow $28) (lambda '($29 $30) (AsStruct '('"column1" $29) '('"o_comment" $30)))))) '('('"_logical_id" '975) '('"_id" '"6cd318f4-ed6cdb59-b150030c-5fa0b528"))))
(let $11 '($4 $8 $10))
(let $12 (DqCnResult (TDqOutput $10 '0) '('"o_comment" '"column1")))
(return (KqpPhysicalQuery '((KqpPhysicalTx $11 '($12) '() '('('"type" '"generic")))) '((KqpTxResultBinding (ListType $6) '0 '0)) '('('"type" '"query"))))
)

No WideCombiner.

Query with spilling enabled (default):

SELECT a.o_comment, count(*) FROM `/local/tpch/string_s1/orders` a group by a.o_comment order by a.o_comment;

AST:

(
(let $1 (DataType 'Uint64))
(let $2 '('"o_comment" (DataType 'String)))
(let $3 '('('"_logical_id" '564) '('"_id" '"9d1544ef-ab4c0bf1-a5bff85a-9b645cd3") '('"_wide_channels" (StructType '('_yql_agg_0 $1) $2))))
(let $4 (DqPhyStage '() (lambda '() (block '(
  (let $13 (KqpTable '"/local/tpch/string_s1/orders" '"72075186232723360:7" '"" '1))
  (let $14 '('"o_comment"))
  (let $15 (KqpBlockReadOlapTableRanges $13 (Void) $14 '() '() (lambda '($16) (TKqpOlapAgg $16 '('('_yql_agg_0 '"count" '"*")) $14))))
  (return (FromFlow $15))
))) $3))
(let $5 (DqCnHashShuffle (TDqOutput $4 '0) '('1)))
(let $6 (StructType '('"column1" $1) $2))
(let $7 '('('"_logical_id" '951) '('"_id" '"1c5e332e-350a26f0-5095a76f-f026fbab") '('"_wide_channels" $6)))
(let $8 (DqPhyStage '($5) (lambda '($17) (block '(
  (let $18 (lambda '($29 $30) $30 $29))
  (let $19 (WideCombiner (WideFromBlocks (ToFlow $17)) '"" (lambda '($20 $21) $21) (lambda '($22 $23 $24) $23) (lambda '($25 $26 $27 $28) (AggrAdd $26 $28)) $18))
  (return (FromFlow (WideSort $19 '('('1 (Bool 'true))))))
))) $7))
(let $9 (DqCnMerge (TDqOutput $8 '0) '('('1 '"Asc"))))
(let $10 (DqPhyStage '($9) (lambda '($31) (FromFlow (NarrowMap (ToFlow $31) (lambda '($32 $33) (AsStruct '('"column1" $32) '('"o_comment" $33)))))) '('('"_logical_id" '963) '('"_id" '"4bec9f97-cf2c703b-8ec5faa0-730c728a"))))
(let $11 '($4 $8 $10))
(let $12 (DqCnResult (TDqOutput $10 '0) '('"o_comment" '"column1")))
(return (KqpPhysicalQuery '((KqpPhysicalTx $11 '($12) '() '('('"type" '"generic")))) '((KqpTxResultBinding (ListType $6) '0 '0)) '('('"type" '"query"))))
)

WideCombiner is presented

More complex request (tpch q4):

https://paste.yandex-team.ru/9d65cdbb-95e4-4b4a-b983-317f6baec409

[github-actions]

⚪ 2024-08-20 09:44:01 UTC Pre-commit check for c8df2e4 has started.
⚪ 2024-08-20 09:47:01 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-20 10:14:20 UTC Build successful.

[github-actions]

⚪ 2024-08-20 09:44:04 UTC Pre-commit check for c8df2e4 has started.
⚪ 2024-08-20 09:46:58 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-20 11:35:15 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
13548	13455	0	25	33	35

🟢 2024-08-20 11:36:27 UTC Build successful.
🟢 2024-08-20 11:36:58 UTC ydbd size 5.5 GiB changed* by +952 Bytes, which is < 100.0 KiB vs main: OK

ydbd size dash	main: bbf4a41	merge: c8df2e4	diff	diff %
ydbd size	5 887 479 744 Bytes	5 887 480 696 Bytes	+952 Bytes	+0.000%
ydbd stripped size	1 478 367 600 Bytes	1 478 367 728 Bytes	+128 Bytes	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

[github-actions]

⚪ 2024-08-20 09:44:18 UTC Pre-commit check for c8df2e4 has started.
⚪ 2024-08-20 09:47:21 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-20 11:34:35 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
75476	61812	0	4	13620	40

⚪ 2024-08-20 11:41:49 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟡 2024-08-20 11:49:06 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
57 (only retried tests)	28	0	1	1	27

⚪ 2024-08-20 11:49:16 UTC Failed tests rerun (try 3) linux-x86_64-relwithdebinfo is running...
🟢 2024-08-20 11:56:12 UTC Tests successful.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
43 (only retried tests)	16	0	0	0	27

🟢 2024-08-20 11:56:20 UTC Build successful.
🟢 2024-08-20 11:57:04 UTC ydbd size 8.1 GiB changed* by -36.8 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: bbf4a41	merge: c8df2e4	diff	diff %
ydbd size	8 748 340 936 Bytes	8 748 303 232 Bytes	-36.8 KiB	-0.000%
ydbd stripped size	475 796 648 Bytes	475 796 840 Bytes	+192 Bytes	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}