Skip to content

Conversation

ChudaykinAlex
Copy link
Contributor

  1. Purpose
    The current implementation has an aggregate function LIST which concatenates multiple row fields into a blob. The SQL standard has a similar function called LISTAGG. The major difference is that it also supports the ordered concatenation.

  2. Syntax and rules

<listagg set function> ::=
LISTAGG <left paren> [ <set quantifier> ] <character value expression> <comma> <listagg separator> [ <listagg overflow clause> ] <right paren> <within group specification>

<listagg separator> ::= 
<character string literal>

<listagg overflow clause> ::=
ON OVERFLOW <overflow behavior>

<overflow behavior> ::=
ERROR | TRUNCATE [ <listagg truncation filler> ] <listagg count indication>

<listagg truncation filler> ::=
<character string literal>

<listagg count indication> ::=
WITH COUNT | WITHOUT COUNT

<within group specification> ::=
WITHIN GROUP <left paren> ORDER BY <sort specification list> <right paren>

The legacy LIST syntax is preserved for backward compatibility, LISTAGG is added to cover the standard features.

There is a <listagg overflow clause> rule in the standard, which is intended to output an error when the output value overflows. Since the LIST function always returns a BLOB, it was decided that this rule would be meaningless. It was not implemented and silently ignored if specified.

If DISTINCT is specified for LISTAGG, then ORDER BY <sort specification list> must fully match <character value expression>

If DISTINCT is specified, the presence of WITHIN GROUP must obey the restriction and will not affect the subsequent code execution.

  1. Examples:
CREATE TABLE TEST_T
	(COL1 INT, COL2 VARCHAR(2), COL3 VARCHAR(2), COL4 VARCHAR(2), COL5 BOOLEAN, COL6 VARCHAR(2)
	CHARACTER SET WIN1251);
COMMIT;
INSERT INTO TEST_T values(1, 'A', 'A', 'J', false, 'П');
INSERT INTO TEST_T values(2, 'B', 'B', 'I', false, 'Д');
INSERT INTO TEST_T values(3, 'C', 'A', 'L', true,  'Ж');
INSERT INTO TEST_T values(4, 'D', 'B', 'K', true,  'Й');
COMMIT;

SELECT LISTAGG (ALL COL4, ':') AS FROM TEST_T;
=======
J:I:L:K

SELECT LISTAGG (DISTINCT COL4, ':') FROM TEST_T;
========
I:J:K:L

SELECT LISTAGG (DISTINCT COL3, ':') FROM TEST_T;
====
A:B

SELECT LISTAGG (DISTINCT COL3, ':') WITHIN GROUP (ORDER BY COL2) FROM TEST_T;
====
A:B

SELECT LISTAGG (DISTINCT COL3, ':') WITHIN GROUP (ORDER BY COL2 DESCENDING) FROM TEST_T;
====
A:B

SELECT LISTAGG (COL2, ':') WITHIN GROUP (ORDER BY COL2 DESCENDING) FROM TEST_T;
=======
D:C:B:A

SELECT LISTAGG (COL4, ':') WITHIN GROUP (ORDER BY COL3 DESC) FROM TEST_T;
=======
I:K:J:L

SELECT LISTAGG (COL3, ':') WITHIN GROUP (ORDER BY COL5 ASCENDING) FROM TEST_T;
=======
A:B:A:B

SELECT LISTAGG (COL4, ':') WITHIN GROUP (ORDER BY COL3 ASC) FROM TEST_T;
=======
J:L:I:K

SELECT LISTAGG (ALL COL2) WITHIN GROUP (ORDER BY COL4) FROM TEST_T;
=======
B,A,D,C

SELECT LISTAGG (COL2, ':') WITHIN GROUP (ORDER BY COL3 DESC, COL4 ASC) FROM TEST_T;
=======
B:D:A:C

SELECT LISTAGG (COL2, ':') WITHIN GROUP (ORDER BY COL3 DESC, COL4 DESC) FROM TEST_T;
=======
D:B:C:A

SELECT LISTAGG (COL2, ':') WITHIN GROUP (ORDER BY COL3 ASC, COL4 DESC) FROM TEST_T;
=======
C:A:D:B

SELECT LISTAGG (ALL COL6, ':')FROM TEST_T;
=======
П:Д:Ж:Й

SELECT LISTAGG (ALL COL6, ':') WITHIN GROUP (ORDER BY COL2 DESC) FROM TEST_T;
=======
Й:Ж:Д:П

SELECT LISTAGG (ALL COL2, ':') WITHIN GROUP (ORDER BY COL6) FROM TEST_T;
=======
B:C:D:A

INSERT INTO TEST_T values(5, 'E', NULL, NULL, NULL, NULL);
INSERT INTO TEST_T values(6, 'F', 'C', 'N', true, 'К');

SELECT LISTAGG (ALL COL2, ':') WITHIN GROUP (ORDER BY COL3) FROM TEST_T;
===========
E:A:C:B:D:F

SELECT LISTAGG (ALL COL2, ':') WITHIN GROUP (ORDER BY COL3 NULLS LAST) FROM TEST_T;
===========
A:C:B:D:F:E

SELECT LISTAGG (ALL COL2, ':') WITHIN GROUP (ORDER BY COL6 NULLS FIRST) FROM TEST_T;
===========
E:B:C:D:F:A

SELECT LISTAGG (DISTINCT COL3, ':') WITHIN GROUP (ORDER BY COL2) FROM TEST_T;
========
Statement failed, SQLSTATE = 42000
SQL error code = -104
-Invalid command
-Sort-key of the ORDER BY specification must match the argument list

@sim1984
Copy link

sim1984 commented Aug 6, 2025

This PR implements one of the aggregate functions (LISTAGG), which depends on the order of the input stream records. The full list can be seen here: #7632

@dyemanov dyemanov self-requested a review August 21, 2025 06:56
INSERT INTO TEST_T values(4, 'D', 'B', 'K', true, 'Й');
COMMIT;

SELECT LISTAGG (ALL COL4, ':') AS FROM TEST_T;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the syntax format section, <within group specification> is mandatory but does not exist in some examples.

=======
C:A:D:B

SELECT LISTAGG (ALL COL6, ':')FROM TEST_T;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
SELECT LISTAGG (ALL COL6, ':')FROM TEST_T;
SELECT LISTAGG (ALL COL6, ':') FROM TEST_T;

for (auto& nodeOrder : sort->expressions)
{
dsc toDesc = *(descOrder++);
toDesc.dsc_address = data + (IPTR)toDesc.dsc_address;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
toDesc.dsc_address = data + (IPTR)toDesc.dsc_address;
toDesc.dsc_address = data + (IPTR) toDesc.dsc_address;

Comment on lines +454 to +456
if (IS_INTL_DATA(fromDsc))
INTL_string_to_key(tdbb, INTL_TEXT_TO_INDEX(fromDsc->getTextType()),
fromDsc, &toDesc, INTL_KEY_UNIQUE);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (IS_INTL_DATA(fromDsc))
INTL_string_to_key(tdbb, INTL_TEXT_TO_INDEX(fromDsc->getTextType()),
fromDsc, &toDesc, INTL_KEY_UNIQUE);
if (IS_INTL_DATA(fromDsc))
{
INTL_string_to_key(tdbb, INTL_TEXT_TO_INDEX(fromDsc->getTextType()),
fromDsc, &toDesc, INTL_KEY_UNIQUE);
}

}

dsc toDesc = asb->desc;
toDesc.dsc_address = data + (IPTR)toDesc.dsc_address;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
toDesc.dsc_address = data + (IPTR)toDesc.dsc_address;
toDesc.dsc_address = data + (IPTR) toDesc.dsc_address;

if (distinct)
desc.dsc_address = data + (asb->intl ? asb->keyItems[1].getSkdOffset() : 0);
else
desc.dsc_address = data + (IPTR)asb->desc.dsc_address;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
desc.dsc_address = data + (IPTR)asb->desc.dsc_address;
desc.dsc_address = data + (IPTR) asb->desc.dsc_address;

if (sort && distinct)
{
ValueExprNode* const sortNode = *sort->expressions.begin();
if (!arg->sameAs(sortNode, false) || sort->expressions.getCount() > 1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should they be identical? Why?
I think it should have like GROUP BY rules.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Currently, we sort only once and this is a good bonus. If we follow your suggestion and allow slightly different expressions, then we should either sort twice or ignore the user-specified ordering after DISTINCT. BTW, in this PR it also seems to be ignored -- LISTAGG(DISTINCT COL) WITHIN GROUP (ORDER BY COL DESC) would produce ASC-ordered output. But I suppose it should work the same way as for plain SELECT DISTINCT(COL) FROM T ORDER BY COL DESC, i.e. respect the ORDER BY ordering and optimize the sorts (merge two sorts into one) only if they fully (expressions / directions / NULLs placement) match each other.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why direction and null placement are important for DISTINCT?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nulls placement is not important, I agree, as NULLs are skipped by all aggregate functions.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By standard, DISTINCT eliminates duplicates in the ordered (if specified) result set, it should not change the user-defined ordering. Why do you think LISTAGG should behave differently?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It shouldn't but I don't quite understand why you said

i.e. respect the ORDER BY ordering and optimize the sorts (merge two sorts into one) only if they fully (expressions / directions / NULLs placement) match each other.

BTW, IMHO, sort->expressions.getCount() > 1 condition here is not needed as it is fine for distinct to use only first sorting segment.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, sorry, my bad. Surely, direction for the combined sort should be taken from ORDER BY -- like we already do in Optimizer::checkSorts().

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nulls placement is not important, I agree, as NULLs are skipped by all aggregate functions.

With my suggestion to use same existing rules, the DISTINCT LISTAGG expression may be something like COALESCE(field, 'z') with a ORDER BY Z`.

const auto keyCount = aggNode->sort->expressions.getCount() * 2;
sort_key_def* sortKey = asb->keyItems.getBuffer(keyCount);

auto const* direction = aggNode->sort->direction.begin();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const auto* for consistency, please.

// per function.
return aggInfo.blr == o->aggInfo.blr && aggInfo.name == o->aggInfo.name &&
distinct == o->distinct && dialect1 == o->dialect1;
distinct == o->distinct && dialect1 == o->dialect1 && sort == o->sort;;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does not look correct the comparation of pointer address here for sort.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ChudaykinAlex IMO, the sort node should be added to the list by ListAggNode::getChildren() -- this way you may remove the sort comparison in dsqlMatch() and also remove doPass2(sort) in pass2(), as the sort node will be processed by the inherited methods automagically.

@ChudaykinAlex
Copy link
Contributor Author

Thanks for the recommendations, I'll fix it soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants