YQL-16402: Implement key cache for Top and TopSort nodes #4761
Conversation
|
⚪
|
|
⚪
|
|
⚪
|
|
⚪
|
|
⚪
|
|
⚪
|
|
⚪
|
|
⚪
|
This patch fixes the issue with the excess calculations of the sorting
keys for TopSort computation node. While compiling TopSort runtime node,
the input source (either stream or flow) is transformed into {key: item}
mapping, and the resulting iterable is processed by both KeepTop and
TopSort nodes using the trivial key extractor, that obtains the key
value as the first component from the item of the resulting iterable.
The result yielded by TopSort is transformed back returning only the
second component from the item of the mapping being processed.
As a result of the changed described above, <keyExtractor> callable is
invoked once for each item of the given input iterable.
A static UDF module is used to test the fix. It provides an echo
function, that increments TLS counter each time being called. When the
processing is finished, the value of this counter is compared with the
number of the items in the Stream/Flow.
This patch fixes the issue with the excess calculations of the sorting
keys for Top computation node. While compiling Top runtime node, the
input source (either stream or flow) is transformed into {key: item}
mapping, and the resulting iterable is processed by both KeepTop and Top
nodes using the trivial key extractor, that obtains the key value as the
first component from the item of the resulting iterable. The result
yielded by Top is transformed back returning only the second component
from the item of the mapping being processed.
As a result of the changed described above, <keyExtractor> callable is
invoked once for each item of the given input iterable.
A static UDF module is used to test the fix. It provides an echo
function, that increments TLS counter each time being called. When the
processing is finished, the value of this counter is compared with the
number of the items in the Stream/Flow.
igormunkin
changed the title
igormunkin
changed the title
|
⚪
|
|
⚪
|
2 participants
This patchset fixes the issue with the excess calculations of the sorting keys for
Top/TopSortcomputation node. While compilingTop/TopSortruntime node, the input source (either stream or flow) is transformed into{key: item}mapping, and the resulting iterable is processed by bothKeepTopandTop/TopSortnodes using the trivial key extractor, that obtains the key value as the first component from the item of the resulting iterable. The result yielded byTop/TopSortis transformed back returning only the second component from the item of the mapping being processed.As a result of the changed described above,
<keyExtractor>callable is invoked once for each item of the given input iterable.A static UDF module is used to test the fix. It provides an echo function, that increments TLS counter each time being called. When the processing is finished, the value of this counter is compared with the number of the items in the Stream/Flow.
Changelog category