-
Notifications
You must be signed in to change notification settings - Fork 739
Closed
Labels
area/queryprocessorQuery processor issues and requestsQuery processor issues and requests
Description
After this PR we're now able to track block (arrow) memory in mkql allocator:
#6731
But at the moment this only works for values, created inside computations graph. We need to also track external values (inputs) for compute actor. There are currently two kind of inputs we need to handle:
- Block sources (column shards). Simple solution is to add mkql header for arrow blocks (with copy) in CA and track them in mkql allocator. More complex and efficient solution is to produce values already with mkql header during reads (scans) in column shards.
- Block channels. For local events, values should already have headers so we can just add them to mkql allocator. For remote events, there are again to solutions. Simple one is to just add header with a copy of data on the receiver side. We already have to reallocate data from IC, so actually there shouldn't be any additional copies. Complex one is to make IC handle data with header and required alignment.
Metadata
Metadata
Assignees
Labels
area/queryprocessorQuery processor issues and requestsQuery processor issues and requests