`ets:lookup_elements`

[ansd]

Feature Request

Do you think it would make sense to add an ets:lookup_elements API function in the Erlang/OTP library, which works similar to the existing ets:lookup_element/3 and ets:lookup_element/4, just that I can pass a list of positions to look up? For example:

ets:lookup_elements(Tab, Key, Positions)

where Positions is a list of integers.

Alternatives

A practical example that prompted this feature request is RabbitMQ looking up destination queue information after routing. Routing happens for every incoming message and there can be multiple queues a message gets routed to. So these destination queue infos need to be looked up hundreds of thousands of times per second. Today, there is a queue ETS table containing a #queue{} record with 21 fields. (Some fields are themselves tuples). To send a message to a destination queue, I need exactly 3 fields out of these 21 fields.

I can think of a few options all having downsides:

1. This is what RabbitMQ does today: Use ets:lookup/2 to get the full record, then just extract the fields I'm interested in. Problem: I don't want to copy the entire queue record out of ETS since that's expensive.
2. Call ets:lookup_element/3 multiple times (in the example above three times). Problem: I don't want pay the penalty of multiple such calls (hashing the Key isn't for free, in the RabbitMQ example the key itself a 4-element tuple {resource, VirtualHostBinary, queue, QueueNameBinary}).
3. Use ets:match/2. From my experiment this is similarly slow as ets:select/2 even if I pass the full Key. The CPU flamegraph shows that the following call stack is very expensive: ets_match_2, ets_select2, db_select_hash, match_traverse.constprop.0, db_match_compile. So it seems ets:match/2 also does some match spec compilation under the hood. This is how I called this function:

 Pattern = {'_', QueueName, '_', '_','_','_','$1','_','_','_','_','_','_','_','_','_','_','_','$2', '$3', '_'},
 ets:match(queue, Pattern)

4. This is the solution I will go for due to the lack of an ets:lookup_elements API function: Keep an additional ETS table (projection in Khepri) with a copy of the few elements of the original ETS table such that I can get all the elements I'm interested in in one go. Downside: Additional memory usage for that ETS table copy and that ETS table also need to be "synced" with its original data source.

[jhogberg]

Thanks for bringing this up, feel free to make a PR. :-)