driver: implement new driver subqueuettl driver for processing tasks from named subqueues in one tube space.

Add subqueuettl driver with consumer groups

Some workloads need to create queues dynamically
with arbitrary names. Creating a separate tube
for each queue requires converting those names
to valid Tarantool space names and granting access
whenever a new tube is created. This adds operational
overhead and makes access management harder.

The utube model is otherwise well suited for this use case:
it stores multiple logical queues in one space
and guarantees sequential processing inside each queue.
However, it does not support taking a task from
a specific utube, so it cannot be used when
consumers must be assigned to particular dynamically named queues.

Added subqueuettl as a TTL-aware alternative.
It stores multiple dynamically named subqueues
in one space, and both put() and take() require
a subqueue name. This keeps task selection isolated
while retaining TTL, TTR, priority, delay, and ready-buffer support.

Author: ThCompiler

CHANGELOG.md

@@ -13,6 +13,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 - Optional `consumer_group(opts, task)` driver API for routing waiting
   consumers to matching tasks.
 - Support extra statistics from driver API.
+- Add the `subqueuettl` driver for processing tasks from named subqueues in one tube space.
 
 ### Fixed
 

README.md

@@ -16,6 +16,7 @@ align="right">
   * [fifottl \- a simple priority queue with support for task time to live](#fifottl---a-simple-priority-queue-with-support-for-task-time-to-live)
   * [utube \- a queue with sub\-queues inside](#utube---a-queue-with-sub-queues-inside)
   * [utubettl \- extension of utube to support ttl](#utubettl---extension-of-utube-to-support-ttl)
+  * [subqueuettl \- TTL subqueues in one space](#subqueuettl---ttl-subqueues-in-one-space)
 * [The underlying spaces](#the-underlying-spaces)
   * [Fields of the \_queue space](#fields-of-the-_queue-space)
   * [Fields of the \_queue\_consumers space](#fields-of-the-_queue_consumers-space)
@@ -302,6 +303,41 @@ seconds; if `ttr` is not specified, it is set to the same as `ttl`
 is changed to 'ready' so another worker may take it)
   * `delay` - time to wait before starting to execute the task, in seconds
 
+## `subqueuettl` - TTL subqueues in one space
+
+`subqueuettl` stores independent TTL subqueues in one Tarantool space. It
+does not provide an operation to take a task from all subqueues. The driver
+supports the `memtx` engine only; `vinyl` is not supported.
+
+The following options can be specified when putting a task in a
+`subqueuettl` queue:
+
+  * `subqueue` - required name of the subqueue.
+  * `pri` - task priority (`0` is the highest priority and is the default).
+  * `ttl` - numeric time to live in seconds. If omitted, it is set to infinity.
+  * `ttr` - numeric time allotted to process a task in seconds. If omitted, it
+    is set to the same value as `ttl`.
+  * `delay` - time in seconds to wait before a task becomes ready.
+
+Both `put()` and `take()` require `opts.subqueue`:
+
+```lua
+local tube = queue.create_tube('sites', 'subqueuettl')
+tube:put('some task', {subqueue = 'example_subqueue'})
+local task = tube:take(10, {subqueue = 'example_subqueue'})
+```
+
+`subqueuettl` adds per-subqueue statistics under the common `extra` field:
+
+```lua
+local stats = queue.statistics('sites')
+local example_stats = stats.extra.subqueues['example_subqueue']
+```
+
+Each subqueue entry contains `ready`, `taken`, `buried`, `delayed`, and `total`
+task counts. The subqueue name registry is best-effort; a registry write error
+does not fail the queue operation, and a stale name can remain with zero counts.
+
 # The underlying spaces
 
 The queue system consists of fibers, IPC channels, functions, and spaces.
@@ -660,6 +696,10 @@ than any other tuple which also has `task_state` = 'r'.
 If there is no such task, and timeout was specified, then
 the job waits until a task becomes ready or the timeout expires.
 
+The options, if specified, must be one or more of the options described
+above
+(`subqueue`, required name of the subqueue to take a task from).
+
 Effect: the value of `task_state` changes to 't' (taken).
 The `take` request tells the system that the task is being worked on.
 It should be followed by an `ack` request when the work is finished.
@@ -860,6 +900,14 @@ queue.statistics('list_of_sites')
      bury: 1
      put: 2
      delete: 1
+   extra:
+     subqueues:
+       example.com:
+         ready: 0
+         taken: 0
+         buried: 0
+         delayed: 0
+         total: 0
 ...
 ```
 
@@ -1007,7 +1055,8 @@ API:
 which is passed on to the user (removes the administrative fields)
 * `tube:put(data[, opts])` - puts a task into the queue.
 Returns a normalized task which represents a tuple in the space
-* `tube:take()` - sets the task state to 'in progress' and returns the task.
+* `tube:take([opts])` - sets the task state to 'in progress' and returns the
+  task. `subqueuettl` requires `opts.subqueue`.
 If there are no 'ready' tasks in the queue, returns nil.
 * `tube:delete(task_id)` - deletes a task from the queue.
 Returns the original task with a state changed to 'done'

queue-scm-1.rockspec

@@ -22,6 +22,7 @@ build = {
         ['queue.abstract.queue_state']       = 'queue/abstract/queue_state.lua',
         ['queue.abstract.driver.fifottl']    = 'queue/abstract/driver/fifottl.lua',
         ['queue.abstract.driver.utubettl']   = 'queue/abstract/driver/utubettl.lua',
+        ['queue.abstract.driver.subqueuettl'] = 'queue/abstract/driver/subqueuettl.lua',
         ['queue.abstract.driver.fifo']       = 'queue/abstract/driver/fifo.lua',
         ['queue.abstract.driver.utube']      = 'queue/abstract/driver/utube.lua',
         ['queue.abstract.driver.limfifottl'] = 'queue/abstract/driver/limfifottl.lua',

queue/CMakeLists.txt

@@ -22,5 +22,7 @@ install(FILES ${CMAKE_CURRENT_SOURCE_DIR}/abstract/driver/fifottl.lua
         DESTINATION ${TARANTOOL_INSTALL_LUADIR}/${PROJECT_NAME}/abstract/driver/)
 install(FILES ${CMAKE_CURRENT_SOURCE_DIR}/abstract/driver/utubettl.lua
         DESTINATION ${TARANTOOL_INSTALL_LUADIR}/${PROJECT_NAME}/abstract/driver/)
+install(FILES ${CMAKE_CURRENT_SOURCE_DIR}/abstract/driver/subqueuettl.lua
+        DESTINATION ${TARANTOOL_INSTALL_LUADIR}/${PROJECT_NAME}/abstract/driver/)
 install(FILES ${CMAKE_CURRENT_SOURCE_DIR}/abstract/driver/limfifottl.lua
         DESTINATION ${TARANTOOL_INSTALL_LUADIR}/${PROJECT_NAME}/abstract/driver/)

queue/abstract.lua

@@ -917,7 +917,7 @@ local function build_stats(space)
         take = 0, touch = 0,
         -- for *ttl queues only
         ttl  = 0, ttr   = 0, delay   = 0,
-    }}
+    }, extra = {}}
 
     local st = rawget(queue.stat, space) or {}
     local idx_tube = 1