Feature Request
Is your feature request related to a problem? Please describe:
In nextgen, a single cluster will be shared by multiple tenants, each user cluster will have its own keyspace, we want all keyspaces share the same resource pool to run resource intensive DXF tasks such as import-into and add-index:
- To make management of resources for DXF tasks easier, scale in/out, etc. No need dedicated DXF nodes for each keyspace.
- To fully utilize resources and reduce cost
- Different steps of different DXF tasks might have different requirement for TiDB resources, by sharing same resource pool we can better utilize those resources
- Dedicated DXF nodes have the drawback that if there not too many load in this keyspace, it's wasting, if we use a scale-out-on-load strategy, it will be less responsive for smaller task, as scale out takes time too.
Describe the feature you'd like:
Describe alternatives you've considered:
Dedicated DXF for each keyspace
Teachability, Documentation, Adoption, Migration Strategy:
As DXF is shared by many user keyspaces, it cannot run on any of the user keyspaces, so we run it in a separate SYSTEM keyspace. This SYSTEM keyspace will be used as the keyspace for all shared services among a nextgen cluster, but currently only DXF service use it.
As user keyspaces depends on SYSTEM keyspace now, we need to make sure SYSTEM keyspace exist before user keyspace start serving any user traffic. So for the deployment of nextgen cluster, we need to:
- Deploy PD/TiKV and other components, wait them to be ready to serve TiDB access.
- Deploy SYSTEM keyspace, wait it fully bootstrapped.
- Deploy user keyspaces, they can be deployed concurrently.
During upgrade, we also need to follow above order, i.e. We need to upgrade the SYSTEM keyspace first, then user keyspace.
In DXF service, we are servicing all user keyspaces using the same DXF meta store, so we need some adaption to task meta abstraction:
- We need add a keyspace column to the task table to identify the source keyspace of the task.
- The task_key of task also need to add keyspace name as prefix to unique identify a task.
We don't need to run DXF components in user keyspace, only run them in SYSTEM keyspace.
Import-into and add-index need integrate DXF service, refer #60418
Subtasks
implementing DXF service and cross keyspace access
syncing info schema for cross keyspace
import-into/add-index integration with DXF service
variable changes and enhancements
use reserved ID for SYSTEM schemas and tables
DXF service SLO and resource auto scale
metering
metrics and alerts
HA or retry related
Fix nextgen CI and test related
Feature Request
Is your feature request related to a problem? Please describe:
In nextgen, a single cluster will be shared by multiple tenants, each user cluster will have its own keyspace, we want all keyspaces share the same resource pool to run resource intensive DXF tasks such as import-into and add-index:
Describe the feature you'd like:
Describe alternatives you've considered:
Dedicated DXF for each keyspace
Teachability, Documentation, Adoption, Migration Strategy:
As DXF is shared by many user keyspaces, it cannot run on any of the user keyspaces, so we run it in a separate SYSTEM keyspace. This SYSTEM keyspace will be used as the keyspace for all shared services among a nextgen cluster, but currently only DXF service use it.
As user keyspaces depends on SYSTEM keyspace now, we need to make sure SYSTEM keyspace exist before user keyspace start serving any user traffic. So for the deployment of nextgen cluster, we need to:
During upgrade, we also need to follow above order, i.e. We need to upgrade the SYSTEM keyspace first, then user keyspace.
In DXF service, we are servicing all user keyspaces using the same DXF meta store, so we need some adaption to task meta abstraction:
We don't need to run DXF components in user keyspace, only run them in SYSTEM keyspace.
Import-into and add-index need integrate DXF service, refer #60418
Subtasks
implementing DXF service and cross keyspace access
syncing info schema for cross keyspace
import-into/add-index integration with DXF service
variable changes and enhancements
use reserved ID for SYSTEM schemas and tables
DXF service SLO and resource auto scale
metering
metrics and alerts
HA or retry related
Fix nextgen CI and test related