Proposal: support prefix matching and index persistence

[robocanic]

English version

Backgroud

Admin currently supports two data storage forms: in-memory and database-backed storage. Both rely on indexes to facilitate search and query operations for upper-layer components. At present, all indexes are maintained in memory. Indexing consists of two phases: index construction and index querying. Index construction is mostly triggered after Discovery and Engine components watch data changes and write them to the database, at which point they rebuild the in-memory indexes. Index querying, on the other hand, primarily originates from frontend query requests.
The current index structure is relatively simple, resembling map[idx_value][resource_key]. Different resources have different index keys, and the logical relationship used in index queries is currently limited to simple equality matches. To support prefix-based searches, the logical relationships in index queries need to be extended.
Meanwhile, when Admin is deployed with multiple replicas, due to the leader-follower architecture of Discovery and Engine components(#1380), only one replica will maintain the in-memory index. Therefore, for database-backed storage, we also need to persist the indexes to ensure that, in a multi-replica deployment, frontend queries can still access the indexes effectively.

Solutions

For memory store

If user choose to use memory for the store, the Master-Slave architecture will not work, and index will be still in memory. In this condition, we can support prefix matching by Trie.

For DB

If user choose to use DB for the store, the Discovery and Engine will run in a Master-Slave architecture. So the index need to persist in the DB to insure the data consistency. In this condition, we need to define a new table in database and performs index query for different conditions composite including prefix matching.

Tips

The current logical relationship when performs index query only includes "Equals", we need to expand the logic operator and refactor the definition of Indexer .
After rebuilding the Indxer, we need to change the usage of upper components such as Manager, Console.

中文版本