Skip to content

roadmap.md

Latest commit

 

History

History
164 lines (157 loc) · 12.9 KB

roadmap.md

File metadata and controls

164 lines (157 loc) · 12.9 KB
Raw

Roadmap

Currently working on..

  • Support for Redis Pub-Sub
  • Committed through FW–GB (ANN HNSW, AOF graph, Redis DUMP/RESTORE, search eviction, Entry TTL retirement, repl publish barrier). Baseline A–E green; main synced with origin.
  • Release 0.7.0 (Batch GC-rel): productization cut (GC–GH, GK, GL) + docs/ops.md. See CHANGELOG.md.
  • Next: optional cluster/Sentinel depth — GN+. See root TODO.md.

Persistence / search letter batches (recent)

Tracked in detail in root TODO.md. High level:

  • FT write classification, aliases, AOF/RDB FT schema (BT–BY, CA)
  • Scratch-load AOF/RDB (preserve target on Err); load quiesce + WATCH bump (CB–CD)
  • FT merge schema/alias equality on clash (CG); multi-DB LOADING gate (CK)
  • Shared FT.CREATE parser; HNSW EF_CONSTRUCTION AOF/RDB; ACL @search (CA, CE)
  • Multi-DB lock-step keyspace install (Batch DR: epoch write + stable-view read; LOADING + SYNC deny)
  • Multi-DB install panic rollback + walk audit (Batch DS: retain discards + Drop restore; AOF rewrite under epoch read; mid-fill single-DB residual)
  • MIGRATE absolute expire + residual closeout (Batch DT: PXAT/PEXPIREAT; LOADING/UI string-contract accepted)
  • Cluster ownership epoch + gossip (Batch DU: per-slot epoch; CLUSTER OWNERS/EPOCH; MEET+heartbeat merge)
  • Dest-first dual-end NODE + epoch fence (Batch DV: no MOVED-to-IMPORTING on dest fail; stale gossip reject)
  • Multi-master fail quorum (Batch DW: pfail→fail votes; FAILREPORTS; ≤2 masters single-observer)
  • Reshard planner (Batch DX: CLUSTER RESHARD PLAN|AUTO; greedy donors; remote RESP execute)
  • Multi-replica failover election (Batch DY: max replica id; ROLEMAP + MEETPEER role)
  • Loser re-point at election winner (Batch DZ: topology REPLICATE + REPLICAOF)
  • Offset-based replica election (Batch EA: ROLEMAP offset; max offset then max id)
  • Replica-priority election (Batch EB: --cluster-replica-priority; 0 never promote)
  • CLUSTER FAILOVER FORCE/TAKEOVER (Batch EC: operator manual promote + claim)
  • Cluster ops helpers (Batch ED: COUNTKEYSINSLOT/GETKEYSINSLOT/REPLICAS/BUMPEPOCH)
  • Slot bootstrap (Batch EE: ADDSLOTS/DELSLOTS/FLUSHSLOTS; unbound empty owner)
  • Slot range bootstrap (Batch EF: ADDSLOTSRANGE/DELSLOTSRANGE)
  • Topology forget/reset (Batch EG: FORGET; RESET SOFT|HARD key wipe)
  • Dual-end NODE partial_source ASK safety (Batch EH: re-assert MIGRATING; COUNT-FAILURE-REPORTS)
  • CLUSTER SHARDS / LINKS (Batch EI: Redis-7 shard view; LINKS empty no bus)
  • Post-commit dual NODE verify + MYSHARDID (Batch EJ; partial_verify status)
  • CLUSTER SET-CONFIG-EPOCH (Batch EK; only if epoch > current)
  • Live cluster CONFIG (Batch EL: cluster-replica-priority + cluster-node-timeout)
  • CLUSTER SAVECONFIG (Batch EM: write nodes.conf)
  • nodes.conf load-on-boot (Batch EN: restore id/peers/slots/epoch; fallback single-node)
  • nodes.conf autosave (Batch EO: topology-mutating CLUSTER ops + failover claim)
  • nodes.conf live flags (Batch FL: require-full / allow-reads / announce-ip|port / replica-priority header; CONFIG SET autosave)
  • Dual-end NODE dest rollback (Batch EP: source NODE fail → rolled_back + IMPORTING)
  • cluster-require-full-coverage (Batch EQ: cluster_state fail + CLUSTERDOWN gate; CONFIG/CLI)
  • Cluster READONLY replica reads (Batch ER: serve master slots; writes still MOVED)
  • cluster-allow-reads-when-down (Batch ES: reads while cluster_state fail; writes blocked)
  • CLUSTER SLOTS lists replicas (Batch ET: master then replica endpoints per range)
  • cluster-announce-ip/port (Batch EU: client-facing addr in NODES/MEET/MOVED myself)
  • CLUSTER SLOT-STATS (Batch EV: SLOTSRANGE key-count + ORDERBY/LIMIT)
  • Sentinel-lite (Batch EW: MONITOR/s_down/GET-MASTER-ADDR/FAILOVER; no multi-sentinel ODOWN)
  • Sentinel ODOWN quorum (Batch EX: MEET + IS-MASTER-DOWN-BY-ADDR votes; failover on o_down)
  • Dual-end NODE preflight (Batch EY: prepare MYID+owner check; failed_preflight)
  • Dual-end NODE wire 2PC slice 1 (Batch FB: SETSLOT PREPARE/ABORTPREPARE + dest-first commit; failed_prepare)
  • Dual-end NODE wire 2PC slice 2 (Batch FH: prepare-epoch + TTL + CHECKPREPARE commit re-check; soft clear; failed_prepare:recheck)
  • NODE 2PC durable prepare (Batch FO: nodes.conf # prepare; wall-clock TTL; COMMITPREPARE atomic check+NODE)
  • Pipeline SET perf investigation (Batch FI: AOF-off unlock + encode/argv/+OK cuts; ~+25% SET P=16 on M3 Pro; residual FI-2 repl backlog)
  • Sentinel conf persistence (Batch EZ: FLUSHCONFIG + load sentinel.conf; autosave)
  • Sentinel hello bus lite (Batch FA: HELLO CSV + PUBLISH + peer exchange)
  • Sentinel promote-success gate (Batch FC: FAILOVER/REPLICAOF/ROLE=master; failover_in_progress)
  • HNSW graph-based ANN search (Batch CQ; layer-0 edges + ef_search)
  • HNSW remove/unlink + insert-time force-keep + update rewire (Batch CS)
  • HNSW hard-delete bridge repair — closest-peer reconnect (Batch CT; 2-chain)
  • HNSW undirected former snapshot + multi-way spanning bridge reconnect (Batch CU)
  • HNSW NN-path bridge branch test + must_keep prune safety (Batch CW)
  • HNSW recall@k unit gate + N=300 indicative micro (Batch CV; not large-N ANN win)
  • HNSW tighter recall gate + optional larger-N median bench (Batch DK; #[ignore] N=5000)
  • HNSW multi-layer insert (Batch FF: geometric level assignment + upper-layer SEARCH-LAYER / connect; query descent)
  • HNSW durable graph in RDB (Batch FV: KORDB v6 levels+edges+entry)
  • HNSW ANN query path (Batch FW: FT.SEARCH / query engine uses dual-written graph)
  • HNSW AOF durable graph (Batch FX: FT._LOADGRAPH rewrite/load; legacy rebuild)

Plans

  • Cluster (kore cluster) — MVP: hash slots + MOVED/CROSSSLOT/ASK + CLUSTER/ASKING stubs (single-node)
  • Cluster gossip / membership + thin failover (RESP MEET/PING, single-observer fail, replica claim slots)
  • Cluster thin slot reshard (CLUSTER MIGRATEKEYS multi-type keys + SETSLOT operator flow)
  • Cluster reshard orchestration slice (Batch DM: CLUSTER RESHARD source-side 4-step flow + range; dual-end NODE best-effort, not atomic)
  • Cluster dual-end NODE harden (Batch DN: verify+retry after RESHARD; CLUSTER RESHARD FINISH NODE-only recovery; still not 2PC)
  • Cluster RESHARD honesty (Batch DO: partial failed_keys counts; range abort-on-partial; FINISH source-keys warning; source-before-dest NODE window docs)
  • Redis key-level MIGRATE (Batch DP: COPY/REPLACE/AUTH/KEYS/timeout; shared recreate path with MIGRATEKEYS; no DUMP/RESTORE)
  • MIGRATE honesty (Batch DQ: multi-key IOERR migrated=/skipped=; typed TTL via SET PX / trailing PEXPIRE)
  • MIGRATE absolute expire (Batch DT: snapshot unix-ms end; string SET PXAT / typed PEXPIREAT; remaining-ms shrink closed)
  • Cluster automatic reshard / multi-type MIGRATE orchestration (MVP complete: PLAN/AUTO DX, epoch gossip DU, dest-first NODE DV, fail quorum DW, EY preflight, EP dest rollback, FB/FH RESP prepare/commit 2PC)
  • Sentinel promote-success gate (Batch FC: real promote required; in-process failover_in_progress)
  • FD measured benchmarks vs Valkey (docs/benchmarks.md; single-host median of 3; no portable-win claims)
  • Sentinel leader election depth (Batch FE: voted-leader / elect gate; election-timeout SM → FT)
  • Sentinel CKQUORUM live probe + probe * honesty (Batch FN; hello SUBSCRIBE residual accepted)
  • HNSW multi-layer insert (Batch FF)
  • HNSW durable graph RDB (Batch FV)
  • HNSW ANN query path (Batch FW)
  • HNSW AOF durable graph (Batch FX)
  • Unified keyspace design + KeyValue facade (Batch FG slice A; multi-map storage)
  • FG-2 physical hashes in ShardedKeyMap<KeyValue> (Cache::key_values)
  • FG-3 remaining typed containers (list/set/zset/geo/stream) + KeyspacePayload collapse; eviction samples typed from one map
  • FG-4 strings into unified KeyValue map; KeyspacePayload single key_values stream; dual-map removed
  • Primary FB–FG-4 queue complete (true single keyspace map)
  • FH NODE 2PC slice 2 (prepare-epoch + TTL + commit re-check + boot clear)
  • FI pipeline SET perf (~+25% P=16 on M3 Pro; residual FI-2 backlog / AOF-off multi-DB)
  • FI-2 AOF-off multi-DB SELECT ordering (propagate_write; backlog serialize residual → partial GB)
  • GB repl backlog write serialization (drop exclusive fullsync Mutex on hot path; atomic barrier; ordered publish under backlog residual)
  • P2 follow-ons complete (FH 2PC slice 2 · FI pipeline SET · FI-2 AOF-off SELECT)
  • FK Sentinel promote ranking (priority then ROLE offset then greatest ip:port; 0 never; closed first-replica-wins)
  • FL nodes.conf live cluster flags (require-full / allow-reads / announce / replica-priority; CONFIG SET autosave; legacy defaults)
  • FM Sentinel residual polish (live INFO slave_priority refresh; auto-failover 15s cooldown; manual FAILOVER force-bypasses)
  • FO durable NODE prepare + COMMITPREPARE (nodes.conf # prepare; dual-end atomic commit)
  • FP expire slot header — typed TTL on KeySlot.expires_at; side typed_expires removed
  • FQ unify string TTL onto KeySlot.expires_at (EXPIRE/TTL/active expire/volatile sample)
  • FR compiler nits + Sentinel suite ephemeral ports (Later/backlog hygiene)
  • FS residual lib warnings (unused imports / replica-feed flag / search _weight)
  • FT Sentinel election-timeout SM (campaign epoch reuse + timed re-campaign)
  • FU expire dual-write cleanup — slot-only string TTL; drop Entry.expires_at RMW mirror
  • FZ search-doc eviction special — proportional allkeys sample; free Search category
  • GA retire Entry.expires_at — full field removal; slot-only TTL
  • FV HNSW durable graph in RDB (KORDB v6; snapshot/apply; AOF residual rebuild)
  • FW HNSW ANN query path (query engine prefers dual-written graph; FLAT exact)
  • FX HNSW AOF graph (FT._LOADGRAPH; edge-identical rewrite/load)
  • FY–GB letter queue (DUMP/RESTORE, search eviction, Entry TTL retire, repl barrier)
  • Next (see TODO.md post-GB): GC+ productization / perf / compat
    • GC pipeline SET standalone skip + hot-path cuts
    • GD CommandId enum dispatch + ACL lower alloc
    • GE ordered deferred repl fan-out
    • GF full re-bench vs Valkey 9
    • GG MIGRATE via DUMP/RESTORE (core types)
    • GH geo/stream Redis-framed DUMP; MIGRATE dump path for all types
    • GK client smoke CI (redis-cli + redis-py)
    • GL TLS mTLS / dual listener / replica TLS
    • GI Redis Functions library (FUNCTION LOAD / real FCALL)
    • GJ redis.setresp + lua-time-limit + SCRIPT KILL
    • GT TF-IDF field-weight scoring + FT.SEARCH WITHSCORES
    • GU adaptive HNSW ef + mid-N large-k recall gate
    • GM admin metrics/UI Bearer·Basic auth + TLS
    • GN per-slot epochs in nodes.conf
    • GO dest CHECKPREPARE folded into COMMITPREPARE
    • GQ parallel Sentinel probes
    • GP Kore peer bus lite for NODE 2PC
    • GR HNSW max_m=1 spanning floor on bridge reconnect
    • GS search-doc access-touch for allkeys-lru/lfu
  • 데드락 감지 고급 기능
    • 크로스 프로세스 감지 (Batch DC–DE snapshot merge MVP; no transport)
    • 비동기(async) 지원
    • 커스텀 희생자 선택 전략
    • 웹 UI 모니터링 (Batch DF–DJ; residual: string-only repaint test — see TODO.md)
  • Export data to file
    • Export to 'RDB' file (Kore KORDB format; SAVE / BGSAVE)
    • Export to 'AOF' file (RESP log; BGREWRITEAOF)
  • Load data from file (init with file)
  • Async replication (SYNC + REPLICAOF / --replicaof)
  • PSYNC partial resync + backlog; replica read path (ROLE / INFO replication)
  • Multi-DB + streams RDB/AOF persistence (KORDB v3)
  • Minimal failover promote (REPLICAOF NO ONE / FAILOVER)
  • Sentinel-lite (Batch EW: subjective-down + manual/auto failover; not full Sentinel)
  • Sentinel multi-instance ODOWN lite (Batch EX: MEET peers + vote quorum)
  • Sentinel FLUSHCONFIG / load-on-boot (Batch EZ)
  • Sentinel hello bus lite (Batch FA: peer HELLO + PUBLISH on master)
  • Sentinel cross-process leader election lite (Batch FE + FT election-timeout SM)
  • Sentinel promote ranking (Batch FK) + INFO slave_priority + failover cooldown (Batch FM)
  • Sentinel CKQUORUM live PING + probe * honesty (Batch FN; hello SUBSCRIBE residual accepted)
  • Blocking XREAD / XREADGROUP (BLOCK)
  • Multi-DB replication parity (SYNC/PSYNC all DBs + SELECT apply)
  • AOF SELECT concurrency fix + atomic SELECT+cmd replication
  • CI (GitHub Actions) + benchmarks runbook
  • ACL MVP + TLS + metrics/HEALTH
  • Coordinated FAILOVER TO
  • Redlock CLI wiring; FT.SEARCH RESP + search memory; pub/sub fan-out limits