Add serializable, lazy-loadable graph stats on Plotter for GFQL optimization

[lmeyerov]

Problem

GFQL WHERE planning and pruning can blow up without cheap selectivity estimates. Today we recompute domain stats per query, and we have no persistent, reusable stats tied to a graph/Plotter instance.

Proposal

Introduce a first-class stats layer on the Plotter stack (e.g., g.stats) that is serializable and lazy-loadable. Stats should be computed on demand, cached, and optionally persisted alongside a graph bundle or stored separately for reuse across sessions.

Scope / Requirements

• Serializable: JSON (or msgpack) friendly; versioned schema; safe to round-trip across Python versions.
• Lazy-loadable: compute only when requested; allow a cache-backed mode (in-memory + optional disk).
• First-class on Plotter: accessible via g.stats with explicit compute APIs; carried with Plotter/Graphistry objects and optionally included in uploads.
• DF-native: pandas + cuDF compatible; avoid .to_pandas() in hot paths.
• Optional: zero overhead unless enabled/asked for.

Candidate Stats (common in graph engines like Neo4j, TigerGraph, etc.)

• Table cardinalities (nodes/edges)
• Per-column NDV (approx OK; HLL-style)
• Per-column min/max + null fraction
• Degree distributions or summary stats (min/max/mean/quantiles)
• Optional: per-label or per-type stats (if labels/types exist)

Why (GFQL priorities)

• Clause ordering / gating based on selectivity
• Inequality bounds pruning (min/max or quantiles)
• Semijoin thresholds for domain intersections
• Query diagnostics (explain-style stats)

Deliverables

1. Stats data model + serialization format
2. Plotter.stats API + lazy compute + caching hooks
3. Integration points in GFQL WHERE planner/executor (behind feature flags)
4. Tests for parity, persistence, and cudf/pandas compatibility

Non-goals (initial)

• Full cost-based optimizer
• Cross-graph/global stats registry

Notes

This is intended to unblock GFQL planning and pruning work without baking in a full optimizer.

[lmeyerov]

Bench findings from new q1–q9 graph-benchmark harness (pandas, graph-benchmark 100k):

• Preindexed mode (prefilter by node type + edge rel before GFQL) improved most queries vs baseline (e.g., q3 946ms→416ms, q4 841ms→288ms, q7 1.23s→411ms). Others small wins or flat.
• Presorted global views (rel/src/dst, node_type/node_id) regressed across most queries (hash joins dominate; sorting is overhead).

Implications for stats/planner:

• We need cheap selectivity stats to decide when to prefilter per-type/per-rel subgraphs vs full-graph filtering.
• Useful stats: counts by node_type / rel_type; distinct counts for key props; null-rate; maybe per-(node_type, rel_type) degree stats (avg/min/max or quantiles).
• These can be lazy-computed + cached on Plotter, and serialized for reuse across runs.

This should feed auto-mode decisions (e.g., build prefiltered subgraph only when selectivity is high).

[lmeyerov]

Additional note on preindexing representation:

• For cudf-safe/GPU-friendly prefiltering, prefer storing domain Series/Index (e.g., node_id per node_type, edge_id or src/dst per rel) and apply via .isin(domain_series)/semijoin.
• Full materialized subgraphs (filtered nodes/edges DataFrames) are only worthwhile when reused across multiple queries; otherwise prep dominates.
• Planner should decide between (a) reusing cached domains, (b) materializing cached subgraphs, or (c) in-query filters, based on selectivity + expected reuse.