Skip to content

README.md

Latest commit

 

History

History
131 lines (87 loc) · 3.69 KB

README.md

File metadata and controls

131 lines (87 loc) · 3.69 KB

Scripts Guide

Scope and Responsibilities

Production-validation-related scripts currently under scripts/:

  • prod_test.sh: correctness and crash-recovery test orchestration (fast / stress / chaos / all)
  • perf_gate.sh: performance snapshot and regression gate (snapshot / compare)
  • perf_thresholds.env: default template for performance gate thresholds and quick benchmark knobs

1) prod_test.sh

Purpose

Unified layered execution for production-style integration tests:

  • fast: quick regression (default)
  • stress: long-run/high-pressure cases (#[ignore])
  • chaos: failpoint crash/fault-injection cases (#[ignore])
  • all: fast + stress + chaos

Basic Usage

./scripts/prod_test.sh [mode] [threads]
  • mode: fast|stress|chaos|all, default fast
  • threads: passed to cargo test -- --test-threads=..., default 8

Examples:

./scripts/prod_test.sh fast 8
./scripts/prod_test.sh all 8

Key Environment Variables

  • FEATURES: default failpoints
  • FAST_TIMEOUT / STRESS_TIMEOUT / CHAOS_TIMEOUT
  • CHAOS_RETRY: retry count for chaos cases
  • MACE_PROD_BUCKET_STRESS_ROUNDS
  • MACE_PROD_BUCKET_CHURN_ROUNDS
  • MACE_PROD_BUCKET_CHURN_WORKERS
  • MACE_PROD_EVICTOR_STRESS_ROUNDS

Result Semantics

  • Exit code 0: full pass
  • Non-zero: failed items are listed in failures at the end

Note: in chaos mode, child-process failure logs can be expected; rely on final script exit code and failures summary as ground truth.

2) perf_gate.sh

Purpose

Provides baseline sampling and regression gating:

  • snapshot: capture performance snapshot for current branch only
  • compare: compare current branch against a baseline (default origin/master) and enforce threshold rules

Basic Usage

./scripts/perf_gate.sh snapshot
./scripts/perf_gate.sh compare

Key Environment Variables

  • MACE_PERF_THRESHOLD_FILE: threshold file, default scripts/perf_thresholds.env
  • MACE_PERF_BASE_REF: baseline ref, default origin/master
  • MACE_PERF_BENCH_ARGS: arguments forwarded to Criterion
  • MACE_PERF_COMPARE_FILTERS: case filters for compare mode (comma-separated)
  • MACE_PERF_REGRESS_PCT: per-case regression threshold (%)
  • MACE_PERF_MAX_REGRESS_CASES: max allowed regressed case count
  • MACE_PERF_MIN_COMPARE_CASES: minimum valid comparable case count
  • MACE_PERF_MIN_BASE_NS: ignores ultra-small baseline noise

Outputs and Metric Definitions

Default output directory: target/perf_gate

Common files:

  • base_summary.json / head_summary.json
  • head_snapshot.json
  • base.bench.log / head.bench.log / snapshot.bench.log

Core metric fields:

  • median_ns: Criterion median point estimate in nanoseconds (lower is better)
  • mean_ns: Criterion mean point estimate in nanoseconds (lower is better)
  • delta_pct (from compare output):
    • formula: (head - base) / base * 100%
    • positive means slower, negative means faster

Gate rule:

  • compare fails when count of cases with delta_pct > MACE_PERF_REGRESS_PCT is greater than MACE_PERF_MAX_REGRESS_CASES.

3) perf_thresholds.env

Purpose

Central place for default perf gate and quick benchmark parameters, reused locally and in CI.

Maintenance Recommendations

  • First run 3–5 rounds on target production-equivalent hardware to establish a stable baseline
  • Then tighten:
    • MACE_PERF_REGRESS_PCT
    • MACE_PERF_MAX_REGRESS_CASES
    • MACE_PERF_COMPARE_FILTERS
  • If CI noise is high, start with looser thresholds and narrower case filters, then tighten gradually

CI Mapping

  • .github/workflows/ci.yml
    • test-prod-all: ./scripts/prod_test.sh all 8
    • perf-regression: ./scripts/perf_gate.sh compare