inference-sim

Discrete event simulator for inference platforms.

Getting started

Go >= 1.21 is required.

Testing Pipeline

Install and build inference-sim

git clone git@github.com:inference-sim/inference-sim.git
cd inference-sim
go build -o simulation_worker main.go

Run test data collection and collect results

Collect data from vllm-data-collection and post process results to get vllm aggregate metrics. Remember to use mode=test.

Tokenize prompts and run BLIS sweep

python offline_tokenizer.py --results_path <path to results_new/scenario in vllm-data-collection>

Next, modify experiment_constants_test.py as required to specify test configs to BLIS. Finally, run BLIS sweeps.

python run_blis_sweep.py --mode test

Analyze results for accuracy, MAE

python analyze_sim_results.py

Inference Pipeline

Install inference-sim

pip install git+https://github.com/inference-sim/inference-sim.git

Import and utilize BLIS related modules

spec = importlib.util.find_spec("run_blis_sweep")
SIMULATION_BASE_DIR = Path(spec.origin).parent
CONSTANTS_PATH = SIMULATION_BASE_DIR / "experiment_constants_inference.py"
RESULTS_PATH = SIMULATION_BASE_DIR / "results/sweep_params/simulator_inference_results.csv"

Set the env variable expected by the BLIS script

os.environ["SIMULATION_BASE_DIR"] = str(SIMULATION_BASE_DIR)
runpy.run_module("run_blis_sweep", run_name="__main__")

Run BLIS sweep

Modify experiment_constants_inference.py as required to specify inference configs to BLIS. Next, run the simulator through:

python run_blis_sweep.py

Help

go run main.go run --help