Author: Joao Felipe De Souza Hardware: NVIDIA RTX 2070 · WSL2 · Ubuntu 22.04
This project models and predicts distributed LLM inference scaling behavior across tensor parallelism (TP), pipeline parallelism (PP), and hybrid TP x PP strategies.
It combines real GPU compute measurements on an RTX 2070 with analytical alpha-beta communication models calibrated from hardware specifications and validated with real Gloo AllReduce benchmarks.
The central question:
Given a model, GPU count, and interconnect type, which parallelism strategy maximizes efficiency while staying within VRAM constraints?
For architecture and methodology details, see DESIGN.md.
comm-cost-modeling/
|-- comm_cost_model.py # Phases 1-4: compute + alpha-beta model
|-- advanced_analysis.py # Phase 5: TP vs PP vs Hybrid comparison
|-- final_analysis.py # Phase 6: corrected PP bubble + VRAM model
|-- nccl_benchmark.py # Real AllReduce microbenchmark (Gloo)
|-- validate_model.py # Empirical alpha-beta validation
|-- ultra_optimization.py # Stochastic latency (Monte Carlo)
|-- optimal_serving.py # Optimal configuration finder v1
|-- final_results_v2.py # Final consolidated results (main script)
|-- DESIGN.md
|-- LICENSE
|-- requirements.txt
|-- results/
|-- plots/
Real measurements on RTX 2070 across hidden sizes 768 to 8192, batch sizes 1 to 8, and sequence lengths 512 and 2048.
When exact data is unavailable, a power-law model is fit:
T_layer = a x (hidden^2 x batch x seq)^b
Calibration result:
b = 0.7232
Mean absolute error: 20.2%
The exponent b < 1.0 reflects GPU memory bandwidth saturation. Doubling FLOPs does not double wall-clock time.
Alpha-beta model:
T_comm = alpha + beta x message_bytes
Ring AllReduce (tensor parallelism):
T = 2 x (n-1) x (alpha + beta x msg / n)
Point-to-point (pipeline parallelism):
T = alpha + beta x activation_bytes
Parameters from hardware specifications:
Interconnect alpha (ms) Bandwidth
PCIe 3.0 x16 0.005 12 GB/s
PCIe 4.0 x16 0.003 25 GB/s
NVLink v3 A100 0.001 150 GB/s
NVLink v4 H100 0.0005 450 GB/s
Gloo (measured) 0.269 1.77 GB/s
Real Gloo AllReduce loopback benchmark (2 processes, single host) confirms the alpha-beta functional form with R^2 = 0.9996.
The 54x alpha overhead and 6.8x bandwidth gap between Gloo and PCIe 3.0 hardware quantifies the OS and TCP software stack cost. Production NCCL bypasses this with GPUDirect RDMA.
Top-ranked configuration per model with total_gpus >= 2:
Model Strategy GPUs Efficiency P50 ms P99 ms VRAM/GPU
GPT-2 (117M) TP 2 83.8% 8.3 8.5 0.21 GB
LLaMA-7B TP 2 82.0% 198.1 200.7 8.14 GB
LLaMA-13B TP 2 96.9% 369.4 373.1 14.76 GB
LLaMA-70B TP 4 90.5% 942.2 960.0 40.50 GB
Falcon-180B TP 8 87.8% 950.7 987.0 54.97 GB
TP consistently outperforms PP when VRAM allows. LLaMA-13B achieves 96.9% efficiency at TP=2 on NVLink v3, meaning near-perfect linear speedup.
Model PCIe 3.0 PCIe 4.0 NVLink v3 NVLink v4
GPT-2 (117M) 9.5% 14.6% 25.4% 28.6%
LLaMA-7B 19.6% 32.7% 68.0% 79.8%
LLaMA-13B 26.3% 42.6% 81.2% 92.7%
LLaMA-70B 29.9% 42.0% 61.1% 65.1%
Falcon-180B 36.9% 54.8% 87.8% 95.5%
PCIe 3.0 at 8 GPUs makes TP essentially useless for all models (below 37% efficiency). NVLink v4 makes TP viable even for mid-size models such as LLaMA-7B at 79.8%.
Ratio = compute_time / comm_time at 8 GPUs. Values above 1.0 mean communication is not the bottleneck.
Model PCIe 3.0 PCIe 4.0 NVLink v3 NVLink v4
GPT-2 (117M) 0.44 (M) 0.90 (M) 4.64 (C) 12.23 (C)
LLaMA-7B 0.29 (M) 0.59 (M) 3.45 (C) 10.02 (C)
LLaMA-70B 0.80 (M) 1.66 (C) 9.76 (C) 28.79 (C)
Falcon-180B 0.58 (M) 1.21 (C) 7.21 (C) 21.43 (C)
C = compute dominated M = communication dominated
Notable crossover: LLaMA-70B sits at ratio=1.66 on PCIe 4.0, right at the compute/comm boundary. The same model on PCIe 3.0 falls to 0.80 (fully communication dominated).
LLaMA-7B, NVLink v3. Reference: TP 8 GPUs = 59.7 ms.
Strategy GPUs Bubble Efficiency Total ms vs TP
PP(m=4) 2 20.0% 45.9% 353.9 5.92x
PP(m=4) 4 42.9% 32.8% 247.8 4.15x
PP(m=8) 8 46.7% 15.3% 265.7 4.45x
Even the best PP configuration is 4 to 6x slower than TP. PP is viable only when VRAM does not allow TP.
Minimum GPU count to serve each model under TP with fp16 weights:
Model 8 GB GPU 24 GB GPU 40 GB GPU 80 GB GPU
GPT-2 (117M) >= 1 >= 1 >= 1 >= 1
LLaMA-7B >= 4 >= 1 >= 1 >= 1
LLaMA-13B >= 8 >= 2 >= 1 >= 1
LLaMA-70B OOM >= 8 >= 8 >= 2
Falcon-180B OOM OOM >= 16 >= 8
Cheapest configurations by cost per 1M tokens:
Strategy Interconnect $/1M tokens Efficiency
TP 2 PCIe 3.0 $0.092 57.9%
TP 2 PCIe 4.0 $0.114 69.4%
PP(m=4) PCIe 3.0 $0.116 45.7%
TP 4 PCIe 3.0 $0.153 34.6%
TP 2 NVLink v3 $0.220 82.0%
PCIe 3.0 with 2 GPUs is the cheapest option for LLaMA-7B serving. NVLink is 2.4x more expensive per token but delivers 82% efficiency versus 57.9% for PCIe. The optimal choice depends on whether you optimize for cost or for latency.
Monte Carlo simulation with log-normal jitter at sigma=12%. LLaMA-7B TP, 8 GPUs:
Interconnect P50 ms P99 ms P99/P50
PCIe 3.0 200.9 254.8 1.27x
NVLink v3 59.7 64.0 1.07x
NVLink v4 45.3 46.7 1.03x
NVLink v4 is near-deterministic (P99/P50 = 1.03x). PCIe 3.0 has 27% tail overhead, which is critical for SLA-bound serving where P99 latency governs contracts.
Real Gloo AllReduce benchmark confirms alpha-beta model:
Measured: alpha = 0.269 ms, BW = 1.77 GB/s, R^2 = 0.9996
PCIe 3.0: alpha = 0.005 ms, BW = 12.0 GB/s
NVLink v4: alpha = 0.0005 ms, BW = 450.0 GB/s
Software stack overhead versus PCIe 3.0 hardware:
- Alpha: 54x higher (OS scheduler + TCP + Python runtime)
- Bandwidth: 6.8x lower (RAM loopback vs PCIe DMA)
Scenario Recommendation
LLaMA-7B on consumer GPUs TP=4 with 4x 8GB cards (minimum VRAM)
LLaMA-7B cost-optimized TP=2, PCIe 3.0 ($0.092 per 1M tokens)
LLaMA-7B latency-optimized TP=2 to 4, NVLink v3 (82 to 73% eff)
LLaMA-70B production TP=4, NVLink v3 (4x 40GB, 90.5% eff)
Falcon-180B TP=8, NVLink v3 (8x 80GB A100, 87.8% eff)
Any model on PCIe at 8 GPUs Avoid — below 37% efficiency
Memory constrained Use PP for weight sharding, accept loss
SLA-bound production NVLink only — PCIe tail adds 27%
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python3 comm_cost_model.py # phases 1-4: compute + comm model
python3 nccl_benchmark.py # real AllReduce benchmark
python3 final_results_v2.py # final consolidated results
results/compute_sweep.csv Real compute measurements Phase 1
results/comm_model.csv Communication model predictions
results/scaling_sim.csv TP scaling simulation
results/crossover_analysis.csv Comm dominance crossover points
results/efficiency_summary.csv Efficiency by interconnect and GPUs
results/nccl_benchmark.csv Real Gloo AllReduce measurements
results/final_scaling_v2.csv Final scaling efficiency table
results/final_cost_v2.csv Final cost efficiency table
results/final_optimal_v2.csv Final optimal configurations
results/final_bubble_v2.csv Pipeline bubble analysis
plots/calibration_quality.png Compute model fit quality
plots/tp_scaling_efficiency.png TP efficiency vs GPU count
plots/regime_heatmap.png Compute vs comm regime map
plots/tp_vs_pp.png TP vs PP efficiency comparison
plots/pipeline_bubble.png Pipeline bubble vs microbatch count
plots/gloo_validation.png Alpha-beta empirical validation
plots/cost_efficiency.png Cost per 1M tokens vs GPU count
plots/vram_frontier.png VRAM per GPU vs model size
- Compute calibrated on RTX 2070 only
- Communication modeled analytically without a multi-GPU NCCL cluster
- GPipe schedule only — 1F1B would reduce PP bubble by approximately 50%
- No data parallelism or expert parallelism modeled
- Inference only — no optimizer states or gradients
- Homogeneous topology assumed throughout
- Shoeybi et al., Megatron-LM (2019)
- Narayanan et al., Efficient Large-Scale LLM Training on GPU Clusters (2021)
- Huang et al., GPipe (2019)
- Hockney, The communication challenge for MPP (1994)
- NVIDIA A100 Tensor Core GPU Architecture Whitepaper (2020)
- NVIDIA H100 Tensor Core GPU Architecture Whitepaper (2022)
- Alizadeh et al., Data Center TCP DCTCP (2010)
MIT — see LICENSE.