WAMP Message Serialization Benchmarks

Comprehensive performance benchmarking suite for WAMP message serialization/deserialization across multiple serializers, payload modes, and payload sizes.

Overview

This benchmark suite measures the serialization and deserialization performance of WAMP messages using real-world vehicle telemetry data. It provides detailed performance metrics and CPU profiling with flamegraph visualization.

Key Features

• Multi-dimensional testing matrix:

  • 7 serializers: json, ujson, msgpack, cbor, cbor2, ubjson, flatbuffers
  • 2 payload modes: normal (WAMP args), transparent (WAMP payload)
  • 6 payload sizes: empty, small, medium, large, xl (16KB), xxl (128KB)
  • 2 Python implementations: CPython, PyPy

• Performance profiling:

  • vmprof statistical profiling (0.01s period)
  • Flamegraph visualization
  • Warm-up phase before measurement

• Real-world data:

  • Vehicle telemetry from CSV datasets (7MB)
  • GPS coordinates, sensor data, timestamps
  • Deterministic pseudo-random pothole sensor data

• Comprehensive reporting:

  • JSON results with msgs/sec, bytes/sec, avg message size
  • HTML report index with tabular results
  • Individual flamegraph HTML pages per configuration

Usage

Prerequisites

Install Autobahn|Python with development dependencies:

pip install -e ".[dev,compress,nvx]"

Required dependencies:

• vmprof>=0.4.15 - Statistical profiling
• jinja2>=3.0.0 - HTML template rendering
• humanize>=4.0.0 - Human-readable formatting
• cbor2 - CBOR serializer (batteries-included)

Running Benchmarks

1. Run a Single Benchmark

python main.py run \
    --serializer cbor \
    --payload_mode normal \
    --payload_size small \
    --iterations 10 \
    --profile build/profile.dat \
    --results build

Parameters:

• --serializer: Choose from json, cbor, msgpack, ubjson, flatbuffers
  • Use AUTOBAHN_USE_UJSON=1 env var to select ujson instead of json
  • Use AUTOBAHN_USE_CBOR2=1 env var to select cbor2 instead of cbor
• --payload_mode: normal (WAMP args) or transparent (WAMP payload)
• --payload_size: empty, small, medium, large, xl, xxl
• --iterations: Number of benchmark iterations (default: 10)
• --profile: Output path for vmprof profile data (.dat file)
• --results: Output directory for JSON results

Output files:

• build/profile.dat - vmprof profiling data
• build/results_cpy_cbor_normal_small.json - JSON results with metrics

2. Generate HTML Report

After running multiple benchmarks:

python main.py index --output build

This generates:

• build/index.html - Main report with tabular results
• build/vmprof_cpy_cbor_normal_small.html - Individual flamegraph pages

View the report:

# Open in browser
xdg-open build/index.html
# or
python -m http.server 8000 -d build
# then visit http://localhost:8000

Example: Full Benchmark Run

Run benchmarks for all serializers with small payload:

mkdir -p build

for serializer in json msgpack cbor ubjson flatbuffers; do
    for mode in normal transparent; do
        echo "Running: $serializer, $mode, small"
        python main.py run \
            --serializer $serializer \
            --payload_mode $mode \
            --payload_size small \
            --iterations 10 \
            --profile build/profile_${serializer}_${mode}.dat \
            --results build
    done
done

# Test ujson variant
AUTOBAHN_USE_UJSON=1 python main.py run \
    --serializer json \
    --payload_mode normal \
    --payload_size small \
    --iterations 10 \
    --profile build/profile_ujson_normal.dat \
    --results build

# Generate HTML report
python main.py index --output build

Using Just Recipes (Recommended)

If you have Just installed:

# Run benchmark with specific configuration
just benchmark-serialization-run cbor normal small

# Run full benchmark suite
just benchmark-serialization-suite

# Generate HTML report
just benchmark-serialization-report

# Clean benchmark artifacts
just benchmark-serialization-clean

See justfile in repository root for recipe definitions.

Payload Sizes

The benchmark suite tests 6 different payload sizes:

Size	Description	Approximate Size	Event Limit	Total Data
empty	Minimal empty payload	~0 bytes	42,039	~0 MB
small	Base vehicle telemetry (GPS, sensors)	~200-300 bytes	42,039	~12 MB
medium	Small + JSON_DATA1 (widget config)	~500-800 bytes	42,039	~33 MB
large	Medium + JSON_DATA2 (actors) + JSON_DATA3 (nested donuts)	~1-2 KB	42,039	~84 MB
xl	Small + 16KB binary frame	~16 KB	1,000	~16 MB
xxl	Small + 128KB binary frame	~128 KB	100	~12.8 MB

Note: xl and xxl payloads are automatically limited to prevent memory exhaustion:

• xl: Limited to 1,000 events (from 42K total) to keep memory usage reasonable
• xxl: Limited to 100 events (from 42K total) for the same reason
• Binary frame data is cached per event to avoid repeated regeneration
• These limits still provide statistically valid benchmark samples

Performance Metrics

For each configuration, the benchmark measures:

• msgs_per_sec: Throughput in messages per second
• bytes_per_sec: Bandwidth in bytes per second
• msg_bytes: Average serialized message size in bytes

Results are saved in JSON format:

{
    "python_version": "3.11.2 (main, ...)",
    "python": "cpy",
    "events": 1234,
    "sample": { ... },
    "iterations": 10,
    "msg_bytes": 256,
    "msgs_per_sec": 123456,
    "bytes_per_sec": 31604736
}

CPU Profiling with vmprof

The benchmark uses vmprof for statistical profiling during the measurement phase:

• Sampling period: 0.01 seconds (100 samples/sec)
• Output format: vmprof .dat files
• Visualization: Convert to flamegraph SVG using vmprof web interface or tooling

Flamegraph HTML pages show:

• Function call hierarchy
• Time spent in each function
• Hot paths in serialization code

This helps identify performance bottlenecks in:

• WAMP message construction
• Serializer encode/decode operations
• Data structure traversal
• Memory allocation patterns

Data Sources

Vehicle Telemetry CSV Datasets

The benchmark uses two CSV datasets with real vehicle telemetry:

• data/dataset1.csv (~1MB) - Fleet 1 vehicles
• data/dataset2.csv (~6MB) - Fleet 2 vehicles

CSV columns:

• vehicleID - Vehicle identifier
• ts - Timestamp (YYYY-MM-DD HH:MM:SS)
• lon, lat - GPS coordinates
• speed - Vehicle speed
• rain - Rain sensor value
• dyn_wiper - Wiper status

Derived fields:

• xtile, ytile - Tile coordinates (zoom level 18)
• pothole_depth - Deterministic pseudo-random (0.0-1.0)
• pothole_type - Random choice from type-a through type-f

Sample JSON Data Structures

Additional JSON structures from sample.py are used to increase payload sizes:

• JSON_DATA1 - JSON:API article structure with widget configuration
• JSON_DATA2 - Actor information (Tom Cruise, Robert Downey Jr.)
• JSON_DATA3 - Nested donut menu with batters and toppings

Directory Structure

examples/benchmarks/serialization/
├── README.md                  # This file
├── main.py                    # Benchmark runner with vmprof profiling
├── loader.py                  # CSV data loader and VehicleEvent class
├── sample.py                  # Sample JSON data structures
├── data/
│   ├── dataset1.csv          # Fleet 1 vehicle telemetry (1MB)
│   └── dataset2.csv          # Fleet 2 vehicle telemetry (6MB)
├── templates/
│   ├── base.html             # Jinja2 base template
│   ├── index.html            # Main report template
│   └── flamegraph.html       # Flamegraph page template
├── crossbarfx_black.svg      # Logo for HTML reports
└── build/                     # Generated artifacts (gitignored)
    ├── *.json                # JSON results per configuration
    ├── *.dat                 # vmprof profile data
    ├── *.html                # HTML report and flamegraphs
    └── *.svg                 # Flamegraph SVG (if generated)

Integration with CI/CD

This benchmark suite is integrated into the GitHub Actions workflow at .github/workflows/benchmark-serialization.yml:

• Runs on CPython and PyPy (x86-64)
• Tests all serializers and payload configurations
• Publishes results as workflow artifacts
• Integrates artifacts into Read the Docs documentation

See workflow file for configuration details.

Tips and Best Practices

For Accurate Results

1. Close other applications to minimize CPU contention
2. Run multiple iterations (at least 10) for statistical validity
3. Use production-like data - the CSV datasets simulate real workloads
4. Compare apples to apples - same payload_mode and payload_size across serializers
5. Test both CPython and PyPy - performance characteristics differ significantly

Expected Performance Characteristics

• PyPy typically shows 2-10x higher throughput than CPython due to JIT compilation
• Binary serializers (msgpack, cbor, flatbuffers) are usually faster than JSON
• Transparent payload mode may show different characteristics than normal mode
• Larger payloads (xl, xxl) stress different code paths than small payloads

Troubleshooting

ImportError: No module named 'vmprof'

pip install vmprof

FileNotFoundError: data/dataset1.csv

# Ensure you're running from examples/benchmarks/serialization/
cd examples/benchmarks/serialization
python main.py run ...

PermissionError on vmprof profile file

# Ensure build directory exists and is writable
mkdir -p build
chmod 755 build

Further Reading

• WAMP Specification
• Autobahn|Python Documentation
• vmprof Documentation
• Performance Optimization Guide

License

MIT License - See LICENSE file in repository root.

Copyright (c) typedef int GmbH