HyperStream

High-performance, header-only C++ library for Hyperdimensional Computing (HDC). HyperStream provides binary hypervectors, SIMD-accelerated kernels (SSE2/AVX2), adaptive runtime backend selection, associative memory data structures, and a streaming classification framework.

• Header-only, C++17
• Cross-platform: Windows, Linux, macOS
• Runtime capability detection and policy-driven backend selection
• Safe-by-default configurations with embedded/desktop profiles

Getting started

Requirements

• CMake 3.16+
• A C++17 compiler
  • MSVC 2022, Clang 12+, or GCC 10+

Build and test

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 4
ctest --test-dir build -C Release --output-on-failure

Quick config report

# Print profile, CPU features, selected backends, and footprint estimates
./build/benchmarks/config_bench        # Unix
./build/benchmarks/Release/config_bench.exe  # Windows

Optional: quick auto-tuning of Hamming backend threshold for your host.

./build/benchmarks/config_bench --auto-tune

Core concepts

• Binary hypervectors with bit-packed storage
• Operations: bind (XOR), bundle (majority), permute (rotate), similarity (Hamming)
• SIMD backends for bind and Hamming
• Policy: runtime CPU detection + dimension-based heuristics

Policy and capability

HyperStream chooses backends at runtime based on the CPU and simple heuristics. You can query decisions via the policy report API.

#include "hyperstream/backend/policy.hpp"

constexpr std::size_t D = 10000;
auto rep = hyperstream::backend::Report<D>();
// rep.bind_kind, rep.hamming_kind, and reason strings

Environment overrides

• Adjust Hamming SSE2 preference threshold (default: 16384) without rebuilding:

export HYPERSTREAM_HAMMING_SSE2_THRESHOLD=32768  # Unix
set HYPERSTREAM_HAMMING_SSE2_THRESHOLD=32768     # Windows

• Force scalar backends at compile time (e.g., portability checks):

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="-DHYPERSTREAM_FORCE_SCALAR"

Runtime dispatch architecture

HyperStream builds universal binaries and selects SIMD backends at runtime to avoid illegal-instruction traps while still delivering optimized SSE2/AVX2 performance.

See Runtime Dispatch Architecture (Docs/Runtime_Dispatch.md) for details on SIMD backend selection and adding new ISA support.

Design and Performance

• Runtime dispatch (portable by default)

  • Universal binaries select SIMD backends at runtime; see Docs/Runtime_Dispatch.md for details.
  • Bind prefers AVX2 → SSE2 → Scalar when available; Hamming prefers AVX2 at small/medium dims and SSE2 beyond a threshold.

• Safety invariants

  • No illegal-instruction hazards: code paths are guarded by feature checks.
  • Scalar fallback: set HYPERSTREAM_FORCE_SCALAR to force portable paths end-to-end.

• Memory access and tails

  • Unaligned IO: SIMD kernels use loadu/storeu; no special alignment is required at API boundaries.
  • Tail safety: remaining bits/words are handled safely using scalar operations/masking.

• Performance guidelines

  • Binary ops are word-wise; sustained throughput is often memory-bound at large dimensions.
  • Hamming threshold: default 16384; override via HYPERSTREAM_HAMMING_SSE2_THRESHOLD.
  • Non-temporal stores (where applicable) are guarded by conservative heuristics; defaults favor stability across hosts.

• Inspect selection and profile

  • Use benchmarks/config_bench to print CPU features, selected backends, thresholds, and footprints.
  • CI includes checks for default policy, env override, scalar-forced mode, and AVX2 preference on x86.

For a deeper dive (ISA codegen strategies, adding new ISAs, CI matrix), see Docs/Runtime_Dispatch.md.

Memory footprint helpers

Constexpr helpers estimate storage requirements for common structures.

#include "hyperstream/config.hpp"
using namespace hyperstream::config;

constexpr std::size_t d = 10000;
auto hv_bytes = BinaryHyperVectorStorageBytes(d);

Benchmarks

• config_bench: configuration, capability, and policy report; optional --auto-tune
• am_bench: associative memory microbenchmark
• cluster_bench: clustering microbenchmark

./build/benchmarks/config_bench --auto-tune
./build/benchmarks/am_bench
./build/benchmarks/cluster_bench

Project status

• Policy supports SSE2/AVX2 backends with runtime selection
• Env-var threshold override for Hamming is available; auto-tuning provides recommendations
• Tests validate correctness across awkward dimensions and backends

Contributing

Issues and pull requests are welcome. Please run the full test suite and ensure Release builds are clean with strict warnings.

CI

This repository is set up with GitHub Actions to build on Windows, Linux, and macOS, run tests, validate policy output, and verify scalar fallback.