[Nexthop] Use run_tests script to run benchmarks

[anna-nexthop]

Pre-submission checklist

• I've ran the linters locally and fixed lint errors related to the files I modified in this PR. You can install the linters by running pip install -r requirements-dev.txt && pre-commit install
• pre-commit run

Summary

Adds support for running FBOSS benchmark test binaries as test suites using the existing run_test.py script. Benchmark tests measure performance metrics like throughput, latency, and speed of various FBOSS operations. Users can now easily run benchmark suites with automated CSV output containing detailed metrics for analysis.

Key Features

Benchmark Suite Management

• Created benchmark suite configuration files in fboss/oss/hw_benchmark_tests/:
  • t1_benchmarks.conf - T1 agent benchmark suite (9 benchmarks)
  • t2_benchmarks.conf - T2 agent benchmark suite (13 benchmarks)
  • additional_benchmarks.conf - Remaining benchmarks (15 benchmarks)
• Configuration files packaged to ./share/hw_benchmark_tests/ following existing test config patterns

Performance Metrics Collection

• Parses benchmark output to extract:
  • Benchmark test name
  • Relative time per iteration
  • Iterations per second
  • CPU time (microseconds)
  • Maximum RSS memory usage
• Generates timestamped CSV files with results for tracking and analysis
• Three status values: OK (success), FAILED (incomplete output), TIMEOUT (exceeded 1200s limit)

Command-Line Interface

• Added "benchmark" subcommand to run_test.py
• Supports custom filter files via --filter_file argument
• Filters out unavailable binaries automatically (e.g., DNX vs. XGS chips)
• Provides helpful messages when no benchmark binaries are found

Implementation Details

BenchmarkTestRunner Architecture

Implemented as a standalone class (does not extend TestRunner) with:

• Class-level constants: BENCHMARK_CONFIG_DIR, T1_BENCHMARKS_CONF, T2_BENCHMARKS_CONF, ALL_BENCHMARKS_CONF
• Public methods:
  • add_subcommand_arguments() - Register command-line arguments
  • run_test(args) - Main entry point for running benchmarks
• Private methods:
  • _parse_benchmark_output() - Extract metrics from benchmark output
  • _run_benchmark_binary() - Execute a single benchmark binary

Design rationale: Unlike other test runners that extend the abstract TestRunner base class, BenchmarkTestRunner is standalone because:

• Benchmark tests are standalone binaries, not gtest-based tests
• No warmboot/coldboot variants needed
• Standard test filtering mechanisms not applicable
• Simpler, more maintainable implementation

Packaging Updates

• Updated package-fboss.py to copy benchmark configuration files to share/hw_benchmark_tests/
• Benchmark binaries packaged to bin/ directory

Documentation Updates

• Added Option 1: Using the run_test.py Script (Recommended) section with examples for running all benchmarks, T1 suite, T2 suite, and additional benchmarks
• Kept existing manual execution as Option 2: Running Individual Binaries
• Updated T1 Tests → Agent Benchmark Tests and T2 Tests → Agent Benchmark Tests sections to use run_test.py commands
• Added note about CSV output with timestamped filenames

Testing

Unit Test Coverage

Added comprehensive unit tests using pytest (25 test cases total):

test_run_test.py - BenchmarkTestRunner coverage:

• Loading from custom filter files
• Handling nonexistent/empty files
• Loading from default T1, T2, and additional configs
• Handling missing default configs
• Successful parsing with all metrics
• Missing JSON metrics and benchmark lines
• Empty output and different time units
• Successful execution, timeout handling, execution failures
• Exception handling
• Execution with config arguments
• List tests mode
• Full execution with CSV writing
• No existing binaries and some missing binaries
• End-to-end workflow from default configs to list tests

test_benchmark_conf_files.py - Configuration validation:

• Ensures no overlaps between benchmark suite lists

Test features:

• All tests use mocking to avoid side effects
• Tests verify both success and error cases
• Temporary files properly cleaned up
• Pythonic code style with list comprehensions

Integration: Tests integrated into CMake and run with ctest

Manual Verification

Verified benchmark subcommand, configuration files, and execution on device with T1 suite showing proper handling of successful benchmarks, failures, and timeouts with correct CSV output.

Visual Inspection of Documentation

Verified documentation formatting is correct by serving Docusaurus locally: