Skip to content

Conversation

@WangzJi
Copy link
Contributor

@WangzJi WangzJi commented Dec 15, 2025

Ⅰ. Describe what this PR did

Closes: #7646
Add a new seata-benchmark-cli module under test-suite for stress testing Seata transaction modes.

Features:

  • Support for AT, TCC, and SAGA transaction modes
  • Dual execution modes:
    • Empty mode (--branches 0): Measures pure Seata protocol overhead without database operations
    • Real mode (--branches N): Executes actual distributed transactions with database operations
  • Configurable TPS (Transactions Per Second) rate limiting
  • Multi-threaded workload generation with configurable thread pool
  • Fault injection with configurable rollback percentage
  • Window-based progress reporting (every 10 seconds)
  • Performance metrics collection: latency percentiles (P50/P95/P99), success rate, TPS
  • CSV export for post-analysis
  • Warmup support: exclude initial ramp-up period from final statistics
  • YAML configuration file support with priority: CLI args > env var > system property > classpath

Module Structure:

test-suite/seata-benchmark-cli/
├── src/main/java/org/apache/seata/benchmark/
│   ├── BenchmarkApplication.java       # Main entry with picocli CLI
│   ├── BenchmarkConstants.java         # Constants definition
│   ├── config/                         # Configuration classes
│   ├── executor/                       # Transaction executors (AT/TCC/SAGA)
│   ├── model/                          # Data models (metrics, records)
│   ├── monitor/                        # Metrics collection
│   ├── saga/                           # SAGA mode services
│   └── util/                           # Utility classes
└── src/main/resources/
    ├── seata/saga/statelang/           # SAGA state machine definitions
    └── *.conf                          # Seata configuration

Ⅱ. Does this pull request fix one issue?

No, this is a new feature.

Ⅲ. Why don't you add test cases (unit test/integration test)?

This is a benchmark/stress testing tool. The primary validation is manual testing against a running Seata Server. Integration tests would require a full Seata Server setup which is better suited for the existing integration test infrastructure.

Ⅳ. Describe how to verify it

  1. Build the module:

    cd test-suite/seata-benchmark-cli
    ../../mvnw clean package
  2. Start Seata Server (ensure it's running on 127.0.0.1:8091)

  3. Run benchmark (empty mode - no database required):

    java -jar target/seata-benchmark-cli.jar \
      --server 127.0.0.1:8091 \
      --mode AT \
      --tps 100 \
      --duration 60
  4. Run benchmark (real mode - requires Docker):

    java -jar target/seata-benchmark-cli.jar \
      --server 127.0.0.1:8091 \
      --mode AT \
      --tps 100 \
      --duration 60 \
      --branches 3
  5. Verify the output shows progress every 10 seconds and final report with metrics.

Ⅴ. Special notes for reviews

  1. Latency Sampling: To prevent OOM on large-scale tests, latencies are sampled (max 500K samples) - inspired by Kafka ProducerPerformance.

  2. Empty vs Real Mode: Empty mode (--branches 0) is useful for measuring pure Seata Server capacity without database overhead.

  3. SAGA Implementation: Real SAGA mode uses Seata's state machine engine with predefined state machine definitions in src/main/resources/seata/saga/statelang/.

  4. Dependencies: The module uses:

    • picocli for CLI argument parsing
    • snakeyaml for YAML configuration
    • testcontainers for MySQL in real mode
    • Seata's existing infrastructure (TM, RM, Saga engine)

@WangzJi WangzJi changed the title feat: benchmark cli feature: benchmark cli Dec 15, 2025
@WangzJi WangzJi added the type: feature Category issues or prs related to feature request. label Dec 18, 2025
@codecov
Copy link

codecov bot commented Dec 24, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 71.17%. Comparing base (340aa3c) to head (447b8e4).

Additional details and impacted files
@@             Coverage Diff              @@
##                2.x    #7865      +/-   ##
============================================
- Coverage     71.20%   71.17%   -0.04%     
+ Complexity      797      785      -12     
============================================
  Files          1300     1300              
  Lines         49620    49620              
  Branches       5874     5874              
============================================
- Hits          35331    35315      -16     
- Misses        11371    11387      +16     
  Partials       2918     2918              

see 11 files with indirect coverage changes

Impacted file tree graph

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: feature Category issues or prs related to feature request.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Seata Benchmark 1.0 Development Task

1 participant