HSU TwinFlow Benchmark

This repository contains the benchmark material for HSU TwinFlow, a living digital-twin-based benchmark for evaluating data-driven methods in cyber-physical production systems (CPPS).

The benchmark is designed for research on:

• Anomaly detection
• Fault diagnosis
• Resilient systems
• AI-based fault handling in production systems
• Digital-twin-generated industrial datasets

Reference

Paper is available at here.

@InProceedings{hranisavljevic_et_al:OASIcs.DX.2026.7,
  author = {Hranisavljevic, Nemanja and Diedrich, Alexander and Moddemann, Lukas and
            Schaeffer, Domenic and Marek, Frank and Pill, Ingo and Niggemann, Oliver},
  title = {{HSU TwinFlow: A Living Benchmark for Evaluating Anomaly Detection and
            Diagnosis Methods in Cyber-Physical Production Systems Using
            Digital-Twin-Generated Data}},
  booktitle = {37th International Conference on Principles of Diagnosis and
               Resilient Systems (DX 2026)},
  pages = {7:1--7:14},
  series = {Open Access Series in Informatics (OASIcs)},
  ISBN = {978-3-95977-455-0},
  ISSN = {2190-6807},
  year = {2026},
  volume = {148},
  editor = {Pill, Ingo and Zanella, Marina and Provan, Gregory},
  publisher = {Schloss Dagstuhl -- Leibniz-Zentrum f{\"u}r Informatik},
  address = {Dagstuhl, Germany},
  URL = {https://drops.dagstuhl.de/entities/document/10.4230/OASIcs.DX.2026.7},
  URN = {urn:nbn:de:0030-drops-278212},
  doi = {10.4230/OASIcs.DX.2026.7},
  annote = {Keywords: Benchmark, Digital Twin, Manufacturing, Production Systems,
            Cyber-Physical Systems, Artificial Intelligence, Anomaly Detection,
            Diagnosis, System Reconfiguration}
}

Overview

HSU TwinFlow is based on a realistic digital twin of a production facility containing more than 100 interacting components, including conveyors, robots, autonomous mobile robots, robot stations, workstations, and goods input/output areas.

The benchmark provides scenario-based simulation data for both nominal and faulty operation. Fault scenarios are generated by injecting configurable failures into selected assets and recording the resulting system behavior.

Repository Contents

This repository is intended to store:

• Digital twin system knowledge
• Scenario definitions
• Nominal operation datasets
• Fault-labeled evaluation datasets
• Ground-truth fault labels
• Benchmark scripts
• Evaluation protocols
• Example anomaly detection and diagnosis baselines
• Documentation for reproducing experiments

System Knowledge Files

The repository contains several machine-readable files that describe the benchmark system independently from the generated time-series data:

• component_type_knowledge.json defines reusable knowledge per component type prefix, such as CC, RC, CLT, AMR, PR, RB, and RS. Each entry separates observable variables, configurable parameters, and injectable fault types. Numeric configurable parameters include their valid min and max range, while fault types include the value used to repair or reset the injected fault.
• conveyor_segments.json groups adjacent conveyor components into named logical segments, for example consecutive CC_* or RC_* transport paths. These groups are useful for plotting, aggregation, and interpreting faults that affect a local transport section rather than a single isolated conveyor.
• structural_hierarchy.json defines the physical hierarchy of the digital twin as area -> component type -> component identifiers. It assigns components to functional areas such as AMR, PAR, VZ, WA, and WE, and intentionally keeps variable and fault definitions separate from the hierarchy.

Benchmark Tasks

Anomaly Detection

The included baselines use all training scenarios, including faulty runs, and are evaluated on held-out test sequences containing injected faults. The benchmark provides point-wise anomaly labels for evaluating detection performance.

Fault Diagnosis

Diagnosis methods are evaluated by their ability to identify the affected component and the corresponding fault type. The benchmark includes structured system information to relate variables, assets, and functional areas.

Dataset Design

The benchmark uses high-level scenario definitions to generate reproducible simulation runs. Faults are introduced, kept active for defined intervals, and then repaired. Ground truth is derived directly from the scenario actions, including the affected component, fault type, and timestamps.

Initial fault categories include failures affecting conveyors, autonomous mobile robots, and six-axis robots.

Scenarios

Scenario files are stored in scenarios/training and scenarios/test. They are the simulator input definitions used to generate the datasets in data/training and data/test.

Each scenario JSON contains metadata under scenario and a time-ordered actions list. Actions create pallets with the PalletCreator or inject and repair faults by setting a component property at a specific startsAt timestamp. Fault actions are paired with repair actions, so the scenarios define the ground truth for anomaly labels and fault-diagnosis targets.

The training scenarios contain repeated nominal or low-fault-rate runs, such as 0_0pct_faults_rep_1.json, 1_0pct_faults_rep_1.json, and 10_0pct_faults_rep_1.json. The test scenarios contain area-specific and multi-fault evaluations, such as AMR_30pct_faults.json, VZ_multi_2_30pct_faults.json, and WE_multi_2_30pct_faults.json.

Parquet datasets

Simulation data is stored in Apache Parquet files, a column-oriented format that preserves numeric, boolean, and categorical values. The dataset is available on Zenodo.

Each scenario has its own folder under data/training or data/test:

data/
  training/
    0_0pct_faults_rep_1/
      measurements.parquet
      faults.parquet
      parameters.parquet
  test/
    AMR_30pct_faults/
      measurements.parquet
      faults.parquet
      parameters.parquet

File	Contents	Use
measurements.parquet	Recorded component signals, including numeric measurements, boolean states, and categorical states.	Model inputs and dashboard time-series plots.
faults.parquet	Recorded fault values for each component, with columns such as CC_1.MotorFault. Values can be boolean or numeric.	Training diagnosis models and deriving evaluation targets.
parameters.parquet	Recorded configurable component parameters.	Inspecting the system configuration over time.

Rows are indexed by simulationTime in seconds. The three files in a scenario share the same timestamps, so measurements, faults, and parameters can be aligned directly. Columns identify component properties, typically as component.property; vector quantities are split into .X, .Y, and .Z columns. Missing observations remain missing values.

Training contains repeated nominal and low-fault-rate scenarios (0_0pct_faults_rep_*, 1_0pct_faults_rep_*, and 10_0pct_faults_rep_*). Test folders contain area-specific and simultaneous-fault scenarios, such as AMR_30pct_faults and AMR_multi_2_30pct_faults.

Load a scenario directly with pandas:

from pathlib import Path
import pandas as pd

scenario = Path("data/test/AMR_30pct_faults")
measurements = pd.read_parquet(scenario / "measurements.parquet")
faults = pd.read_parquet(scenario / "faults.parquet")
parameters = pd.read_parquet(scenario / "parameters.parquet")

assert measurements.index.equals(faults.index)
assert measurements.index.equals(parameters.index)

The saved index is restored automatically. To read selected signals only, pass columns=["component.property"] to pd.read_parquet, using names present in the file. Pandas requires a Parquet engine such as pyarrow.

Use extract_fault_labels(faults, scenario.name) from scripts.evaluate_methods to derive boolean targets consistently: nonzero values indicate active faults, missing values are treated as inactive, and IncreasedDampingFault=0.1 is also normal for the listed benchmark scenarios.

Training and evaluation

Each method script exposes train() and predict(measurements). Running the script calls train() and saves its model as JSON at the module's MODEL_PATH. Each predict() loads that file itself; the evaluator only supplies measurements and calculates metrics. Anomaly detectors return a boolean array, while diagnosis methods return a DataFrame with component and fault_type columns. Structural diagnosis also loads its range thresholds from THRESHOLDS_PATH.

The evaluator discovers Python scripts directly under methods/ad and methods/dx, skipping filenames starting with _. No evaluator changes are needed when adding a method. Optional METHOD_NAME sets its benchmark label. Missing MODEL_PATH or THRESHOLDS_PATH files print a warning and skip the method during evaluation.

Both diagnosis methods use every folder under data/training and the same nonzero fault-label rules as evaluation, including the damping exception. Case-based training encodes numeric, boolean (0/1), and categorical (one-hot) measurements. It builds one fault case from the last active sample of each fault segment. Nominal cases are excluded from diagnosis training and prediction.

Train and evaluate with:

python methods/ad/range_monitoring.py
python methods/ad/autoencoder.py
python methods/dx/case_based.py
python methods/dx/structural_knowledge.py
python scripts/evaluate_methods.py

Range monitoring and the autoencoder load measurements from all scenario folders under data/training, including faulty training scenarios. The autoencoder retains its configurable row subsampling (ROW_STRIDE = 5). It includes numeric measurements, boolean values encoded as 0/1, and one-hot categorical features. The model JSON stores the encoding and category order; categories unseen during training produce an all-zero one-hot group. Retrain existing numeric-only models to include these additional features. All four models are plain JSON; scaler parameters, autoencoder weights, and case vectors are stored as lists. Evaluation uses the saved models without retraining and the same test scenarios, fault labels, and metrics for both anomaly detectors. Nonzero fault values are active; for the listed current scenarios, IncreasedDampingFault=0.1 is normal. Structural diagnosis learns component fault candidates from all training faults.parquet files using the evaluator's label rules, and keeps component/area mappings from the structural hierarchy. Diagnosis methods and dashboard confusion matrices use the last active sample of each individual fault segment; ambiguous structural fault types are reported as unknown, without access to the true fault type. AD results are saved to benchmark_ad.json and DX results to benchmark_dx.json. Detection results include per-scenario and overall metrics; composite F1 keeps event boundaries separate between scenarios. Existing method entries are skipped on subsequent runs; only new evaluations are added. Remove a method's entry from its benchmark JSON to evaluate it again.

Dashboard

Run the following commands from the repository root. Create a virtual environment and install the dependencies once:

python -m venv .venv

Activate it in Windows PowerShell with .\.venv\Scripts\Activate.ps1, or on Linux/macOS with source .venv/bin/activate, then run:

python -m pip install -r requirements.txt
python dashboard/run_dashboard.py

Open http://localhost:8050/selfx/ in your browser. Keep the terminal running; press Ctrl+C to stop the dashboard.

For scenario plots, place the prepared dataset in data/training/<scenario>/ and data/test/<scenario>/. Each scenario folder contains measurements.parquet, faults.parquet, and parameters.parquet. See Parquet datasets for file contents and loading examples.

Use the sidebar to browse system knowledge, measurements, anomaly detection, diagnosis, and benchmark results. Select a scenario using the dropdown at the top. Detection and diagnosis views load saved JSON models from models/; run the commands in Training and evaluation to generate them. Diagnosis plots and confusion matrices use the last faulty sample of each fault segment. The benchmark tables read benchmark_ad.json and benchmark_dx.json from the repository root each time the page renders.

After retraining models, restart the dashboard and refresh your browser to clear cached models. If a page is blank, verify that the server is running and that the URL includes /selfx/; check the terminal for errors.

Pallet configurations

The benchmark includes several predefined pallet configurations used by the scenario generator. These configurations define the product types placed on a pallet and their production routes through the robot stations. They are defined in pallet_configurations.json.

Currently, defined configurations include:

• PalletConfiguration-NOVA-6-DTF-6
• PalletConfiguration-NOVA-DTF-HSU-Mix
• HSU-1Box

For details on product layouts and production routes, see docs/pallet_configurations.md.

Known Problems

• In pallet_configurations.json, PalletConfiguration-NOVA-DTF-HSU-Mix assigns two products to the same position: HeightPosition=1, LengthPosition=2, WidthPosition=1. As a result, its 12 products occupy only 11 distinct configured positions.
• IncreasedDampingFault is assigned a default value of 0.1 by the simulation in some cases even when this value was not explicitly present in the scenario definition. Because that default is not a scenario-specified fault and is used as a neutral/normal baseline, the dashboard treats it as non-faulty and only flags values different from both 0 and 0.1.

Citation

This repository accompanies the paper:

HSU TwinFlow: A Living Benchmark for Evaluating Anomaly Detection and Diagnosis Methods in Cyber-Physical Production Systems Using Digital-Twin-Generated Data

submitted at the The 37th International Conference on Principles of Diagnosis and Resilient Systems (DX'26).

Please cite the paper when using this benchmark material.

License

This repository uses separate licenses for code and non-code material.

Code

All source code, scripts, baseline implementations, and software utilities in this repository are licensed under the MIT License.

See LICENSE for details.

Data and Benchmark Material

All datasets, scenario definitions, labels, documentation, figures, and other non-code benchmark material are licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0).

See LICENSE-DATA for details.

Summary

• Code: MIT License
• Datasets and benchmark material: CC BY 4.0