Skip to content

Commit de59ef5

Browse files
amilcarlucasamilcarlucas
committed
test(backend): Initial usage of ArduCopter SITL to test the flightcontroller backend
1 parent 0f8ff72 commit de59ef5

File tree

10 files changed

+788
-2
lines changed

10 files changed

+788
-2
lines changed

‎.github/instructions/SITL_TESTING.md‎

Lines changed: 191 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,191 @@
1+
# SITL Testing Setup
2+
3+
This document describes how to set up and run integration tests using ArduPilot SITL (Software In The Loop) for testing the `backend_flightcontroller.py` module.
4+
5+
## Overview
6+
7+
SITL testing provides real MAVLink communication validation instead of mocked tests.
8+
This ensures the flight controller backend works correctly with actual ArduPilot firmware.
9+
10+
## Architecture
11+
12+
The SITL testing setup consists of:
13+
14+
1. **Direct Download**: Tests download pre-built ArduCopter SITL binaries directly from the official ArduPilot firmware server (`firmware.ardupilot.org`)
15+
2. **Pytest Fixtures**: Session-scoped SITLManager class manages SITL process lifecycle
16+
3. **TCP Connection**: SITL runs on TCP port 5760 with MAVLink protocol
17+
4. **Parameter Configuration**: SITL uses `sitl/copter.param` with battery monitoring enabled
18+
19+
## Prerequisites
20+
21+
### For CI/CD (GitHub Actions)
22+
23+
- No additional setup required - SITL binaries are downloaded automatically during tests
24+
25+
### For Local Development
26+
27+
#### Download Pre-built SITL (Recommended)
28+
29+
Download the latest pre-built SITL binary directly from the official ArduPilot firmware server:
30+
31+
```bash
32+
./scripts/run_sitl_tests.sh download
33+
```
34+
35+
This downloads ArduCopter SITL from `https://firmware.ardupilot.org/Copter/latest/SITL_x86_64_linux_gnu/arducopter`
36+
37+
## Usage
38+
39+
### CI/CD Testing
40+
41+
SITL tests run automatically in GitHub Actions when SITL artifacts are available. The test workflow:
42+
43+
1. Downloads the latest SITL artifact
44+
2. Extracts and sets up SITL binary
45+
3. Runs tests marked with `@pytest.mark.sitl`
46+
4. Falls back to mocked tests if SITL is unavailable
47+
48+
### Local Development
49+
50+
Use the provided script for local SITL testing. You can either download pre-built SITL or use a locally built version:
51+
52+
#### Using Downloaded SITL (Recommended)
53+
54+
```bash
55+
# Download ArduCopter SITL from official firmware server
56+
./scripts/run_sitl_tests.sh download
57+
58+
# Download and run tests in one command
59+
./scripts/run_sitl_tests.sh download-test
60+
61+
# Check if downloaded SITL is available
62+
./scripts/run_sitl_tests.sh check
63+
```
64+
65+
#### Using Locally Built SITL
66+
67+
```bash
68+
# Set up environment for locally built SITL
69+
export ARDUPILOT_DIR="$HOME/ardupilot-sitl"
70+
71+
# Check if locally built SITL is available
72+
./scripts/run_sitl_tests.sh check
73+
74+
# Set up SITL for testing
75+
./scripts/run_sitl_tests.sh setup
76+
77+
# Run SITL integration tests
78+
./scripts/run_sitl_tests.sh test
79+
```
80+
81+
#### General Commands
82+
83+
```bash
84+
# Clean up SITL processes and cache
85+
./scripts/run_sitl_tests.sh cleanup
86+
87+
# Show help
88+
./scripts/run_sitl_tests.sh help
89+
```
90+
91+
### Manual Testing
92+
93+
Run specific SITL tests:
94+
95+
```bash
96+
# Run all SITL tests
97+
python -m pytest tests/test_backend_flightcontroller_sitl.py -v
98+
99+
# Run only SITL tests (skip if SITL unavailable)
100+
python -m pytest -m sitl -v
101+
102+
# Run SITL tests or fallback to mocked tests
103+
python -m pytest -m "sitl or not sitl" -v
104+
```
105+
106+
## Test Coverage
107+
108+
SITL tests cover:
109+
110+
- **Real MAVLink Connection**: Validates actual protocol communication on TCP port 5760
111+
- **Parameter Management**: Download, set, and verify parameters with real firmware
112+
- **Motor Testing**: Test motor commands against actual ArduPilot firmware
113+
- **Battery Monitoring**: Test battery status reporting with enabled monitoring
114+
- **Frame Information**: Validate vehicle configuration queries
115+
116+
## Implementation Details
117+
118+
### SITL Configuration
119+
120+
SITL runs with the following command line parameters:
121+
122+
```bash
123+
arducopter --model quad --home "40.071374,-105.229930,1440,0" --defaults ./sitl/copter.param --sysid 1 --speedup 10
124+
```
125+
126+
### Connection Details
127+
128+
- **Protocol**: MAVLink over TCP
129+
- **Port**: 5760
130+
- **Connection String**: "tcp:127.0.0.1:5760"
131+
- **Vehicle Type**: ArduCopter (Quadcopter)
132+
- **System ID**: 1
133+
134+
### Parameter Requirements
135+
136+
Some tests require specific parameters to be set in `sitl/copter.param`:
137+
138+
- `BATT_MONITOR = 4` (Analog voltage and current)
139+
- `BATT_VOLT_PIN = 1`
140+
- `BATT_CURR_PIN = 2`
141+
- `BATT_VOLT_MULT = 10.0`
142+
- `BATT_AMP_PERVOLT = 17.0`
143+
144+
## Configuration
145+
146+
### Environment Variables
147+
148+
- `SITL_BINARY`: Path to ArduCopter SITL binary (auto-detected in CI)
149+
- `ARDUPILOT_DIR`: Path to ArduPilot directory for local development
150+
151+
### Test Markers
152+
153+
- `@pytest.mark.sitl`: Marks tests requiring SITL
154+
- Tests automatically skip if SITL is unavailable
155+
156+
## Troubleshooting
157+
158+
### SITL Not Found
159+
160+
- **For downloaded SITL**: Run `./scripts/run_sitl_tests.sh download` to download from ArduPilot website
161+
- **For locally built SITL**: Ensure ArduPilot is built with `./waf configure --board=sitl && ./waf copter`
162+
- Check `ARDUPILOT_DIR` environment variable for locally built SITL
163+
- Verify SITL binary exists at expected path
164+
165+
### Connection Failures
166+
167+
- SITL may take time to start - tests include startup delays
168+
- Check for port conflicts on TCP port 5760
169+
- Verify MAVLink heartbeat detection
170+
- Ensure connection string format is "tcp:127.0.0.1:5760"
171+
172+
### Test Timeouts
173+
174+
- SITL tests are slower than mocked tests
175+
- Increase timeout values if needed
176+
- Check system performance for SITL simulation
177+
178+
## Benefits
179+
180+
1. **Real Validation**: Tests actual MAVLink protocol implementation
181+
2. **Regression Detection**: Catches firmware compatibility issues
182+
3. **CI/CD Integration**: Automated testing with pre-built artifacts
183+
4. **Development Flexibility**: Local testing with fallback to mocks
184+
5. **Cost Efficiency**: Monthly builds reduce CI resource usage
185+
186+
## Future Enhancements
187+
188+
- Multiple vehicle types (ArduPlane, Rover, etc.)
189+
- SITL version pinning for reproducible tests
190+
- Performance optimization for faster test execution
191+
- Multi-SITL instance testing for complex scenarios

‎.github/workflows/pytest.yml‎

Lines changed: 54 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,53 @@ jobs:
6060
run: |
6161
uv pip install --editable .[dev,ci_headless_tests]
6262
63+
- name: Download ArduCopter SITL (if available)
64+
run: |
65+
# Create cache key based on current quarter (YYYY-Q)
66+
CURRENT_YEAR=$(date +%Y)
67+
CURRENT_MONTH=$(date +%m)
68+
QUARTER=$(( (CURRENT_MONTH-1)/3 + 1 ))
69+
CACHE_KEY="${CURRENT_YEAR}-Q${QUARTER}"
70+
71+
echo "Cache key: ${CACHE_KEY}"
72+
73+
# Check if we have cached SITL files for this quarter
74+
if [ -d "sitl-cache/${CACHE_KEY}" ] && [ -f "sitl-cache/${CACHE_KEY}/arducopter" ]; then
75+
echo "Using cached SITL files from ${CACHE_KEY}"
76+
mkdir -p sitl/
77+
cp sitl-cache/${CACHE_KEY}/* sitl/
78+
else
79+
echo "Downloading fresh SITL files for ${CACHE_KEY}"
80+
mkdir -p sitl/ sitl-cache/${CACHE_KEY}/
81+
82+
# Download latest ArduCopter SITL from official firmware server
83+
curl -L -o sitl/arducopter https://firmware.ardupilot.org/Copter/latest/SITL_x86_64_linux_gnu/arducopter
84+
curl -L -o sitl/firmware-version.txt https://firmware.ardupilot.org/Copter/latest/SITL_x86_64_linux_gnu/firmware-version.txt
85+
curl -L -o sitl/git-version.txt https://firmware.ardupilot.org/Copter/latest/SITL_x86_64_linux_gnu/git-version.txt
86+
87+
# Cache the downloaded files
88+
cp sitl/* sitl-cache/${CACHE_KEY}/
89+
fi
90+
91+
# Make executable and verify
92+
chmod +x sitl/arducopter
93+
ls -la sitl/
94+
95+
# Set environment variables
96+
echo "SITL_BINARY=$(pwd)/sitl/arducopter" >> $GITHUB_ENV
97+
echo "SITL_AVAILABLE=true" >> $GITHUB_ENV
98+
echo "SITL version: $(cat sitl/git-version.txt)"
99+
echo "Firmware version: $(cat sitl/firmware-version.txt)"
100+
continue-on-error: true
101+
102+
- name: Cache SITL files
103+
uses: actions/cache@v4
104+
with:
105+
path: sitl-cache/
106+
key: sitl-cache-${{ github.run_id }}
107+
restore-keys: |
108+
sitl-cache-
109+
63110
- name: Test with pytest
64111
id: pytest
65112
continue-on-error: false
@@ -72,7 +119,13 @@ jobs:
72119
Xvfb :99 -screen 0 1024x768x16 -ac &
73120
# ensure Xvfb is fully started before running tests
74121
sleep 2
75-
uv run pytest --cov=ardupilot_methodic_configurator --cov-report=xml:tests/coverage.xml --md=tests/results-${{ matrix.python-version }}.md --junit-xml=tests/results-junit.xml
122+
if [ "$SITL_AVAILABLE" = "true" ]; then
123+
echo "Running tests with SITL support"
124+
uv run pytest --cov=ardupilot_methodic_configurator --cov-report=xml:tests/coverage.xml --md=tests/results-${{ matrix.python-version }}.md --junit-xml=tests/results-junit.xml -m "sitl or not sitl"
125+
else
126+
echo "Running tests without SITL (mocked tests only)"
127+
uv run pytest --cov=ardupilot_methodic_configurator --cov-report=xml:tests/coverage.xml --md=tests/results-${{ matrix.python-version }}.md --junit-xml=tests/results-junit.xml -m "not sitl"
128+
fi
76129
77130
- name: Fix coverage paths
78131
run: |

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,7 @@ git_hash.txt
1313
venv/
1414
test.txt
1515
test.xml
16+
17+
sitl/arducopter
18+
sitl/firmware-version.txt
19+
sitl/git-version.txt

‎REUSE.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,7 @@ SPDX-FileCopyrightText = "2024-2025 Amilcar Lucas, 2025 Jonas Vermeulen"
5858
SPDX-License-Identifier = "GPL-3.0-or-later"
5959

6060
[[annotations]]
61-
path = ["pyrightconfig.json", "pyproject.toml", "pytest.ini", "test_pip_package.sh", "ardupilot_methodic_configurator_command_line_completion.psm1"]
61+
path = ["pyrightconfig.json", "pyproject.toml", "pytest.ini", "test_pip_package.sh", "ardupilot_methodic_configurator_command_line_completion.psm1", "sitl/copter.parm"]
6262
SPDX-FileCopyrightText = "2024-2025 Amilcar Lucas"
6363
SPDX-License-Identifier = "GPL-3.0-or-later"
6464

‎ardupilot_methodic_configurator/backend_flightcontroller.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -572,6 +572,11 @@ def __process_autopilot_version(self, m: MAVLink_autopilot_version_message, bann
572572
firmware_type_banner_substrings = banner_msgs[os_custom_version_index + 1].split(" ")
573573
if len(firmware_type_banner_substrings) >= 3:
574574
firmware_type = firmware_type_banner_substrings[0]
575+
elif banner_msgs and not firmware_type:
576+
# For SITL or systems without ChibiOS version, try to parse from first banner message
577+
firmware_type_banner_substrings = banner_msgs[0].split(" ")
578+
if len(firmware_type_banner_substrings) >= 1:
579+
firmware_type = firmware_type_banner_substrings[0]
575580
if firmware_type and firmware_type != self.info.firmware_type:
576581
logging_debug(
577582
_("FC firmware type mismatch: %s (BANNER) != %s (AUTOPILOT_VERSION)"), firmware_type, self.info.firmware_type

‎pytest.ini‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,3 +8,4 @@ addopts = -v --strict-config --continue-on-collection-errors
88
markers =
99
integration: mark a test as an integration test
1010
slow: mark a test as slow running
11+
sitl: mark a test as requiring SITL (real ArduPilot simulation)

0 commit comments

Comments
 (0)