Edge-BehAV

Benchmark and evaluation code for "Offline Vision-Language Navigation with Geometric Goal Localization for Outdoor Environments."

Edge-BehAV runs the full vision-language navigation stack on the robot: instruction decomposition, open-vocabulary goal detection, prompted segmentation, and LiDAR-based metric goal estimation, with no cloud services and no network connectivity.

Watch the supplementary video (2 min 42 s)  ·  Project page

_{Four closed-loop outdoor trials, each shown from the robot's onboard view and a third-person view. Silent and captioned.}

Note (under review). This repository is linked from a manuscript currently under double-anonymous review, so author and institution details are omitted for now. They will be added, along with the citation entry, once the review process concludes.

---

What is here

Path	Contents
benchmark/	The 350-instruction navigation decomposition benchmark with human annotations, as CSV and JSON
results/	Per-instruction model outputs for 21 models across three embedded platforms and four cloud APIs
results/leaderboard.csv	Headline accuracy and latency for every model reported in the paper
results/cross_platform.csv	Identical INT4 checkpoints measured on all three platforms
scripts/	Scoring and cloud-benchmark harness for instruction decomposition
detection/	Open-vocabulary landmark localization benchmark on a nuImages subset, with per-image results
docs/	Source of the project page, including the supplementary video at docs/assets/video.mp4

The two evaluations are independent: benchmark/, results/, and scripts/ cover instruction decomposition, while detection/ is self-contained and covers goal detection.

The benchmark

350 natural-language navigation instructions, evenly split across seven semantic categories (50 each):

Category	Focus
Surface/Terrain	ground types, surface constraints
Human Interaction	pedestrians, social constraints
Static Obstacles	fixed objects to avoid or approach
Compound Instructions	multiple chained constraints
Contextual/Adversarial	ambiguous or misleading phrasing
Off-Road & Natural Terrain	unstructured outdoor settings
Wet & Winter Terrain	puddles, snow, ice

Every instruction is annotated by hand with four target fields: landmarks, navigation actions, behavioral actions, and behavioral targets, plus a per-target behavioral cost. Field definitions are in benchmark/SCHEMA.md.

python3 - <<'PY'
import json
data = json.load(open("benchmark/instructions.json"))
print(len(data), "instructions")
print(data[0]["Instruction"])
print(data[0]["GT_Landmarks"], data[0]["GT_Behavioral_Actions"])
PY

Results

Decomposition quality is scored as Macro-F1, the mean set-level F1 over the four annotated fields, with 95% bootstrap confidence intervals. Latency is the median per call.

Model	Type	Macro-F1	Latency (s)
Qwen2.5-7B (recommended)	offline LLM	0.775	1.13
GPT-5.5	cloud API	0.765	9.95
Qwen2.5-3B	offline LLM	0.718	0.53
GPT-5.4-mini	cloud API	0.700	2.19
GPT-4	cloud API	0.512	4.05

The strongest offline model is statistically on par with the best cloud API, with overlapping confidence intervals, while running roughly 9x faster and entirely on-board. Full results for all models are in results/leaderboard.csv.

Cloud models use BehAV's original prompt with a type-tolerant field extractor (Prompt_Variant = legacy, Extractor = robust).

On goal detection, Florence-2 running on-board reaches an F-score of 0.974 at 307 ms per image, against 0.904 at 3198 ms for the FastSAM and GPT-4o pipeline of the reference system. See detection/ for the full evaluation.

Geometric goal localization reduces mean goal-distance error from 2.05 m to 0.20 m, and the integrated system completes 31 of 32 closed-loop outdoor trials on a Clearpath Husky A200.

Reproducing the scores

pip install openpyxl
python3 scripts/score_macro_f1.py --results results/laptop_rtx4070/qwen2.5_7b.csv

To re-run the cloud comparison you need your own API key:

export OPENAI_API_KEY=...
python3 scripts/harmonized_cloud_benchmark.py --model gpt-5.5 --prompt harmonized --extractor robust

Platforms

Offline models were evaluated on an NVIDIA Jetson Orin NX, an NVIDIA Jetson Thor, and a laptop RTX 4070, using identical INT4 checkpoints. Closed-loop experiments ran on a Clearpath Husky A200 with an Ouster OS1-64 LiDAR and an Intel RealSense L515 depth camera.

License

The benchmark and annotations in benchmark/ and results/ are released under CC BY 4.0. All code, in scripts/ and detection/, is released under the MIT License.

One exception: the result files in detection/results/ contain ground-truth columns derived from nuImages and therefore stay under that dataset's CC BY-NC-SA 4.0 terms, which restrict them to non-commercial use. See detection/README.md.

Citation

The citation is withheld while the manuscript is under double-anonymous review. It will be added here once the review process concludes.

@article{edgebehav,
  title  = {Offline Vision-Language Navigation with Geometric Goal Localization
            for Outdoor Environments},
  note   = {Under review},
  year   = {2026}
}