Skip to content

Open devices.yml with utf-8-sig to handle UTF-8 BOM - #193

Merged
PeterKraus merged 5 commits into
dgbowl:mainfrom
gulakbulut:fix/devicefile-bom-encoding
Oct 6, 2026
Merged

PeterKraus merged 5 commits into
dgbowl:mainfrom
gulakbulut:fix/devicefile-bom-encoding

Conversation

@gulakbulut

@gulakbulut gulakbulut commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #191

DeviceFile.populate_attrs opened the devices file with a plain self.filename.open("r"), so on Windows it read with the system locale codepage instead of UTF-8. A UTF-8 BOM at the start of devices.yml (added automatically by some editors when saving as "UTF-8") then gets mis-decoded under that codepage, corrupting the very first key in the file devices: no longer matches, so jsdata.get("devices", {}) silently falls back to empty, and every declared device is treated as missing. This surfaces as a confusing pydantic_core.ValidationError: Assertion failed, device '' is not specified, even though the device is clearly declared

Switching to encoding="utf-8-sig" strips a BOM if one is present and is a no-op otherwise, so it's safe for files with or without one
Found and reproduced while live-testing dgbowl/marinara's #47 against tomato-3.0rc3 on Windows

Verified locally: parsed a devices.yml with a real UTF-8 BOM directly via DeviceFile(filename=...) using the patched code it now loads successfully (previously raised pydantic_core.ValidationError: device '' is not specified)

@PeterKraus

Copy link
Copy Markdown
Contributor

Could you please add a test, e.g. by changing one of the existing yml files in the tests/common folder to include BOM?

@gulakbulut

Copy link
Copy Markdown
Contributor Author

Added a test in test_04_tomato_reload.py: it writes devices_counter.json out as a devices.yml with a BOM and checks that DeviceFile still loads all devices and pipelines. On Windows it fails without the fix (device 'dev-counter' is not specified) and passes with it.

I didn't put the BOM in one of the tests/common yml files, because those are payloads read by ketchup, not device files. ketchup opens payloads the same way (payload.open() without an encoding), so a payload with a BOM would break on Windows too.

Should I fix that here as well, or in a separate PR?

The failing Windows job was timeouts in the daemon/job tests (wait_until_tomato_components etc.), not the devices file, which loaded fine in test_01. I think #194 covers that, so I can merge main back in once it's in.

@PeterKraus

Copy link
Copy Markdown
Contributor

If you could fix the payload issue too in this PR, that would be great. Perhaps we never noticed since I've been using Notepad++ on the windows PCs.

@gulakbulut

gulakbulut commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor Author

ketchup now opens payload files with utf-8-sig, the same change as in DeviceFile

Also I added a BOM to tests/common/counter_1_0.1.yml. The ketchup tests that submit it now cover payloads with a BOM

@PeterKraus

Copy link
Copy Markdown
Contributor

LGTM, thanks!

@PeterKraus
PeterKraus merged commit d89fb36 into dgbowl:main Oct 6, 2026
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DeviceFile fails to parse devices.yml files saved with a UTF-8 BOM on Windows

2 participants