Repository navigation
Open devices.yml with utf-8-sig to handle UTF-8 BOM - #193
Conversation
|
Could you please add a test, e.g. by changing one of the existing |
|
Added a test in test_04_tomato_reload.py: it writes devices_counter.json out as a devices.yml with a BOM and checks that DeviceFile still loads all devices and pipelines. On Windows it fails without the fix (device 'dev-counter' is not specified) and passes with it. I didn't put the BOM in one of the tests/common yml files, because those are payloads read by ketchup, not device files. ketchup opens payloads the same way (payload.open() without an encoding), so a payload with a BOM would break on Windows too. Should I fix that here as well, or in a separate PR? The failing Windows job was timeouts in the daemon/job tests (wait_until_tomato_components etc.), not the devices file, which loaded fine in test_01. I think #194 covers that, so I can merge main back in once it's in. |
|
If you could fix the payload issue too in this PR, that would be great. Perhaps we never noticed since I've been using Notepad++ on the windows PCs. |
…encoding # Conflicts: # tests/test_04_tomato_reload.py
|
ketchup now opens payload files with utf-8-sig, the same change as in DeviceFile Also I added a BOM to tests/common/counter_1_0.1.yml. The ketchup tests that submit it now cover payloads with a BOM |
|
LGTM, thanks! |
Fixes #191
DeviceFile.populate_attrs opened the devices file with a plain self.filename.open("r"), so on Windows it read with the system locale codepage instead of UTF-8. A UTF-8 BOM at the start of devices.yml (added automatically by some editors when saving as "UTF-8") then gets mis-decoded under that codepage, corrupting the very first key in the file devices: no longer matches, so jsdata.get("devices", {}) silently falls back to empty, and every declared device is treated as missing. This surfaces as a confusing pydantic_core.ValidationError: Assertion failed, device '' is not specified, even though the device is clearly declared
Switching to encoding="utf-8-sig" strips a BOM if one is present and is a no-op otherwise, so it's safe for files with or without one
Found and reproduced while live-testing dgbowl/marinara's #47 against tomato-3.0rc3 on Windows
Verified locally: parsed a devices.yml with a real UTF-8 BOM directly via DeviceFile(filename=...) using the patched code it now loads successfully (previously raised pydantic_core.ValidationError: device '' is not specified)