Align dict/JSON/YAML import to headers by key, not by position - #663
Open
chuenchen309 wants to merge 1 commit into
Open
Align dict/JSON/YAML import to headers by key, not by position#663chuenchen309 wants to merge 1 commit into
chuenchen309 wants to merge 1 commit into
Conversation
_set_dict built each row with list(row.values()), so importing a list of
mappings whose keys appear in a different order across records silently
misaligned the data:
Dataset().load('[{"name":"a","age":30},{"age":25,"name":"b"}]', 'json')
# row 1 became (25, 'b') instead of ('b', 25)
The dict-property setter's own docstring example is exactly this shape.
JSON object member order is not significant (RFC 8259) and YAML mapping
keys are unordered, so producers may legitimately vary key order. Look
each value up by header name instead. Missing keys fill with None rather
than raising, which is a strict improvement over the previous silent
misalignment.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
Importing a list of mappings (JSON objects / YAML mappings, or the
Dataset.dictsetter) builds each row fromdict.values()positionally rather than by header name. When records have their keys in a different order — which is valid input — the data is silently misaligned:The
Dataset.dictsetter is affected too, and its own docstring example is exactly the shape that breaks:No error is raised — the columns are silently swapped. YAML import (
load(text, 'yaml')) has the same behavior.Why this is a bug, not a limitation
JSON object member order is explicitly not significant (RFC 8259 §4: "the ordering of the members is not significant"), and YAML mapping keys are unordered. Any producer — an external API,
json.dumps(sort_keys=...), another language — may legitimately emit records with differing key order. Treating key order as positionally significant contradicts the format's own data model, and a user reading thedict-setter docstring expects alignment by name.Root cause
src/tablib/core.py,_set_dict:The fix
Look each value up by header name:
I used
row.get(header)so a record missing a key fillsNonerather than raising — a strict improvement over the previous silent misalignment. If you'd prefer to fail loud on inconsistent records,row[header](raisingKeyError) is the one-character alternative; happy to switch.Verification
dictsetter (('bob', 25),('Bessie','Monke',40)).None(d.dict = [{'a':1,'b':2},{'a':3}]→[(1,2),(3,None)]), no crash.test_json_import_heterogeneous_key_order; it fails onmaster((25, 'bob') != ('bob', 25)) and passes with the fix.179 passedwith the fix (the one unrelatedtest_getDate_datetime_timestampdbf failure is pre-existing and timezone-dependent — it fails identically without this change).ruff check src/tablib/core.pyclean.Not a duplicate of #255 / #149 — those concerned tablib's own consistently-ordered output being reordered by pre-3.7 dict-hash randomization; that reasoning doesn't cover heterogeneous key order across records, which this fixes.
This PR was authored by an AI coding agent (Claude Code) running on this account: the AI found the bug, ran the repro, wrote the test, and wrote this description. The human account holder reviews every change and is accountable for it. The verification above is real and re-runnable from the diff. If this isn't the kind of contribution you want, say so and I'll close it.