Skip to content

Align dict/JSON/YAML import to headers by key, not by position - #663

Open
chuenchen309 wants to merge 1 commit into
jazzband:masterfrom
chuenchen309:fix/dict-import-align-by-key
Open

Align dict/JSON/YAML import to headers by key, not by position#663
chuenchen309 wants to merge 1 commit into
jazzband:masterfrom
chuenchen309:fix/dict-import-align-by-key

Conversation

@chuenchen309

Copy link
Copy Markdown

The bug

Importing a list of mappings (JSON objects / YAML mappings, or the Dataset.dict setter) builds each row from dict.values() positionally rather than by header name. When records have their keys in a different order — which is valid input — the data is silently misaligned:

import tablib
d = tablib.Dataset().load('[{"name":"alice","age":30},{"age":25,"name":"bob"}]', 'json')
list(d)   # [('alice', 30), (25, 'bob')]   <- row 1 should be ('bob', 25)

The Dataset.dict setter is affected too, and its own docstring example is exactly the shape that breaks:

d = tablib.Dataset()
d.dict = [{'first_name': 'Kenneth', 'last_name': 'Reitz', 'age': 90},
          {'age': 40, 'first_name': 'Bessie', 'last_name': 'Monke'}]
list(d)   # [('Kenneth','Reitz',90), (40,'Bessie','Monke')]  <- row 1 names/age swapped

No error is raised — the columns are silently swapped. YAML import (load(text, 'yaml')) has the same behavior.

Why this is a bug, not a limitation

JSON object member order is explicitly not significant (RFC 8259 §4: "the ordering of the members is not significant"), and YAML mapping keys are unordered. Any producer — an external API, json.dumps(sort_keys=...), another language — may legitimately emit records with differing key order. Treating key order as positionally significant contradicts the format's own data model, and a user reading the dict-setter docstring expects alignment by name.

Root cause

src/tablib/core.py, _set_dict:

self.headers = list(pickle[0].keys())
for row in pickle:
    self.append(Row(list(row.values())))   # positional

The fix

Look each value up by header name:

self.append(Row([row.get(header) for header in self.headers]))

I used row.get(header) so a record missing a key fills None rather than raising — a strict improvement over the previous silent misalignment. If you'd prefer to fail loud on inconsistent records, row[header] (raising KeyError) is the one-character alternative; happy to switch.

Verification

  • All three entry points now align correctly: JSON, YAML, and the dict setter (('bob', 25), ('Bessie','Monke',40)).
  • A record missing a header key fills None (d.dict = [{'a':1,'b':2},{'a':3}][(1,2),(3,None)]), no crash.
  • Added test_json_import_heterogeneous_key_order; it fails on master ((25, 'bob') != ('bob', 25)) and passes with the fix.
  • Full suite: 179 passed with the fix (the one unrelated test_getDate_datetime_timestamp dbf failure is pre-existing and timezone-dependent — it fails identically without this change). ruff check src/tablib/core.py clean.

Not a duplicate of #255 / #149 — those concerned tablib's own consistently-ordered output being reordered by pre-3.7 dict-hash randomization; that reasoning doesn't cover heterogeneous key order across records, which this fixes.


This PR was authored by an AI coding agent (Claude Code) running on this account: the AI found the bug, ran the repro, wrote the test, and wrote this description. The human account holder reviews every change and is accountable for it. The verification above is real and re-runnable from the diff. If this isn't the kind of contribution you want, say so and I'll close it.

_set_dict built each row with list(row.values()), so importing a list of
mappings whose keys appear in a different order across records silently
misaligned the data:

    Dataset().load('[{"name":"a","age":30},{"age":25,"name":"b"}]', 'json')
    # row 1 became (25, 'b') instead of ('b', 25)

The dict-property setter's own docstring example is exactly this shape.
JSON object member order is not significant (RFC 8259) and YAML mapping
keys are unordered, so producers may legitimately vary key order. Look
each value up by header name instead. Missing keys fill with None rather
than raising, which is a strict improvement over the previous silent
misalignment.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant