feat: update auto.partition() function to recognize Unstructured json #337

natygyoon · 2023-03-02T23:15:38Z

unstructured/partition/json.py

tomaarsen · 2023-03-03T07:46:10Z

unstructured/partition/json.py

+        if file_text_whitespace_removed[0] == "[" and file_text_whitespace_removed[-1] == "]":
+            if len(file_text_whitespace_removed) == 2 or \
+                (file_text_whitespace_removed[1] == "{" and file_text_whitespace_removed[-2] == "}"):


This is not a good way to verify if text is indeed parsable as JSON. Consider e.g.:

try: unstructured_json = json.loads(file_text) except json.JSONDecodeError: raise ... return dict_to_elements(unstructured_json)

I've originally put these conditions to quickly raise for invalid input format as optimization, since we call json_loads in elements_from_json anyways.

unstructured/partition/json.py

…eat/auto-partition

unstructured/partition/json.py

test_unstructured/partition/test_json.py

added partition_json

0cf541f

tomaarsen suggested changes Mar 3, 2023

View reviewed changes

natygyoon added 15 commits March 3, 2023 13:49

added unit tests for partition_json

4e503e1

Merge branch 'main' of github.com:Unstructured-IO/unstructured into f…

0f8e246

…eat/auto-partition

use regex to check file format

9525280

simplified regex and added JSONDecodeError exception

d9cdf2e

make tidy

8bdc8bf

re-formatted import blocks

9f628cd

re-formatted import blocks

4561709

added missing import

86b55de

make tidy

182363d

handle cases where file or text is not None

c26d1c8

added missing import

ba1a5a9

make check

c05662f

added test files

5835756

deleted test files for unsupported file types

e02b228

deleted test files for unsupported file types

f3500f4