Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add --fields-include to unstructured-ingest #376

Merged
merged 52 commits into from
Mar 22, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
3ef6265
added metadata in/exclude params
natygyoon Mar 15, 2023
a084a5c
updated process_file
natygyoon Mar 15, 2023
499280d
existing tests
natygyoon Mar 15, 2023
3b8d369
remove default behavior
natygyoon Mar 15, 2023
e8319f5
changelog and ci
natygyoon Mar 16, 2023
2d01c7a
line length
natygyoon Mar 16, 2023
d6ab499
import
natygyoon Mar 16, 2023
ef788f6
import
natygyoon Mar 16, 2023
297ff5f
import sorted
natygyoon Mar 16, 2023
780f7ff
import
natygyoon Mar 16, 2023
252654b
type
natygyoon Mar 16, 2023
59d7a00
line length
natygyoon Mar 16, 2023
b5393c3
version sync
natygyoon Mar 16, 2023
ef07f40
main
natygyoon Mar 16, 2023
1283a84
ci
natygyoon Mar 16, 2023
97d04e6
json
natygyoon Mar 16, 2023
dd0319a
dict
natygyoon Mar 16, 2023
bdd4a1e
type ignore
natygyoon Mar 16, 2023
79333a5
lint
natygyoon Mar 16, 2023
fc1db61
unit tests for process_file
natygyoon Mar 17, 2023
4f92b68
lint
natygyoon Mar 17, 2023
203e1b2
added --fields-include
natygyoon Mar 17, 2023
53b7aea
lint
natygyoon Mar 17, 2023
e35a04f
line length
natygyoon Mar 17, 2023
b5d7b3e
fix version
natygyoon Mar 17, 2023
3905681
type changed to Optional(str)
natygyoon Mar 20, 2023
c690284
ci
natygyoon Mar 20, 2023
5fab790
line length
natygyoon Mar 20, 2023
46b7550
merge `feat/metadata` into `feat/fields-include`
natygyoon Mar 20, 2023
c0f2ad0
merge conflict
natygyoon Mar 20, 2023
dbf7774
merge conflict
natygyoon Mar 20, 2023
ed5869d
line length
natygyoon Mar 20, 2023
45c9b86
type check
natygyoon Mar 20, 2023
84b831a
line length
natygyoon Mar 20, 2023
c0dbcbe
default
natygyoon Mar 20, 2023
a18d91d
subclass type
natygyoon Mar 20, 2023
7a131b7
fixed dict iter error
natygyoon Mar 20, 2023
054fdc4
Merge branch 'main' into feat/fields-include
natygyoon Mar 20, 2023
bc758bc
Merge branch 'main' into feat/fields-include
natygyoon Mar 21, 2023
6c42cf4
code refactor
natygyoon Mar 21, 2023
f1f90d2
added unit tests for fields_include
natygyoon Mar 21, 2023
245266b
version sync
natygyoon Mar 21, 2023
73f0f6d
Merge branch 'main' into feat/fields-include
natygyoon Mar 21, 2023
509457b
remove custom class
natygyoon Mar 21, 2023
33faf50
unit tests
natygyoon Mar 21, 2023
ec46090
changelog
natygyoon Mar 21, 2023
4e7da4f
version bump
natygyoon Mar 22, 2023
d98c8c2
nit
natygyoon Mar 22, 2023
7fd4ad5
Merge branch 'main' into feat/fields-include
natygyoon Mar 22, 2023
3b46996
remove duplicate
natygyoon Mar 22, 2023
840a1ae
nit
natygyoon Mar 22, 2023
f29bf65
Merge branch 'main' into feat/fields-include
natygyoon Mar 22, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
changelog and ci
  • Loading branch information
natygyoon committed Mar 16, 2023
commit e8319f52e089e78ed4f9fa0e63d1559977652485
3 changes: 2 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
## 0.5.5-dev0
## 0.5.5-dev1

### Enhancements

### Features

* Add `--metadata-include` and `--metadata-exclude` parameters to `unstructured-ingest`
* Add `clean_non_ascii_chars` to remove non-ascii characters from unicode string

### Fixes
Expand Down
10 changes: 5 additions & 5 deletions unstructured/ingest/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,14 +53,14 @@ def handle_parse_result(self, ctx, opts, args):
"Illegal usage: `{}` is mutually exclusive with "
"arguments `{}`.".format(
self.name,
', '.join(self.mutually_exclusive)
)
', '.join(self.mutually_exclusive),
),
)

return super(MutuallyExclusiveOption, self).handle_parse_result(
ctx,
opts,
args
args,
)


Expand Down Expand Up @@ -137,14 +137,14 @@ def run(self):
cls=MutuallyExclusiveOption,
default=None,
help="If set, include the specified metadata fields if they exist and drop all other fields.",
mutually_exclusive=["--metadata-exclude"]
mutually_exclusive=["--metadata-exclude"],
)
@click.option(
"--metadata-exclude",
cls=MutuallyExclusiveOption,
default=None,
help="If set, drop the specified metadata fields if they exist.",
mutually_exclusive=["--metadata-include"]
mutually_exclusive=["--metadata-include"],
)
@click.option(
"--remote-url",
Expand Down