[WIP] Staging/dev/profile serialization by taylorfturner · Pull Request #908 · capitalone/DataProfiler

taylorfturner · 2023-06-26T19:40:15Z

Updating dev with feature/profile-serialization work in preparation for next release
Main updates through the feature/profile-serialization branch are all low-level profilers, compiler, Structured Col Profiler, Structured Profiler, and Unstructured Profiler (pending PR) encode / decode

…filer (#852)

* hot fixes for encode and decode of numeric stats mixin and intcol profiler * cleaned up type checking and updated numericstatsmixin readin helper to give type conversions to more attributes * Added docstring to the _load_stats_helper function * Update dataprofiler/profilers/numerical_column_stats.py Co-authored-by: Taylor Turner <taylorfturner@gmail.com> * Update dataprofiler/profilers/numerical_column_stats.py * fix for nan values issue in pytesting * Implementation of float profiler encode and decode process --------- Co-authored-by: Taylor Turner <taylorfturner@gmail.com>

* more verbose error log with types for easy debug * add load_from_dict to handle tiimestamps * add json decode tests * include DateTimeColumn class

…erent ordering of values is introduced (#868)

…piler (#885) * feat: add test and compiler serialization * fix: move primitive tests to own class * feat: add primitive col compiler save tests * fix: float serializers asserts

… fixes numerical deserialization (#886) * feat: add test and compiler serialization * fix: move primitive tests to own class * feat: add primitive col compiler save tests * fix: float serializers asserts * feat: add tests and allow primitive compiler to deserialize * fix: bug in numeric stats deserial * fix: missing `)` after conflict resolution

…refactors for order Typing (#887) * fix: organize categorical and add get function * refactor: reorganize tests and add stats test * feat: order typing * feat: add serial and deserial for stats compiler * fix: bug when sample_size == 0

…n for datalabeler (#879)

* Added initial profiler decoding for datalabeler column (WIP) * Intialial implementation for deserialization of datalabelercolumn * Fix LSP violations (#840) * Make profiler superclasses generic Makes the superclasses BaseColumnProfiler, NumericStatsMixin, and BaseCompiler generic, to avoid casting in subclass diff() methods and violating LSP in principle. * Add needed cast import --------- Co-authored-by: Junho Lee <53921230+junholee6a@users.noreply.github.com>

* encode testing * encode dataLabeler testing * encode structuredOptions testing * cleaned up datalabeler test * added text options

* formatting * update formatting * setting up full test suite for DataLabelerCompiler * update isort * updates to test -- still failing * update

* update * string in list * formatting

* refactored options encode testing * updated test name * updated class names * fixing test * initial base option decode * inital tests

* refactor: allow options to go through all * fix: bug

* refactor: allow options to go through all * fix: bug * update * update * update * updates * update * Fixes for taylors StructuredCol Issue * update * update * remove try/except --------- Co-authored-by: Jeremy Goodsitt <jeremy.goodsitt@gmail.com> Co-authored-by: ksneab7 <ksneab7@gmail.com>

* fix: bug and add tests * fix: limit scipy requirements till problem understood and fixed

taylorfturner · 2023-06-26T21:20:23Z

rebased

taylorfturner · 2023-06-27T12:37:34Z

dataprofiler/profilers/categorical_column_profile.py

rebase addition

taylorfturner · 2023-06-27T12:37:53Z

dataprofiler/profilers/order_column_profile.py

rebase addition

taylorfturner · 2023-06-27T13:03:38Z

dataprofiler/profilers/base_column_profilers.py

actually concerned there may be an issue with the rebase here.... I thought there was or should be another small class here... but don't see it in main, dev, or feature/profile-serialization. Just take your time reviewing

disregard -- the thing I thought would be here is actually properly in order_column_profile.py

taylorfturner · 2023-06-27T13:07:09Z

requirements.txt

fixing scipy issue @JGSweets

taylorfturner · 2023-06-27T14:50:35Z

dataprofiler/tests/profilers/profiler_options/test_categorical_options.py

rebase change to include feature/memory-optimization work

* refactor: allow options to go through all * fix: bug in loading options * update * update * Fixes for taylors StructuredCol Issue * Created load and save code from structuredprofiler * intermidiate commit for fixing structured profile --------- Co-authored-by: Jeremy Goodsitt <jeremy.goodsitt@gmail.com> Co-authored-by: taylorfturner <taylorfturner@gmail.com>

* refactor: allow options to go through all * fix: bug in loading options * update * update * Fixes for taylors StructuredCol Issue * Created load and save code from structuredprofiler * intermidiate commit for fixing structured profile * test fix * mypy fixes for typing issues * fix for none case of the datalabler in options * Added mock of datalabeler to structured profile test * Added tests for encoding of the Structured profiler * Update dataprofiler/profilers/json_decoder.py Co-authored-by: Michael Davis <36012613+micdavis@users.noreply.github.com> * Update dataprofiler/profilers/profile_builder.py Co-authored-by: Michael Davis <36012613+micdavis@users.noreply.github.com> * Update dataprofiler/profilers/profiler_options.py Co-authored-by: Michael Davis <36012613+micdavis@users.noreply.github.com> * Pr fixes * Fixed typo in test * Update dataprofiler/profilers/json_decoder.py Co-authored-by: Taylor Turner <taylorfturner@gmail.com> * Update dataprofiler/profilers/profile_builder.py Co-authored-by: Michael Davis <36012613+micdavis@users.noreply.github.com> * Update dataprofiler/tests/profilers/utils.py Co-authored-by: Taylor Turner <taylorfturner@gmail.com> * Update dataprofiler/profilers/profile_builder.py Co-authored-by: Michael Davis <36012613+micdavis@users.noreply.github.com> * Fixes for unneeeded callout for _profile check * small change --------- Co-authored-by: Jeremy Goodsitt <jeremy.goodsitt@gmail.com> Co-authored-by: taylorfturner <taylorfturner@gmail.com> Co-authored-by: ksneab7 <ksneab7@gmail.com> Co-authored-by: ksneab7 <91956551+ksneab7@users.noreply.github.com>

taylorfturner · 2023-06-28T13:03:23Z

Pending #923 and #924

* refactor: loading labeler for reuse and abstract loading * refactor: use for DataLabelerColumn as well * fix: don't error if doesn't exist * refactor: allow for config dict to be passed entire way * fix: compiler tests * fix: structCol tests * fix: test

* added save for top level and tests * small refactor * small fix

* refactor: use seed for sample for consistency * fix: formatting and variables

* added load_method * updated tests

* update example data profiler demo save/load * update notebook cells * Update examples/data_profiler_demo.ipynb * Update examples/data_profiler_demo.ipynb

micdavis and others added 22 commits May 16, 2023 09:36

initial changes to categoricalColumn decoder (#818)

7cd3ae4

Implemented decoding for numerical stats mixin and integer profiles (#…

b4ad93d

…844)

hot fixes for encode and decode of numeric stats mixin and intcol pro…

620f0d9

…filer (#852)

Json decode date time column (#861)

8b4d6e2

* more verbose error log with types for easy debug * add load_from_dict to handle tiimestamps * add json decode tests * include DateTimeColumn class

Added decoding for encoding of ordered column profiles (#864)

42ad4a4

Added ordered col test to ensure correct response to update when diff…

f4f7e47

…erent ordering of values is introduced (#868)

added decode text_column_profiler functionality and tests (#870)

2390004

Created encoder for the datalabelercolumn (#869)

6eb6852

feat: add test and compiler serialization (#884)

6c51368

[WIP] Adds tests validating serialization with Primitive type for com…

8285293

…piler (#885) * feat: add test and compiler serialization * fix: move primitive tests to own class * feat: add primitive col compiler save tests * fix: float serializers asserts

ready datalabeler for deserialization and improvement on serializatio…

f46b8a9

…n for datalabeler (#879)

Encode Options (#875)

3eb7f75

* encode testing * encode dataLabeler testing * encode structuredOptions testing * cleaned up datalabeler test * added text options

[WIP] ColumnDataLabelerCompiler: serialize / deserialize (#888)

3ebadb4

* formatting * update formatting * setting up full test suite for DataLabelerCompiler * update isort * updates to test -- still failing * update

Quick Test update (#893)

31f2a7a

* update * string in list * formatting

Decode options (#894)

96ca39f

* refactored options encode testing * updated test name * updated class names * fixing test * initial base option decode * inital tests

refactor: allow options to go through all (#902)

ee1f602

* refactor: allow options to go through all * fix: bug

fix: bug and add tests for structuredcolprofiler (#904)

1e03f90

* fix: bug and add tests * fix: limit scipy requirements till problem understood and fixed

taylorfturner self-assigned this Jun 26, 2023

taylorfturner requested review from JGSweets, ksneab7, micdavis and tyfarnan as code owners June 26, 2023 19:40

taylorfturner added the New Feature A feature addition not currently in the library label Jun 26, 2023

taylorfturner changed the title ~~WIP Staging/dev/profile serialization~~ [WIP] Staging/dev/profile serialization Jun 26, 2023

taylorfturner added the do_not_merge label Jun 26, 2023

taylorfturner force-pushed the staging/dev/profile-serialization branch from f927d52 to f33b3fa Compare June 26, 2023 21:18

taylorfturner commented Jun 27, 2023

View reviewed changes

dataprofiler/profilers/categorical_column_profile.py Outdated

Comment on lines +392 to +283

Copy link

Contributor Author

taylorfturner Jun 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rebase addition

taylorfturner commented Jun 27, 2023

View reviewed changes

dataprofiler/profilers/order_column_profile.py Outdated

Comment on lines +15 to +26

Copy link

Contributor Author

taylorfturner Jun 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rebase addition

taylorfturner commented Jun 27, 2023

View reviewed changes

requirements.txt Outdated

Copy link

Contributor Author

taylorfturner Jun 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixing scipy issue @JGSweets

taylorfturner removed the do_not_merge label Jun 27, 2023

taylorfturner commented Jun 27, 2023

View reviewed changes

ksneab7 and others added 3 commits June 27, 2023 12:26

Added testing for values for test_json_decode_after_update (#915)

7f6dffa

taylorfturner added the Work In Progress Solution is being developed label Jun 27, 2023

JGSweets and others added 8 commits June 28, 2023 08:37

Reuse passed labeler (#924)

2320718

* refactor: loading labeler for reuse and abstract loading * refactor: use for DataLabelerColumn as well * fix: don't error if doesn't exist * refactor: allow for config dict to be passed entire way * fix: compiler tests * fix: structCol tests * fix: test

BaseProfiler save() for json (#923)

1a59a33

* added save for top level and tests * small refactor * small fix

refactor: use seed for sample for consistency (#927)

5ec8907

* refactor: use seed for sample for consistency * fix: formatting and variables

WIP top level load (#925)

52f54f6

quick hot fix for input validation on save() save_metho (#931)

55202ac

BaseProfiler: load_method hotfix (#932)

bd04cd5

* added load_method * updated tests

fix: null_rep mat should calculate even if datetime (#933)

bcf9eeb

Notebook Example save/load Profile (#930)

423bc0a

* update example data profiler demo save/load * update notebook cells * Update examples/data_profiler_demo.ipynb * Update examples/data_profiler_demo.ipynb

micdavis force-pushed the staging/dev/profile-serialization branch from aa4795f to 423bc0a Compare June 29, 2023 01:08

taylorfturner closed this Jun 29, 2023

taylorfturner deleted the staging/dev/profile-serialization branch June 29, 2023 01:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Staging/dev/profile serialization#908

[WIP] Staging/dev/profile serialization#908
taylorfturner wants to merge 33 commits intodevfrom
staging/dev/profile-serialization

taylorfturner commented Jun 26, 2023 •

edited

Loading

Uh oh!

taylorfturner commented Jun 26, 2023

Uh oh!

taylorfturner Jun 27, 2023

Uh oh!

taylorfturner Jun 27, 2023

Uh oh!

taylorfturner Jun 27, 2023 •

edited

Loading

Uh oh!

taylorfturner Jun 27, 2023 •

edited

Loading

Uh oh!

taylorfturner Jun 27, 2023

Uh oh!

taylorfturner Jun 27, 2023

Uh oh!

taylorfturner commented Jun 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

taylorfturner commented Jun 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

taylorfturner commented Jun 26, 2023

Uh oh!

taylorfturner Jun 27, 2023

Choose a reason for hiding this comment

Uh oh!

taylorfturner Jun 27, 2023

Choose a reason for hiding this comment

Uh oh!

taylorfturner Jun 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

taylorfturner Jun 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

taylorfturner Jun 27, 2023

Choose a reason for hiding this comment

Uh oh!

taylorfturner Jun 27, 2023

Choose a reason for hiding this comment

Uh oh!

taylorfturner commented Jun 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

taylorfturner commented Jun 26, 2023 •

edited

Loading

taylorfturner Jun 27, 2023 •

edited

Loading

taylorfturner Jun 27, 2023 •

edited

Loading