Skip to content

Commit

Permalink
Pydantic Transformer V2 (#2792)
Browse files Browse the repository at this point in the history
* Pydantic Transformer V2

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* add __init__.py

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* add json schema

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* convert float to int

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* change gitsha in test script mode

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* change gitsha

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* use strict map=false

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* Test flytefile console input + attr access

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* add conditional branch

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* better rx

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* Add flytedir generic -> flytedir

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* support enum

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* update

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* add tests for input from flyte console

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* Add Tests for dataclass in BaseModel and pydantic.dataclass in BaseModel

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* update

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* update thomas's advice

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* change tree file structure

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* update Niel's advice

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* > to >=

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* try monodoc build again

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* add pydantic README.md

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* revert -vvv in monodocs

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* use model_validate_json to turn protobuf struct to python val

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* fix issue

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* handle flyte types in dict transformer from protobuf struct input (e.g FlyteConsole)

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* Add print

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* expected python type

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* switch call function order

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* try msgpack to handle protobug struct

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* Better Comment in Dict Transformer

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* Propeller -> FlytePropeller

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* dict_to_flyte_types

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* remove comments

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* add attr for protobuf struct . dict

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* Add Life Cycle for Flyte Types

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* better comments for derializing flyteschema and sd

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* nit

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* add back pv._remote_path = None to flytefile and flytedir

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* experiment

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* experiment

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* experiment

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* Add comments

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* update Yee's advice

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* code dc -> bm

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* Example dc -> bm and Example all flyte types

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* fix union dataclass, not yet add comments

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* add pydantic and dataclass optional test

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* NoneType=type(None)

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* fix union transformer none case with Eduardo

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* add comments for none type transformer + union transformer

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* add TODO

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* use deserailize = True

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* add all deserialize

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* better comments

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* better comments

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* test

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* Fix flyte directory issue by discussion with Kevin

Signed-off-by: Future-Outlier <eric901201@gmail.com>
Co-authored-by: pingsutw  <pingsutw@apache.org>

* add tests for providing conext when doing serialization

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* lint

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* test

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* move setattr to core

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* remove comments

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* lint

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* remove

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* testing

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* nit

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* flytekit/core/type_engine.py

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* add tests for Union

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* Trigger CI

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* remove nonetype

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* raw=Fasle as default

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* pydantic move to core test

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* move to core

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* log

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* update

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* lint

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* test

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* nit

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* nit

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* lint

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* move to type_engine

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* move back to init

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* update kevin's advice

Signed-off-by: Future-Outlier <eric901201@gmail.com>
Co-authored-by: pingsutw  <pingsutw@apache.org>

* wip

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* use decorator

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* decorator

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* fix syntax to support python 3.9

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* add Eduardo's advice

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* warning

Signed-off-by: Future-Outlier <eric901201@gmail.com>

* Update Yee's advice

Signed-off-by: Future-Outlier <eric901201@gmail.com>
Co-authored-by: Yee Hing Tong  <wild-endeavor@users.noreply.github.com>

* Show traceback by default (#2862)

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Support Identifier in generate_console_url (#2868)

Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>

* Support overriding node metadata for array node (#2865)

* set parent node metadata on arraynode subnode

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

* clean up setting passed in metadata for array node

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

* add comment

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

* test

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

---------

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

* Fix Jupyter Versioning  (#2866)

* Update versioning

Signed-off-by: Mecoli1219 <michaellai901026@gmail.com>

* Fix unit test

Signed-off-by: Mecoli1219 <michaellai901026@gmail.com>

* Update path if jupyter file actually exist

Signed-off-by: Mecoli1219 <michaellai901026@gmail.com>

* Fix pickled twice problem

Signed-off-by: Mecoli1219 <michaellai901026@gmail.com>

---------

Signed-off-by: Mecoli1219 <michaellai901026@gmail.com>

* improved output handling in notebooks (#2869)

---------

Signed-off-by: Future-Outlier <eric901201@gmail.com>
Co-authored-by: pingsutw <pingsutw@apache.org>
Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
  • Loading branch information
3 people authored Oct 28, 2024
1 parent 6d3d3e7 commit ff2d0da
Show file tree
Hide file tree
Showing 28 changed files with 3,082 additions and 447 deletions.
1 change: 0 additions & 1 deletion .github/workflows/pythonbuild.yml
Original file line number Diff line number Diff line change
Expand Up @@ -356,7 +356,6 @@ jobs:
- flytekit-pandera
- flytekit-papermill
- flytekit-polars
- flytekit-pydantic
- flytekit-ray
- flytekit-snowflake
- flytekit-spark
Expand Down
1 change: 1 addition & 0 deletions dev-requirements.in
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ torch<=1.12.1; python_version<'3.11'
# pytorch 2 supports python 3.11
# pytorch 2 does not support 3.12 yet: https://github.com/pytorch/pytorch/issues/110436
torch; python_version<'3.12'
pydantic

# TODO: Currently, the python-magic library causes build errors on Windows due to its dependency on DLLs for libmagic.
# We have temporarily disabled this feature on Windows and are using python-magic for Mac OS and Linux instead.
Expand Down
Loading

0 comments on commit ff2d0da

Please sign in to comment.