Skip to content

Commit

Permalink
Add polars DataFrame support for Dataset (#2029)
Browse files Browse the repository at this point in the history
* initial commit to try implement from_dataframes for a burn dataset

* added the beginnings of tests. removed ref to self in utility method

* added unit test for dataframe module. added utility methods to convert polars rows to burn dataset values

* putting polars and dataframe mod behind a fearure flag

* testing both methods

* added a if let OK so that it doesn't panic. if we can't convert serde map to json string. added comments

* using polars serializer, renaming vars

* removed prints. just unwrapping

* setting feature flags back

* return Value::Null rather than panic if we can't serialize list value. no longer convert to object before converting to string. no longer using serde_json to_string method

* Use native deserializer instead of serde_json

* added support for lazyframes. added support to deserialize a few more data. added a few more tests

* Remove lazy, add more testing and other fixes

* Update the book

* Remove lazy feature

* Put back lazy feature for polars

---------

Co-authored-by: Dilshod Tadjibaev <939125+antimora@users.noreply.github.com>
  • Loading branch information
ragyabraham and antimora authored Jul 31, 2024
1 parent f673721 commit 04d7ff2
Show file tree
Hide file tree
Showing 6 changed files with 1,296 additions and 50 deletions.
Loading

0 comments on commit 04d7ff2

Please sign in to comment.