Skip to content

feat: Add Spark-compatible implementation of SchemaAdapterFactory #1169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Dec 17, 2024
Merged

feat: Add Spark-compatible implementation of SchemaAdapterFactory #1169

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
9111902
Add Spark-compatible SchemaAdapterFactory implementation
andygrove Dec 13, 2024
861ef1d
remove prototype code
andygrove Dec 13, 2024
0bc7e82
fix
andygrove Dec 13, 2024
5d912ff
refactor
andygrove Dec 13, 2024
fb53796
implement more cast logic
andygrove Dec 13, 2024
51ddea1
implement more cast logic
andygrove Dec 13, 2024
690b0de
add basic test
andygrove Dec 13, 2024
940dadd
improve test
andygrove Dec 13, 2024
6b108c7
cleanup
andygrove Dec 13, 2024
f4d76f1
fmt
andygrove Dec 13, 2024
37bb93a
add support for casting unsigned int to signed int
andygrove Dec 13, 2024
e077008
clippy
andygrove Dec 13, 2024
295f468
address feedback
andygrove Dec 17, 2024
8417e1f
fix test
andygrove Dec 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 63 additions & 4 deletions native/Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion native/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ arrow-buffer = { version = "53.2.0" }
arrow-data = { version = "53.2.0" }
arrow-schema = { version = "53.2.0" }
parquet = { version = "53.2.0", default-features = false, features = ["experimental"] }
datafusion-common = { version = "43.0.0" }
datafusion = { version = "43.0.0", default-features = false, features = ["unicode_expressions", "crypto_expressions"] }
datafusion-common = { version = "43.0.0" }
datafusion-functions = { version = "43.0.0", features = ["crypto_expressions"] }
datafusion-functions-nested = { version = "43.0.0", default-features = false }
datafusion-expr = { version = "43.0.0", default-features = false }
Expand Down
3 changes: 1 addition & 2 deletions native/core/src/parquet/util/test_common/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,9 @@
// specific language governing permissions and limitations
// under the License.

pub mod file_util;
pub mod page_util;
pub mod rand_gen;

pub use self::rand_gen::{random_bools, random_bytes, random_numbers, random_numbers_range};

pub use self::file_util::{get_temp_file, get_temp_filename};
pub use datafusion_comet_spark_expr::test_common::file_util::{get_temp_file, get_temp_filename};
4 changes: 3 additions & 1 deletion native/spark-expr/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ arrow-buffer = { workspace = true }
arrow-data = { workspace = true }
arrow-schema = { workspace = true }
chrono = { workspace = true }
datafusion = { workspace = true }
datafusion = { workspace = true, features = ["parquet"] }
datafusion-common = { workspace = true }
datafusion-expr = { workspace = true }
datafusion-physical-expr = { workspace = true }
Expand All @@ -43,9 +43,11 @@ regex = { workspace = true }
thiserror = { workspace = true }
futures = { workspace = true }
twox-hash = "2.0.0"
rand = { workspace = true }

[dev-dependencies]
arrow-data = {workspace = true}
parquet = { workspace = true, features = ["arrow"] }
criterion = "0.5.1"
rand = { workspace = true}
tokio = { version = "1", features = ["rt-multi-thread"] }
Expand Down
Loading
Loading