Skip to content

chore: Merge remote-tracking branch 'apache/main' into comet-parquet-exec - 20240121 #1316

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 81 commits into from
Jan 21, 2025
Merged

chore: Merge remote-tracking branch 'apache/main' into comet-parquet-exec - 20240121 #1316

Changes from all commits
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
9657b75
feat: support array_append (#1072)
NoeB Nov 13, 2024
c32bf0c
chore: Simplify CometShuffleMemoryAllocator to use Spark unified memo…
viirya Nov 14, 2024
f3da844
docs: Update benchmarking.md (#1085)
rluvaton-flarion Nov 14, 2024
2c832b4
feat: Require offHeap memory to be enabled (always use unified memory…
andygrove Nov 14, 2024
7cec285
test: Restore one test in CometExecSuite by adding COMET_SHUFFLE_MODE…
viirya Nov 15, 2024
10ef62a
Add changelog for 0.4.0 (#1089)
andygrove Nov 15, 2024
0c9a403
chore: Prepare for 0.5.0 development (#1090)
andygrove Nov 15, 2024
406ffef
build: Skip installation of spark-integration and fuzz testing modul…
parthchandra Nov 15, 2024
bfd7054
Add hint for finding the GPG key to use when publishing to maven (#1093)
andygrove Nov 15, 2024
59da6ce
docs: Update documentation for 0.4.0 release (#1096)
andygrove Nov 18, 2024
ca3a529
fix: Unsigned type related bugs (#1095)
kazuyukitanimura Nov 19, 2024
b64c13d
chore: Include first ScanExec batch in metrics (#1105)
andygrove Nov 20, 2024
19dd58d
chore: Improve CometScan metrics (#1100)
andygrove Nov 20, 2024
e602305
chore: Add custom metric for native shuffle fetching batches from JVM…
andygrove Nov 21, 2024
9990b34
feat: support array_insert (#1073)
SemyonSinchenko Nov 22, 2024
500895d
feat: enable decimal to decimal cast of different precision and scale…
himadripal Nov 22, 2024
7b1a290
docs: fix readme FGPA/FPGA typo (#1117)
gstvg Nov 24, 2024
5400fd7
fix: Use RDD partition index (#1112)
viirya Nov 25, 2024
ebdde77
fix: Various metrics bug fixes and improvements (#1111)
andygrove Dec 2, 2024
9b250c4
fix: Don't create CometScanExec for subclasses of ParquetFileFormat (…
Kimahriman Dec 2, 2024
95727aa
fix: Fix metrics regressions (#1132)
andygrove Dec 3, 2024
36a2307
docs: Add more technical detail and new diagram to Comet plugin overv…
andygrove Dec 3, 2024
2671e0c
Stop passing Java config map into native createPlan (#1101)
andygrove Dec 4, 2024
8d7bcb8
feat: Improve ScanExec native metrics (#1133)
andygrove Dec 6, 2024
587c29b
chore: Remove unused StringView struct (#1143)
andygrove Dec 6, 2024
b95dc1d
docs: Add some documentation explaining how shuffle works (#1148)
andygrove Dec 6, 2024
1c6c7a9
test: enable more Spark 4.0 tests (#1145)
kazuyukitanimura Dec 6, 2024
8d83cc1
chore: Refactor cast to use SparkCastOptions param (#1146)
andygrove Dec 6, 2024
21503ca
Enable more scenarios in CometExecBenchmark. (#1151)
mbutrovich Dec 7, 2024
73f1405
chore: Move more expressions from core crate to spark-expr crate (#1152)
andygrove Dec 9, 2024
5c45fdc
remove dead code (#1155)
andygrove Dec 10, 2024
2c1a6b9
fix: Spark 4.0-preview1 SPARK-47120 (#1156)
kazuyukitanimura Dec 11, 2024
49cf0d7
chore: Move string kernels and expressions to spark-expr crate (#1164)
andygrove Dec 12, 2024
7db9aa6
chore: Move remaining expressions to spark-expr crate + some minor re…
andygrove Dec 12, 2024
f1d0879
chore: Add ignored tests for reading complex types from Parquet (#1167)
andygrove Dec 12, 2024
b9ac78b
feat: Add Spark-compatible implementation of SchemaAdapterFactory (#1…
andygrove Dec 17, 2024
46a28db
fix: Document enabling comet explain plan usage in Spark (4.0) (#1176)
parthchandra Dec 17, 2024
655081b
test: enabling Spark tests with offHeap requirement (#1177)
kazuyukitanimura Dec 18, 2024
e297d23
feat: Improve shuffle metrics (second attempt) (#1175)
andygrove Dec 18, 2024
8f4a8a5
fix: stddev_pop should not directly return 0.0 when count is 1.0 (#1184)
viirya Dec 19, 2024
ea6d205
feat: Make native shuffle compression configurable and respect `spark…
andygrove Dec 20, 2024
053b7cc
minor: move shuffle classes from common to spark (#1193)
andygrove Dec 22, 2024
639fa2f
minor: refactor decodeBatches to make private in broadcast exchange (…
andygrove Dec 22, 2024
58dee73
minor: refactor prepare_output so that it does not require an Executi…
andygrove Dec 22, 2024
5432e03
fix: fix missing explanation for then branch in case when (#1200)
rluvaton Dec 27, 2024
103f82f
minor: remove unused source files (#1202)
andygrove Dec 28, 2024
5d2c909
chore: Upgrade to DataFusion 44.0.0-rc2 (#1154)
andygrove Dec 28, 2024
4f8ce75
feat: add support for array_contains expression (#1163)
dharanad Jan 2, 2025
9320aed
feat: Add a `spark.comet.exec.memoryPool` configuration for experimen…
Kontinuation Jan 3, 2025
2e0f00a
feat: Reenable tests for filtered SMJ anti join (#1211)
comphead Jan 3, 2025
4333dce
chore: Add safety check to CometBuffer (#1050)
viirya Jan 3, 2025
4b56c52
remove unreachable code (#1213)
andygrove Jan 4, 2025
5f1e998
test: Enable Comet by default except some tests in SparkSessionExten…
kazuyukitanimura Jan 4, 2025
e39ffa6
extract struct expressions to folders based on spark grouping (#1216)
rluvaton Jan 6, 2025
5c389d1
chore: extract static invoke expressions to folders based on spark gr…
rluvaton Jan 6, 2025
e72beb1
chore: Follow-on PR to fully enable onheap memory usage (#1210)
andygrove Jan 6, 2025
74a6a8d
feat: Move shuffle block decompression and decoding to native code an…
andygrove Jan 7, 2025
3f0d442
chore: extract agg_funcs expressions to folders based on spark groupi…
rluvaton Jan 7, 2025
4cf840f
extract datetime_funcs expressions to folders based on spark grouping…
rluvaton Jan 7, 2025
508db06
chore: use datafusion from crates.io (#1232)
rluvaton Jan 7, 2025
c19202c
chore: extract strings file to `strings_func` like in spark grouping …
rluvaton Jan 8, 2025
fbcf025
chore: extract predicate_functions expressions to folders based on sp…
rluvaton Jan 8, 2025
ca7b4a8
build(deps): bump protobuf version to 3.21.12 (#1234)
wForget Jan 8, 2025
c6acc9d
extract json_funcs expressions to folders based on spark grouping (#1…
rluvaton Jan 8, 2025
0a68f1c
test: Enable shuffle by default in Spark tests (#1240)
kazuyukitanimura Jan 9, 2025
e731b6e
chore: extract hash_funcs expressions to folders based on spark group…
rluvaton Jan 9, 2025
be48839
fix: Fall back to Spark for unsupported partition or sort expressions…
andygrove Jan 9, 2025
d15d051
perf: Improve query planning to more reliably fall back to columnar s…
andygrove Jan 9, 2025
d52038e
fix regression (#1259)
andygrove Jan 10, 2025
c25060e
feat: add support for array_remove expression (#1179)
jatin510 Jan 12, 2025
e8261fb
fix: Fall back to Spark for distinct aggregates (#1262)
andygrove Jan 13, 2025
d7a7812
feat: Implement custom RecordBatch serde for shuffle for improved per…
andygrove Jan 13, 2025
1eb932a
docs: Update TPC-H benchmark results (#1257)
andygrove Jan 13, 2025
9fe5420
fix: disable initCap by default (#1276)
kazuyukitanimura Jan 14, 2025
cbe50e1
chore: Add changelog for 0.5.0 (#1278)
andygrove Jan 14, 2025
08d892a
update TPC-DS results for 0.5.0 (#1277)
andygrove Jan 14, 2025
9c1f0ee
fix: cast timestamp to decimal is unsupported (#1281)
wForget Jan 14, 2025
d36e8d7
chore: Start 0.6.0 development (#1286)
andygrove Jan 14, 2025
3eced67
docs: Fix links and provide complete benchmarking scripts (#1284)
andygrove Jan 14, 2025
82022af
feat: Add HasRowIdMapping interface (#1288)
viirya Jan 15, 2025
d782d29
Merge branch 'main' into comet-parquet-exec-merge-20240121
parthchandra Jan 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

These merge commits were added into this branch cleanly.

There are no new changes to show.