Add ExecutionEngine abstraction #687
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
andygrove
requested review from
thinkharderdev,
tustvold,
avantgardnerio,
yahoNanJing and
Dandandan
andygrove
commented
Feb 26, 2023
Dandandan
approved these changes
Feb 27, 2023
| @@ -288,6 +305,22 @@ impl ShuffleWriterExec { | |||
| } | |||
| } | |||
|
|
|||
| #[async_trait] | |||
| impl ShuffleWriter for ShuffleWriterExec { | |||
There was a problem hiding this comment.
This seems inverted to me. Shouldn't ShuffleWriterExec have a Arc<dyn ShuffleWriter> instead of impl the trait?
I know this is a step toward the pluggable execution engine, but also thinking about how this fits with allowing for different shuffle methods (write to disk, write to object store, stream to network, etc.) and seems like we would need a second layer of indirection for that.
There was a problem hiding this comment.
That's a good point. Maybe want I want for now is just an ExecutionEngine trait with an execute method that returns the metadata with shuffle file locations. I will give this some more thought.
There was a problem hiding this comment.
@thinkharderdev I have renamed the trait and made some other naming changes. I also added the ExecutionEngine abstraction in this PR. PTAL when you have time.
|
@Dandandan I know you already approved this PR, but it has now changed significantly. Could you take another look? |
ch-sc
added a commit
to coralogix/arrow-ballista
that referenced
this pull request
Mar 31, 2023
* configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Prepare 0.10.0 Release (apache#522) * bump version * CHANGELOG * Ballista gets a docker image!!! (apache#521) * Ballista gets a docker image!!! * Enable flight sql * Allow executing startup script * Allow executing executables * Clippy * Remove capture group (apache#527) * fix python build in CI (apache#528) * fix python build in CI * save progress * use same min rust version in all crates * fix * use image from pyo3 * use newer image from pyo3 * do not require protoc * wheels now generated * rat - exclude generated file * Update docs for simplified instructions (apache#532) * Update docs for simplified instructions * Fix whoopsie * Update docs/source/user-guide/flightsql.md Co-authored-by: Andy Grove <andygrove73@gmail.com> Co-authored-by: Andy Grove <andygrove73@gmail.com> * remove --locked (apache#533) * Bump actions/labeler from 4.0.2 to 4.1.0 (apache#525) * Provide a memory StateBackendClient (apache#523) * Rename StateBackend::Standalone to StateBackend:Sled * Copy utility files from sled crate since they cannot be used directly * Provide a memory StateBackendClient * Fix dashmap deadlock issue * Fix for the comments Co-authored-by: yangzhong <yangzhong@ebay.com> * only build docker images on rc tags (apache#535) * docs: fix style in the Helm readme (apache#551) * Fix Helm chart's image format (apache#550) * Update datafusion requirement from 14.0.0 to 15.0.0 (apache#552) * Update datafusion requirement from 14.0.0 to 15.0.0 * Fix UT * Fix python * Fix python * Fix Python Co-authored-by: yangzhong <yangzhong@ebay.com> * Make it concurrently to launch tasks to executors (apache#557) * Make it concurrently to launch tasks to executors * Refine for comments Co-authored-by: yangzhong <yangzhong@ebay.com> * fix(ui): fix last seen (apache#562) * Support Alibaba Cloud OSS with ObjectStore (apache#567) * Fix cargo clippy (apache#571) Co-authored-by: yangzhong <yangzhong@ebay.com> * Super minor spelling error (apache#573) * Update env_logger requirement from 0.9 to 0.10 (apache#539) Updates the requirements on [env_logger](https://github.com/rust-cli/env_logger) to permit the latest version. - [Release notes](https://github.com/rust-cli/env_logger/releases) - [Changelog](https://github.com/rust-cli/env_logger/blob/main/CHANGELOG.md) - [Commits](rust-cli/env_logger@v0.9.0...v0.10.0) --- updated-dependencies: - dependency-name: env_logger dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update graphviz-rust requirement from 0.4.0 to 0.5.0 (apache#574) Updates the requirements on [graphviz-rust](https://github.com/besok/graphviz-rust) to permit the latest version. - [Release notes](https://github.com/besok/graphviz-rust/releases) - [Changelog](https://github.com/besok/graphviz-rust/blob/master/CHANGELOG.md) - [Commits](https://github.com/besok/graphviz-rust/commits) --- updated-dependencies: - dependency-name: graphviz-rust dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * updated readme to contain correct versions of dependencies. (apache#580) * Fix benchmark image link (apache#596) * Add support for Azure (apache#599) * Remove outdated script and use evergreen version of rust (apache#597) * Remove outdated script and use evergreen version of rust * Use debian protobuf * feat: update script such that ballista-cli image is built as well (apache#601) * Fix Cargo.toml format issue (apache#616) * Refactor executor main (apache#614) * Refactor executor main * copy all configs * toml fmt * Refactor scheduler main (apache#615) * refactor scheduler main * toml fmt * Python: add method to get explain output as a string (apache#593) * Update contributor guide (apache#617) * Cluster state refactor part 1 (apache#560) * Customize session builder * Add setter for executor slots policy * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <martins.purins@coralogix.com> * configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Add ClusterState trait * Refactor slightly for clarity * Revert "Use node-level local limit (#20)" This reverts commit ff96bcd. * Revert "Public method for stage metrics" This reverts commit a802315. * Revert "Public method for getting execution graph" This reverts commit 490bda5. * Revert "Add public methods to SchedulerServer" This reverts commit 5ad27c0. * Revert "Add queued and completed timestamps to successful job status" This reverts commit c615fce. * Revert "Construct Executor with functions" This reverts commit 24d4830. * Always forget the apache header Co-authored-by: Martins Purins <martins.purins@coralogix.com> Co-authored-by: Brent Gardner <brent.gardner@spaceandtime.io> * replace master with main (apache#621) * implement new release process (apache#623) * add docs on who can release (apache#632) * Upgrade to DataFusion 16 (again) (apache#636) * Update datafusion dependency to the latest version (apache#612) * Update datafusion dependency to the latest version * Fix python * Skip ut of test_window_lead due to apache/datafusion-python#135 * Fix clippy --------- Co-authored-by: yangzhong <yangzhong@ebay.com> * Upgrade to DataFusion 17 (apache#639) * Upgrade to DF 17 * Restore original error handling functionality * check in benchmark image (apache#647) * Remove `python` dir & python-related workflows (apache#654) * refactor: remove python dir & python-related workflows * remove brackets * Handle job resubmission (apache#586) * Handle job resubmission * Make resubmission configurable and add test * Fix debug log * Add executor self-registration mechanism in the heartbeat service (apache#649) Co-authored-by: yangzhong <yangzhong@ebay.com> * Cluster state refactor Part 2 (apache#658) * Customize session builder * Add setter for executor slots policy * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <martins.purins@coralogix.com> * configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Add ClusterState trait * Refactor slightly for clarity * Revert "Use node-level local limit (#20)" This reverts commit ff96bcd. * Revert "Public method for stage metrics" This reverts commit a802315. * Revert "Public method for getting execution graph" This reverts commit 490bda5. * Revert "Add public methods to SchedulerServer" This reverts commit 5ad27c0. * Revert "Add queued and completed timestamps to successful job status" This reverts commit c615fce. * Revert "Construct Executor with functions" This reverts commit 24d4830. * Always forget the apache header * WIP * Implement JobState * Tests and fixes * do not hold ref across await point * Fix clippy warnings * Fix tomlfmt github action * uncomment test --------- Co-authored-by: Martins Purins <martins.purins@coralogix.com> Co-authored-by: Brent Gardner <brent.gardner@spaceandtime.io> * Upgrade to DataFusion 18.0.0-rc1 (apache#664) * Minor refactor to reduce duplicate code (apache#659) * move test_util to ballista-examples package (apache#661) * Upgrade to DataFusion 18 (apache#668) * Enable physical plan round-trip tests (apache#666) * Prep 0.11 (apache#682) * Change version to 0.11.0 * changelog * update react-timeago version * yarn upgrade * fix * fix * revert yarn change * Print versions * Print locations * Avoid github shenanigans * Try to get runners running * Try to get runners running * already root --------- Co-authored-by: Andy Grove <andygrove73@gmail.com> * [minor] remove todo (apache#683) * Add executor terminating status for graceful shutdown (apache#667) * Add executor terminating status for graceful shutdown * Remove empty file * Update ballista/executor/src/executor_process.rs Co-authored-by: Brent Gardner <brent.gardner@spaceandtime.io> --------- Co-authored-by: Brent Gardner <brent.gardner@spaceandtime.io> * Allow `BallistaContext::read_*` methods to read multiple paths. (apache#679) * updated dependency in cargo, added read_json method, modified read_* methods to read multiple paths. * ran cargo fmt * Added revision for proper builds. * Update scheduler.md (apache#657) * Mark `SchedulerState` as pub (apache#688) * Mark as pub * Fmt --------- Co-authored-by: Daniël Heres <daniel.heres@coralogix.com> * Update graphviz-rust requirement from 0.5.0 to 0.6.1 (apache#651) Updates the requirements on [graphviz-rust](https://github.com/besok/graphviz-rust) to permit the latest version. - [Release notes](https://github.com/besok/graphviz-rust/releases) - [Changelog](https://github.com/besok/graphviz-rust/blob/master/CHANGELOG.md) - [Commits](https://github.com/besok/graphviz-rust/commits) --- updated-dependencies: - dependency-name: graphviz-rust dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Upgrade DataFusion to 19.0.0 (apache#691) * update release notes (apache#692) * Make task launcher pub (apache#695) Co-authored-by: Daniël Heres <daniel.heres@coralogix.com> * Make task_manager pub (apache#698) Co-authored-by: Daniël Heres <daniel.heres@coralogix.com> * Add ExecutionEngine abstraction (apache#687) * Allow accessing s3 locations in client mode (apache#700) * Allow accessing s3 locations in client mode * Removed s3 feature from test dependencies. * fixed cargo-tomlfmt issues * deployment/docker-compose.md incorrect remote ref (apache#699) * Fix for error message during testing (apache#707) * Fix cargo clippy * Fix for error message during testing * Remove unwrap for dealing with JobQueued event * log task ids when launch tasks --------- Co-authored-by: yangzhong <yangzhong@ebay.com> * Upgrade datafusion to 20.0.0 & sqlparser to to 0.32.0 (apache#711) * Upgrade datafusion & sqlparser * Move ballista_round_trip tests of benchmark into a separate feature to avoid stack overflow * Fix failed tests of scheduler * Update README.md (apache#729) * Update link to proto file in dev docs (apache#713) * Fix `show tables` fails (apache#715) * Remove cancelled jobs from active cache (#36) * Downgrade expected error to warning (#37) * Downgrade expected error to warning * add context * Serialize configoptions and pass them to executor (#34) * serialize configoptions and pass them to executor and allow extensions for TaskContext * use ConfigOptions::with_extensions * fix usage of ConfigOptions * clippy * Add wait_drained to SchedulerServer and Executor (#41) * Add missing code from previous commits * Fixes after merging from master * Reintroduce Executor::with_functions * Adapt prometheus histogram buckets * cargo tomlfmt * cargo fmt --all * Allow too_many_arguments lint * Cargo tomlfmt --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Brent Gardner <brent.gardner@spaceandtime.io> Co-authored-by: Andy Grove <andygrove73@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: yahoNanJing <90197956+yahoNanJing@users.noreply.github.com> Co-authored-by: yangzhong <yangzhong@ebay.com> Co-authored-by: Xin Hao <haoxinst@gmail.com> Co-authored-by: Duyet Le <5009534+duyet@users.noreply.github.com> Co-authored-by: r.4ntix <antix.blue@antix.blue> Co-authored-by: Jeremy Dyer <jdye64@gmail.com> Co-authored-by: Sai Krishna Reddy Lakkam <86965352+saikrishna1-bidgely@users.noreply.github.com> Co-authored-by: Aidan Kovacic <95648995+aidankovacic-8451@users.noreply.github.com> Co-authored-by: Tim Van Wassenhove <tim@timvw.be> Co-authored-by: Dan Harris <1327726+thinkharderdev@users.noreply.github.com> Co-authored-by: Martins Purins <martins.purins@coralogix.com> Co-authored-by: Brent Gardner <bgardner@squarelabs.net> Co-authored-by: Ian Alexander Joiner <14581281+iajoiner@users.noreply.github.com> Co-authored-by: jiangzhx <jiangzhx@gmail.com> Co-authored-by: Yang Jiang <yangjiang@ebay.com> Co-authored-by: Lakkam Sai Krishna Reddy <lakkam.saikrishnareddy@gmail.com> Co-authored-by: Vrishabh <psvrishabh@gmail.com> Co-authored-by: Daniël Heres <danielheres@gmail.com> Co-authored-by: Daniël Heres <daniel.heres@coralogix.com> Co-authored-by: Joe Williams <7463219+BubbaJoe@users.noreply.github.com> Co-authored-by: Jaap Aarts <JAicewizard@users.noreply.github.com> Co-authored-by: mpurins-coralogix <101556208+mpurins-coralogix@users.noreply.github.com>
fsdvh
added a commit
to coralogix/arrow-ballista
that referenced
this pull request
Apr 26, 2023
* configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Prepare 0.10.0 Release (apache#522) * bump version * CHANGELOG * Ballista gets a docker image!!! (apache#521) * Ballista gets a docker image!!! * Enable flight sql * Allow executing startup script * Allow executing executables * Clippy * Remove capture group (apache#527) * fix python build in CI (apache#528) * fix python build in CI * save progress * use same min rust version in all crates * fix * use image from pyo3 * use newer image from pyo3 * do not require protoc * wheels now generated * rat - exclude generated file * Update docs for simplified instructions (apache#532) * Update docs for simplified instructions * Fix whoopsie * Update docs/source/user-guide/flightsql.md Co-authored-by: Andy Grove <andygrove73@gmail.com> Co-authored-by: Andy Grove <andygrove73@gmail.com> * remove --locked (apache#533) * Bump actions/labeler from 4.0.2 to 4.1.0 (apache#525) * Provide a memory StateBackendClient (apache#523) * Rename StateBackend::Standalone to StateBackend:Sled * Copy utility files from sled crate since they cannot be used directly * Provide a memory StateBackendClient * Fix dashmap deadlock issue * Fix for the comments Co-authored-by: yangzhong <yangzhong@ebay.com> * only build docker images on rc tags (apache#535) * docs: fix style in the Helm readme (apache#551) * Fix Helm chart's image format (apache#550) * Update datafusion requirement from 14.0.0 to 15.0.0 (apache#552) * Update datafusion requirement from 14.0.0 to 15.0.0 * Fix UT * Fix python * Fix python * Fix Python Co-authored-by: yangzhong <yangzhong@ebay.com> * Make it concurrently to launch tasks to executors (apache#557) * Make it concurrently to launch tasks to executors * Refine for comments Co-authored-by: yangzhong <yangzhong@ebay.com> * fix(ui): fix last seen (apache#562) * Support Alibaba Cloud OSS with ObjectStore (apache#567) * Fix cargo clippy (apache#571) Co-authored-by: yangzhong <yangzhong@ebay.com> * Super minor spelling error (apache#573) * Update env_logger requirement from 0.9 to 0.10 (apache#539) Updates the requirements on [env_logger](https://github.com/rust-cli/env_logger) to permit the latest version. - [Release notes](https://github.com/rust-cli/env_logger/releases) - [Changelog](https://github.com/rust-cli/env_logger/blob/main/CHANGELOG.md) - [Commits](rust-cli/env_logger@v0.9.0...v0.10.0) --- updated-dependencies: - dependency-name: env_logger dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update graphviz-rust requirement from 0.4.0 to 0.5.0 (apache#574) Updates the requirements on [graphviz-rust](https://github.com/besok/graphviz-rust) to permit the latest version. - [Release notes](https://github.com/besok/graphviz-rust/releases) - [Changelog](https://github.com/besok/graphviz-rust/blob/master/CHANGELOG.md) - [Commits](https://github.com/besok/graphviz-rust/commits) --- updated-dependencies: - dependency-name: graphviz-rust dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * updated readme to contain correct versions of dependencies. (apache#580) * Fix benchmark image link (apache#596) * Add support for Azure (apache#599) * Remove outdated script and use evergreen version of rust (apache#597) * Remove outdated script and use evergreen version of rust * Use debian protobuf * feat: update script such that ballista-cli image is built as well (apache#601) * Fix Cargo.toml format issue (apache#616) * Refactor executor main (apache#614) * Refactor executor main * copy all configs * toml fmt * Refactor scheduler main (apache#615) * refactor scheduler main * toml fmt * Python: add method to get explain output as a string (apache#593) * Update contributor guide (apache#617) * Cluster state refactor part 1 (apache#560) * Customize session builder * Add setter for executor slots policy * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <martins.purins@coralogix.com> * configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Add ClusterState trait * Refactor slightly for clarity * Revert "Use node-level local limit (#20)" This reverts commit ff96bcd. * Revert "Public method for stage metrics" This reverts commit a802315. * Revert "Public method for getting execution graph" This reverts commit 490bda5. * Revert "Add public methods to SchedulerServer" This reverts commit 5ad27c0. * Revert "Add queued and completed timestamps to successful job status" This reverts commit c615fce. * Revert "Construct Executor with functions" This reverts commit 24d4830. * Always forget the apache header Co-authored-by: Martins Purins <martins.purins@coralogix.com> Co-authored-by: Brent Gardner <brent.gardner@spaceandtime.io> * replace master with main (apache#621) * implement new release process (apache#623) * add docs on who can release (apache#632) * Upgrade to DataFusion 16 (again) (apache#636) * Update datafusion dependency to the latest version (apache#612) * Update datafusion dependency to the latest version * Fix python * Skip ut of test_window_lead due to apache/datafusion-python#135 * Fix clippy --------- Co-authored-by: yangzhong <yangzhong@ebay.com> * Upgrade to DataFusion 17 (apache#639) * Upgrade to DF 17 * Restore original error handling functionality * check in benchmark image (apache#647) * Remove `python` dir & python-related workflows (apache#654) * refactor: remove python dir & python-related workflows * remove brackets * Handle job resubmission (apache#586) * Handle job resubmission * Make resubmission configurable and add test * Fix debug log * Add executor self-registration mechanism in the heartbeat service (apache#649) Co-authored-by: yangzhong <yangzhong@ebay.com> * Cluster state refactor Part 2 (apache#658) * Customize session builder * Add setter for executor slots policy * Construct Executor with functions * Add queued and completed timestamps to successful job status * Add public methods to SchedulerServer * Public method for getting execution graph * Public method for stage metrics * Use node-level local limit (#20) * Use node-level local limit * serialize limit in shuffle writer * Revert "Merge pull request #19 from coralogix/sc-5792" This reverts commit 08140ef, reversing changes made to a7f1384. * add log * make sure we don't forget limit for shuffle writer * update accum correctly and try to break early * Check local limit accumulator before polling for more data * fix build Co-authored-by: Martins Purins <martins.purins@coralogix.com> * configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520) * configure_me_codegen retroactively reserved on our `bind_host` parameter name * Add label and pray * Add more labels why not * Add ClusterState trait * Refactor slightly for clarity * Revert "Use node-level local limit (#20)" This reverts commit ff96bcd. * Revert "Public method for stage metrics" This reverts commit a802315. * Revert "Public method for getting execution graph" This reverts commit 490bda5. * Revert "Add public methods to SchedulerServer" This reverts commit 5ad27c0. * Revert "Add queued and completed timestamps to successful job status" This reverts commit c615fce. * Revert "Construct Executor with functions" This reverts commit 24d4830. * Always forget the apache header * WIP * Implement JobState * Tests and fixes * do not hold ref across await point * Fix clippy warnings * Fix tomlfmt github action * uncomment test --------- Co-authored-by: Martins Purins <martins.purins@coralogix.com> Co-authored-by: Brent Gardner <brent.gardner@spaceandtime.io> * Upgrade to DataFusion 18.0.0-rc1 (apache#664) * Minor refactor to reduce duplicate code (apache#659) * move test_util to ballista-examples package (apache#661) * Upgrade to DataFusion 18 (apache#668) * Enable physical plan round-trip tests (apache#666) * Prep 0.11 (apache#682) * Change version to 0.11.0 * changelog * update react-timeago version * yarn upgrade * fix * fix * revert yarn change * Print versions * Print locations * Avoid github shenanigans * Try to get runners running * Try to get runners running * already root --------- Co-authored-by: Andy Grove <andygrove73@gmail.com> * [minor] remove todo (apache#683) * Add executor terminating status for graceful shutdown (apache#667) * Add executor terminating status for graceful shutdown * Remove empty file * Update ballista/executor/src/executor_process.rs Co-authored-by: Brent Gardner <brent.gardner@spaceandtime.io> --------- Co-authored-by: Brent Gardner <brent.gardner@spaceandtime.io> * Allow `BallistaContext::read_*` methods to read multiple paths. (apache#679) * updated dependency in cargo, added read_json method, modified read_* methods to read multiple paths. * ran cargo fmt * Added revision for proper builds. * Update scheduler.md (apache#657) * Mark `SchedulerState` as pub (apache#688) * Mark as pub * Fmt --------- Co-authored-by: Daniël Heres <daniel.heres@coralogix.com> * Update graphviz-rust requirement from 0.5.0 to 0.6.1 (apache#651) Updates the requirements on [graphviz-rust](https://github.com/besok/graphviz-rust) to permit the latest version. - [Release notes](https://github.com/besok/graphviz-rust/releases) - [Changelog](https://github.com/besok/graphviz-rust/blob/master/CHANGELOG.md) - [Commits](https://github.com/besok/graphviz-rust/commits) --- updated-dependencies: - dependency-name: graphviz-rust dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Upgrade DataFusion to 19.0.0 (apache#691) * update release notes (apache#692) * Make task launcher pub (apache#695) Co-authored-by: Daniël Heres <daniel.heres@coralogix.com> * Make task_manager pub (apache#698) Co-authored-by: Daniël Heres <daniel.heres@coralogix.com> * Add ExecutionEngine abstraction (apache#687) * Allow accessing s3 locations in client mode (apache#700) * Allow accessing s3 locations in client mode * Removed s3 feature from test dependencies. * fixed cargo-tomlfmt issues * deployment/docker-compose.md incorrect remote ref (apache#699) * Fix for error message during testing (apache#707) * Fix cargo clippy * Fix for error message during testing * Remove unwrap for dealing with JobQueued event * log task ids when launch tasks --------- Co-authored-by: yangzhong <yangzhong@ebay.com> * Upgrade datafusion to 20.0.0 & sqlparser to to 0.32.0 (apache#711) * Upgrade datafusion & sqlparser * Move ballista_round_trip tests of benchmark into a separate feature to avoid stack overflow * Fix failed tests of scheduler * Update README.md (apache#729) * Update link to proto file in dev docs (apache#713) * Fix `show tables` fails (apache#715) * Remove cancelled jobs from active cache (#36) * Downgrade expected error to warning (#37) * Downgrade expected error to warning * add context * Serialize configoptions and pass them to executor (#34) * serialize configoptions and pass them to executor and allow extensions for TaskContext * use ConfigOptions::with_extensions * fix usage of ConfigOptions * clippy * Add wait_drained to SchedulerServer and Executor (#41) * Add missing code from previous commits * Fixes after merging from master * Reintroduce Executor::with_functions * Adapt prometheus histogram buckets * cargo tomlfmt * cargo fmt --all * Allow too_many_arguments lint * sc-16350: introducing notion of external and internal error in the failed job status * Cargo tomlfmt * sc-16350: small test fix * sc-16350: partially implemented ballista error serialization * sc-16350: update failed job proto definition * sc-16350: cleanup * sc-16350: update protoc * sc-16350: update action * sc-16350: more action update * sc-16350: update test * sc-16350: allow optional fields * VTX-522: fix models * VTX-522: cleanup * Change error to warn * VTX-522: fixes --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Brent Gardner <brent.gardner@spaceandtime.io> Co-authored-by: Andy Grove <andygrove73@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: yahoNanJing <90197956+yahoNanJing@users.noreply.github.com> Co-authored-by: yangzhong <yangzhong@ebay.com> Co-authored-by: Xin Hao <haoxinst@gmail.com> Co-authored-by: Duyet Le <5009534+duyet@users.noreply.github.com> Co-authored-by: r.4ntix <antix.blue@antix.blue> Co-authored-by: Jeremy Dyer <jdye64@gmail.com> Co-authored-by: Sai Krishna Reddy Lakkam <86965352+saikrishna1-bidgely@users.noreply.github.com> Co-authored-by: Aidan Kovacic <95648995+aidankovacic-8451@users.noreply.github.com> Co-authored-by: Tim Van Wassenhove <tim@timvw.be> Co-authored-by: Dan Harris <1327726+thinkharderdev@users.noreply.github.com> Co-authored-by: Martins Purins <martins.purins@coralogix.com> Co-authored-by: Brent Gardner <bgardner@squarelabs.net> Co-authored-by: Ian Alexander Joiner <14581281+iajoiner@users.noreply.github.com> Co-authored-by: jiangzhx <jiangzhx@gmail.com> Co-authored-by: Yang Jiang <yangjiang@ebay.com> Co-authored-by: Lakkam Sai Krishna Reddy <lakkam.saikrishnareddy@gmail.com> Co-authored-by: Vrishabh <psvrishabh@gmail.com> Co-authored-by: Daniël Heres <danielheres@gmail.com> Co-authored-by: Daniël Heres <daniel.heres@coralogix.com> Co-authored-by: Joe Williams <7463219+BubbaJoe@users.noreply.github.com> Co-authored-by: Jaap Aarts <JAicewizard@users.noreply.github.com> Co-authored-by: mpurins-coralogix <101556208+mpurins-coralogix@users.noreply.github.com> Co-authored-by: Christoph Schulze <christoph.schulze@coralogix.com> Co-authored-by: Dan Harris <dan@coralogix.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
N/A
Rationale for this change
Add an abstraction for shuffle writer as a step towards supporting pluggable execution engines. See apache/datafusion-ballista-python#19 for more context on how I intend to use this.
What changes are included in this PR?
ExecutionEngineandQueryStageExecutortraitsAre there any user-facing changes?
No