Upstream rebase (#29)

* configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520)

* configure_me_codegen retroactively reserved on our `bind_host` parameter name

* Add label and pray

* Add more labels why not

* Prepare 0.10.0 Release (apache#522)

* bump version

* CHANGELOG

* Ballista gets a docker image!!! (apache#521)

* Ballista gets a docker image!!!

* Enable flight sql

* Allow executing startup script

* Allow executing executables

* Clippy

* Remove capture group (apache#527)

* fix python build in CI (apache#528)

* fix python build in CI

* save progress

* use same min rust version in all crates

* fix

* use image from pyo3

* use newer image from pyo3

* do not require protoc

* wheels now generated

* rat - exclude generated file

* Update docs for simplified instructions (apache#532)

* Update docs for simplified instructions

* Fix whoopsie

* Update docs/source/user-guide/flightsql.md

Co-authored-by: Andy Grove <andygrove73@gmail.com>

Co-authored-by: Andy Grove <andygrove73@gmail.com>

* remove --locked (apache#533)

* Bump actions/labeler from 4.0.2 to 4.1.0 (apache#525)

* Provide a memory StateBackendClient (apache#523)

* Rename StateBackend::Standalone to StateBackend:Sled

* Copy utility files from sled crate since they cannot be used directly

* Provide a memory StateBackendClient

* Fix dashmap deadlock issue

* Fix for the comments

Co-authored-by: yangzhong <yangzhong@ebay.com>

* only build docker images on rc tags (apache#535)

* docs: fix style in the Helm readme (apache#551)

* Fix Helm chart's image format (apache#550)

* Update datafusion requirement from 14.0.0 to 15.0.0 (apache#552)

* Update datafusion requirement from 14.0.0 to 15.0.0

* Fix UT

* Fix python

* Fix python

* Fix Python

Co-authored-by: yangzhong <yangzhong@ebay.com>

* Make it concurrently to launch tasks to executors (apache#557)

* Make it concurrently to launch tasks to executors

* Refine for comments

Co-authored-by: yangzhong <yangzhong@ebay.com>

* fix(ui): fix last seen (apache#562)

* Support Alibaba Cloud OSS with ObjectStore (apache#567)

* Fix cargo clippy (apache#571)

Co-authored-by: yangzhong <yangzhong@ebay.com>

* Super minor spelling error (apache#573)

* Update env_logger requirement from 0.9 to 0.10 (apache#539)

Updates the requirements on [env_logger](https://github.com/rust-cli/env_logger) to permit the latest version.
- [Release notes](https://github.com/rust-cli/env_logger/releases)
- [Changelog](https://github.com/rust-cli/env_logger/blob/main/CHANGELOG.md)
- [Commits](rust-cli/env_logger@v0.9.0...v0.10.0)

---
updated-dependencies:
- dependency-name: env_logger
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update graphviz-rust requirement from 0.4.0 to 0.5.0 (apache#574)

Updates the requirements on [graphviz-rust](https://github.com/besok/graphviz-rust) to permit the latest version.
- [Release notes](https://github.com/besok/graphviz-rust/releases)
- [Changelog](https://github.com/besok/graphviz-rust/blob/master/CHANGELOG.md)
- [Commits](https://github.com/besok/graphviz-rust/commits)

---
updated-dependencies:
- dependency-name: graphviz-rust
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* updated readme to contain correct versions of dependencies. (apache#580)

* Fix benchmark image link (apache#596)

* Add support for Azure (apache#599)

* Remove outdated script and use evergreen version of rust (apache#597)

* Remove outdated script and use evergreen version of rust

* Use debian protobuf

* Customize session builder

* Add setter for executor slots policy

* Construct Executor with functions

* Add queued and completed timestamps to successful job status

* Add public methods to SchedulerServer

* Public method for getting execution graph

* Public method for stage metrics

* Use node-level local limit (#20)

* Use node-level local limit

* serialize limit in shuffle writer

* Revert "Merge pull request #19 from coralogix/sc-5792"

This reverts commit 08140ef, reversing
changes made to a7f1384.

* add log

* make sure we don't forget limit for shuffle writer

* update accum correctly and try to break early

* Check local limit accumulator before polling for more data

* fix build

Co-authored-by: Martins Purins <martins.purins@coralogix.com>

* Add ClusterState trait

* Expose active job count

* Remove println

* Resubmit jobs when no resources available for scheduling

* Make parse_physical_expr public

* Reduce log spam

* Fix job submitted metric by ignoring resubmissions

* Record when job is queued in scheduler metrics (#28)

* Record when job is queueud in scheduler metrics

* add additional buckets for exec times

* fmt

* clippy

* tomlfmt

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Brent Gardner <brent.gardner@spaceandtime.io>
Co-authored-by: Andy Grove <andygrove73@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: yahoNanJing <90197956+yahoNanJing@users.noreply.github.com>
Co-authored-by: yangzhong <yangzhong@ebay.com>
Co-authored-by: Xin Hao <haoxinst@gmail.com>
Co-authored-by: Duyet Le <5009534+duyet@users.noreply.github.com>
Co-authored-by: r.4ntix <antix.blue@antix.blue>
Co-authored-by: Jeremy Dyer <jdye64@gmail.com>
Co-authored-by: Sai Krishna Reddy Lakkam <86965352+saikrishna1-bidgely@users.noreply.github.com>
Co-authored-by: Aidan Kovacic <95648995+aidankovacic-8451@users.noreply.github.com>
Co-authored-by: Dan Harris <dan@thinkharder.dev>
Co-authored-by: Dan Harris <1327726+thinkharderdev@users.noreply.github.com>
Co-authored-by: Martins Purins <martins.purins@coralogix.com>
Co-authored-by: Dan Harris <dan@coralogix.com>

ballista/core/Cargo.toml

@@ -55,9 +55,9 @@ datafusion-objectstore-hdfs = { version = "0.1.1", default-features = false, opt
 datafusion-proto = "17.0.0"
 futures = "0.3"
 hashbrown = "0.13"
+itertools = "0.10"
 
 lazy_static = "1.4.0"
-itertools = "0.10"
 libloading = "0.7.3"
 log = "0.4"
 lru = "0.8.1"

ballista/core/build.rs

@@ -26,7 +26,7 @@ fn main() -> Result<(), String> {
     println!("cargo:rerun-if-env-changed=FORCE_REBUILD");
 
     let version = rustc_version::version().unwrap();
-    println!("cargo:rustc-env=RUSTC_VERSION={version}");
+    println!("cargo:rustc-env=RUSTC_VERSION={}", version);
 
     // TODO: undo when resolved: https://github.com/intellij-rust/intellij-rust/issues/9402
     #[cfg(feature = "docsrs")]
@@ -42,7 +42,7 @@ fn main() -> Result<(), String> {
         tonic_build::configure()
             .extern_path(".datafusion", "::datafusion_proto::protobuf")
             .compile(&["proto/ballista.proto"], &["proto"])
-            .map_err(|e| format!("protobuf compilation failed: {e}"))?;
+            .map_err(|e| format!("protobuf compilation failed: {}", e))?;
         let generated_source_path = out.join("ballista.protobuf.rs");
         let code = std::fs::read_to_string(generated_source_path).unwrap();
         let mut file = std::fs::OpenOptions::new()

ballista/core/proto/ballista.proto

@@ -95,6 +95,7 @@ message AvroScanExecNode {
 enum PartitionMode {
   COLLECT_LEFT = 0;
   PARTITIONED = 1;
+  AUTO = 2;
 }
 
 message HashJoinExecNode {

ballista/core/src/serde/physical_plan/from_proto.rs

@@ -411,6 +411,8 @@ impl TryInto<FileScanConfig> for &protobuf::FileScanExecConf {
             projection,
             limit: self.limit.as_ref().map(|sl| sl.limit as usize),
             table_partition_cols: vec![],
+            // TODO add ordering info to the ballista proto file
+            output_ordering: None,
         })
     }
 }

ballista/core/src/serde/physical_plan/mod.rs

@@ -331,6 +331,8 @@ impl AsExecutionPlan for PhysicalPlanNode {
                     physical_window_expr,
                     input,
                     Arc::new((&input_schema).try_into()?),
+                    vec![],
+                    None,
                 )?))
             }
             PhysicalPlanType::Aggregate(hash_agg) => {
@@ -528,6 +530,7 @@ impl AsExecutionPlan for PhysicalPlanNode {
                 let partition_mode = match partition_mode {
                     protobuf::PartitionMode::CollectLeft => PartitionMode::CollectLeft,
                     protobuf::PartitionMode::Partitioned => PartitionMode::Partitioned,
+                    protobuf::PartitionMode::Auto => PartitionMode::Auto,
                 };
                 Ok(Arc::new(HashJoinExec::try_new(
                     left,
@@ -800,7 +803,7 @@ impl AsExecutionPlan for PhysicalPlanNode {
                         input: Some(Box::new(input)),
                         skip: limit.skip() as u32,
                         fetch: match limit.fetch() {
-                            Some(n) => *n as i64,
+                            Some(n) => n as i64,
                             _ => -1, // no limit
                         },
                     },
@@ -875,6 +878,7 @@ impl AsExecutionPlan for PhysicalPlanNode {
             let partition_mode = match exec.partition_mode() {
                 PartitionMode::CollectLeft => protobuf::PartitionMode::CollectLeft,
                 PartitionMode::Partitioned => protobuf::PartitionMode::Partitioned,
+                PartitionMode::Auto => protobuf::PartitionMode::Auto,
             };
 
             Ok(protobuf::PhysicalPlanNode {
@@ -1285,6 +1289,7 @@ fn decode_scan_config(
         projection,
         limit: proto.limit.as_ref().map(|sl| sl.limit as usize),
         table_partition_cols: vec![],
+        output_ordering: None,
     })
 }
 
@@ -1612,6 +1617,7 @@ mod roundtrip_tests {
             projection: None,
             limit: None,
             table_partition_cols: vec![],
+            output_ordering: None,
         };
 
         let predicate = datafusion::prelude::col("col").eq(datafusion::prelude::lit("1"));