Upstream rebase (#29)

* configure_me_codegen retroactively reserved on our `bind_host` parame… (apache#520)

* configure_me_codegen retroactively reserved on our `bind_host` parameter name

* Add label and pray

* Add more labels why not

* Prepare 0.10.0 Release (apache#522)

* bump version

* CHANGELOG

* Ballista gets a docker image!!! (apache#521)

* Ballista gets a docker image!!!

* Enable flight sql

* Allow executing startup script

* Allow executing executables

* Clippy

* Remove capture group (apache#527)

* fix python build in CI (apache#528)

* fix python build in CI

* save progress

* use same min rust version in all crates

* fix

* use image from pyo3

* use newer image from pyo3

* do not require protoc

* wheels now generated

* rat - exclude generated file

* Update docs for simplified instructions (apache#532)

* Update docs for simplified instructions

* Fix whoopsie

* Update docs/source/user-guide/flightsql.md

Co-authored-by: Andy Grove <andygrove73@gmail.com>

Co-authored-by: Andy Grove <andygrove73@gmail.com>

* remove --locked (apache#533)

* Bump actions/labeler from 4.0.2 to 4.1.0 (apache#525)

* Provide a memory StateBackendClient (apache#523)

* Rename StateBackend::Standalone to StateBackend:Sled

* Copy utility files from sled crate since they cannot be used directly

* Provide a memory StateBackendClient

* Fix dashmap deadlock issue

* Fix for the comments

Co-authored-by: yangzhong <yangzhong@ebay.com>

* only build docker images on rc tags (apache#535)

* docs: fix style in the Helm readme (apache#551)

* Fix Helm chart's image format (apache#550)

* Update datafusion requirement from 14.0.0 to 15.0.0 (apache#552)

* Update datafusion requirement from 14.0.0 to 15.0.0

* Fix UT

* Fix python

* Fix python

* Fix Python

Co-authored-by: yangzhong <yangzhong@ebay.com>

* Make it concurrently to launch tasks to executors (apache#557)

* Make it concurrently to launch tasks to executors

* Refine for comments

Co-authored-by: yangzhong <yangzhong@ebay.com>

* fix(ui): fix last seen (apache#562)

* Support Alibaba Cloud OSS with ObjectStore (apache#567)

* Fix cargo clippy (apache#571)

Co-authored-by: yangzhong <yangzhong@ebay.com>

* Super minor spelling error (apache#573)

* Update env_logger requirement from 0.9 to 0.10 (apache#539)

Updates the requirements on [env_logger](https://github.com/rust-cli/env_logger) to permit the latest version.
- [Release notes](https://github.com/rust-cli/env_logger/releases)
- [Changelog](https://github.com/rust-cli/env_logger/blob/main/CHANGELOG.md)
- [Commits](rust-cli/env_logger@v0.9.0...v0.10.0)

---
updated-dependencies:
- dependency-name: env_logger
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update graphviz-rust requirement from 0.4.0 to 0.5.0 (apache#574)

Updates the requirements on [graphviz-rust](https://github.com/besok/graphviz-rust) to permit the latest version.
- [Release notes](https://github.com/besok/graphviz-rust/releases)
- [Changelog](https://github.com/besok/graphviz-rust/blob/master/CHANGELOG.md)
- [Commits](https://github.com/besok/graphviz-rust/commits)

---
updated-dependencies:
- dependency-name: graphviz-rust
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* updated readme to contain correct versions of dependencies. (apache#580)

* Fix benchmark image link (apache#596)

* Add support for Azure (apache#599)

* Remove outdated script and use evergreen version of rust (apache#597)

* Remove outdated script and use evergreen version of rust

* Use debian protobuf

* Customize session builder

* Add setter for executor slots policy

* Construct Executor with functions

* Add queued and completed timestamps to successful job status

* Add public methods to SchedulerServer

* Public method for getting execution graph

* Public method for stage metrics

* Use node-level local limit (#20)

* Use node-level local limit

* serialize limit in shuffle writer

* Revert "Merge pull request #19 from coralogix/sc-5792"

This reverts commit 08140ef, reversing
changes made to a7f1384.

* add log

* make sure we don't forget limit for shuffle writer

* update accum correctly and try to break early

* Check local limit accumulator before polling for more data

* fix build

Co-authored-by: Martins Purins <martins.purins@coralogix.com>

* Add ClusterState trait

* Expose active job count

* Remove println

* Resubmit jobs when no resources available for scheduling

* Make parse_physical_expr public

* Reduce log spam

* Fix job submitted metric by ignoring resubmissions

* Record when job is queued in scheduler metrics (#28)

* Record when job is queueud in scheduler metrics

* add additional buckets for exec times

* fmt

* clippy

* tomlfmt

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Brent Gardner <brent.gardner@spaceandtime.io>
Co-authored-by: Andy Grove <andygrove73@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: yahoNanJing <90197956+yahoNanJing@users.noreply.github.com>
Co-authored-by: yangzhong <yangzhong@ebay.com>
Co-authored-by: Xin Hao <haoxinst@gmail.com>
Co-authored-by: Duyet Le <5009534+duyet@users.noreply.github.com>
Co-authored-by: r.4ntix <antix.blue@antix.blue>
Co-authored-by: Jeremy Dyer <jdye64@gmail.com>
Co-authored-by: Sai Krishna Reddy Lakkam <86965352+saikrishna1-bidgely@users.noreply.github.com>
Co-authored-by: Aidan Kovacic <95648995+aidankovacic-8451@users.noreply.github.com>
Co-authored-by: Dan Harris <dan@thinkharder.dev>
Co-authored-by: Dan Harris <1327726+thinkharderdev@users.noreply.github.com>
Co-authored-by: Martins Purins <martins.purins@coralogix.com>
Co-authored-by: Dan Harris <dan@coralogix.com>

ballista/core/Cargo.toml

@@ -55,9 +55,9 @@ datafusion-objectstore-hdfs = { version = "0.1.1", default-features = false, opt
 datafusion-proto = "18.0.0"
 futures = "0.3"
 hashbrown = "0.13"
+itertools = "0.10"
 
 lazy_static = "1.4.0"
-itertools = "0.10"
 libloading = "0.7.3"
 log = "0.4"
 lru = "0.8.1"

ballista/core/src/serde/physical_plan/from_proto.rs

@@ -411,6 +411,8 @@ impl TryInto<FileScanConfig> for &protobuf::FileScanExecConf {
             projection,
             limit: self.limit.as_ref().map(|sl| sl.limit as usize),
             table_partition_cols: vec![],
+            // TODO add ordering info to the ballista proto file
+            output_ordering: None,
         })
     }
 }

ballista/scheduler/src/cluster/storage/etcd.rs

@@ -34,7 +34,7 @@ use crate::state::backend::{
     Keyspace, Lock, Operation, StateBackendClient, Watch, WatchEvent,
 };
 
-/// A [`StateBackendClient`] implementation that uses etcd to save cluster configuration.
+/// A [`StateBackendClient`] implementation that uses etcd to save cluster state.
 #[derive(Clone)]
 pub struct EtcdClient {
     namespace: String,