Skip to content

Latest commit

 

History

History
611 lines (412 loc) · 57.9 KB

CHANGELOG.md

File metadata and controls

611 lines (412 loc) · 57.9 KB

Changelog

v2.1.0 (2024-12-06)

New Features

  • Upgrade to Spark 3.5.3 (#2202 by @jacobsalway)
  • feat: support archives param for spark-submit (#2256 by @kaka-zb)
  • Allow --ingress-class-name to be specified in chart (#2278 by @jacobsalway)
  • Update default container security context (#2265 by @ChenYi015)
  • Support pod template for Spark 3.x applications (#2141 by @ChenYi015)
  • Allow setting automountServiceAccountToken (#2298 by @Aranch)
  • Allow the Controller and Webhook Containers to run with the securityContext: readOnlyRootfilesystem: true (#2282 by @npgretz)
  • Use NSS_WRAPPER_PASSWD instead of /etc/passwd as in spark-operator image entrypoint.sh (#2312 by @Aakcht)

Bug Fixes

Misc

Full Changelog

v2.0.2 (2024-10-10)

Bug Fixes

Misc

Full Changelog

v2.0.1 (2024-09-26)

New Features

  • FEATURE: build operator image as non-root (#2171 by @ImpSy)

Bug Fixes

  • Update controller RBAC for ConfigMap and PersistentVolumeClaim (#2187 by @ChenYi015)

Misc

Full Changelog

v2.0.0 (2024-09-23)

Breaking Changes

  • Use controller-runtime to reconsturct spark operator (#2072 by @ChenYi015)
  • feat: support driver and executor pod use different priority (#2146 by @Kevinz857)

New Features

Bug Fixes

  • fix: Add default values for namespaces to match usage descriptions (#2128 by @snappyyouth)
  • Fix: Spark role binding did not render properly when setting spark service account name (#2135 by @ChenYi015)
  • fix: unable to set controller/webhook replicas to zero (#2147 by @ChenYi015)
  • Adding support for setting spark job namespaces to all namespaces (#2123 by @ChenYi015)
  • Fix: e2e test failes due to webhook not ready (#2149 by @ChenYi015)
  • fix: webhook not working when settings spark job namespaces to empty (#2163 by @ChenYi015)
  • fix: The logger had an odd number of arguments, making it panic (#2166 by @tcassaert)
  • fix the make kind-delete-custer to avoid accidental kubeconfig deletion (#2172 by @ImpSy)
  • Add specific error in log line when failed to create web UI service (#2170 by @tcassaert)
  • Account for spark.executor.pyspark.memory in Yunikorn gang scheduling (#2178 by @jacobsalway)
  • Fix: spark application does not respect time to live seconds (#2165 by @ChenYi015)

Misc

What's Changed

  • Cherry pick #2081 #2046 #2091 #2072 by @ChenYi015 in #2108
  • Cherry pick #2089 #2109 #2111 by @ChenYi015 in #2110
  • Release v2.0.0-rc.0 by @ChenYi015 in #2115
  • Cherry pick commits for releasing v2.0.0 by @ChenYi015 in #2156
  • Release v2.0.0 by @ChenYi015 in #2182

Full Changelog

v2.0.0-rc.0 (2024-08-09)

Breaking Changes

  • Use controller-runtime to reconsturct spark operator (#2072 by @ChenYi015)

Misc

What's Changed

Full Changelog

Full Changelog

  • Update the process to build api-docs, generate CRD manifests and code (#2046 by @ChenYi015)
  • Add workflow for closing stale issues and PRs (#2073 by @ChenYi015)

Full Changelog

Full Changelog

Full Changelog

Full Changelog

Full Changelog

Full Changelog

  • Bump appVersion to v1beta2-1.5.0-3.5.0 (#2044 by @ChenYi015)
  • Add restartPolicy field to SparkApplication Driver/Executor initContainers CRDs (#2022 by @mschroering)
  • 📝 Add Inter&Co to who-is-using.md (#2040 by @ignitz)

Full Changelog

  • Chart: add POD_NAME env for leader election (#2039 by @Aakcht)

Full Changelog

  • Support exposing extra TCP ports in Spark Driver via K8s Ingress (#1998 by @hiboyang)
  • Fixes a bug with dynamic allocation forcing the executor count to be 1 even when minExecutors is set to 0 (#1979 by @peter-mcclonski)
  • Remove outdated PySpark experimental warning in example (#2014 by @andrejpk)
  • Update Spark Job Namespace docs (#2000 by @matthewrossi)

Full Changelog

Full Changelog

  • feat: add support for service labels on driver-svc (#1985 by @Cian911)

Full Changelog

  • fix(chart): remove operator namespace default for job namespaces value (#1989 by @t3mi)
  • Fix Docker Hub Credentials in CI (#2003 by @andreyvelich)

Full Changelog

Full Changelog

Full Changelog

  • fix: upgraded k8s deps (#1983 by @AndrewChubatiuk)
  • chore: remove k8s.io/kubernetes replaces and adapt to v1.29.3 apis (#1968 by @ajayk)
  • Add some helm chart unit tests and fix spark service account render failure when extra annotations are specified (#1967 by @ChenYi015)
  • feat: Doc updates, Issue and PR templates are added (#1970 by @vara-bonthu)

Full Changelog

Full Changelog

Full Changelog

Full Changelog

  • update go to 1.19 + k8s.io libs to v0.25.3 (#1630 by @ImpSy)
  • Update README - secrets and sidecars need mutating webhooks (#1550 by @djdillon)

Full Changelog

  • Webhook init and cleanup should respect nodeSelector (#1545 by @erikcw)
  • rename unit tests to integration tests in Makefile#integration-test (#1539 by @dcoliversun)

Full Changelog

  • Fix: use V1 api for CRDs for volcano integration (#1540 by @Aakcht)

Full Changelog

  • fix: add pre-upgrade hook to rbac resources (#1511 by @cwyl02)

Full Changelog

  • Fixes issue #1467 (issue when deleting SparkApplication without metrics server) (#1530 by @aneagoe)
  • Implement --logs and --delete flags on 'sparkctl create' and a timeout on 'sparkctl log' to wait a pod startup (#1506 by @alaurentinoofficial)
  • Fix Spark UI URL in app status (#1518 by @gtopper)
  • remove quotes from yaml file (#1524 by @zencircle)
  • Added missing manifest yaml, point the manifest to the right direction (#1504 by @RonZhang724)

Full Changelog

  • Ensure that driver is deleted prior to sparkapplication resubmission (#1521 by @khorshuheng)

Full Changelog

Full Changelog

  • Add Operator volumes and volumeMounts in chart (#1475 by @ocworld)

Full Changelog

  • Updated default registry to ghcr.io (#1454 by @aneagoe)
  • Github actions workflow fix for Helm chart deployment (#1456 by @vara-bonthu)
  • Kubernetes v1.22 extensions/v1beta1 API removal (#1427 by @aneagoe)
  • Fixes an issue with github action in job build-spark-operator (#1452 by @aneagoe)
  • use github container registry instead of gcr.io for releases (#1422 by @TomHellier)
  • Fixes an error that was preventing the pods from being mutated (#1421 by @ssullivan)
  • Make github actions more feature complete (#1418 by @TomHellier)
  • Resolves an error when deploying the webhook where the k8s api indica… (#1413 by @ssullivan)

Full Changelog

Full Changelog

Full Changelog

  • delete-service-accounts-and-roles-before-creation (#1384 by @TiansuYu)

Full Changelog

Full Changelog

Full Changelog

  • Add custom toleration support for webhook jobs (#1383 by @korjek)
  • fix container name in addsecuritycontext patch (#1377 by @lybavsky)

Full Changelog

  • Role and RoleBinding not installed for webhook-init in Helm pre-hook (#1379 by @zzvara)

Full Changelog

Full Changelog

Full Changelog

  • Add hook deletion policy for spark-operator service account (#1313 by @pdrastil)

Full Changelog

Full Changelog

  • Migrate CRDs from v1beta1 to v1. Add additionalPrinterColumns (#1298 by @drazul)
  • Explain "signal: kill" errors during submission (#1292 by @zzvara)
  • fix the invalid repo address (#1291 by @william-wang)
  • add failure context to recordExecutorEvent (#1280 by @ImpSy)
  • Update pythonVersion to fix example (#1284 by @stratus)
  • add crds drift check between chart/ and manifest/ (#1272 by @ImpSy)

Full Changelog

  • Allow user to specify service annotation on Spark UI service (#1264 by @khorshuheng)

Full Changelog

  • implement shareProcessNamespace in SparkPodSpec (#1262 by @ImpSy)

Full Changelog

Full Changelog

Full Changelog

  • Add support for blue/green deployments (#1230 by @flupke)
  • Update who-is-using.md: Fossil is using Spark Operator for Production (#1244 by @duyet)

Full Changelog

Full Changelog

Full Changelog

Full Changelog

Full Changelog

Full Changelog

Full Changelog

Full Changelog