25 Pull requests opened by 24 people

• [SPARK-52104][CONNECT][SCALA] Validate column name eagerly in Spark Connect Scala Client

  #50873 opened May 13, 2025

• [SPARK-52110][SDP][SQL] Implement SQL syntax support for pipelines

  #50875 opened May 13, 2025

• [SPARK-52124] Actively Releasing Disk Space After Application Completio…

  #50888 opened May 14, 2025

• [SPARK-52153][SQL] Fix from_json and to_json with variant

  #50901 opened May 15, 2025

• [SPARK-52166] [SDP] Add support for PipelineEvents

  #50906 opened May 15, 2025

• [DO-NOT-MERGE][SPARK-52179][INFRA] GitHub workflow for releasing the official Apache Spark

  #50911 opened May 16, 2025

• [Spark-50873][SQL] Prune column for ExistsSubquery in DSV2

  #50916 opened May 16, 2025

• [SPARK-52186][SQL] Rewrite EXISTS to add Scalar Project

  #50917 opened May 16, 2025

• [SPARK-52185][CORE] Automate the thread dump collection for Spark applications

  #50919 opened May 16, 2025

• [SPARK-52187][SQL] Introduce Join pushdown for DSv2

  #50921 opened May 16, 2025

• [SPARK-52195][PYTHON][SS] Fix initial state column dropping issue for Python TWS

  #50926 opened May 16, 2025

• [SPARK-52212][PYTHON][INFRA] Upgrade linter image to python 3.11

  #50931 opened May 19, 2025

• [SPARK-52032][SQL] Fix orc filter pushdown with null-safe equality operator

  #50932 opened May 19, 2025

• [WIP] [SPARK-47547] BloomFilter fpp degradation

  #50933 opened May 19, 2025

• [SPARK-52216][SQL][CONNECT] Make InvalidCommandInput a user-facing error

  #50934 opened May 19, 2025

• [SPARK-52218][SQL] Make current datetime functions evaluable again

  #50936 opened May 19, 2025

• [SPARK-52219][SQL] Schema level collation support for tables

  #50937 opened May 19, 2025

• [MINOR][COMPILER] Fix error on non-string input to schema_of_csv/xml

  #50940 opened May 19, 2025

• [SPARK-52223][CONNECT] Add SDP Spark Connect Protos

  #50942 opened May 19, 2025

• [SPARK-51984][SQL] Support Check Constraint enforcement in UpdateTable and MergeIntoTable

  #50943 opened May 19, 2025

• [SPARK-52224][CONNECT][PYTHON] Introduce pyyaml as a dependency for the Python client

  #50944 opened May 19, 2025

• [SPARK-52226] [SQL] Strengthen unusual equality checks in three operators

  #50949 opened May 20, 2025

• [SPARK-52228][SS][PYSPARK] Construct the benchmark purposed TWS state server with in-memory state impls and the benchmark code in python

  #50952 opened May 20, 2025

• [SPARK-52079][SQL][Followup] Normalize internal projections under Aggregate as well

  #50953 opened May 20, 2025

• [SPARK-52232][SQL] Fix non-deterministic queries to produce different results at every step

  #50957 opened May 20, 2025

20 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• [SPARK-52095][SQL] Alter table alter column to pass V2Expression to DSV2

  #50864 commented on May 19, 2025 • 24 new comments

• [SPARK-52060][SQL] Make `OneRowRelationExec` node

  #50849 commented on May 20, 2025 • 9 new comments

• [SPARK-52215][PYTHON][CONNECT] Implement Scalar Arrow UDF

  #50759 commented on May 20, 2025 • 6 new comments

• [SPARK-45926][SQL] Implementing equals and hashCode which takes into account pushed runtime filters , in InMemoryTable related scans

  #49153 commented on May 20, 2025 • 1 new comment

• [SPARK-52103][SQL] Fallback complex expression whole-stage codegen

  #50871 commented on May 14, 2025 • 0 new comments

• [SPARK-52082][PYTHON] Improve ExtractPythonUDF docs

  #50867 commented on May 14, 2025 • 0 new comments

• [SPARK-52065][SQL] Produce another plan tree with output columns (name, data type, nullability) in plan change logging

  #50852 commented on May 14, 2025 • 0 new comments

• [SPARK-52024][SQL] Support cancel ShuffleQueryStage when propagate empty relations

  #50814 commented on May 16, 2025 • 0 new comments

• [SPARK-51867][ML][FOLLOW-UP] Make funtions in `ReadWriteUtils` private

  #50811 commented on May 14, 2025 • 0 new comments

• [SPARK-51927][SPARK-52213][BUILD] Upgrade jackson to 2.19.0 and upgrade kubernetes-client to version 7.3.0

  #50730 commented on May 20, 2025 • 0 new comments

• [Python] (PySpark) Support for subclasses in type_verifier

  #50726 commented on May 15, 2025 • 0 new comments

• [SPARK-51919][PYTHON] Allow overwriting statically registered Python Data Source

  #50716 commented on May 16, 2025 • 0 new comments

• Test rockdbsjni 10

  #50688 commented on May 19, 2025 • 0 new comments

• [SPARK-51883][DOCS][PYTHON] Python Data Source user guide for filter pushdown

  #50684 commented on May 15, 2025 • 0 new comments

• [SPARK-51745] Enforce State Machine for RocksDBStateStore

  #50497 commented on May 15, 2025 • 0 new comments

• Update supported_api_gen.py: remove an invalid escape sequence "\_" using a raw string

  #50243 commented on May 14, 2025 • 0 new comments

• [SPARK-51430][PYTHON] Stop PySpark context logger from propagating logs to stdout

  #50198 commented on May 19, 2025 • 0 new comments

• [SPARK-51268][SQL] Make `TreeNode` lock-free

  #50019 commented on May 15, 2025 • 0 new comments

• [WIP][SPARK-51180][BUILD] Upgrade Arrow to 19.0.0

  #49909 commented on May 14, 2025 • 0 new comments

• [SPARK-50137][HIVE] Avoid fallback to Hive-incompatible ways when table creation fails by thrift exception

  #48668 commented on May 13, 2025 • 0 new comments