Skip to content

Commit

Permalink
[MINOR][PYTHON][DOCS] Remove duplicated versionchanged per versionadded
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This PR addresses all the cases of duplicated `versionchanged` directives with `versionadded` directives, see also apache#42597.

Also, this PR mentions that all functions support Spark Connect from Apache Spark 3.5.0.

### Why are the changes needed?

To remove duplicated information in docstring.

### Does this PR introduce _any_ user-facing change?

Yes, it removes duplicated information in PySpark API Reference page.

### How was this patch tested?

CI in this PR should validate them.

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes apache#42602 from HyukjinKwon/minor-versionchanges.

Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
  • Loading branch information
HyukjinKwon authored and ragnarok56 committed Mar 2, 2024
1 parent 907be59 commit 7fe517e
Show file tree
Hide file tree
Showing 5 changed files with 67 additions and 74 deletions.
3 changes: 3 additions & 0 deletions python/docs/source/reference/pyspark.sql/functions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ Functions
=========
.. currentmodule:: pyspark.sql.functions

A collections of builtin functions available for DataFrame operations.
From Apache Spark 3.5.0, all functions support Spark Connect.

Normal Functions
----------------
.. autosummary::
Expand Down
18 changes: 2 additions & 16 deletions python/pyspark/sql/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,11 +39,7 @@ def __init__(self, jconf: JavaObject) -> None:

@since(2.0)
def set(self, key: str, value: Union[str, int, bool]) -> None:
"""Sets the given Spark runtime configuration property.
.. versionchanged:: 3.4.0
Supports Spark Connect.
"""
"""Sets the given Spark runtime configuration property."""
self._jconf.set(key, value)

@since(2.0)
Expand All @@ -52,9 +48,6 @@ def get(
) -> Optional[str]:
"""Returns the value of Spark runtime configuration property for the given key,
assuming it is set.
.. versionchanged:: 3.4.0
Supports Spark Connect.
"""
self._checkType(key, "key")
if default is _NoValue:
Expand All @@ -66,11 +59,7 @@ def get(

@since(2.0)
def unset(self, key: str) -> None:
"""Resets the configuration property for the given key.
.. versionchanged:: 3.4.0
Supports Spark Connect.
"""
"""Resets the configuration property for the given key."""
self._jconf.unset(key)

def _checkType(self, obj: Any, identifier: str) -> None:
Expand All @@ -84,9 +73,6 @@ def _checkType(self, obj: Any, identifier: str) -> None:
def isModifiable(self, key: str) -> bool:
"""Indicates whether the configuration property with the given key
is modifiable in the current session.
.. versionchanged:: 3.4.0
Supports Spark Connect.
"""
return self._jconf.isModifiable(key)

Expand Down
33 changes: 18 additions & 15 deletions python/pyspark/sql/dataframe.py
Original file line number Diff line number Diff line change
Expand Up @@ -2268,9 +2268,6 @@ def to(self, schema: StructType) -> "DataFrame":
.. versionadded:: 3.4.0
.. versionchanged:: 3.4.0
Supports Spark Connect.
Parameters
----------
schema : :class:`StructType`
Expand Down Expand Up @@ -2299,6 +2296,8 @@ def to(self, schema: StructType) -> "DataFrame":
* Fail if the nullability is not compatible. For example, the column and/or inner field
is nullable but the specified schema requires them to be not nullable.
Supports Spark Connect.
Examples
--------
>>> from pyspark.sql.types import StructField, StringType
Expand Down Expand Up @@ -3577,9 +3576,6 @@ def unpivot(
.. versionadded:: 3.4.0
.. versionchanged:: 3.4.0
Supports Spark Connect.
Parameters
----------
ids : str, Column, tuple, list
Expand All @@ -3599,6 +3595,10 @@ def unpivot(
:class:`DataFrame`
Unpivoted DataFrame.
Notes
-----
Supports Spark Connect.
Examples
--------
>>> df = spark.createDataFrame(
Expand Down Expand Up @@ -3663,9 +3663,6 @@ def melt(
.. versionadded:: 3.4.0
.. versionchanged:: 3.4.0
Supports Spark Connect.
Parameters
----------
ids : str, Column, tuple, list, optional
Expand All @@ -3688,6 +3685,10 @@ def melt(
See Also
--------
DataFrame.unpivot
Notes
-----
Supports Spark Connect.
"""
return self.unpivot(ids, values, variableColumnName, valueColumnName)

Expand Down Expand Up @@ -4265,9 +4266,6 @@ def dropDuplicatesWithinWatermark(self, subset: Optional[List[str]] = None) -> "
.. versionadded:: 3.5.0
.. versionchanged:: 3.5.0
Supports Spark Connect.
Parameters
----------
subset : List of column names, optional
Expand All @@ -4278,6 +4276,10 @@ def dropDuplicatesWithinWatermark(self, subset: Optional[List[str]] = None) -> "
:class:`DataFrame`
DataFrame without duplicates.
Notes
-----
Supports Spark Connect.
Examples
--------
>>> from pyspark.sql import Row
Expand Down Expand Up @@ -5243,9 +5245,6 @@ def withColumnsRenamed(self, colsMap: Dict[str, str]) -> "DataFrame":
.. versionadded:: 3.4.0
Added support for multiple columns renaming
.. versionchanged:: 3.4.0
Supports Spark Connect.
Parameters
----------
colsMap : dict
Expand All @@ -5261,6 +5260,10 @@ def withColumnsRenamed(self, colsMap: Dict[str, str]) -> "DataFrame":
--------
:meth:`withColumnRenamed`
Notes
-----
Support Spark Connect
Examples
--------
>>> df = spark.createDataFrame([(2, "Alice"), (5, "Bob")], schema=["age", "name"])
Expand Down
Loading

0 comments on commit 7fe517e

Please sign in to comment.