[SPARK-35382][PYTHON] Fix lambda variable name issues in nested DataFrame functions in Python APIs. #32523

ueshin · 2021-05-12T18:55:51Z

What changes were proposed in this pull request?

This PR fixes the same issue as #32424.

from pyspark.sql.functions import flatten, struct, transform
df = spark.sql("SELECT array(1, 2, 3) as numbers, array('a', 'b', 'c') as letters")
df.select(flatten(
    transform(
        "numbers",
        lambda number: transform(
            "letters",
            lambda letter: struct(number.alias("n"), letter.alias("l"))
        )
    )
).alias("zipped")).show(truncate=False)

Before:

+------------------------------------------------------------------------+
|zipped                                                                  |
+------------------------------------------------------------------------+
|[{a, a}, {b, b}, {c, c}, {a, a}, {b, b}, {c, c}, {a, a}, {b, b}, {c, c}]|
+------------------------------------------------------------------------+

After:

+------------------------------------------------------------------------+
|zipped                                                                  |
+------------------------------------------------------------------------+
|[{1, a}, {1, b}, {1, c}, {2, a}, {2, b}, {2, c}, {3, a}, {3, b}, {3, c}]|
+------------------------------------------------------------------------+

Why are the changes needed?

To produce the correct results.

Does this PR introduce any user-facing change?

Yes, it fixes the results to be correct as mentioned above.

How was this patch tested?

Added a unit test as well as manually.

…on APIs.

ueshin · 2021-05-12T18:56:27Z

cc @maropu @zero323 @HyukjinKwon

.github/workflows/build_and_test.yml

SparkQA · 2021-05-12T19:47:38Z

Test build #138470 has finished for PR 32523 at commit 0e66385.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-05-12T20:11:22Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42991/

SparkQA · 2021-05-12T20:11:23Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42991/

maropu · 2021-05-13T00:26:10Z

The fix itself looks fine. Thank you, @ueshin ~

HyukjinKwon

The changes look good (except separating GA fix)

### What changes were proposed in this pull request? This PR allows the PR source branch to include slashes. ### Why are the changes needed? There are PRs whose source branches include slashes, like `issues/SPARK-35119/gha` here or #32523. Before the fix, the PR build fails in `Sync the current branch with the latest in Apache Spark` phase. For example, at #32523, the source branch is `issues/SPARK-35382/nested_higher_order_functions`: ``` ... fatal: couldn't find remote ref nested_higher_order_functions Error: Process completed with exit code 128. ``` (https://github.com/ueshin/apache-spark/runs/2569356241) ### Does this PR introduce _any_ user-facing change? No, this is a dev-only change. ### How was this patch tested? This PR source branch includes slashes and #32525 doesn't. Closes #32524 from ueshin/issues/SPARK-35119/gha. Authored-by: Takuya UESHIN <ueshin@databricks.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

This reverts commit 0e66385.

…ctions

SparkQA · 2021-05-13T03:04:01Z

Test build #138483 has finished for PR 32523 at commit a8f169e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-05-13T03:26:07Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43003/

SparkQA · 2021-05-13T03:26:08Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43003/

HyukjinKwon

LGTM

HyukjinKwon · 2021-05-13T05:58:00Z

Merged to master and branch-3.1.

…rame functions in Python APIs ### What changes were proposed in this pull request? This PR fixes the same issue as #32424. ```py from pyspark.sql.functions import flatten, struct, transform df = spark.sql("SELECT array(1, 2, 3) as numbers, array('a', 'b', 'c') as letters") df.select(flatten( transform( "numbers", lambda number: transform( "letters", lambda letter: struct(number.alias("n"), letter.alias("l")) ) ) ).alias("zipped")).show(truncate=False) ``` **Before:** ``` +------------------------------------------------------------------------+ |zipped | +------------------------------------------------------------------------+ |[{a, a}, {b, b}, {c, c}, {a, a}, {b, b}, {c, c}, {a, a}, {b, b}, {c, c}]| +------------------------------------------------------------------------+ ``` **After:** ``` +------------------------------------------------------------------------+ |zipped | +------------------------------------------------------------------------+ |[{1, a}, {1, b}, {1, c}, {2, a}, {2, b}, {2, c}, {3, a}, {3, b}, {3, c}]| +------------------------------------------------------------------------+ ``` ### Why are the changes needed? To produce the correct results. ### Does this PR introduce _any_ user-facing change? Yes, it fixes the results to be correct as mentioned above. ### How was this patch tested? Added a unit test as well as manually. Closes #32523 from ueshin/issues/SPARK-35382/nested_higher_order_functions. Authored-by: Takuya UESHIN <ueshin@databricks.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit 17b59a9) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

…rame functions in Python APIs ### What changes were proposed in this pull request? This PR fixes the same issue as apache#32424. ```py from pyspark.sql.functions import flatten, struct, transform df = spark.sql("SELECT array(1, 2, 3) as numbers, array('a', 'b', 'c') as letters") df.select(flatten( transform( "numbers", lambda number: transform( "letters", lambda letter: struct(number.alias("n"), letter.alias("l")) ) ) ).alias("zipped")).show(truncate=False) ``` **Before:** ``` +------------------------------------------------------------------------+ |zipped | +------------------------------------------------------------------------+ |[{a, a}, {b, b}, {c, c}, {a, a}, {b, b}, {c, c}, {a, a}, {b, b}, {c, c}]| +------------------------------------------------------------------------+ ``` **After:** ``` +------------------------------------------------------------------------+ |zipped | +------------------------------------------------------------------------+ |[{1, a}, {1, b}, {1, c}, {2, a}, {2, b}, {2, c}, {3, a}, {3, b}, {3, c}]| +------------------------------------------------------------------------+ ``` ### Why are the changes needed? To produce the correct results. ### Does this PR introduce _any_ user-facing change? Yes, it fixes the results to be correct as mentioned above. ### How was this patch tested? Added a unit test as well as manually. Closes apache#32523 from ueshin/issues/SPARK-35382/nested_higher_order_functions. Authored-by: Takuya UESHIN <ueshin@databricks.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit 17b59a9) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

…bda functions ### What changes were proposed in this pull request? 1, #39068 reused the `UnresolvedAttribute` for the `UnresolvedNamedLambdaVariable`, but then `Column('x')` and `UnresolvedNamedLambdaVariable('x')` are mixed in `lambda x: x + cdf.x` (since we use `x/y/z` as augment names); this PR adds the `UnresolvedNamedLambdaVariable` back to distinguish between `Column('x')` and `UnresolvedNamedLambdaVariable('x')`; 2, the `refreshVarName` logic in PySpark was added in #32523 to address similar issue in PySpark's Lambda Function, this PR adds a similar function in the Python Client to avoid rewriting the function expression in the server side, which is unnecessary and prone to error . ### Why are the changes needed? before this PR, the nested lambda function doesn't work properly ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? enabled UT and added UT Closes #39619 from zhengruifeng/connect_fix_nested_lambda. Authored-by: Ruifeng Zheng <ruifengz@apache.org> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>

Fix lambda variable name issues in nested DataFrame functions in Pyth…

61f3c00

…on APIs.

Test GitHub Actions.

0e66385

dongjoon-hyun reviewed May 12, 2021

View reviewed changes

.github/workflows/build_and_test.yml Outdated Show resolved Hide resolved

github-actions bot added CORE INFRA PYTHON SQL labels May 12, 2021

ueshin marked this pull request as draft May 12, 2021 19:18

ueshin mentioned this pull request May 12, 2021

[SPARK-35388][INFRA] Allow the PR source branch to include slashes. #32524

Closed

HyukjinKwon approved these changes May 13, 2021

View reviewed changes

ueshin added 2 commits May 12, 2021 19:37

Revert "Test GitHub Actions."

846ca23

This reverts commit 0e66385.

Merge branch 'master' into issues/SPARK-35382/nested_higher_order_fun…

a8f169e

…ctions

maropu approved these changes May 13, 2021

View reviewed changes

ueshin marked this pull request as ready for review May 13, 2021 05:27

ueshin requested a review from HyukjinKwon May 13, 2021 05:36

HyukjinKwon approved these changes May 13, 2021

View reviewed changes

HyukjinKwon closed this in 17b59a9 May 13, 2021

zhengruifeng mentioned this pull request Jan 17, 2023

[SPARK-42089][CONNECT][PYTHON] Fix variable name issues in nested lambda functions #39619

Closed

zhengruifeng mentioned this pull request Mar 6, 2023

[SPARK-42562][CONNECT] UnresolvedNamedLambdaVariable in python do not need unique names #40287

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-35382][PYTHON] Fix lambda variable name issues in nested DataFrame functions in Python APIs. #32523

[SPARK-35382][PYTHON] Fix lambda variable name issues in nested DataFrame functions in Python APIs. #32523

Uh oh!

ueshin commented May 12, 2021

Uh oh!

ueshin commented May 12, 2021

Uh oh!

Uh oh!

SparkQA commented May 12, 2021

Uh oh!

SparkQA commented May 12, 2021

Uh oh!

SparkQA commented May 12, 2021

Uh oh!

maropu commented May 13, 2021

Uh oh!

HyukjinKwon left a comment

Uh oh!

SparkQA commented May 13, 2021

Uh oh!

SparkQA commented May 13, 2021

Uh oh!

SparkQA commented May 13, 2021

Uh oh!

HyukjinKwon left a comment

Uh oh!

HyukjinKwon commented May 13, 2021

Uh oh!

Uh oh!

[SPARK-35382][PYTHON] Fix lambda variable name issues in nested DataFrame functions in Python APIs. #32523

[SPARK-35382][PYTHON] Fix lambda variable name issues in nested DataFrame functions in Python APIs. #32523

Uh oh!

Conversation

ueshin commented May 12, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

ueshin commented May 12, 2021

Uh oh!

Uh oh!

SparkQA commented May 12, 2021

Uh oh!

SparkQA commented May 12, 2021

Uh oh!

SparkQA commented May 12, 2021

Uh oh!

maropu commented May 13, 2021

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 13, 2021

Uh oh!

SparkQA commented May 13, 2021

Uh oh!

SparkQA commented May 13, 2021

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented May 13, 2021

Uh oh!

Uh oh!