Skip to content

Commit b2602c7

Browse files
ueshinHyukjinKwon
authored andcommitted
[SPARK-42338][CONNECT] Add details to non-fatal errors to raise a proper exception in the Python client
### What changes were proposed in this pull request? Adds details to non-fatal errors to raise a proper exception in the Python client, which makes `df.sample` raise `IllegalArgumentException` as same as PySpark, except for the timing that is delayed to call actions. ### Why are the changes needed? Currently `SparkConnectService` does not add details for `NonFatal` exceptions to the `PRCStatus`, so the Python client can't detect the exception properly and raises `SparkConnectGrpcException` instead. It also should have the details for the Python client. ### Does this PR introduce _any_ user-facing change? Users will see a proper exception when they call `df.sample` with illegal arguments, but in a different timing. ### How was this patch tested? Enabled `DataFrameParityTests.test_sample`. Closes #39957 from ueshin/issues/SPARK-42338/sample. Authored-by: Takuya UESHIN <ueshin@databricks.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit ced6750) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
1 parent 9e2fc64 commit b2602c7

File tree

2 files changed

+8
-2
lines changed

2 files changed

+8
-2
lines changed

connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,13 @@ class SparkConnectService(debug: Boolean)
109109
val status = RPCStatus
110110
.newBuilder()
111111
.setCode(RPCCode.INTERNAL_VALUE)
112+
.addDetails(
113+
ProtoAny.pack(
114+
ErrorInfo
115+
.newBuilder()
116+
.setReason(nf.getClass.getName)
117+
.setDomain("org.apache.spark")
118+
.build()))
112119
.setMessage(nf.getLocalizedMessage)
113120
.build()
114121
observer.onError(StatusProto.toStatusRuntimeException(status))

python/pyspark/sql/tests/connect/test_parity_dataframe.py

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -85,8 +85,7 @@ def test_require_cross(self):
8585
def test_same_semantics_error(self):
8686
super().test_same_semantics_error()
8787

88-
# TODO(SPARK-42338): Different exception in DataFrame.sample
89-
@unittest.skip("Fails in Spark Connect, should enable.")
88+
# Spark Connect throws `IllegalArgumentException` when calling `collect` instead of `sample`.
9089
def test_sample(self):
9190
super().test_sample()
9291

0 commit comments

Comments
 (0)