[SPARK-58684][PYTHON] Make PySpark error sub-conditions inherit the parent condition error state - #57887
Open
j1wonpark wants to merge 2 commits into
Open
[SPARK-58684][PYTHON] Make PySpark error sub-conditions inherit the parent condition error state#57887j1wonpark wants to merge 2 commits into
j1wonpark wants to merge 2 commits into
Conversation
…lass's sqlState get_sqlstate reads a sub-class level sqlState (SPARK-54532) but returns None when the sub-class does not declare one, even if the main class does: NEAREST_BY_JOIN.UNSUPPORTED_MODE reported no SQLSTATE despite NEAREST_BY_JOIN declaring 42604. get_breaking_change_info already falls back to the main class for the same shape. Make the resolution match the JVM framework: a sub-class sqlState overrides the main class's, and a sub-class without one inherits it. Signed-off-by: Jiwon Park <jpark92@outlook.kr>
uros-b
approved these changes
Aug 10, 2026
Member
|
Thank you @j1wonpark! |
nchammas
reviewed
Aug 10, 2026
The sub-class lookup allowed a sub-class to override the main class's sqlState, which no error condition does and which the JVM side does not support. Drop it so the Python and JVM readers resolve the same way. Signed-off-by: Jiwon Park <jpark92@outlook.kr>
nchammas
approved these changes
Aug 11, 2026
nchammas
left a comment
nchammas
left a comment
Contributor
There was a problem hiding this comment.
LGTM. Nitpick but: I would tweak the PR title to something a bit clearer (at least to me).
For example: "Make PySpark error sub-conditions inherit the parent condition error state"
j1wonpark
changed the title
Contributor
Author
|
Renamed, thanks. Also updated the description to use the same terminology. |
Member
|
so this one is a standalone bugfix, should land all active branches? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
ErrorClassesReader.get_sqlstatelooks the error state up under the sub-condition, so a sub-condition reportsNoneeven when its condition declares one. This makes the lookup resolve on the condition only, matchingErrorClassesJsonReader.getSqlStateon the JVM side.Split out of #57831; the two are unrelated, as
python/pyspark/errors/error-conditions.jsonis a separate file and does not contain the condition that PR is about.Why are the changes needed?
getSqlState()is public API onPySparkException.NEAREST_BY_JOINdeclares42604, but:The JVM reader resolves on the condition here, so the two sides report different error states for the same condition.
Does this PR introduce any user-facing change?
Yes.
getSqlState()returns the condition's error state for a sub-condition instead ofNone:NEAREST_BY_JOIN.*goes fromNoneto42604, the only condition affected today. Where the condition itself declares no error state, it still returnsNone.How was this patch tested?
Added a case to
test_sqlstatepinningNEAREST_BY_JOIN.UNSUPPORTED_MODEto42604, written test-first and confirmed failing before the fix. Addedtest_sqlstate_is_taken_from_the_main_classcovering a sub-condition that declares a different error state, an unknown sub-condition name, and an unknown condition. All 7 tests inpyspark.errors.tests.test_errorspass.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-fable-5, claude-opus-5)