[SPARK-35343][PYTHON] Make the conversion from/to pandas data-type-based for non-ExtensionDtypes #32592

xinrong-meng · 2021-05-19T17:41:29Z

What changes were proposed in this pull request?

Make the conversion from/to pandas (for non-ExtensionDtype) data-type-based.
NOTE: Ops class per ExtensionDtype and its data-type-based from/to pandas will be implemented in a separate PR as https://issues.apache.org/jira/browse/SPARK-35614.

Why are the changes needed?

The conversion from/to pandas includes logic for checking data types and behaving accordingly.
That makes code hard to change or maintain.
Since we have introduced the Ops class per non-ExtensionDtype data type, we ought to make the conversion from/to pandas data-type-based for non-ExtensionDtypes.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Unit tests.

Keyword: SPARK-35337

SparkQA · 2021-05-19T17:55:09Z

Test build #138714 has finished for PR 32592 at commit a8c7f8f.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-05-19T18:51:27Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43236/

SparkQA · 2021-05-19T18:55:00Z

Test build #138717 has finished for PR 32592 at commit b2a6442.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-05-19T19:31:21Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43236/

SparkQA · 2021-05-19T19:38:42Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43239/

SparkQA · 2021-05-19T20:12:31Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43239/

SparkQA · 2021-06-01T18:00:05Z

Test build #139177 has finished for PR 32592 at commit 6af8165.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-06-01T18:46:27Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43696/

SparkQA · 2021-06-01T19:18:26Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43696/

SparkQA · 2021-06-01T22:58:54Z

Test build #139184 has finished for PR 32592 at commit 69b47f0.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

xinrong-meng · 2021-06-01T23:04:02Z

python/pyspark/pandas/data_type_ops/base.py

@@ -87,7 +92,6 @@ def __init__(self, dtype: Dtype, spark_type: DataType):
        self.spark_type = spark_type

    @property
-    @abstractmethod


@abstractmethod is removed in order to pass mypy checks.
Otherwise,

mypy checks failed: python/pyspark/pandas/internal.py:1050: error: Cannot instantiate abstract class 'DataTypeOps' with abstract attribute 'pretty_name' python/pyspark/pandas/internal.py:1441: error: Cannot instantiate abstract class 'DataTypeOps' with abstract attribute 'pretty_name'

Reference mypy issue: python/mypy#1843.

python/pyspark/pandas/tests/indexes/test_base.py

SparkQA · 2021-06-01T23:14:15Z

Test build #139186 has finished for PR 32592 at commit 6f0bc5e.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-06-01T23:40:35Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43704/

SparkQA · 2021-06-01T23:45:39Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43705/

SparkQA · 2021-06-01T23:52:51Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43706/

SparkQA · 2021-06-02T00:11:39Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43704/

SparkQA · 2021-06-02T00:19:08Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43705/

SparkQA · 2021-06-02T01:22:55Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43706/

SparkQA · 2021-06-02T01:27:50Z

Test build #139185 has finished for PR 32592 at commit 5966987.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2021-06-02T02:29:53Z

Test build #139190 has finished for PR 32592 at commit ba1207f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-06-02T02:38:57Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43710/

SparkQA · 2021-06-02T04:08:37Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43710/

SparkQA · 2021-06-05T00:50:28Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43882/

SparkQA · 2021-06-05T01:22:23Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43882/

SparkQA · 2021-06-05T02:01:34Z

Test build #139360 has finished for PR 32592 at commit de83d86.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-06-07T17:31:03Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43948/

SparkQA · 2021-06-07T18:14:25Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43952/

SparkQA · 2021-06-07T18:50:18Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43952/

SparkQA · 2021-06-07T19:10:27Z

Test build #139426 has finished for PR 32592 at commit 555209b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ueshin

LGTM.

ueshin · 2021-06-07T20:11:48Z

Thanks! merging to master.

github-actions bot added BUILD CORE PYTHON labels May 19, 2021

xinrong-meng force-pushed the datatypeop_pd_conversion branch from b2a6442 to 6af8165 Compare June 1, 2021 17:30

xinrong-meng commented Jun 1, 2021

View reviewed changes

python/pyspark/pandas/tests/indexes/test_base.py Outdated Show resolved Hide resolved

xinrong-meng force-pushed the datatypeop_pd_conversion branch from 5966987 to 6f0bc5e Compare June 1, 2021 23:08

xinrong-meng changed the title ~~[WIP][SPARK-35343][PYTHON] Make conversion from/to pandas data-type-based~~ [WIP][SPARK-35343][PYTHON] Make the conversion from/to pandas (for non-ExtensionDtype) data-type-based. Jun 2, 2021

xinrong-meng changed the title ~~[WIP][SPARK-35343][PYTHON] Make the conversion from/to pandas (for non-ExtensionDtype) data-type-based.~~ [WIP][SPARK-35343][PYTHON] Make the conversion from/to pandas data-type-based for non-ExtensionDtypes Jun 2, 2021

xinrong-meng added 16 commits June 7, 2021 09:16

restore and prepare in data_type_ops

b74ed3b

Make conversion data-type-based

788c5a6

Default args at the end

df4fe85

Comments

0b78ded

Introduce NullOps

afbd214

For mypy checks

69a1c4e

Avoid circular improts

1a63822

Introduce UDTOps

af3bffc

test_null_ops

9fa2a40

test_udt_ops

1848976

pd_conversion unit tests

64045d1

Add to test modules

85b32d0

Address comments

7c627e4

Lint

7420a92

Lint

a75f5b4

Black lint

7c3efca

xinrong-meng force-pushed the datatypeop_pd_conversion branch from de83d86 to 7c3efca Compare June 7, 2021 16:33

Remove import

555209b

xinrong-meng requested a review from ueshin June 7, 2021 16:36

ueshin approved these changes Jun 7, 2021

View reviewed changes

ueshin closed this in 04a8d2c Jun 7, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-35343][PYTHON] Make the conversion from/to pandas data-type-based for non-ExtensionDtypes #32592

[SPARK-35343][PYTHON] Make the conversion from/to pandas data-type-based for non-ExtensionDtypes #32592

xinrong-meng commented May 19, 2021 •

edited

Loading

SparkQA commented May 19, 2021

SparkQA commented May 19, 2021

SparkQA commented May 19, 2021

SparkQA commented May 19, 2021

SparkQA commented May 19, 2021

SparkQA commented May 19, 2021

SparkQA commented Jun 1, 2021

SparkQA commented Jun 1, 2021

SparkQA commented Jun 1, 2021

SparkQA commented Jun 1, 2021

xinrong-meng Jun 1, 2021

SparkQA commented Jun 1, 2021

SparkQA commented Jun 1, 2021

SparkQA commented Jun 1, 2021

SparkQA commented Jun 1, 2021

SparkQA commented Jun 2, 2021

SparkQA commented Jun 2, 2021

SparkQA commented Jun 2, 2021

SparkQA commented Jun 2, 2021

SparkQA commented Jun 2, 2021

SparkQA commented Jun 2, 2021

SparkQA commented Jun 2, 2021

SparkQA commented Jun 5, 2021

SparkQA commented Jun 5, 2021

SparkQA commented Jun 5, 2021

SparkQA commented Jun 7, 2021

SparkQA commented Jun 7, 2021

SparkQA commented Jun 7, 2021

SparkQA commented Jun 7, 2021

ueshin left a comment

ueshin commented Jun 7, 2021

[SPARK-35343][PYTHON] Make the conversion from/to pandas data-type-based for non-ExtensionDtypes #32592

[SPARK-35343][PYTHON] Make the conversion from/to pandas data-type-based for non-ExtensionDtypes #32592

Conversation

xinrong-meng commented May 19, 2021 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented May 19, 2021

SparkQA commented May 19, 2021

SparkQA commented May 19, 2021

SparkQA commented May 19, 2021

SparkQA commented May 19, 2021

SparkQA commented May 19, 2021

SparkQA commented Jun 1, 2021

SparkQA commented Jun 1, 2021

SparkQA commented Jun 1, 2021

SparkQA commented Jun 1, 2021

xinrong-meng Jun 1, 2021

Choose a reason for hiding this comment

SparkQA commented Jun 1, 2021

SparkQA commented Jun 1, 2021

SparkQA commented Jun 1, 2021

SparkQA commented Jun 1, 2021

SparkQA commented Jun 2, 2021

SparkQA commented Jun 2, 2021

SparkQA commented Jun 2, 2021

SparkQA commented Jun 2, 2021

SparkQA commented Jun 2, 2021

SparkQA commented Jun 2, 2021

SparkQA commented Jun 2, 2021

SparkQA commented Jun 5, 2021

SparkQA commented Jun 5, 2021

SparkQA commented Jun 5, 2021

SparkQA commented Jun 7, 2021

SparkQA commented Jun 7, 2021

SparkQA commented Jun 7, 2021

SparkQA commented Jun 7, 2021

ueshin left a comment

Choose a reason for hiding this comment

ueshin commented Jun 7, 2021

xinrong-meng commented May 19, 2021 •

edited

Loading