Work around astype with columns in Pandas < 0.19.0 #7

HyukjinKwon · 2017-06-22T06:53:57Z

What changes were proposed in this pull request?

Currently the codes below fails in Pandas < 0.19.0

from pyspark.sql.types import *

schema = StructType().add("a", IntegerType()).add("b", StringType())\
                     .add("c", BooleanType()).add("d", FloatType())
data = [
    (1, "foo", True, 3.0,), (2, "foo", True, 5.0),
    (3, "bar", False, -1.0), (4, "bar", False, 6.0),
]
spark.createDataFrame(data, schema).toPandas().dtypes

This looks because this was added from 0.19.0 (astype with dict). This PR proposes to manually work around by for loop.

Before (pandas < 0.19.0):

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/hyukjinkwon/Desktop/workspace/repos/forked/spark/python/pyspark/sql/dataframe.py", line 1731, in toPandas
    return pd.DataFrame.from_records(self.collect(), columns=self.columns).astype(dtype)
  File "/Library/Python/2.7/site-packages/pandas/core/generic.py", line 2950, in astype
    raise_on_error=raise_on_error, **kwargs)
  File "/Library/Python/2.7/site-packages/pandas/core/internals.py", line 2938, in astype
    return self.apply('astype', dtype=dtype, **kwargs)
  File "/Library/Python/2.7/site-packages/pandas/core/internals.py", line 2890, in apply
    applied = getattr(b, f)(**kwargs)
  File "/Library/Python/2.7/site-packages/pandas/core/internals.py", line 434, in astype
    values=values, **kwargs)
  File "/Library/Python/2.7/site-packages/pandas/core/internals.py", line 449, in _astype
    dtype = np.dtype(dtype)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/core/_internal.py", line 61, in _usefields
    names, formats, offsets, titles = _makenames_list(adict, align)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/core/_internal.py", line 26, in _makenames_list
    n = len(obj)
TypeError: object of type 'type' has no len()

After (tested on Pandas 0.18.1 and 0.20.2):

a      int32
b     object
c       bool
d    float32
dtype: object

How was this patch tested?

Manually tested.

viirya · 2017-06-22T06:57:26Z

python/pyspark/sql/dataframe.py

+        pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns)
+
+        for f, t in dtype.items():
+            pdf[f] = pdf[f].astype(t)


I think we don't need copying data?

viirya · 2017-06-22T07:01:00Z

LGTM

cloud-fan · 2017-06-22T07:28:29Z

great, thanks! @HyukjinKwon @viirya

…types ### What changes were proposed in this pull request? This PR intends to fix a bug that occurs when comparing null types to decimal types in master/branch-3.0; ``` scala> Seq(BigDecimal(10)).toDF("v1").selectExpr("v1 = NULL").explain(true) org.apache.spark.sql.AnalysisException: cannot resolve '(`v1` = NULL)' due to data type mismatch: differing types in '(`v1` = NULL)' (decimal(38,18) and null).; line 1 pos 0; 'Project [(v1#5 = null) AS (v1 = NULL)#7] +- Project [value#2 AS v1#5] +- LocalRelation [value#2] ... ``` The query above passed in v2.4.5. ### Why are the changes needed? bugfix ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Added tests. Closes apache#28241 from maropu/SPARK-31468. Authored-by: Takeshi Yamamuro <yamamuro@apache.org> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

…types ### What changes were proposed in this pull request? This PR intends to fix a bug that occurs when comparing null types to decimal types in master/branch-3.0; ``` scala> Seq(BigDecimal(10)).toDF("v1").selectExpr("v1 = NULL").explain(true) org.apache.spark.sql.AnalysisException: cannot resolve '(`v1` = NULL)' due to data type mismatch: differing types in '(`v1` = NULL)' (decimal(38,18) and null).; line 1 pos 0; 'Project [(v1#5 = null) AS (v1 = NULL)#7] +- Project [value#2 AS v1#5] +- LocalRelation [value#2] ... ``` The query above passed in v2.4.5. ### Why are the changes needed? bugfix ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Added tests. Closes apache#28241 from maropu/SPARK-31468. Authored-by: Takeshi Yamamuro <yamamuro@apache.org> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit a7fb330) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

…n properly ### What changes were proposed in this pull request? Make `ResolveRelations` handle plan id properly ### Why are the changes needed? bug fix for Spark Connect, it won't affect classic Spark SQL before this PR: ``` from pyspark.sql import functions as sf spark.range(10).withColumn("value_1", sf.lit(1)).write.saveAsTable("test_table_1") spark.range(10).withColumnRenamed("id", "index").withColumn("value_2", sf.lit(2)).write.saveAsTable("test_table_2") df1 = spark.read.table("test_table_1") df2 = spark.read.table("test_table_2") df3 = spark.read.table("test_table_1") join1 = df1.join(df2, on=df1.id==df2.index).select(df2.index, df2.value_2) join2 = df3.join(join1, how="left", on=join1.index==df3.id) join2.schema ``` fails with ``` AnalysisException: [CANNOT_RESOLVE_DATAFRAME_COLUMN] Cannot resolve dataframe column "id". It's probably because of illegal references like `df1.select(df2.col("a"))`. SQLSTATE: 42704 ``` That is due to existing plan caching in `ResolveRelations` doesn't work with Spark Connect ``` === Applying Rule org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations === '[#12]Join LeftOuter, '`==`('index, 'id) '[#12]Join LeftOuter, '`==`('index, 'id) !:- '[#9]UnresolvedRelation [test_table_1], [], false :- '[#9]SubqueryAlias spark_catalog.default.test_table_1 !+- '[#11]Project ['index, 'value_2] : +- 'UnresolvedCatalogRelation `spark_catalog`.`default`.`test_table_1`, [], false ! +- '[#10]Join Inner, '`==`('id, 'index) +- '[#11]Project ['index, 'value_2] ! :- '[#7]UnresolvedRelation [test_table_1], [], false +- '[#10]Join Inner, '`==`('id, 'index) ! +- '[#8]UnresolvedRelation [test_table_2], [], false :- '[#9]SubqueryAlias spark_catalog.default.test_table_1 ! : +- 'UnresolvedCatalogRelation `spark_catalog`.`default`.`test_table_1`, [], false ! +- '[#8]SubqueryAlias spark_catalog.default.test_table_2 ! +- 'UnresolvedCatalogRelation `spark_catalog`.`default`.`test_table_2`, [], false Can not resolve 'id with plan 7 ``` `[#7]UnresolvedRelation [test_table_1], [], false` was wrongly resolved to the cached one ``` :- '[#9]SubqueryAlias spark_catalog.default.test_table_1 +- 'UnresolvedCatalogRelation `spark_catalog`.`default`.`test_table_1`, [], false ``` ### Does this PR introduce _any_ user-facing change? yes, bug fix ### How was this patch tested? added ut ### Was this patch authored or co-authored using generative AI tooling? ci Closes apache#45214 from zhengruifeng/connect_fix_read_join. Authored-by: Ruifeng Zheng <ruifengz@apache.org> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

…plan properly ### What changes were proposed in this pull request? Make `ResolveRelations` handle plan id properly cherry-pick bugfix apache#45214 to 3.5 ### Why are the changes needed? bug fix for Spark Connect, it won't affect classic Spark SQL before this PR: ``` from pyspark.sql import functions as sf spark.range(10).withColumn("value_1", sf.lit(1)).write.saveAsTable("test_table_1") spark.range(10).withColumnRenamed("id", "index").withColumn("value_2", sf.lit(2)).write.saveAsTable("test_table_2") df1 = spark.read.table("test_table_1") df2 = spark.read.table("test_table_2") df3 = spark.read.table("test_table_1") join1 = df1.join(df2, on=df1.id==df2.index).select(df2.index, df2.value_2) join2 = df3.join(join1, how="left", on=join1.index==df3.id) join2.schema ``` fails with ``` AnalysisException: [CANNOT_RESOLVE_DATAFRAME_COLUMN] Cannot resolve dataframe column "id". It's probably because of illegal references like `df1.select(df2.col("a"))`. SQLSTATE: 42704 ``` That is due to existing plan caching in `ResolveRelations` doesn't work with Spark Connect ``` === Applying Rule org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations === '[#12]Join LeftOuter, '`==`('index, 'id) '[#12]Join LeftOuter, '`==`('index, 'id) !:- '[#9]UnresolvedRelation [test_table_1], [], false :- '[#9]SubqueryAlias spark_catalog.default.test_table_1 !+- '[#11]Project ['index, 'value_2] : +- 'UnresolvedCatalogRelation `spark_catalog`.`default`.`test_table_1`, [], false ! +- '[#10]Join Inner, '`==`('id, 'index) +- '[#11]Project ['index, 'value_2] ! :- '[#7]UnresolvedRelation [test_table_1], [], false +- '[#10]Join Inner, '`==`('id, 'index) ! +- '[#8]UnresolvedRelation [test_table_2], [], false :- '[#9]SubqueryAlias spark_catalog.default.test_table_1 ! : +- 'UnresolvedCatalogRelation `spark_catalog`.`default`.`test_table_1`, [], false ! +- '[#8]SubqueryAlias spark_catalog.default.test_table_2 ! +- 'UnresolvedCatalogRelation `spark_catalog`.`default`.`test_table_2`, [], false Can not resolve 'id with plan 7 ``` `[#7]UnresolvedRelation [test_table_1], [], false` was wrongly resolved to the cached one ``` :- '[#9]SubqueryAlias spark_catalog.default.test_table_1 +- 'UnresolvedCatalogRelation `spark_catalog`.`default`.`test_table_1`, [], false ``` ### Does this PR introduce _any_ user-facing change? yes, bug fix ### How was this patch tested? added ut ### Was this patch authored or co-authored using generative AI tooling? ci Closes apache#46291 from zhengruifeng/connect_fix_read_join_35. Authored-by: Ruifeng Zheng <ruifengz@apache.org> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>

Work around astype with columns in Pandas < 0.19.0

e903cd2

HyukjinKwon mentioned this pull request Jun 22, 2017

[SPARK-21163][SQL] DataFrame.toPandas should respect the data type apache/spark#18378

Closed

viirya reviewed Jun 22, 2017

View reviewed changes

No copy

6702ad1

cloud-fan merged commit 357a798 into cloud-fan:to_pandas Jun 22, 2017

HyukjinKwon deleted the pandas-astype branch January 2, 2018 03:38

cloud-fan pushed a commit that referenced this pull request Mar 13, 2019

Fix UT (#7)

41f3a90

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Work around astype with columns in Pandas < 0.19.0 #7

Work around astype with columns in Pandas < 0.19.0 #7

Uh oh!

HyukjinKwon commented Jun 22, 2017 •

edited

Loading

Uh oh!

viirya Jun 22, 2017

Uh oh!

HyukjinKwon Jun 22, 2017

Uh oh!

viirya commented Jun 22, 2017

Uh oh!

cloud-fan commented Jun 22, 2017

Uh oh!

Uh oh!

Work around astype with columns in Pandas < 0.19.0 #7

Work around astype with columns in Pandas < 0.19.0 #7

Uh oh!

Conversation

HyukjinKwon commented Jun 22, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

viirya Jun 22, 2017

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon Jun 22, 2017

Choose a reason for hiding this comment

Uh oh!

viirya commented Jun 22, 2017

Uh oh!

cloud-fan commented Jun 22, 2017

Uh oh!

Uh oh!

HyukjinKwon commented Jun 22, 2017 •

edited

Loading