[SPARK-5195][sql]Update HiveMetastoreCatalog.scala(override the MetastoreRelation's sameresult method only compare databasename and table name) #3898

seayi · 2015-01-05T09:09:07Z

override the MetastoreRelation's sameresult method only compare databasename and table name

because in previous :
cache table t1;
select count() from t1;
it will read data from memory but the sql below will not,instead it read from hdfs:
select count() from t1 t;

because cache data is keyed by logical plan and compare with sameResult ,so when table with alias the same table 's logicalplan is not the same logical plan with out alias so modify the sameresult method only compare databasename and table name

in previous : cache table t1; select count(*) from t1; it will read data from memory but the sql below will not,instead it read from hdfs: select count(*) from t1 t; because cache data is keyed by logical plan and compare with sameResult ,so when with alias the same table 's logicalplan is not the same logical plan so modify the sameresult method only compare databasename and table name

AmplabJenkins · 2015-01-05T09:12:11Z

Can one of the admins verify this patch?

seayi · 2015-01-05T09:13:58Z

i test with hive table,after modify the sameresult method it is ok

marmbrus · 2015-01-06T07:02:49Z

Can you also add a regression test to CachedTableSuite?

marmbrus · 2015-01-06T07:03:02Z

ok to test

marmbrus · 2015-01-06T07:04:55Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala

@@ -427,6 +427,13 @@ private[hive] case class MetastoreRelation
          .getOrElse(sqlContext.defaultSizeInBytes))
    }
  )
+  override def sameResult(plan: LogicalPlan): Boolean = {
+    val new_plan = plan.asInstanceOf[MetastoreRelation];


what about when its not a metastore relation? how about:

plan match { case mr: MetastoreRelation => mr.databaseName == databaseName && mr.tableName == tableName case _ => false }

ok i will add thanks

marmbrus · 2015-01-06T07:06:18Z

Finally, please file a JIRA and add it to the PR title. Thanks!

SparkQA · 2015-01-06T07:07:39Z

Test build #25091 has started for PR 3898 at commit 8d910aa.

This patch merges cleanly.

SparkQA · 2015-01-06T08:57:26Z

Test build #25091 has finished for PR 3898 at commit 8d910aa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-06T08:57:32Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25091/
Test PASSed.

marmbrus · 2015-01-11T01:10:28Z

ping :)

seayi · 2015-01-11T15:11:42Z

sorry i am busy these days i will file a JIRA and add it to the PR title and pull a new request as soon as possibly.:) thanks again for your advice and attention!

SparkQA · 2015-01-11T15:27:39Z

Test build #25377 has started for PR 3898 at commit a277120.

This patch merges cleanly.

SparkQA · 2015-01-11T15:42:45Z

Test build #25378 has started for PR 3898 at commit 8f0c7d2.

This patch merges cleanly.

SparkQA · 2015-01-11T17:20:06Z

Test build #25377 has finished for PR 3898 at commit a277120.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-11T17:20:09Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25377/
Test PASSed.

SparkQA · 2015-01-11T17:35:01Z

Test build #25378 has finished for PR 3898 at commit 8f0c7d2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-11T17:35:05Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25378/
Test PASSed.

…toreRelation's sameresult method only compare databasename and table name) override the MetastoreRelation's sameresult method only compare databasename and table name because in previous : cache table t1; select count(*) from t1; it will read data from memory but the sql below will not,instead it read from hdfs: select count(*) from t1 t; because cache data is keyed by logical plan and compare with sameResult ,so when table with alias the same table 's logicalplan is not the same logical plan with out alias so modify the sameresult method only compare databasename and table name Author: seayi <405078363@qq.com> Author: Michael Armbrust <michael@databricks.com> Closes #3898 from seayi/branch-1.2 and squashes the following commits: 8f0c7d2 [seayi] Update CachedTableSuite.scala a277120 [seayi] Update HiveMetastoreCatalog.scala 8d910aa [seayi] Update HiveMetastoreCatalog.scala

marmbrus · 2015-02-03T00:10:16Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala

@@ -427,6 +427,13 @@ private[hive] case class MetastoreRelation
          .getOrElse(sqlContext.defaultSizeInBytes))
    }
  )
+  override def sameResult(plan: LogicalPlan): Boolean = {
+      plan match {
+           case mr: MetastoreRelation =>


nit: 2 spaces for intents.

I'll fix while merging.

thanks, marmbrus but i don't know how to make PRs against master,can you tell me? thanks again for your attension

When you open the PR make sure the base branch says master not branch-1.X. https://help.github.com/articles/comparing-commits-across-time/#comparing-branches

marmbrus · 2015-02-03T00:10:29Z

Thanks for fixing this! Merging to master.

marmbrus · 2015-02-03T00:13:50Z

Ahhh, I just noticed this was not made against master. Please make all PRs against master in the future.

…toreRelation's sameresult method only compare databasename and table name) override the MetastoreRelation's sameresult method only compare databasename and table name because in previous : cache table t1; select count(*) from t1; it will read data from memory but the sql below will not,instead it read from hdfs: select count(*) from t1 t; because cache data is keyed by logical plan and compare with sameResult ,so when table with alias the same table 's logicalplan is not the same logical plan with out alias so modify the sameresult method only compare databasename and table name Author: seayi <405078363@qq.com> Author: Michael Armbrust <michael@databricks.com> Closes apache#3898 from seayi/branch-1.2 and squashes the following commits: 8f0c7d2 [seayi] Update CachedTableSuite.scala a277120 [seayi] Update HiveMetastoreCatalog.scala 8d910aa [seayi] Update HiveMetastoreCatalog.scala

…toreRelation's sameresult method only compare databasename and table name) override the MetastoreRelation's sameresult method only compare databasename and table name because in previous : cache table t1; select count(*) from t1; it will read data from memory but the sql below will not,instead it read from hdfs: select count(*) from t1 t; because cache data is keyed by logical plan and compare with sameResult ,so when table with alias the same table 's logicalplan is not the same logical plan with out alias so modify the sameresult method only compare databasename and table name Author: seayi <405078363@qq.com> Author: Michael Armbrust <michael@databricks.com> Closes #3898 from seayi/branch-1.2 and squashes the following commits: 8f0c7d2 [seayi] Update CachedTableSuite.scala a277120 [seayi] Update HiveMetastoreCatalog.scala 8d910aa [seayi] Update HiveMetastoreCatalog.scala

…toreRelation's sameresult method only compare databasename and table name) override the MetastoreRelation's sameresult method only compare databasename and table name because in previous : cache table t1; select count(*) from t1; it will read data from memory but the sql below will not,instead it read from hdfs: select count(*) from t1 t; because cache data is keyed by logical plan and compare with sameResult ,so when table with alias the same table 's logicalplan is not the same logical plan with out alias so modify the sameresult method only compare databasename and table name Author: seayi <405078363@qq.com> Author: Michael Armbrust <michael@databricks.com> Closes apache#3898 from seayi/branch-1.2 and squashes the following commits: 8f0c7d2 [seayi] Update CachedTableSuite.scala a277120 [seayi] Update HiveMetastoreCatalog.scala 8d910aa [seayi] Update HiveMetastoreCatalog.scala

seayi changed the title ~~Update HiveMetastoreCatalog.scala~~ Update HiveMetastoreCatalog.scala(override the MetastoreRelation's sameresult method only compare databasename and table name) Jan 5, 2015

marmbrus reviewed Jan 6, 2015
View reviewed changes

Update HiveMetastoreCatalog.scala

a277120

Update CachedTableSuite.scala

8f0c7d2

marmbrus reviewed Feb 3, 2015
View reviewed changes

asfgit closed this in dca6faa Feb 3, 2015

[SPARK-5195][sql]Update HiveMetastoreCatalog.scala(override the MetastoreRelation's sameresult method only compare databasename and table name) #3898

[SPARK-5195][sql]Update HiveMetastoreCatalog.scala(override the MetastoreRelation's sameresult method only compare databasename and table name) #3898

Uh oh!

Conversation

seayi commented Jan 5, 2015

Uh oh!

AmplabJenkins commented Jan 5, 2015

Uh oh!

seayi commented Jan 5, 2015

Uh oh!

marmbrus commented Jan 6, 2015

Uh oh!

marmbrus commented Jan 6, 2015

Uh oh!

marmbrus Jan 6, 2015

Choose a reason for hiding this comment

Uh oh!

seayi Jan 9, 2015

Choose a reason for hiding this comment

Uh oh!

marmbrus commented Jan 6, 2015

Uh oh!

SparkQA commented Jan 6, 2015

Uh oh!

SparkQA commented Jan 6, 2015

Uh oh!

AmplabJenkins commented Jan 6, 2015

Uh oh!

marmbrus commented Jan 11, 2015

Uh oh!

seayi commented Jan 11, 2015

Uh oh!

SparkQA commented Jan 11, 2015

Uh oh!

SparkQA commented Jan 11, 2015

Uh oh!

SparkQA commented Jan 11, 2015

Uh oh!

AmplabJenkins commented Jan 11, 2015

Uh oh!

SparkQA commented Jan 11, 2015

Uh oh!

AmplabJenkins commented Jan 11, 2015

Uh oh!

marmbrus Feb 3, 2015

Choose a reason for hiding this comment

Uh oh!

seayi Feb 7, 2015

Choose a reason for hiding this comment

Uh oh!

marmbrus Feb 8, 2015

Choose a reason for hiding this comment

Uh oh!

marmbrus commented Feb 3, 2015

Uh oh!

marmbrus commented Feb 3, 2015

Uh oh!

Uh oh!