[SPARK-3299][SQL] add to SQLContext API to show tables #3872

bbejeck · 2015-01-01T18:44:39Z

SPARK-3299 Add to the SQLContext the ability to show tables, updated the Catalog, OverrideCatalog Traits, SimpleCatalog class, EmptyCatalog object, SQLContext class and the HiveMetastoreCatalog class. Added tests in ListTablesSuite. The method 'showTables' has been added, takes one parameter of type Optional[String] used for designating a database. I tried to include the HiveMetastoreCatalog class in the unit testing but was unsuccessful in being able to get a test to run. I will follow up on the developer list with questions on how to do this and update accordingly.

AmplabJenkins · 2015-01-01T18:47:09Z

Can one of the admins verify this patch?

andrewor14 · 2015-01-08T23:51:07Z

@bbejeck Could you change the title to include [SPARK-3299][SQL] instead? This will help us sort our PRs.
@marmbrus

andrewor14 · 2015-01-08T23:51:14Z

ok to test

SparkQA · 2015-01-08T23:52:30Z

Test build #25276 has started for PR 3872 at commit c5609fa.

This patch merges cleanly.

SparkQA · 2015-01-09T00:59:52Z

Test build #25276 has finished for PR 3872 at commit c5609fa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-09T00:59:55Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25276/
Test PASSed.

marmbrus · 2015-01-10T21:59:01Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Catalog.scala

@@ -41,6 +41,8 @@ trait Catalog {

  def unregisterAllTables(): Unit

+  def showTables(databaseName: Option[String]): immutable.List[String]


I think we should probably stick to generic Seq in public interfaces like this.

marmbrus · 2015-01-10T22:21:34Z

Thanks for working on this!

/cc @rxin

SparkQA · 2015-01-11T04:02:37Z

Test build #25371 has started for PR 3872 at commit a21b732.

This patch does not merge cleanly.

bbejeck · 2015-01-11T04:50:52Z

I need to merge in new changes then I will push the branch again

SparkQA · 2015-01-11T05:38:16Z

Test build #25371 has finished for PR 3872 at commit a21b732.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-11T05:38:20Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25371/
Test FAILed.

OopsOutOfMemory · 2015-01-11T06:09:27Z

sql/core/src/test/scala/org/apache/spark/sql/ListTablesSuite.scala

+class ListTablesSuite extends FunSuite with Matchers with BeforeAndAfter {
+
+  val simpleCatalog = new SimpleCatalog(true)
+  val expectedTablesOne = List("org.apache.spark.sql.ListTablesSuite.foo", "org.apache.spark.sql.ListTablesSuite.bar", "org.apache.spark.sql.ListTablesSuite.baz")


Beyond the line limit.

i.e. limit lines to 100 chars wide

SparkQA · 2015-01-12T02:57:37Z

Test build #25393 has started for PR 3872 at commit a42c963.

This patch merges cleanly.

SparkQA · 2015-01-12T02:58:30Z

Test build #25393 has finished for PR 3872 at commit a42c963.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-12T02:58:31Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25393/
Test FAILed.

bbejeck · 2015-01-12T03:00:25Z

Fixing style tests, thought I ran it locally...

SparkQA · 2015-01-12T03:12:38Z

Test build #25394 has started for PR 3872 at commit 90cbd43.

This patch merges cleanly.

bbejeck · 2015-01-12T03:19:29Z

I believe I've made all suggested changes, branch was out of date so it's been rebased, just waiting for tests to finish. Thanks to @marmbrus and @rxin for the guidance.

OopsOutOfMemory · 2015-01-12T03:45:04Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Catalog.scala

@@ -60,6 +60,10 @@ trait Catalog {
  protected def getDBTable(tableIdent: Seq[String]) : (Option[String], String) = {
    (tableIdent.lift(tableIdent.size - 2), tableIdent.last)
  }
+


remove empty line?

OopsOutOfMemory · 2015-01-12T03:45:16Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Catalog.scala

@@ -152,6 +160,16 @@ trait OverrideCatalog extends Catalog {
  override def unregisterAllTables(): Unit = {
    overrides.clear()
  }
+


bbejeck · 2015-01-12T04:07:30Z

@OopsOutOfMemory removed all empty lines, thanks for checking!

SparkQA · 2015-01-12T04:07:37Z

Test build #25396 has started for PR 3872 at commit 88c410d.

This patch merges cleanly.

SparkQA · 2015-01-12T04:19:01Z

Test build #25394 has finished for PR 3872 at commit 90cbd43.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-12T04:19:04Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25394/
Test PASSed.

SparkQA · 2015-01-12T05:14:49Z

Test build #25396 has finished for PR 3872 at commit 88c410d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-12T05:14:52Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25396/
Test PASSed.

rxin · 2015-01-16T07:39:27Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala

+
+  // Necessary helper method so showTables can return Array[String] vs List
+  private def convertToArray(tables: java.util.List[String]): Array[String] = {
+    new Array[String](tables.length)


This doesn't actually convert the list to an array, does it?

bbejeck · 2015-01-17T05:02:05Z

Addressed latest comments, rebased branch.

SparkQA · 2015-01-17T05:02:39Z

Test build #25696 has started for PR 3872 at commit 4ef099b.

This patch merges cleanly.

SparkQA · 2015-01-17T06:11:08Z

Test build #25696 has finished for PR 3872 at commit 4ef099b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-17T06:11:11Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25696/
Test PASSed.

yhuai · 2015-02-12T01:57:04Z

@bbejeck Thank you for working on this. I made another version based on yours (see #4547). What do you think about that approach (returning a DataFrame containing names of existing tables)?

bbejeck · 2015-02-12T04:12:40Z

@yhuai I think it's a good idea, nice work.

bbejeck · 2015-02-12T17:04:18Z

@yhuai , @marmbrus , @rxin should I close out my pull request, or would both of these be combined somehow?

yhuai · 2015-02-12T18:29:35Z

@bbejeck Yes, please close yours. We can continue the work in #4547. Also, feel free to leave comments in that PR. Thank you for working on it!

bbejeck · 2015-02-12T19:19:44Z

Closing this pull request in deference to the work being done in PR #4547

bbejeck changed the title ~~Spark 3299 add to SQLContext API to show tables~~ [SPARK-3299][SQL] add to SQLContext API to show tables Jan 9, 2015

marmbrus reviewed Jan 10, 2015
View reviewed changes

OopsOutOfMemory reviewed Jan 11, 2015
View reviewed changes

bbejeck force-pushed the spark-3299-SQLContext-list_tables branch from a21b732 to a42c963 Compare January 12, 2015 02:52

OopsOutOfMemory reviewed Jan 12, 2015
View reviewed changes

rxin reviewed Jan 16, 2015
View reviewed changes

bbejeck added 5 commits January 16, 2015 22:47

SPARK-3299 corrected covertToArray method implementation, rebased branch

da36273

SPARK-3299 fixing changes from merge

c5d9c39

SPARK-3299 corrected comment

e77c26b

SPARK-3299 fixing formatting after rebase

d6f9162

SPARK-3299 fixing scala style, file ends with newline character

4ef099b

bbejeck force-pushed the spark-3299-SQLContext-list_tables branch from 88c410d to 4ef099b Compare January 17, 2015 04:59

bbejeck closed this Feb 12, 2015

		@@ -41,6 +41,8 @@ trait Catalog {

		def unregisterAllTables(): Unit

		def showTables(databaseName: Option[String]): immutable.List[String]

[SPARK-3299][SQL] add to SQLContext API to show tables #3872

[SPARK-3299][SQL] add to SQLContext API to show tables #3872

Uh oh!

Conversation

bbejeck commented Jan 1, 2015

Uh oh!

AmplabJenkins commented Jan 1, 2015

Uh oh!

andrewor14 commented Jan 8, 2015

Uh oh!

andrewor14 commented Jan 8, 2015

Uh oh!

SparkQA commented Jan 8, 2015

Uh oh!

SparkQA commented Jan 9, 2015

Uh oh!

AmplabJenkins commented Jan 9, 2015

Uh oh!

marmbrus Jan 10, 2015

Choose a reason for hiding this comment

Uh oh!

marmbrus commented Jan 10, 2015

Uh oh!

SparkQA commented Jan 11, 2015

Uh oh!

bbejeck commented Jan 11, 2015

Uh oh!

SparkQA commented Jan 11, 2015

Uh oh!

AmplabJenkins commented Jan 11, 2015

Uh oh!

OopsOutOfMemory Jan 11, 2015

Choose a reason for hiding this comment

Uh oh!

rxin Jan 11, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jan 12, 2015

Uh oh!

SparkQA commented Jan 12, 2015

Uh oh!

AmplabJenkins commented Jan 12, 2015

Uh oh!

bbejeck commented Jan 12, 2015

Uh oh!

SparkQA commented Jan 12, 2015

Uh oh!

bbejeck commented Jan 12, 2015

Uh oh!

OopsOutOfMemory Jan 12, 2015

Choose a reason for hiding this comment

Uh oh!

OopsOutOfMemory Jan 12, 2015

Choose a reason for hiding this comment

Uh oh!

bbejeck commented Jan 12, 2015

Uh oh!

SparkQA commented Jan 12, 2015

Uh oh!

SparkQA commented Jan 12, 2015

Uh oh!

AmplabJenkins commented Jan 12, 2015

Uh oh!

SparkQA commented Jan 12, 2015

Uh oh!

AmplabJenkins commented Jan 12, 2015

Uh oh!

rxin Jan 16, 2015

Choose a reason for hiding this comment

Uh oh!

bbejeck commented Jan 17, 2015

Uh oh!

SparkQA commented Jan 17, 2015

Uh oh!

SparkQA commented Jan 17, 2015

Uh oh!

AmplabJenkins commented Jan 17, 2015

Uh oh!

yhuai commented Feb 12, 2015

Uh oh!