[SPARK-4502][SQL] Parquet nested column pruning #16578

mallman · 2017-01-13T18:30:59Z

(Link to Jira: https://issues.apache.org/jira/browse/SPARK-4502)

What changes were proposed in this pull request?

One of the hallmarks of a column-oriented data storage format is the ability to read data from a subset of columns, efficiently skipping reads from other columns. Spark has long had support for pruning unneeded top-level schema fields from the scan of a parquet file. For example, consider a table, contacts, backed by parquet with the following Spark SQL schema:

root
 |-- name: struct
 |    |-- first: string
 |    |-- last: string
 |-- address: string

Parquet stores this table's data in three physical columns: name.first, name.last and address. To answer the query

select address from contacts

Spark will read only from the address column of parquet data. However, to answer the query

select name.first from contacts

Spark will read name.first and name.last from parquet.

This PR modifies Spark SQL to support a finer-grain of schema pruning. With this patch, Spark reads only the name.first column to answer the previous query.

Implementation

There are three main components of this patch. First, there is a ParquetSchemaPruning optimizer rule for gathering the required schema fields of a PhysicalOperation over a parquet file, constructing a new schema based on those required fields and rewriting the plan in terms of that pruned schema. The pruned schema fields are pushed down to the parquet requested read schema. ParquetSchemaPruning uses a new ProjectionOverSchema extractor for rewriting a catalyst expression in terms of a pruned schema.

Second, the ParquetRowConverter has been patched to ensure the ordinals of the parquet columns read are correct for the pruned schema. ParquetReadSupport has been patched to address a compatibility mismatch between Spark's built in vectorized reader and the parquet-mr library's reader.

Third, we introduce two new catalyst query transformations, AggregateFieldExtractionPushdown and JoinFieldExtractionPushdown, to support schema pruning in aggregation and join query plans. These rules extract field references in aggregations and joins respectively, push down aliases to those references and replace them with references to the pushed down aliases. They use a new SelectedField extractor that transforms a catalyst complex type extractor (the "selected field") into a corresponding StructField.

Performance

The performance difference in executing queries with this patch compared to master is related to the depth of the table schema and the query itself. At VideoAmp, one of our biggest tables stores OpenRTB bid requests we receive from our exchange partners. Our bid request table's schema closely follows the OpenRTB bid request object schema. Additionally, when we bid we save our response along with the request in the same table. We store these two objects as two top-level fields in our table. Therefore, all bid request and response data are contained within nested fields.

For the purposes of measuring the performance impact of this patch, we ran some queries on our bid request table with the un-patched and patched master. We measured query execution time and the amount of data read from the underlying parquet files. I'll focus on a couple of benchmarks. (All benchmarks were run on an AWS EC2 cluster with four c3.8xl workers.) The first query I'll highlight is

select count(request.device.ip) from event.bid_request where ds=20161128 and h=0

(Hopefully it's obvious what this query means.) On the un-patched master, this query ran in 2.7 minutes and read 34.3 GB of data. On the patched master, this query ran in 4 seconds and read 987.3 MB of data.

We also ran a reporting-oriented query benchmark. I won't reproduce the query here, but it reads a larger subset of the bid request fields and joins against another table with a deeply nested schema. In addition to a join, we perform several aggregations in this query. On the un-patched master, this query ran in 3.4 minutes and read 34.6 GB of data. On the patched master, this query ran in 59 seconds and read 2.6 GB of data.

Limitation

Among the complex Spark SQL data types, this patch supports parquet column pruning of nested sequences of struct fields only.

How was this patch tested?

Care has been taken to ensure correctness and prevent regressions. This patch introduces over two dozen new unit tests and has been running on a production Spark 1.5 cluster at VideoAmp for about a year. In that time, one bug was found and fixed early on, and we added a regression test for that bug.

We forward-ported this patch to Spark master in June 2016 and have been running this patch against Spark 2.0 and 2.1 branches on ad-hoc clusters since then.

gatorsmile · 2017-01-13T18:40:23Z

...st/scala/org/apache/spark/sql/catalyst/optimizer/AggregateFieldExtractionPushdownSuite.scala

+    val originalQuery =
+      testRelation
+        .select('a)
+        .groupBy('a getField "a1")('a getField "a1" as 'a1, Count('*))


replace it by Count("*")?

mallman · 2017-01-13T19:10:42Z

cc @rxin @ericl @cloud-fan

@marmbrus I would love to get your feedback on this if you have the time.

SparkQA · 2017-01-13T21:02:44Z

Test build #71339 has finished for PR 16578 at commit 8925372.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class FieldExtractionPushdown extends Rule[LogicalPlan]
case class ProjectionOverSchema(schema: StructType)

SparkQA · 2017-01-13T21:32:25Z

Test build #71343 has finished for PR 16578 at commit efc4d76.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

lw-lin · 2017-01-15T07:05:22Z

Maybe we also want to get feedback from @liancheng ?

HyukjinKwon · 2017-01-15T07:40:57Z

Does this take over #14957? If so, we might need Closes #14957 in the PR description for the merge script to close that one or let the author know this takes over that.

mallman · 2017-01-16T20:53:37Z

Does this take over #14957? If so, we might need Closes #14957 in the PR description for the merge script to close that one or let the author know this takes over that.

I don't know. @xuanyuanking, how do you feel about this?

xuanyuanking · 2017-01-20T03:15:29Z

@mallman Thanks for let me know. I'll try your patch and check #14957 take over or not.
I also think we need getting feedback from @liancheng , from our last discussion, liancheng may do some work based on the old PR.

viirya · 2017-02-01T01:04:09Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/GetStructField2.scala

+ * This is in contrast to the [[GetStructField]] case class extractor which returns the field
+ * ordinal instead of the field itself.
+ */
+private[planning] object GetStructField2 {


Can we have a better name for this?

Or combine it with GetStructField case class extractor?

What do you mean by combining it with the existing case class extractor?

oh, nvm, I thought GetStructField is another Scala extractor. Actually it is a case class.

But we can have a better name.

Agreed. I think the best name in this context is GetStructField, but that's already taken. I'll keep thinking about a good alternative.

How about GetStructFieldObject? Or GetStructFieldRef?

GetStructFieldObject or GetStructFieldExtractor?

Let's go with GetStructFieldObject.

viirya · 2017-02-01T08:12:28Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/SelectedField.scala

+  }
+
+  /**
+   * Converts some chain of complex type extractors into a [[StructField]].


It is better to add few example input and output in the comment.

viirya · 2017-02-01T08:17:55Z

...src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala

+        // return [[op]]
+        if (requestedFields.exists { case (_, optAtt) => optAtt.isEmpty }) {
+          val prunedSchema = requestedFields
+            .map { case (field, _) => field }


Those two map can be done one map.

viirya · 2017-02-01T08:22:50Z

...src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala

+          // If the original Parquet relation data fields are different from the
+          // pruned data fields, continue. Otherwise, return [[op]]
+          if (parquetDataFields != prunedDataFields) {
+            val dataSchemaFieldNames = hadoopFsRelation.dataSchema.fieldNames


I think dataSchemaFieldNames is just parquetDataColumnNames above?

viirya · 2017-02-01T08:25:07Z

...src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala

+          if (parquetDataFields != prunedDataFields) {
+            val dataSchemaFieldNames = hadoopFsRelation.dataSchema.fieldNames
+            val newDataSchema =
+              StructType(prunedSchema.filter(f => dataSchemaFieldNames.contains(f.name)))


Is newDataSchema actually prunedDataSchema, if dataSchemaFieldNames is parquetDataColumnNames?

viirya · 2017-02-01T08:31:31Z

...src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala

+
+            logDebug("New projects:\n" + newProjects.map(_.treeString).mkString("\n"))
+
+            require(prunedDataSchema == prunedParquetRelation.dataSchema,


Could we break this require? I think it is not.

viirya · 2017-02-01T09:57:06Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/SelectedField.scala

+        val childField = fieldOpt.map(field => StructField(name, ArrayType(
+          StructType(Array(field)), arrayNullable), fieldNullable)).getOrElse(field)
+        selectField(child, Some(childField))
+      case GetArrayStructFields(child,


If we have a chain of GetArrayStructFields, looks like this will produce wrong result.

For example, if we add a "field5" like the following to the schema in SelectedFieldSuite.

StructField("field5", ArrayType(StructType( StructField("subfield1", StructType( StructField("subsubfield1", IntegerType) :: StructField("subsubfield2", IntegerType) :: Nil)) :: StructField("subfield2", StructType( StructField("subsubfield1", IntegerType) :: StructField("subsubfield2", IntegerType) :: Nil)) :: Nil)), false)

The extractor like col2.field5.subfield1.subsubfield1 will get:

StructField(col2, StructType( StructField(field5, ArrayType( ArrayType( StructType( StructField(subfield1, StructType( StructField(subsubfield1,IntegerType,true)) ,true) ), true), true), false) ), true)

I believe I have a fix for this, but I probably won't be able to post a new commit until early next week—I'm working on a proposal for the Spark Summit RFP.

Cheers.

I've spent some time this week developing a few different solutions to this problem, however none of them are very easy to understand or verify. I'm going to spend some more time working on a simpler solution before posting something back.

SparkQA · 2017-02-14T01:25:24Z

Test build #72836 has finished for PR 16578 at commit 948b584.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

mallman · 2017-02-14T18:17:35Z

@viirya I've added a commit to address some of your feedback. I will have another commit to address the others, but I'm not sure when I'll have it in. Hopefully by the end of next week.

mallman · 2017-03-16T23:16:14Z

@viirya A month has gone by since my last update. I've added much more comprehensive coverage to the SelectedFieldSuite, however I haven't yet fixed the SelectedField extractor to pass all of the tests. All of the failures are related to handling path expressions including GetArrayStructFields extractors. There are many complicated cases, and they are proving quite a challenge to resolve comprehensively.

I hope to spend some more time on this by the end of next week. I would love to push an update by then. After next week I will be away for two weeks.

Cheers.

mallman · 2017-03-24T20:45:58Z

I'm still working actively on this PR (as I have time), but I wanted to share that I will be away and unavailable from tonight, March 24th until Tuesday, April 11th. If you post a comment to this PR during that timeframe I'll respond as soon as I can after I return.

Cheers.

mallman · 2017-03-24T21:01:36Z

Rebased to latest master.

SparkQA · 2017-03-24T21:09:52Z

Test build #75178 has finished for PR 16578 at commit b20da5c.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-24T23:53:42Z

Test build #75181 has finished for PR 16578 at commit 30bc7e9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Gauravshah · 2017-03-30T14:21:09Z

can I do something to help this pull request ?

mallman · 2017-04-12T16:31:56Z

can I do something to help this pull request?

Hi @Gauravshah. Thanks for asking. Right now I need to fix a broken piece of the code, or reimplement it. At the moment this is something I'm dedicating myself to. Once I have that completed, a code review would be very helpful.

saulshanabrook · 2017-05-24T22:26:37Z

@mallman Would it be helpful for me to rebase this off the head? I am interested in helping with this too.

mallman · 2017-05-24T23:31:10Z

@saulshanabrook I'm sorry this effort has stalled. I actually have a lot of new code for this PR in my own clone, including a lot of new tests, but I haven't convinced myself that the code is actually correct and complete. I actually wrote a lot of new code to satisfy failing test cases that I found. I don't know that there aren't more test cases that would fail that I simply haven't uncovered.

I will take some time right now to re-examine my current code-to-date. If I push something tonight, it's likely to be something I have not established full confidence in.

mallman · 2017-05-25T02:28:49Z

@saulshanabrook I've pushed my latest work (rebased off the latest master). I'm not sure it's quite complete, but feel free to review and comment. FYI, the one piece of code that's given me the most trouble is the SelectedField extractor. It's passing all of the test cases I've thrown at it, but I can't quite convince myself that it's complete/correct. If you have some ideas for how to simplify it to make it easier to understand I welcome them.

If you'd like to submit actual changes to the code in this PR, please submit a PR against our https://github.com/VideoAmp/spark-public/tree/spark-4502-parquet_column_pruning branch, and I will review it there.

Thanks for your help.

API changes

constraints on complex type extractors

ParquetSchemaPruning.scala when deciding whether to perform schema pruning

ParquetSchemaPruningJoinSuite.scala to match revised pruning behavior

more idiomatic data. Also, add coverage to test for appropriate application of isnotnull, parquet-mr read support compatibility and the absence of schema pruning when the requested schema and the file schema differ only in the order of their columns

"nested field"

leaf fields in a catalyst data type. Use a method which counts the leaf fields directly

AggregateFieldExtractionPushdown and JoinFieldExtractionPushdown optimizations if there are no aliases

…uite

spark.sql.nestedSchemaPruning.enabled

SQL config flag that Parquet is currently the only supported data source. Also, set default value of this flag to "false" to bias toward correctness

in ParquetSchemaPruning.scala

turned on

SparkQA · 2018-04-24T21:18:56Z

Test build #89794 has finished for PR 16578 at commit dd4f2d8.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AggregateFieldExtractionPushdownSuite extends SchemaPruningTest
class JoinFieldExtractionPushdownSuite extends SchemaPruningTest

gatorsmile · 2018-05-02T00:54:20Z

To ensure the PR and review quality, we normally avoid doing everything in a single huge PR. It would be much better if you can cut it to a few smaller PRs. Both @cloud-fan and I think separating the optimizer rules makes sense. WDYT?

mallman · 2018-05-05T00:25:33Z

To ensure the PR and review quality, we normally avoid doing everything in a single huge PR. It would be much better if you can cut it to a few smaller PRs.

I'll have a go at it. Of course this will rewrite most of the commits, but I assume you don't mind that.

gatorsmile · 2018-05-05T00:32:05Z

Yeah. That is fine. Will try to review the relevant PRs ASAP. Please ping me. Thanks again!

mallman · 2018-05-11T04:11:23Z

BTW I’ve been and am currently traveling with a busy itinerary. I haven’t started work on this and probably won’t get to work on it until Monday at the very earliest.

…

On May 5, 2018, at 8:32 AM, Xiao Li ***@***.***> wrote: Yeah. That is fine. Will try to review the relevant PRs ASAP. Please ping me. Thanks again! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

mallman · 2018-05-14T10:02:46Z

I'm closing this PR in favor of #21320. That PR deals with simple projection and filter queries only. I will submit subsequent PRs for aggregation and join queries following the acceptance of #21320.

(Link to Jira: https://issues.apache.org/jira/browse/SPARK-4502) _N.B. This is a restart of PR #16578 which includes a subset of that code. Relevant review comments from that PR should be considered incorporated by reference. Please avoid duplication in review by reviewing that PR first. The summary below is an edited copy of the summary of the previous PR._ ## What changes were proposed in this pull request? One of the hallmarks of a column-oriented data storage format is the ability to read data from a subset of columns, efficiently skipping reads from other columns. Spark has long had support for pruning unneeded top-level schema fields from the scan of a parquet file. For example, consider a table, `contacts`, backed by parquet with the following Spark SQL schema: ``` root |-- name: struct | |-- first: string | |-- last: string |-- address: string ``` Parquet stores this table's data in three physical columns: `name.first`, `name.last` and `address`. To answer the query ```SQL select address from contacts ``` Spark will read only from the `address` column of parquet data. However, to answer the query ```SQL select name.first from contacts ``` Spark will read `name.first` and `name.last` from parquet. This PR modifies Spark SQL to support a finer-grain of schema pruning. With this patch, Spark reads only the `name.first` column to answer the previous query. ### Implementation There are two main components of this patch. First, there is a `ParquetSchemaPruning` optimizer rule for gathering the required schema fields of a `PhysicalOperation` over a parquet file, constructing a new schema based on those required fields and rewriting the plan in terms of that pruned schema. The pruned schema fields are pushed down to the parquet requested read schema. `ParquetSchemaPruning` uses a new `ProjectionOverSchema` extractor for rewriting a catalyst expression in terms of a pruned schema. Second, the `ParquetRowConverter` has been patched to ensure the ordinals of the parquet columns read are correct for the pruned schema. `ParquetReadSupport` has been patched to address a compatibility mismatch between Spark's built in vectorized reader and the parquet-mr library's reader. ### Limitation Among the complex Spark SQL data types, this patch supports parquet column pruning of nested sequences of struct fields only. ## How was this patch tested? Care has been taken to ensure correctness and prevent regressions. A more advanced version of this patch incorporating optimizations for rewriting queries involving aggregations and joins has been running on a production Spark cluster at VideoAmp for several years. In that time, one bug was found and fixed early on, and we added a regression test for that bug. We forward-ported this patch to Spark master in June 2016 and have been running this patch against Spark 2.x branches on ad-hoc clusters since then. Closes #21320 from mallman/spark-4502-parquet_column_pruning-foundation. Lead-authored-by: Michael Allman <msa@allman.ms> Co-authored-by: Adam Jacques <adam@technowizardry.net> Co-authored-by: Michael Allman <michael@videoamp.com> Signed-off-by: Xiao Li <gatorsmile@gmail.com>

(Link to Jira: https://issues.apache.org/jira/browse/SPARK-4502) _N.B. This is a restart of PR apache#16578 which includes a subset of that code. Relevant review comments from that PR should be considered incorporated by reference. Please avoid duplication in review by reviewing that PR first. The summary below is an edited copy of the summary of the previous PR._ One of the hallmarks of a column-oriented data storage format is the ability to read data from a subset of columns, efficiently skipping reads from other columns. Spark has long had support for pruning unneeded top-level schema fields from the scan of a parquet file. For example, consider a table, `contacts`, backed by parquet with the following Spark SQL schema: ``` root |-- name: struct | |-- first: string | |-- last: string |-- address: string ``` Parquet stores this table's data in three physical columns: `name.first`, `name.last` and `address`. To answer the query ```SQL select address from contacts ``` Spark will read only from the `address` column of parquet data. However, to answer the query ```SQL select name.first from contacts ``` Spark will read `name.first` and `name.last` from parquet. This PR modifies Spark SQL to support a finer-grain of schema pruning. With this patch, Spark reads only the `name.first` column to answer the previous query. There are two main components of this patch. First, there is a `ParquetSchemaPruning` optimizer rule for gathering the required schema fields of a `PhysicalOperation` over a parquet file, constructing a new schema based on those required fields and rewriting the plan in terms of that pruned schema. The pruned schema fields are pushed down to the parquet requested read schema. `ParquetSchemaPruning` uses a new `ProjectionOverSchema` extractor for rewriting a catalyst expression in terms of a pruned schema. Second, the `ParquetRowConverter` has been patched to ensure the ordinals of the parquet columns read are correct for the pruned schema. `ParquetReadSupport` has been patched to address a compatibility mismatch between Spark's built in vectorized reader and the parquet-mr library's reader. Among the complex Spark SQL data types, this patch supports parquet column pruning of nested sequences of struct fields only. Care has been taken to ensure correctness and prevent regressions. A more advanced version of this patch incorporating optimizations for rewriting queries involving aggregations and joins has been running on a production Spark cluster at VideoAmp for several years. In that time, one bug was found and fixed early on, and we added a regression test for that bug. We forward-ported this patch to Spark master in June 2016 and have been running this patch against Spark 2.x branches on ad-hoc clusters since then. Closes apache#21320 from mallman/spark-4502-parquet_column_pruning-foundation. Lead-authored-by: Michael Allman <msa@allman.ms> Co-authored-by: Adam Jacques <adam@technowizardry.net> Co-authored-by: Michael Allman <michael@videoamp.com> Signed-off-by: Xiao Li <gatorsmile@gmail.com> Ref: LIHADOOP-48531

gatorsmile reviewed Jan 13, 2017

View reviewed changes

viirya reviewed Feb 1, 2017

View reviewed changes

mallman force-pushed the spark-4502-parquet_column_pruning branch from 948b584 to b20da5c Compare March 24, 2017 21:00

saulshanabrook mentioned this pull request May 18, 2017

[SPARK-4502][SQL]Support parquet nested struct pruning and add relevant test #14957

Closed

mallman force-pushed the spark-4502-parquet_column_pruning branch from 30bc7e9 to 9f2f340 Compare May 25, 2017 02:21

Michael Allman and others added 18 commits April 25, 2018 01:10

Refine and (attempt to) clarify SelectedField.scala

106296f

"Fix" scalastyle errors in SelectedFieldSuite.scala

ea1e01a

Adjust some code in ParquetSchemaPruning.scala to conform to external

ab46037

API changes

(Updates from PR review feedback)

96c00d8

Document ColumnarFileFormat and make it private[sql]

17d00ce

Update documentation of QueryPlanConstraints.scala to cover case of

d95cb65

constraints on complex type extractors

Compare the parquet file schema and pruned parquet schema in

d36d380

ParquetSchemaPruning.scala when deciding whether to perform schema pruning

Update order of fields in expected schema in some test cases in

90ae3cd

ParquetSchemaPruningJoinSuite.scala to match revised pruning behavior

Fix Filter null check.

8b5661c

Alter some code docs to change terminology from "proper field" to

750954c

"nested field"

Forgo the use of ParquetSchemaConverter when counting the number of

dfc17a1

leaf fields in a catalyst data type. Use a method which counts the leaf fields directly

Enhance documentation and skip application of the

50ae29a

AggregateFieldExtractionPushdown and JoinFieldExtractionPushdown optimizations if there are no aliases

Don't print the schema before running the tests in the SelectedFieldS…

f4cf8ea

…uite

Put the schema pruning optimizer rules behind a boolean SQLConf flag,

812c70d

spark.sql.nestedSchemaPruning.enabled

Remark in the documentation of the spark.sql.nestedSchemaPruning.enabled

336cd40

SQL config flag that Parquet is currently the only supported data source. Also, set default value of this flag to "false" to bias toward correctness

Don't set catalogTable to None when copying an existing logical relation

0cd85b6

in ParquetSchemaPruning.scala

Add a SchemaPruningTest trait for running tests with schema pruning

dd4f2d8

turned on

mallman force-pushed the spark-4502-parquet_column_pruning branch from 27737a0 to dd4f2d8 Compare April 24, 2018 17:34

mallman mentioned this pull request May 14, 2018

[SPARK-4502][SQL] Parquet nested column pruning - foundation #21320

Closed

mallman closed this May 14, 2018

mallman mentioned this pull request Aug 4, 2018

[SPARK-4502][SQL] Parquet nested column pruning - foundation (2nd attempt) #21889

Closed


		logDebug("New projects:\n" + newProjects.map(_.treeString).mkString("\n"))

		require(prunedDataSchema == prunedParquetRelation.dataSchema,

[SPARK-4502][SQL] Parquet nested column pruning #16578

[SPARK-4502][SQL] Parquet nested column pruning #16578

Uh oh!

Conversation

mallman commented Jan 13, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Implementation

Performance

Limitation

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mallman commented Jan 13, 2017

Uh oh!

SparkQA commented Jan 13, 2017

Uh oh!

SparkQA commented Jan 13, 2017

Uh oh!

lw-lin commented Jan 15, 2017

Uh oh!

HyukjinKwon commented Jan 15, 2017

Uh oh!

mallman commented Jan 16, 2017

Uh oh!

xuanyuanking commented Jan 20, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Feb 14, 2017

Uh oh!

mallman commented Feb 14, 2017

Uh oh!

mallman commented Mar 16, 2017

Uh oh!

mallman commented Mar 24, 2017

Uh oh!

mallman commented Mar 24, 2017

Uh oh!

SparkQA commented Mar 24, 2017

Uh oh!

SparkQA commented Mar 24, 2017

Uh oh!

Gauravshah commented Mar 30, 2017

Uh oh!

mallman commented Jan 13, 2017 •

edited

Loading