[SPARK-30312][SQL] Preserve path permission and acl when truncate table #26956

viirya · 2019-12-19T22:12:43Z

What changes were proposed in this pull request?

This patch proposes to preserve existing permission/acls of paths when truncate table/partition.

Why are the changes needed?

When Spark SQL truncates table, it deletes the paths of table/partitions, then re-create new ones. If permission/acls were set on the paths, the existing permission/acls will be deleted.

We should preserve the permission/acls if possible.

Does this PR introduce any user-facing change?

Yes. When truncate table/partition, Spark will keep permission/acls of paths.

How was this patch tested?

Unit test.

Manual test:

Create a table.
Manually change it permission/acl
Truncate table
Check permission/acl

val df = Seq(1, 2, 3).toDF
df.write.mode("overwrite").saveAsTable("test.test_truncate_table")
val testTable = spark.table("test.test_truncate_table")
testTable.show()
+-----+
|value|
+-----+
|    1|
|    2|
|    3|
+-----+
// hdfs dfs -setfacl ...
// hdfs dfs -getfacl ...
sql("truncate table test.test_truncate_table")
// hdfs dfs -getfacl ...
val testTable2 = spark.table("test.test_truncate_table")
testTable2.show()
+-----+
|value|
+-----+
+-----+

viirya · 2019-12-19T22:23:37Z

cc @cloud-fan @dongjoon-hyun @HyukjinKwon

SparkQA · 2019-12-20T02:26:27Z

Test build #115586 has finished for PR 26956 at commit 73e1bb0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-12-20T03:57:23Z

Test build #115588 has finished for PR 26956 at commit 60134c1.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class FakeLocalFsFileSystem extends RawLocalFileSystem

viirya · 2019-12-27T16:43:25Z

retest this please

SparkQA · 2019-12-27T20:33:06Z

Test build #115869 has finished for PR 26956 at commit 60134c1.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class FakeLocalFsFileSystem extends RawLocalFileSystem

SparkQA · 2019-12-28T02:27:28Z

Test build #115873 has finished for PR 26956 at commit 1df437b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-12-28T04:41:03Z

Test build #115875 has finished for PR 26956 at commit 2c3f1fb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2019-12-31T19:07:43Z

ping @cloud-fan @HyukjinKwon

cloud-fan · 2020-01-02T07:59:46Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala

+          // Not all fs impl. support these APIs.
+          val optPermission = Try(fileStatus.getPermission())
+          val optAcls = Try(fs.getAclStatus(path).getEntries)
+
          fs.delete(path, true)


not familiar with Hadoop FS APIs, but do we have something like rm -rf tablePath/*?. My point is, if we don't delete the parent folder, then we don't have this problem at all.

In the FileSystem API, I didn't find one like that.

But I just search again, there is FileUtil API that seems working like that.

I will test tomorrow to see if it work well to keep permission/acl in a Spark cluster like current approach.

oh, I think FileUtil.fullyDeleteContents only works on local file (java.io.File), not for files on distributed file system like HDFS.

So I think there is no single API to do rm -rf tablePath/* in Hadoop FS, if I haven't missed anything.

We can do listStatus and delete all contents in the given directory. But it is inefficient for DFS and is easier to have error.

Do you know how Hive/Presto implement TRUNCATE TABLE? Are there other file attributes we need to retain?

For Hive, I try to trace the code path for TRUNCATE TABLE:

HiveMetaStore. truncateTableInternal:

Get the locations of table/partitions and call Warehouse.deleteDir for each.

https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L3109

Warehouse.deleteDir delegates to MetaStoreFS.deleteDir.

HiveMetaStoreFsImpl (implements MetaStoreFS) delegates to FileUtils.moveToTrash which calls FileSystem.delete(f, true).

In HiveMetaStore.truncateTableInternal, after the location is deleted, new directory is created and previous file status including permissions, group, and ACLs are set to new directory:

https://github.com/apache/hive/blob/35f86c749cefc2a9972a991deed78a1c3719093d/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/HdfsUtils.java#L288

Now I retain permission and ACLs. I was doing fs.setOwner in first commit for retaining path owner and group. But fs.setOwner throws exception because it is only doable by super user.

In Hive code, since it is code running the metastore server, I think it is running with enough permission to set owner/group. For Spark, we may not running it with super user.

In Presto, looks like it takes the approach by listing all files/directories under the path to delete (i.e., rm -rf tablePath/*), and recursively deleting them:

https://github.com/prestodb/presto/blob/9c21aaa659f9d4927fe12bec4b8015c8cbf4722e/presto-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/SemiTransactionalHiveMetastore.java#L485

https://github.com/prestodb/presto/blob/9c21aaa659f9d4927fe12bec4b8015c8cbf4722e/presto-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/SemiTransactionalHiveMetastore.java#L1796

the presto way seems safer to me.

I was concerned with the presto way on performance regression. For a table which has many files/directories, deleting them one by one could be a bottleneck.

If we add a config for retaining permission/ACLs when truncating table? Users can choose to disable it and directly delete top directory (current behavior) without retaining permission/ACLs.

cloud-fan · 2020-01-03T07:56:55Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala

+              case Failure(e) if e.isInstanceOf[UnsupportedOperationException] =>
+                log.warn(s"Can not set original permission $permission back to " +
+                  s"the created path: $path. Exception: ${e.getMessage}")
+              case Failure(e) => throw e


do we have to fail if we can't set the permission back?

In internal discussion about this issue, we think it is more dangerous to have wrong permission/ACLs silently on a secure cluster when truncate table. This failure tells users what is wrong and they could so something manually to fix it.

Again, we can consider to add one config that users can choose to fail if we cannot preserve original permission/ACLs. For non secure cluster, it can be ignored.

Makes sense to me, I don't think we need a config. Shall we also fail if the FS can't set permission?

We already fail if the FS cannot set permission. No?

case Failure(e) if e.isInstanceOf[UnsupportedOperationException] => log.warn(s"Can not set original permission $permission back to " + s"the created path: $path. Exception: ${e.getMessage}")

it's just a warning

Oh, I see. You meant UnsupportedOperationException case. Hmm, setAcl also has UnsupportedOperationException case. I'm not sure if it is possible that we can get permission/ACLs, but setPermission/setAcl throws UnsupportedOperationException.

Logically, following above reason, it should be safer to fail it too if it is UnsupportedOperationException.

SparkQA · 2020-01-06T08:05:02Z

Test build #116136 has finished for PR 26956 at commit 39fe234.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2020-01-06T16:39:11Z

retest this please

SparkQA · 2020-01-06T20:31:42Z

Test build #116181 has finished for PR 26956 at commit 39fe234.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala

cloud-fan

LGTM except one comment

sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala

viirya · 2020-01-09T00:41:37Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala

+          try {
+            optPermission = Some(fileStatus.getPermission())
+          } catch {
+            case NonFatal(e) =>


@cloud-fan @HyukjinKwon By using try catch, I'm not sure what we should do for exception during getPermission/getAcls. I think we don't/can't do anything?

In this case, I guess case NonFatal(_) => // do nothing is okay.

dongjoon-hyun · 2020-01-09T00:45:53Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala

+          try {
+            optAcls = Some(fs.getAclStatus(path).getEntries)
+          } catch {
+            case NonFatal(e) =>


dongjoon-hyun · 2020-01-09T00:47:19Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala

+              fs.setPermission(path, permission)
+            } catch {
+              case NonFatal(e) =>
+                throw new AnalysisException(


Ur, AnalysisException looks weird a little. This is not Analysis error.

Also, Exception is too risky behavior change. (@gatorsmile ).
I guess warning might be enough?

Ur, AnalysisException looks weird a little. This is not Analysis error.

Ah, this follows throwing AnalysisException at the outside try..catch block. We can change this.

Also, Exception is too risky behavior change. (@gatorsmile ).
I guess warning might be enough?

For security reason, cannot set back original Acl/permission seems serious issue so Exception is thrown now. warning might be ignored and not noticed by users.

I'm ok with a warning if you all think it isn't an issue.

Ur, AnalysisException looks weird a little. This is not Analysis error.

Maybe SparkException or RuntimeException?

If this is for security, what about SecurityException?

If we need to raise an exception, shall we have a fallback configuration?

Based on previous discussion around #26956 (comment), we may not need a config.

SecurityException seems ok. cc @cloud-fan

SecurityException sounds good

dongjoon-hyun · 2020-01-09T00:49:42Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala

+              fs.setAcl(path, acls)
+            } catch {
+              case NonFatal(e) =>
+                throw new AnalysisException(


SparkQA · 2020-01-09T04:37:08Z

Test build #116329 has finished for PR 26956 at commit 7b29f2f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-01-10T19:57:07Z

Thanks!

…e table ### What changes were proposed in this pull request? This patch proposes to preserve existing permission/acls of paths when truncate table/partition. Note that this is backport of #26956 to branch-2.4. ### Why are the changes needed? When Spark SQL truncates table, it deletes the paths of table/partitions, then re-create new ones. If permission/acls were set on the paths, the existing permission/acls will be deleted. We should preserve the permission/acls if possible. ### Does this PR introduce any user-facing change? Yes. When truncate table/partition, Spark will keep permission/acls of paths. ### How was this patch tested? Unit test and manual test as shown in #26956. Closes #27173 from viirya/truncate-table-permission-2.4. Authored-by: Liang-Chi Hsieh <liangchi@uber.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

dongjoon-hyun · 2020-01-11T18:18:34Z

Ur, @viirya . Our Jenkins with Hadoop-2.7/Hive 1.2 job seems to fail. Could you take a look?

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7-hive-1.2/

Also, @shaneknapp , does our system have different umask?

viirya · 2020-01-11T19:34:35Z

@dongjoon-hyun Thanks for pinging me. Opened a followup at #27175.

dongjoon-hyun · 2020-01-11T20:57:30Z

Thanks!

gatorsmile · 2020-01-15T03:22:40Z

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

@@ -1988,6 +1988,14 @@ object SQLConf {
      .booleanConf
      .createWithDefault(false)

+  val TRUNCATE_TABLE_IGNORE_PERMISSION_ACL =
+    buildConf("spark.sql.truncateTable.ignorePermissionAcl")


-> spark.sql.truncateTable.ignorePermissionAcl.enabled?

Ok. I will create a followup for this. Thanks.

### What changes were proposed in this pull request? Based on the [comment](#26956 (comment)), this patch changes the SQL config name from `spark.sql.truncateTable.ignorePermissionAcl` to `spark.sql.truncateTable.ignorePermissionAcl.enabled`. ### Why are the changes needed? Make this config consistent other SQL configs. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Unit test. Closes #27210 from viirya/truncate-table-permission-followup. Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

Based on the [comment](#26956 (comment)), this patch changes the SQL config name from `spark.sql.truncateTable.ignorePermissionAcl` to `spark.sql.truncateTable.ignorePermissionAcl.enabled`. Make this config consistent other SQL configs. No. Unit test. Closes #27210 from viirya/truncate-table-permission-followup. Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit be4d825) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

### What changes were proposed in this pull request? This is a followup of #26956 to add a migration document for 2.4.5. ### Why are the changes needed? New legacy configuration will restore the previous behavior safely. ### Does this PR introduce any user-facing change? This PR updates the doc. <img width="763" alt="screenshot" src="https://user-images.githubusercontent.com/9700541/72639939-9da5a400-391b-11ea-87b1-14bca15db5a6.png"> ### How was this patch tested? Build the document and see the change manually. Closes #27269 from dongjoon-hyun/SPARK-30312. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

This is a followup of #26956 to add a migration document for 2.4.5. New legacy configuration will restore the previous behavior safely. This PR updates the doc. <img width="763" alt="screenshot" src="https://user-images.githubusercontent.com/9700541/72639939-9da5a400-391b-11ea-87b1-14bca15db5a6.png"> Build the document and see the change manually. Closes #27269 from dongjoon-hyun/SPARK-30312. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit fdbded3) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

…ntries when setting up ACLs in truncate table ### What changes were proposed in this pull request? This is a follow-up to the PR #26956. In #26956, the patch proposed to preserve path permission when truncating table. When setting up original ACLs, we need to set user/group/other permission as ACL entries too, otherwise if the path doesn't have default user/group/other ACL entries, ACL API will complain an error `Invalid ACL: the user, group and other entries are required.`. In short this change makes sure: 1. Permissions for user/group/other are always kept into ACLs to work with ACL API. 2. Other custom ACLs are still kept after TRUNCATE TABLE (#26956 did this). ### Why are the changes needed? Without this fix, `TRUNCATE TABLE` will get an error when setting up ACLs if there is no default default user/group/other ACL entries. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Update unit test. Manual test on dev Spark cluster. Set ACLs for a table path without default user/group/other ACL entries: ``` hdfs dfs -setfacl --set 'user:liangchi:rwx,user::rwx,group::r--,other::r--' /user/hive/warehouse/test.db/test_truncate_table hdfs dfs -getfacl /user/hive/warehouse/test.db/test_truncate_table # file: /user/hive/warehouse/test.db/test_truncate_table # owner: liangchi # group: supergroup user::rwx user:liangchi:rwx group::r-- mask::rwx other::r-- ``` Then run `sql("truncate table test.test_truncate_table")`, it works by normally truncating the table and preserve ACLs. Closes #27548 from viirya/fix-truncate-table-permission. Lead-authored-by: Liang-Chi Hsieh <liangchi@uber.com> Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

…ntries when setting up ACLs in truncate table ### What changes were proposed in this pull request? This is a follow-up to the PR #26956. In #26956, the patch proposed to preserve path permission when truncating table. When setting up original ACLs, we need to set user/group/other permission as ACL entries too, otherwise if the path doesn't have default user/group/other ACL entries, ACL API will complain an error `Invalid ACL: the user, group and other entries are required.`. In short this change makes sure: 1. Permissions for user/group/other are always kept into ACLs to work with ACL API. 2. Other custom ACLs are still kept after TRUNCATE TABLE (#26956 did this). ### Why are the changes needed? Without this fix, `TRUNCATE TABLE` will get an error when setting up ACLs if there is no default default user/group/other ACL entries. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Update unit test. Manual test on dev Spark cluster. Set ACLs for a table path without default user/group/other ACL entries: ``` hdfs dfs -setfacl --set 'user:liangchi:rwx,user::rwx,group::r--,other::r--' /user/hive/warehouse/test.db/test_truncate_table hdfs dfs -getfacl /user/hive/warehouse/test.db/test_truncate_table # file: /user/hive/warehouse/test.db/test_truncate_table # owner: liangchi # group: supergroup user::rwx user:liangchi:rwx group::r-- mask::rwx other::r-- ``` Then run `sql("truncate table test.test_truncate_table")`, it works by normally truncating the table and preserve ACLs. Closes #27548 from viirya/fix-truncate-table-permission. Lead-authored-by: Liang-Chi Hsieh <liangchi@uber.com> Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 5b76367) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

…ndle non-existed path ### What changes were proposed in this pull request? This fix #26956 Wrap try-catch on `fs.getFileStatus(path)` within acl/permission in case of the path doesn't exist. ### Why are the changes needed? `truncate table` may fail to re-create path in case of interruption or something else. As a result, next time we `truncate table` on the same table with acl/permission, it will fail due to `FileNotFoundException`. And it also brings behavior change compares to previous Spark version, which could still `truncate table` successfully even if the path doesn't exist. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Added UT. Closes #27923 from Ngone51/fix_truncate. Authored-by: yi.wu <yi.wu@databricks.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…ndle non-existed path ### What changes were proposed in this pull request? This fix #26956 Wrap try-catch on `fs.getFileStatus(path)` within acl/permission in case of the path doesn't exist. ### Why are the changes needed? `truncate table` may fail to re-create path in case of interruption or something else. As a result, next time we `truncate table` on the same table with acl/permission, it will fail due to `FileNotFoundException`. And it also brings behavior change compares to previous Spark version, which could still `truncate table` successfully even if the path doesn't exist. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Added UT. Closes #27923 from Ngone51/fix_truncate. Authored-by: yi.wu <yi.wu@databricks.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit cb26f63) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…ntries when setting up ACLs in truncate table ### What changes were proposed in this pull request? This is a follow-up to the PR apache#26956. In apache#26956, the patch proposed to preserve path permission when truncating table. When setting up original ACLs, we need to set user/group/other permission as ACL entries too, otherwise if the path doesn't have default user/group/other ACL entries, ACL API will complain an error `Invalid ACL: the user, group and other entries are required.`. In short this change makes sure: 1. Permissions for user/group/other are always kept into ACLs to work with ACL API. 2. Other custom ACLs are still kept after TRUNCATE TABLE (apache#26956 did this). ### Why are the changes needed? Without this fix, `TRUNCATE TABLE` will get an error when setting up ACLs if there is no default default user/group/other ACL entries. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Update unit test. Manual test on dev Spark cluster. Set ACLs for a table path without default user/group/other ACL entries: ``` hdfs dfs -setfacl --set 'user:liangchi:rwx,user::rwx,group::r--,other::r--' /user/hive/warehouse/test.db/test_truncate_table hdfs dfs -getfacl /user/hive/warehouse/test.db/test_truncate_table # file: /user/hive/warehouse/test.db/test_truncate_table # owner: liangchi # group: supergroup user::rwx user:liangchi:rwx group::r-- mask::rwx other::r-- ``` Then run `sql("truncate table test.test_truncate_table")`, it works by normally truncating the table and preserve ACLs. Closes apache#27548 from viirya/fix-truncate-table-permission. Lead-authored-by: Liang-Chi Hsieh <liangchi@uber.com> Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

…ndle non-existed path ### What changes were proposed in this pull request? This fix apache#26956 Wrap try-catch on `fs.getFileStatus(path)` within acl/permission in case of the path doesn't exist. ### Why are the changes needed? `truncate table` may fail to re-create path in case of interruption or something else. As a result, next time we `truncate table` on the same table with acl/permission, it will fail due to `FileNotFoundException`. And it also brings behavior change compares to previous Spark version, which could still `truncate table` successfully even if the path doesn't exist. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Added UT. Closes apache#27923 from Ngone51/fix_truncate. Authored-by: yi.wu <yi.wu@databricks.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

Preserve original permission when truncate table.

73e1bb0

Update test case for ACL test.

60134c1

viirya added 2 commits December 27, 2019 14:33

Remove setOwner.

1df437b

Consider the cases that not all fs support these APIs.

2c3f1fb

viirya changed the title ~~[SPARK-30312][SQL] Preserve path attributes including permission when truncate table~~ [SPARK-30312][SQL] Preserve path permission and acl when truncate table Dec 31, 2019

cloud-fan reviewed Jan 2, 2020

View reviewed changes

cloud-fan reviewed Jan 3, 2020

View reviewed changes

Address comment.

39fe234

cloud-fan reviewed Jan 7, 2020

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala Outdated Show resolved Hide resolved

cloud-fan approved these changes Jan 7, 2020

View reviewed changes

HyukjinKwon reviewed Jan 7, 2020

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala Outdated Show resolved Hide resolved

HyukjinKwon approved these changes Jan 7, 2020

View reviewed changes

Use try instead of Try.

7b29f2f

viirya commented Jan 9, 2020

View reviewed changes

dongjoon-hyun reviewed Jan 9, 2020

View reviewed changes

Address comment.

ab69638

viirya mentioned this pull request Jan 11, 2020

[SPARK-30312][SQL][2.4] Preserve path permission and acl when truncate table #27173

Closed

gatorsmile reviewed Jan 15, 2020

View reviewed changes

viirya mentioned this pull request Jan 15, 2020

[SPARK-30312][SQL][FOLLOWUP] Rename conf by adding .enabled #27210

Closed

dongjoon-hyun mentioned this pull request Jan 17, 2020

[SPARK-30312][DOCS][FOLLOWUP] Add a migration guide #27269

Closed

viirya mentioned this pull request Feb 12, 2020

[SPARK-30797][SQL] Set tradition user/group/other permission to ACL entries when setting up ACLs in truncate table #27548

Closed

Ngone51 mentioned this pull request Mar 16, 2020

[SPARK-31163][SQL] TruncateTableCommand with acl/permission should handle non-existent path #27923

Closed

viirya deleted the truncate-table-permission branch December 27, 2023 18:38

[SPARK-30312][SQL] Preserve path permission and acl when truncate table #26956

[SPARK-30312][SQL] Preserve path permission and acl when truncate table #26956

Uh oh!

Conversation

viirya commented Dec 19, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

viirya commented Dec 19, 2019

Uh oh!

SparkQA commented Dec 20, 2019

Uh oh!

SparkQA commented Dec 20, 2019

Uh oh!

viirya commented Dec 27, 2019

Uh oh!

SparkQA commented Dec 27, 2019

Uh oh!

SparkQA commented Dec 28, 2019

Uh oh!

SparkQA commented Dec 28, 2019

Uh oh!

viirya commented Dec 31, 2019

Uh oh!

cloud-fan Jan 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viirya Jan 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jan 6, 2020

Uh oh!

viirya commented Jan 6, 2020

Uh oh!

SparkQA commented Jan 6, 2020

Uh oh!

Uh oh!

cloud-fan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viirya commented Dec 19, 2019 •

edited

Loading

cloud-fan Jan 2, 2020 •

edited

Loading

viirya Jan 2, 2020 •

edited

Loading

dongjoon-hyun Jan 9, 2020 •

edited

Loading

dongjoon-hyun commented Jan 11, 2020 •

edited

Loading