[Storage System] Support for Google Cloud

[mukulmurthy]

This is the official issue for tracking the work for supporting Delta Lake over Google Cloud Storage.

Here are the requirements for the underlying storage system in order to maintain ACID guarantees:
https://github.com/delta-io/delta#requirements-for-underlying-storage-systems

HDFS over Google Cloud will work out of the box. We haven't yet investigated whether Google Cloud Storage satisfies these guarantees. If they don't, we will need a custom LogStore implementation similar to what we did for AWS and Azure (see below).

Related issues:
#39
#40
#134

[brkyvz]

This blog post seems to suggest it works out of the box:
https://cloudblog-withgoogle-com.cdn.ampproject.org/c/s/cloudblog.withgoogle.com/products/data-analytics/getting-started-with-new-table-formats-on-dataproc/amp/

[gkatzioura]

Hi all!

I am interested in using Delta Lake on Google Cloud so I did start to write an implementation for this one. It seems that unlike on S3 as mentioned on #39
Google Cloud Storage Object listing is strongly consistent as mentioned here.
Also provides a put if not present atomic mechanism as mentioned here.
As far as I can see the last one was recently announced.
Is it ok if my implementation imports the Google Cloud Storage Java library and interacts directly with the Google Cloud buckets?

[sap1ens]

Regarding atomic put if not present operation, object holds can be used as mentioned above, but preconditions seem to be a simpler option:

If you use 0 instead of a generation number in ifGenerationMatch, the request only succeeds if there is no live object in your bucket matching the object named in the request.

[fr-ashikaumagiliya]

is GCS supported now ? I found this article : https://cloud.google.com/blog/products/data-analytics/getting-started-with-new-table-formats-on-dataproc

[tdas]

Looks like they allow Delta Lake to operate on DataProc and GCS but it's unclear whether they have customized the LogStore implementation in their platform to get the correct semantics for writing delta log files. Without that, concurrent workloads will not provide ACID guarantees.

[fr-ashikaumagiliya]

Some object stores on other cloud providers do not provide a consistent listing, which could lead to data loss. So, in some cases Delta Lake only allows one single Spark session to write to the transaction log on the object store at a time. For the same reason, Delta Lake also disallows multiple writers, further limiting the concurrency and throughput of Delta Lake.

Cloud Storage allows multiple writers to the transaction log and guarantees consistency , allowing you to use Delta Lake to its full potential.
Delta Lake by default uses HDFS as the transaction log store. You can easily use Cloud Storage to store your transaction log >by pointing your table to a location on Cloud Storage using the format of gs:///.

looks like they have implemented a custom LogStore for CloudStorage ?

[tdas]

Even though they have not explicitly say, it indicates that they have done it. It would be good to get a confirmation from their support staff, etc. If you (or anyone) are in contact with GCS or DataProc support, then please ask them to confirm this.

…

On Sun, Oct 4, 2020 at 8:35 PM fr-ashikaumagiliya ***@***.***> wrote: Some object stores on other cloud providers do not provide a consistent listing, which could lead to data loss. So, in some cases Delta Lake only allows one single Spark session to write to the transaction log on the object store at a time. For the same reason, Delta Lake also disallows multiple writers, further limiting the concurrency and throughput of Delta Lake. *Cloud Storage allows multiple writers to the transaction log and guarantees consistency , allowing you to use Delta Lake to its full potential.* looks like they have implemented a custom LogStore ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#294 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFB5LGOQS5BXG4K2ARZBBLSJEIFDANCNFSM4KCGQXCQ> .

[jaceklaskowski]

What a timing! They've just expressed interest in this "closed" development in spark-bigquery-connector so there seem to be a mutual opportunity for both companies for a small exchange of goods / experience for the benefit of Apache Spark and Delta Lake communities 😎

[fr-ashikaumagiliya]

I received a comment from Google Support :

The internal ticket was routed to the engineer working on Delta Lake support in Dataproc.

GCS provides atomic operation for writing and listing [1], so it may not require custom LogStore. But, I'll double-check the behavior with the product team.
https://cloud.google.com/storage/docs/consistency

[fr-ashikaumagiliya]

And this

A specialist for Delta Lake support in Dataproc confirmed that GCS Connector 2.1.x works with Delta Lake concurrency. Dataproc clusters install GCS connector by default [1], so you don’t need to install additional dependency. You can use HDFSLogStore.

As GCS rename is implemented by “copy/rewrite + delete”, it needs to use GCS connector 2.1.5 and enable fs.gs.overwrite.generation.mismatch.ignore=true [1]. For this, you need to choose dataproc 1.5 [2].

As per the specialist, you can use concurrent Delta Lake with dataproc 1.5 and enabling fs.gs.overwrite.generation.mismatch.ignore=true.

[fr-ashikaumagiliya]

Seems concurrent writes are not supported :

Sorry, I have another update from a specialist of Delta Lake support in Dataproc that “concurrent write in Delta Lake is not supported for GCS in Dataproc” due to the corner cases related to “rename”.

They mentioned that it requires to implement “GcsLogStore” in Delta Lake which may be similar with S3SingleDriverLogStore [1]. A feature request internal ticket was filed with GCS connector team. But, there is no ETA.

[ranu010101]

I have raised a PR to address this.

Kindly review. Please suggest if I'm missing something.