Consider splitting Google Provider

[potiuk]

The Google provider is huge. It's by far the biggest provider out there and it has a number of sometimes more closely coupled and sometimes not coupled at all (leveldb) operators.

We should consider splitting the Google operator into smaller pieces, but that would involve some more decisions:

• what to do with common part
• what should be the split ? Per service? Per cloud ?
• how to manage dependencies between different provider
• how to manage transition and backwards compatibility

This is more of opening the discussion than making decisions yet.

[potiuk]

cc: @dstandish @kaxil @ashb @mik-laj @turbaszek @anitakar

[anitakar]

Hey, To start the discussion. The split that would make logical sense to me would be per Cloud Service (API), like shown be `gcloud services list`, with transfer operators belonging to appropriate Google side package, like Google Cloud Storage for gcs transfer operators. For the common part maybe an apache-airflow-providers-google-core package, similarly to https://pypi.org/project/google-cloud-core/ ? The most problematic part would be keeping common python libraries (but that could be solved by apache-airflow-providers-google-core package) and other cloud provider libraries in sync, especially between transfer operators, so maybe apache-airflow-providers-google-transfer-othercloudprovider for transfer operators in the end? That is what comes to my mind straight away but I am probably not seeing some potential problems. WDYT Anita śr., 19 maj 2021 o 07:59 Jarek Potiuk ***@***.***> napisał(a):

…

cc: @dstandish <https://github.com/dstandish> @kaxil <https://github.com/kaxil> @ashb <https://github.com/ashb> @mik-laj <https://github.com/mik-laj> @turbaszek <https://github.com/turbaszek> @anitakar <https://github.com/anitakar> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15933 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABE6AZEVD5PX22PR6PB7YULTONHTDANCNFSM45D4T6YA> .

[anitakar]

Having looked a little closer ( https://github.com/apache/airflow/tree/master/airflow/providers/google): 1. all the non cloud operators, sensors, hooks (ads, marketing_platform, firebase) could become their own modules (apache-airflow-providers-google-ads etc.) 2. all the files in cloud operators, sensors, hooks ( https://github.com/apache/airflow/tree/master/airflow/providers/google/cloud/operators) could become their own modules (apache-airflow-providers-google-dataflow etc.) 3. all transfer operators would have to move per service With modifying __init__.py in a smart way all operators could preserve the same import. The problem would be only with transfer operators as they would have to be moved not to have the same packages in different pypi modules and that would be a breaking change for the users. Seems like a lot of work. What would be the benefit? Being able to release for example firebase operators separately? śr., 19 maj 2021 o 22:56 Anita Fronczak ***@***.***> napisał(a):

…

Hey, To start the discussion. The split that would make logical sense to me would be per Cloud Service (API), like shown be `gcloud services list`, with transfer operators belonging to appropriate Google side package, like Google Cloud Storage for gcs transfer operators. For the common part maybe an apache-airflow-providers-google-core package, similarly to https://pypi.org/project/google-cloud-core/ ? The most problematic part would be keeping common python libraries (but that could be solved by apache-airflow-providers-google-core package) and other cloud provider libraries in sync, especially between transfer operators, so maybe apache-airflow-providers-google-transfer-othercloudprovider for transfer operators in the end? That is what comes to my mind straight away but I am probably not seeing some potential problems. WDYT Anita śr., 19 maj 2021 o 07:59 Jarek Potiuk ***@***.***> napisał(a): > cc: @dstandish <https://github.com/dstandish> @kaxil > <https://github.com/kaxil> @ashb <https://github.com/ashb> @mik-laj > <https://github.com/mik-laj> @turbaszek <https://github.com/turbaszek> > @anitakar <https://github.com/anitakar> > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#15933 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABE6AZEVD5PX22PR6PB7YULTONHTDANCNFSM45D4T6YA> > . >

[dstandish]

This came up in part because leveldb hooks depend on plyvel library which requires that the machine has leveldb installed.

So, prior to #15770, on mac if you want to install google provider (e.g. because you want to use some GCP hook) then you also have to install leveldb (which has nothing to do with GCP), and if you do not have leveldb installed, then the google provider install will fail.

To resolve this, what we ended up doing was making plyvel an optional dependency of google provider (#15770) so you have to install ...google[leveldb] to get plyvel (and this will only work if your system has leveldb installed). This also required that we make plyvel a "core" dependency of airflow, since CI requires it in order to fully import the google provider.

(Come to think of it perhaps CI could be amended to do ...provider-x[ALL] and then we would not have to add it as a core dep...)

Rather than making leveldb a google provider extra, I think a cleaner solution would be if leveldb was made a distinct provider, either google-leveldb or just leveldb. In the case of leveldb, there's no interaction with GCP hooks -- AFAIK it's completely independent. And this way plyvel would not need to be an extra, and would not need to be a airflow core dep.

The issue with leveldb was my main concern because being forced to brew install this system dependency is a bad user experience. And we have fixed the issue in a way, though it can be debated what the best solution is (i.e. sep provider vs extras). I'm not aware if there are other similar problems within the google provider, or other reasons that there may be to separate certain components of the provider

[turbaszek]

I think a cleaner solution would be if leveldb was made a distinct provider

+1 to that, I think we had similar discussion about salesforce, tableau and slack (Salesforce have bought Tableau and Slack):
https://lists.apache.org/thread.html/rb1828485df78df7649000ff586a749ab685635f606b394386e1be808%40%3Cdev.airflow.apache.org%3E

Also with companies like google it would be nice if they would be able to take care of the providers (even if that would mean extracting them from core). In that way they would be able to split it in whatever seems best.

[kaxil]

Agree with @turbaszek -- The best judge of the split is the company itself -- so according to me that would best. However, if Google does not want to do that, we should separate leveldb to a distinct provider like @dstandish suggested

[anitakar]

Hey, So maybe without big upfront design and trying to solve the problem of backwards compatibility for all cases, let's create another package for leveldb. I could start during the weekend but I would need some help. The way I see it you can not preserve same imports if you move the operator file outside the current folder. You can only nest it deeper and create __init__ .py that will fix the imports. Am I correct? Thanks, Anita

…

On Fri, 21 May 2021, 21:02 Kaxil Naik, ***@***.***> wrote: Agree with @turbaszek <https://github.com/turbaszek> -- The best judge of the split is the company itself -- so according to me that would best. However, if Google does not want to do that, we should separate leveldb to a distinct provider like @dstandish <https://github.com/dstandish> suggested — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15933 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABE6AZD4HMCWUPXMEZXONP3TO2U2ZANCNFSM45D4T6YA> .

[eladkal]

+1 to that, I think we had similar discussion about salesforce, tableau and slack (Salesforce have bought Tableau and Slack)

Thats true but there may be a different case here as leveldb is under Google github account. So the users of leveldb are very aware that this is a google service unlike Tableau/Slack & Salesforce relationship.

[potiuk]

Yeah. I see the case @eladkal. But I kind of see how Plyvel is very different from all other Google "core" components. Which cloud/ads/marketing_platform might be really "core" google business, where Plyvel is more of a "side" thing and could be fully separated out. I think it is much easier to separate out plyvel as a provider than splitting google "core" one.

I am quite torn on this one. I see the benefits of keeping all those as single "provider" - on the other hand the benefits of splitting it are "potentially good" but it makes them also difficult to work together when installed with all the dependencies - it will be easy for example to run in a situation where "cloud" dependencies are different (and conflicting) with "ads" dependencies. I think until we figure out (possibly) some way of separating the dependencies out between tasks, this might be a difficult one to tackle and rather than "origin" (google), the "independence" of a package might be more important.

I think the "extras" approach where plyvel is "optional" google extra dependency is maybe not perfect, but it kind of join both worlds. On one hand we have them together by "ownership", on the other hand we have just a problem of "environment" set properly for plyvel to work (so plyvel db libs installed). I think the current approach is kinda sustainable and does not introduce too much of complexity.

As usually with "compromise" solutions, it's not perfect but maybe it's "good enough"? As long as it works, it could be "OK". I am also quite OK to separate just plyvel as @anitakar suggested.

[anitakar]

@potiuk Thank you for pointing out that leveldb package is outside of cloud package ( https://github.com/apache/airflow/tree/master/airflow/providers/google/leveldb) and also that it is an extra ( https://github.com/apache/airflow/blob/master/airflow/providers/google/provider.yaml#L751 ). I am a little confused though. I see plyvel on the list of requirements on https://pypi.org/project/apache-airflow-providers-google/ and I do not see any extra name there. Ok, the change has been merged 4 days ago ( #15812). Yep, it seems to solve the problem of a problematic dependency on Mac. So, nothing more to be done there, at least in my opinion. Going back to splitting discussion. I can see that having to download and install lots of libraries when you just want to use for example firebase, can lead to dependency conflicts with your code, even though you do not use those other dependencies when you are using Google providers. Also, why download all those binaries and for example make your docker image bigger. But managing dependencies is a complicated and in my opinion quite unsolvable problem. Actually, the way it is now and as Jarek pointed out, when you install Google providers, the package has taken care of solving those conflicts at least for the Google providers package code. If I could I would like to make a poll here if people are unhappy about having to download and solve dependency conflicts against all libraries on Google provider package? Or maybe, they are happy that somebody has fixed those dependencies so that they match each other? When it comes to backwards compatibility. It is quite easy to do a split on the top level of google providers package ( https://github.com/apache/airflow/tree/master/airflow/providers/google). But it seems impossible inside cloud package ( https://github.com/apache/airflow/tree/master/airflow/providers/google/cloud) as packages there are split technically (hooks, sensors, operators ...) not on a domain/service level (automl, bigquery etc.). Operators could keep their imports ( https://github.com/apache/airflow/tree/master/airflow/providers/google/cloud/operators) but matching hooks and sensors couldn't, so a change would be breaking by definition. Personally, in software development I am all for splitting packages by domain not technical function. But I am also against forcing everyone to modify their code if there is no strong reason. I would also like to say those are my personal opinions not my employer's. sob., 22 maj 2021 o 18:41 Jarek Potiuk ***@***.***> napisał(a):

…

Yeah. I see the case @eladkal <https://github.com/eladkal>. But I kind of see how Plyvel is very different from all other Google "core" components. Which cloud/ads/marketing_platform might be really "core" google business, where Plyvel is more of a "side" thing and could be fully separated out. I think it is much easier to separate out plyvel as a provider than splitting google "core" one. I am quite torn on this one. I see the benefits of keeping all those as single "provider" - on the other hand the benefits of splitting it are "potentially good" but it makes them also difficult to work together when installed with all the dependencies - it will be easy for example to run in a situation where "cloud" dependencies are different (and conflicting) with "ads" dependencies. I think until we figure out (possibly) some way of separating the dependencies out between tasks, this might be a difficult one to tackle and rather than "origin" (google), the "independence" of a package might be more important. I think the "extras" approach where plyvel is "optional" google extra dependency is maybe not perfect, but it kind of join both worlds. On one hand we have them together by "ownership", on the other hand we have just a problem of "environment" set properly for plyvel to work (so plyvel db libs installed). I think the current approach is kinda sustainable and does not introduce too much of complexity. As usually with "compromise" solutions, it's not perfect but maybe it's "good enough"? As long as it works, it could be "OK". I am also quite OK to separate just plyvel as @anitakar <https://github.com/anitakar> suggested. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15933 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABE6AZBUE4CM5Z3JL4JJZUTTO7NEZANCNFSM45D4T6YA> .

[potiuk]

I am a little confused though. I see plyvel on the list of requirements

Yeah. It will be released in the next wave of providers. And I also think the [plyvel] extra is a good option especially that this is really just a dependency. The plyvel provider is still there it will just fail with missing import if it's not installed with extra (same as a number of other cases, where we have cross-provider dependency.

Or maybe, they are happy that somebody has fixed
those dependencies so that they match each other?

Yeah. I am a bit torn myself. I see a value in having the "google" provider to get "everything-google". It it 'biggish" but on the other hand, it's also a kind of "good" to promote the whole platform (no need to install anything else to use "another" Google operator - it's the same with AWS (though there are far less number of of operators) and a bit different in Microsoft as there are really different "domains" there (winrm, mssql, azure).

Actually I like the idea of the poll from @anitakar ! Just a thought... Maybe we can use Airflow Summit to run a number of mini-pols where we could prepare a list of such questions and make people more engaged? That could be one of a number of questions that we could ask. I will raise it to Airflow Summit organizers :)

[anitakar]

+1 to the polls on Airflow Summit.