Explore going the route of using Clients ?

[jgeewax]

After our last talk, there have been quite a few different ideas tossed around to make it clear and obvious which credentials and project IDs are in use during a particular API call, some of those have been....

1. Changing the default values globally
2. Making "Connections" a context manager (with connection: # do something)
3. Creating the concept of a client

We do (1), are talking about doing (2), while the others tend to do (3) -- and comparing the code, I think (3) is the nicest.

gcloud-node:

var gcloud = require('gcloud');
var project1_storage = gcloud.storage({projectId: 'project1', keyFilename: '/path/to/key'});
var project2_storage_auto = gcloud.storage();

# Do things with the two "clients"...
bucket1 = project1_storage.get_bucket('bucket-1')
bucket2 = project2_storage_auto.get_bucket('bucket2')

gcloud-ruby

require 'gcloud/storage'
project1_storage = Gcloud.storage "project-id-1" "/path/to/key"
project2_storage_auto = Gcloud.storage  # Magically figure out the project ID and credentials

# Do things with the two "clients"...
bucket1 = project1_storage.find_bucket "bucket-1"
bucket2 = project2_storage_auto.find_bucket "bucket-2"

gcloud-python

from gcloud import storage
from gcloud.credentials import get_for_service_account_json

# Create two different credentials.
credentials1 = get_for_service_account_json('key1.json')
credentials2 = get_for_service_account_json('key2.json')

# Create two different connections.
connection1 = storage.Connection(credentials=credentials1)
connection2 = storage.Connection(credentials=credentials2)

# Get two different buckets
bucket1 = storage.get_bucket('bucket-1', project='project1', connection=connection1)
bucket2 = storage.get_bucket('bucket-2', project='project2', connection=connection2)

gcloud-python if we followed the client pattern:

from gcloud import storage

project1_storage = storage.Client('project1', '/path/to/key')
project2_storage_auto = storage.Client()

# Do things with the two "clients"...
bucket1 = project1_storage.get_bucket('bucket-1')
bucket2 = project2_storage_auto.get_bucket('bucket-2')

Another option for gcloud-python using the client pattern:

import gcloud

project1_storage = gcloud.storage('project1', '/path/to/key')
project2_storage_auto = gcloud.storage()

# Do things with the two "clients"
bucket1 = project1_storage.get_bucket('bucket-1')
bucket2 = project2_storage_auto.get_bucket('bucket-2')

/cc @dhermes @tseaver

[tseaver]

I honestly don't see the benefit to bundling project together with connection: it only gets passed to a couple of constructors (storage.bucket.Bucket, pubsub.topic.Topic). "Burying" it in a client object doesn't feel like a win for such limited use. The datastore.dataset.Dataset case is different, because a non-default dataset_id has to be passed in lots of places.

If we don't bundle the project, then we could just focus on adopting a common pattern for connection passing / lookup everywhere (#825 and related PRs).

[tseaver]

Note that the work toward #825 makes #2 above (using a connection as a context manager) actually feasible.

[dhermes]

It's probably worth it to create a list of all config that could be a candidate for a global (or membership in a long-lived object like a Dataset).

Right now we have

• Datastore
• Dataset ID
• Connection -> Credentials
• Storage
• Project (only used in 2 of 34 methods, methods only required for "startup")
• Bucket (not used anywhere, but could be)
• Connection -> Credentials
• Pubsub
• Project (used in all API requests in the path, but rarely used directly in code)
• Connection -> Credentials

[tseaver]

While the back-end pubsub API has project in all requests, our wrappers only require passing the project ID when constructing a Topic and when listing topics / subscriptions for a project: the other cases all get their project ID from the relevant Topic instance.

[dhermes]

@tseaver Added this to my list

[tseaver]

Issues about the "always use a Client" notion we discussed in Monday's call:

• It interferes with the "stackable" pattern we use to allow batches / transactions to proxy for their underlying connections.
• In every place we just visited in the Add optional 'connection' parameter for any method that hits the wire #825 PRs, we would probably end up replacing the connection=None argument and the _require_connection() inside the method with client=None and _require_client(), all FBO developers who a) don't want to use the default project ID, and b) don't want to pass it to the Bucket / Topic constructors or the list_buckets / list_topics / list_subscriptions queries.
• This line of development could logically be pushed to a point where we move all the API methods onto the Client, dropping all the intermediate classes in favor of pure imperative code.

[jgeewax]

@tseaver, hate to make more work for you, but is it crazy to toss in short examplar code snippets illustrating the problem ?

Some questions I'm having trouble answering based on our conversations...

1. Is it bad form to have objects know about their "parent" (Blob knowing about it's Bucket, which knows about it's Client; or Entity + Key knowing about their Dataset which knows about its Client) ? If so, why?

2. Is it bad form to make things involving the parent happen on that parent rather than the child? For example...

   # Swap
   topic = Topic('topic_name')
   topic.create()
   # for
   client.create_topic('topic_name')

   and then

   subscription = topic.subscribe('subscription name')
   # or
   subscription = topic.get_subscription('subscription_name')

3. Is it terrible if batches only permit a single client at a time? That is...

   clientA = datastore.Client()  # With automatic credentails...
   clientB = datastore.Client(project_id='other-project')  # With other credentials...
   
   entityA = ... # Get an entity from clientA
   entityB = ... # Get an entity from clientB
   
   entityA.age = 5
   entityB.age = 5
   
   with clientA.batch() as batch:
   batch.put(entityA)
   batch.put(entityB)  # Fails?

4. You mentioned that if we go the client route, the "operate on the module" idea is not a good one. I don't have a huge preference, but I'm curious why it'd be horrible to offer an import-time constructed client if they want to "just use the default everything". For example...

   from gcloud.datastore import default_client as datastore
   key = datastore.Key('Product', 123)
   datastore.get([key])

   (I have no problems with us skipping this, but I wanted to ask....)

---

I get that this is an enormous number of questions, so if it's going to make GitHub explode and make everything unreadable, let's pick one of them for now and go with that...

[tseaver]

Is it bad form to have objects know about their "parent" (Blob knowing about it's Bucket, which knows about it's Client; or Entity + Key knowing about their Dataset which knows about its Client) ? If so, why?

I think there are different cases:

• A Bucket instance is tied to a project (server-side), but not necessarily to a connection: e.g., one might have a connection whose credentials / scopes provided only read access to a given bucket, and another connection whose credentials / scopes had owner access. Also note that the project is only needed when instantiating a Bucket which does not yet exist server-side: existing buckets know their project, but the library user won't have to deal with it.
• A Blob instance is necessarily tied to a bucket due to the server-side API. There is no conceivable access pattern which allows using a blob with a foreign bucket. In addition, nearly all the Blob methods are implemented using self.bucket.
• A Key instance knows its dataset_id, but needs nothing else from a "parent" object. Again, one might be using a key across different connections in order to take advantage of different credentials / scopes. An Entity instance holds its Key, but needs nothing beyond it (no connection, for instance).
• Like a bucket, a Topic instance is tied to a project at creation time. No other methods require / allow the project, however. Also like a bucket, a topic could be used across multiple connections.
• As a blob is tied to its bucket, a Subscription instance is intrinsically tied to its topic, and uses it to implement its methods (mostly to compute its path).

[tseaver]

[Can you] toss in short examplar code snippets illustrating the problem?

["Always use a client"] interferes with the "stackable" pattern we use to allow batches / transactions to proxy for their underlying connections.

At the moment, one can use a batch or a transaction as a context manager an not pass it everywhere, or call methods through it. E.g.:

    from gcloud import datastore
    with datastore.Transaction():
        keys = [entity.key for entity in some_list]
        datastore.delete(keys)

This works because the _require_connection code is willing to look up the current batch / transaction from a thread-local stack, falling back to use the global default. Calling API functions through the notional Client instance will defeat that stacking. Note that a object methods will be also lose that stacking.

In every place we just visited in the #825 PRs, we would probably end up replacing the connection=None argument and the _require_connection() inside the method with client=None and _require_client(), all FBO developers who a) don't want to use the default project ID, and b) don't want to pass it to the Bucket / Topic constructors or the list_buckets / list_topics / list_subscriptions queries.

Per #825, all of the methods / functions that trigger remote API calls now take an optional connection argument (modulo #858 which isn't yet merged). They all fall back to using either the "stack" (for batch/transaction support) or the global default. If we implement "always use a client", then we would have to undo all that work (which cleaned up the code a lot, and go back to using self.connection attributes / properties.

This line of development could logically be pushed to a point where we move all the API methods onto the Client, dropping all the intermediate classes in favor of pure imperative code.

It is a bit of a strawman, but having "always use a client" as the access pattern reduces the desirability of the various client-side objects: they become basically thin wrappers over the client.

[jgeewax]

At the moment, one can use a batch or a transaction as a context manager an not pass it everywhere, or call methods through it. E.g.:

from gcloud import datastore
with datastore.Transaction():
    keys = [entity.key for entity in some_list]
    datastore.delete(keys)

This works because the _require_connection code is willing to look up the current batch / transaction from a thread-local stack, falling back to use the global default. Calling API functions through the notional Client instance will defeat that stacking. Note that a object methods will be also lose that stacking.

I'm proposing that we make people be a bit more explicit for batches and transactions:

from gcloud import datastore
client = datastore.Client() #  This guesses the credentials and project ID magically
# or
client = datastore.Client(project='project-id', key_path='/path/to/key.json')

keys = [ .... ]

with client.Transaction() as t:
  t.delete(keys)

# or

with client.Transaction():
  client.delete(keys)

Wouldn't this make the "require connection" stuff just go away all together? We never guess, or look to the global scope, or anything like that...

[dhermes]

Wasn't the original desire for a client so that people would type less? So we sacrifice usability elsewhere for the sake of usability of a feature which no one has asked for (easy context switching).

[jgeewax]

Wasn't the original desire for a client so that people would type less?

I don't see a huge difference in the amount of typing. What's the delta here?

So we sacrifice usability elsewhere ...

Where are we sacrificing usability? This is where I'm so confused. I see the client pattern as far more usable than the module-level actions and module-level global-only configuration pattern we (and App Engine) have...

[dhermes]

RE: "Where are we sacrificing usability?", I was referring to

I'm proposing that we make people be a bit more explicit for batches and transactions

The client pattern is not "far more usable than the module-level actions", it is exactly identical (given our proposal). We are just swapping out

foo.method(...)

for

bar.method(...)

As for the actions, they are driven by the objects.

[jgeewax]

be a bit more explicit for batches and transactions

Sure, the explicit case here is one extra line however I believe we could make that go away if we wanted to. Tres didn't seem to think that was a good idea.

from gcloud import datastore
client = datastore.Client()
with client.Transaction():
  client.delete(keys)

versus

from gcloud import datastore
with datastore.Transaction():
  datastore.delete(keys)

---

The client pattern is not "far more usable than the module-level actions", it is exactly identical (given our proposal).

I don't think it's identical when you look at dealing with multiple projects or multiple sets of credentials -- is it?

Assuming it's not in those cases, I'm having tons of trouble with the idea of that we should design for usability for the one-and-only-one project and set of credentials. Making the multi-project or multi-credential use cases less usable (IMO).

If we did the client pattern, wouldn't we get both ? Multiple clients with multiple projects with multiple sets of credentials are all really nice, and the one-and-only-one-everything is also really nice... No?

[dhermes]

But we already support multiple connections, it just requires more typing, which we just shuffle from passing arguments to instantiating client objects.

When #864 is in, creating multiple Connections will be possible with a single line (rather than the 3 or 4 it used to take).

[jgeewax]

But we already support multiple connections, it just requires more typing, which we just shuffle from passing arguments to instantiating client objects.

Doesn't it seem weird and repetitive to mix in the configuration details at the time when you're trying to accomplish the action? Shouldn't we split the configure stage apart from the "make an API call" ?

This is my understanding of what we've been aiming for since the beginning, sorry if it's wrong (or long).

The simple case: one-and-only-one everything. Cool.

from gcloud import datastore
key = datastore.Key('Product', '0')
product = datastore.get([key])

The multi-credential, multi-project case. Repetitive.

from gcloud import datastore

# Configure:
connectionA = datastore.Connection(...)
connectionB = datastore.Connection(...)

# Do stuff (and also configure):
keyA = datastore.Key('Product', 'a', dataset_id='dataset-a')
keyB = datastore.Key('Product', 'b', dataset_id='dataset-b')
productA = datastore.get([keyA], connection=connectionA, dataset_id='dataset-a')
productB = datastore.get([keyB], connection=connectionB, dataset_id='dataset-b')
productA.value = 1
datastore.put(productA, connection=connectionA, dataset_id='dataset-a')
datastore.delete([keyA], connection=connectionA, dataset_id='dataset-a')
datastore.delete([keyB], connection=connectionB, dataset_id='dataset-b')

---

Here's the direction I'm trying to push us...

The simple case: one-and-only-one everything (without any magic). Somewhat cool.

from gcloud import datastore
client = datastore.Client()
key = client.Key('Product', '0') # (or datastore.Key(), I'm not sure)
product = client.get(key)

The simple case: one-and-only-one everything (with the magic). Mostly cool.

from gcloud.datastore import default_client as datastore
key = datastore.Key('Product', '0')
product = datastore.get(key)

The simple case: one-and-only-one everything (with different magic). Cool.

from glcoud import datastore  # Module-level methods just proxy to default_client
key = datastore.Key('Product', '0')
product = datastore.get(key)

The multi-credential, multi-project case. Cool.

from gcloud import datastore

# Configure:
clientA = datastore.Client(project='projectA')
clientB = datastore.Client(project='projectB')
datasetA = clientA.Dataset('dataset-a')
datasetB = clientB.Dataset('dataset-b')

# Do stuff:
keyA = datasetA.Key('Product', 'a')
keyB = datasetB.Key('Product', 'b')
productA = datasetA.get(keyA)
productB = datasetB.get(keyB)
productA.value = 1
datasetA.put(productA)
datasetA.delete(keyA)
datasetB.delete(keyB)

[dhermes]

RE:

productA = datastore.get([keyA], connection=connectionA, dataset_id='dataset-a')
productB = datastore.get([keyB], connection=connectionB, dataset_id='dataset-b')
productA.value = 1
datastore.put(productA, connection=connectionA, dataset_id='dataset-a')
datastore.delete([keyA], connection=connectionA, dataset_id='dataset-a')
datastore.delete([keyB], connection=connectionA, dataset_id='dataset-a')

We've been trying to point out that only the connection is the only config object that gets passed on a regular basis. The dataset ID is encoded in the key and is not needed in any of those method calls.

---

So in the above proposal we go from the concept of Noun+Verb (Key+delete) to

client -> dataset -> verb + client -> dataset -> noun

and we consider

datastore.Client(project='projectA').Dataset('dataset-a').delete([keyA])

to be superior to

datastore.delete([keyA], connection=connectionA)

I don't see an improvement.

---

(Also note that the project is as-of-yet un-needed for datastore, but as you've mentioned the landscape is changing, though at an unknown date.)

[jgeewax]

Wait -- datastore.delete(keyA, connection=connectionA) -- where is the dataset ID there?

Sorry missed "The dataset ID is encoded in the key and is not needed in any of those method calls.":

OK, so then datastore.Client(project='project-a').delete(keyA) should work just fine. So the comparison is:

• datastore.Client(project='project-a').delete(keyA)
• datastore.delete(keyA, connection=datastore.Connection(project='project-a'))

And the multiple method calls, we're comparing:

client = datastore.Client(project='project-a')
client.delete(keyA)
if client.get(keyA2).whatever == 42:
  client.delete(keyA2)

to

connection = datastore.Connection(project='project-a')
datastore.delete(keyA, connection=connection)
if datastore.get(keyA2, connection=connection).whatever == 42:
  datastore.delete(keyA2, connection=connection)

[dhermes]

It's in the key.

[jgeewax]

Sorry, updated my note -- misread.

[dhermes]

I still maintain that we are adding more concepts / classes to gain functionality which we already have. The only repetitive use is connection. All other config get encoded in the objects (as eloquently put forth above #861 (comment) by @tseaver).

There is no way to save the repetitive typing of connection, we just exchange it for client (and in the process add a new concept).

[jgeewax]

You're saying the first example with client is using it repetitively? We're not swapping client for connection -- we're swapping client for datastore. I agree I don't think we could get rid of client in the first, or datastore in the second.

I do think we could get rid of connection in the second:

connection = datastore.Connection(project='project-a')
connection.delete(keyA)
if connection.get(keyB).whatever == 42:
  connection.delete(keyB)

... but then s/connection/client/ and you have the first example...

[jgeewax]

I still maintain that we are adding more concepts / classes to gain functionality which we already have.

Functionality -- yes. It is certainly possible to do these things with what we have.

Usability (aka, not repeating ourselves) -- I don't think so. It looks like we're typing connection far too often when we need to be specific about it, right?

[dhermes]

RE: "It looks like we're typing connection far too often " But we exchange typing connection for typing client. There is no gain.

Two simultaneous contexts require typing more to keep them distinct. That is something we can't escape.

I'd prefer we just do nothing until we have users say "it's important for me to have multiple credentials / config active at once". And then we can hear their requirements and design based on real feedback.

[pcostell]

@dhermes I don't like implicit changing behavior. I.e.:

datastore.put(e)  # to default project
set_client(client)
... lots of non datastore code
datastore.put(e)  # now to some other project

Us being able to do the smart thing for users that don't need or want extra configuration is fine.

yes idiomatic is important. But are you saying that passing in the connection everywhere is more idiomatic in python? ISTM that if there isn't a clear more idiomatic way to do things, being consistent is a better policy.

@jgeewax fair point, maybe client is the better name (but then you could still have a Database/Dataset which contained a client that you could interact with directly).

[dhermes]

@pcostell I wasn't trying to connect the idiomatic comment to the above discussion, just to point out that doing something to match the other runtimes isn't a good reason to do it.

[marcja]

Interesting.

My take on this is that simply focusing on the quantity of typing or the relative aesthetic elegance of the code is distracting from more important issues. To me, there are three: 1) consistency between gcloud API projections in different languages; 2) consistency between gcloud-python API and other APIs that users might use; and 3) user expectations about what should be "remembered".

Consistency between gcloud API projects in different languages

Why does it matter? In a polyglot world, it's not unreasonable that developers have to move between languages, from client-side JavaScript to server-side front-end Python to server-side back-end Java (as just one example). If I've used the Datastore API in one language, it should feel familiar when using it in another language on another part of the stack. Further (and perhaps more importantly), when I search the web for answers to my Datastore usage questions, if I find an answer on StackOverflow that is in Ruby, I should be able to immediately adapt that answer to my code in Python. The examples from Ruby and Node that JJ listed at the top of this thread made sense to me and as a user I would be surprised if Python worked differently. If the document on cloud.google.com had a "how to connect to Datastore" inset that had tabs for each language, I would be annoyed if I were flipping back and forth between the various languages and the Python sample deviated from the norm established by the rest.

Consistency between gcloud-python and other APIs that users might use

Why does it matter? Our users do not have access to Datastore outside of GCP (with the exception of the Datastore fake when doing local App Engine development) and they will not have experience working with Datastore until they become GCP customers. But our users will have experience running MySQL clients, SQLlite, CouchDB, MongoDB, and many others. And the bulk of the first-party Python APIs for those products employ the model of a client/connection object with operations hanging off that object. When I read the samples for the Datastore API that did NOT use the client/connection model, I was surprised as a user because I could not relate it to my prior experience with these other APIs. That created a cognitive load because I now needed to "figure out" how this API worked because all I knew upon first reading was that it didn't work like the others I was already familiar with. I have years of experience working with many different databases using different languages on different platforms. I want my APIs to tap into my knowledge, not set it aside.

User expectations about what should be "remembered"

Why does it matter? As I user I rely on the fact that computers should have a better memory than I do. I put things into computers so that I don't have to remember them. I put things into computers because there is more than I could possibly remember. The point of this is that I hate it when I tell I computer something and it doesn't remember it when I expected it to. It's bad user experience. When I read an API like:

from gcloud.datastore import Connection, delete
connection = Connection(...)
delete(key0, connection=connection)
delete(key1, connection=connection)

it feels "forgetful" to me. I've told you my connection. Why do you keep forgetting it? Why do I have to keep reminding you of it? On the other hand, though functionally equivalent, this:

from gcloud import datastore
client = datastore.Client(...)
client.delete(key1)
client.delete(key2)

does not create the same reaction for me. I've told the client about my connection, and without having to remind it anything about my connection, I can simply tell the client what I want it to do. In a sense, I have a delegated the responsibility of remembering those details to something and now I want to give my instructions to that something to "just take care of it". The client idiom meets that expectation, while the parameterized idiom does not.

To be clear, I have read and understood the more technical points about "stackability" and the implementation considerations and I also totally get that the "forgetfulness" argument is only at play when the user is working with multiple connection/datasets. That said, I am more concerned about the user experience of this API (in the context of the universe of gcloud and other APIs it lives in). There will no doubt be users for whom multiple connections/datasets is a requirement. Their user experience should not deviate from the mainstream user experience. Their user experience should not turn into "forgetful" mode just because their business problem got a little more complex.

[dhermes]

@marcja RE: "User expectations about what should be "remembered"" We have made it very convenient for users that don't need to context switch to have the library detect automatically and remember their configuration details.

For a user running in GCE

from gcloud import datastore
key = datastore.Key('Kind', 1234)  # Dataset ID is inferred from GCE environ
datastore.delete(key)  # Connection credentials inferred from GCE service account

For a user running in a custom environment

$ export GCLOUD_DATASET_ID="foo"
$ export GOOGLE_APPLICATION_CREDENTIALS="/path/to/credentials/file.json"

and then

from gcloud import datastore
key = datastore.Key('Kind', 1234)  # Dataset ID is inferred from env. var.
datastore.delete(key)  # Connection credentials inferred from env. var.

---

The snippets

delete(key0, connection=connection)
delete(key1, connection=connection)

were meant for a case where a user has two separate connections, in which case being explicit is important.

[marcja]

@dhermes Yes, exactly as I said in my final paragraph above, I understand that it only applies to the multiple connections case, but in that case it feels "forgetful".

[jgeewax]

Yea - I think we all agree that the module-level case is most and common and nice to have. The client-pattern doesn't preclude us from doing that though... We can totally still provide that style while still making the multiple-connection case nice...

[tmatsuo]

Just a thought.

Are we really sure that people want to use multiple credentials? That means one single application has multiple identities. I think that is maintenance-wise cumbersome.

I would just attach one single identity to an application, and add necessary permissions to that identity. For example, I would add a service account to the second project, instead of having multiple service accounts. I understand there is a strong need for multi-project access though.

Maybe I miss something, what's the use case for multi credentials access?

[jgeewax]

One use case is moving and reconciling data between two apps: get data out of one dataset, move into another dataset. Configuring the project1/project2 stuff with a client is a common pattern. Passing in arguments, to use @marcja 's term, seems "forgetful".

Regardless, of that, the fact that configuring is always done in the global scope is an issue for me. I'll let @Alfus chime in on this one as well.

[dhermes]

@jgeewax RE: "use case is moving and reconciling data between two apps" @tmatsuo point is that you could just add a service account from project 1 as an admin for project 2 and use the same credential.

As for global scope, that has bothered me in some ways too. However, if you don't do it within the package, then the "it just works" with no set-up (that I discussed above) is never possible.

[jgeewax]

As for global scope, that has bothered me in some ways too.

Cool.. so let's fix it... Here's a reason (or excuse) to do that... And the "it just works" stuff can still exist too.

[tmatsuo]

@dhermes
Yes!
Re: copying data. I would temporary add the service account A to the permission list of project B. It is an easier and safer option than temporary pushing another JSON key to the application. NO?

The use cases I can come up with are following 2:

1. Use the service account and 3 legged OAuth in a single app. For example, a storage application, where the management tasks are done with the service account, but when the end users access their files, they use their own OAuth2 credentials.
2. Thread/process safety

For the first case, I might store the users' credentials and re-use them. Anyways, I think I have to change credentials in a single application.

For the second case, I would prefer the library handling thread/process safety for me.

Re: the fact that configuring is always done in the global scope
I also see the problem. If multi-thread/process application change the global configuration, what happens?

[dhermes]

@jgeewax

How can

And the "it just works" stuff can still exist too.

work? We manage this by having the get_default_foo() methods as fallbacks if the user doesn't pass the corresponding config value. They fall back to globals (having get_default_foo() re-compute it's output every time is a bad idea, and using a memoizing decorator on the methods is still using globals).

[marcja]

This is reasonable, but I encourage you all to write the help article first
to guide folks through this use case. While your approach is logical, I
can't say all users would find it obvious. As a user I would immediately
try to do dual credentials, only to discover it doesn't work and then
search the web to find out how to make it work. And then I find your
article that says, "you can't do it with code; you have to do it at least
partly through administration".
On Thu, May 14, 2015 at 9:19 PM Takashi Matsuo notifications@github.com
wrote:

@dhermes https://github.com/dhermes
Yes!
Re: copying data. I would temporary add the service account A to the
permission list of project B. It is an easier and safer option than
temporary pushing another JSON key to the application. NO?

The use cases I can come up with are following 2:

Use the service account and 3 legged OAuth in a single app. For
example, a storage application, where the management tasks are done with
the service account, but when the end users access their files, they use
their own OAuth2 credentials.
2.

Thread/process safety

For the first case, I might store the users' credentials and re-use them.
Anyways, I think I have to change credentials in a single application.

For the second case, I would prefer the library handling thread/process
safety for me.

Re: the fact that configuring is always done in the global scope
I also see the problem. If multi-thread/process application change the
global configuration, what happens?

—
Reply to this email directly or view it on GitHub
#861 (comment)
.

[dhermes]

@tmatsuo Right now the global is not threadsafe but we do pay some attention to thread-safety in batching / transactions, so it wouldn't be hard to move over.

The case you lay out above (user creds. + service account) seems like such a rare case and potentially something we should discourage users from building.

---

@marcja

As a user I would immediately try to do dual credentials

You mean this is something you would immediately experiment with? It's not obvious to me that anyone who wasn't very familiar with Google APIs would think it even possible to have more than one way to authenticate an application.

[marcja]

I think I'm failing to make my point.

When I buy AV receiver, I expect to leverage my prior knowledge and be able
to setup my home theater system without cracking the manual. When I get an
AV receiver that deviates from my experience, that thwarts my expectations,
I have to resort to cracking the manual and suddenly I get very annoyed.
Seriously, did you (the AV receiver product designers) even try to make it
obvious for me (an experienced techie) to use? Or did you think being
fancy/cute/novel/innovative was helpful/valuable to me? I like technology
and have no fear of learning, but sometimes I just want to get my home
theater setup without being given an object lesson in how in the way
current home theater systems are "backward" or "ripe for innovation".

To make the analogy clear, I know that, as an experienced developer, I have
the skills to recover from failure to complete my task at my first attempt.
I know what Google searches to make, I know how to make inferences from
documentation, I know how to keep hacking on it until it works. But
"Seriously, Google", did you even try to make it obvious for me (an
experience developer) to use? You couldn't help me get it to work without
searching Google or mining StackOverflow?

As a senior person at Google, I have heard the customer feedback that a)
our APIs seem like they were written for what's easy to implement
versus what's easy to use and that b) we often don't seem like our left
hand knows what our right hand is doing. I care deeply about how obvious
and consistent our GCP APIs are. Having API issues that are "one search
away" from an answer means that our customers are also "one search away"
from switching to Amazon, Microsoft, or some other competitor.

[jgeewax]

How can

And the "it just works" stuff can still exist too.

work? We manage this by having the get_default_foo() methods as fallbacks if the user doesn't pass the corresponding config value. They fall back to globals (having get_default_foo() re-compute it's output every time is a bad idea, and using a memoizing decorator on the methods is still using globals).

The way I'd suggested before was...

Code I'd write

from gcloud import datastore
datastore.get(datsatore.Key('Product', 123))

or

from gcloud.datastore import default_client as datastore
datastore.get(datastore.Key('Product', 123))

Code under the hood:

# Maybe this is in api.py?
from gcloud import datastore

# Create a global default client that users may or may not use.
default_client = datastore.Client()

# Bring that default client's methods into global scope.
get = default_client.get
delete = default_client.delete
set_project = default_client.set_project
set_credentials = default_client.set_credentials
# ...

Hm, I want to change the settings on the globally shared default connection...

datastore.set_project('...')
datastore.set_credentials('...')
# or
default_client.set_project('...')
default_client.set_credentials('...')

I want to be explicit now about using the default client...

from gcloud import datastore
client = datastore.default_client
other_client = datastore.Client(credentials=other_credentials)

---

We manage this by having the get_default_foo() methods as fallbacks if the user doesn't pass the corresponding config value.

Not sure I understand what get_default_foo() is here, but in the example I'm saying that the methods themselves never "try to figure out the client". The client is either explicitly created by the user, or the "special one" that we explicitly created for the user. No more DefaultsContainer or _implicit_environ stuff... All code everywhere is always using a client -- we just happen to have created one with no arguments for you as a convenience.

having get_default_foo() re-compute it's output every time is a bad idea

Again, not 100% certain on what you mean by that method, but I'm going to take a guess. If you mean that we shouldn't re-compute the magic values every time (and we also shouldn't compute them for the first time until someone needs them), I agree. So a Client shouldn't try to figure out what the magic things are until methods are called on it. That is:

# datastore client

class Client(object):
  def __init__(self, credentials=None, ...):
    self._credentials = credentials

  def get(self, key, ...):
    self.connection._rpc('...')

  @property
  def connection(self):
    if self._connection is None:
      self._connection = Connection(credentials=self.credentials, ...)
    return self._connection

  @property
  def credentials(self):
    if self._credentials is None:
      self._credentials = discovery_module.magically_figure_out_my_creds()
    return self._credentials

So instantiating a Client() doesn't try to figure anything out. Once I call a method that needs those properties, we trigger the "figure out everything please, using magic if necessary, throwing errors if necessary". The memo-ized state is at the object level (it's all inside the Client object) and the client is globally available.

[jgeewax]

So... looks like this conversation has gone silent... but there's some talk on PR #886.. Here's the summary of what I see so far

Pattern	Voters	Total
Client	@jgeewax, @marcja, @Alfus, @pcostell	4
Global config	@tseaver, @dhermes	2
Not voted yet?	@tmatsuo	1

Given that scenario, I really want to push that unless someone shows why going the client route is a horrible idea, we should get started on moving that direction (being a lot of work doesn't count).

[tseaver]

Seems to me that voting isn't quite what we need, especially without a more concrete thing to be considering. Rather than talking further past one another, can we work on exploring the idea more fully by writing up the "developer experience" documents which show all the ways clients will be used, similar to the one I wrote for pubsub?

[jgeewax]

This whole thread has been about "what code a developer has to write" with many examples (I think at least), so I'm not sure what else we should explore. @dhermes mentioned that we're not stating any new facts, just debating "which is better".

Given that, we've had several people chime in with feelings that the globally-configured way of doing things "seems forgetful" -- the majority preferring the client pattern. I've not heard anyone say "if we go the client pattern way, we can't still make the exact same code we have today continue to work just fine". So it seems to me, that if we wanted to offer both, we could if we went the client route, and we can't if we go the globally configured route.

Overall, I'm having trouble seeing what the argument is against going the route of client pattern. Can anyone chime in and explain why it's a bad idea?

I think the feedback has been "for multi-client multi-credential multi-project code, we prefer the client pattern", so I'm looking for other reasons of why it's a bad choice.

[dhermes]

1. (Most importantly) I have lost the energy to fight on this, but have not changed my opinion. I'm ready to implement clients.
2. The reason not to write clients is because it is more code to maintain and "more ways to do the same thing" to confuse users.
3. Keeping "the exact same code we have today continue to work just fine" doesn't address @marcja concern that we "thwarts my expectations" or @pcostell concerns about implicit globals.
4. The "seems forgetful" concerns were about multi-tenant code. We've still yet to hear from any users that need multiple credentials at once and @tmatsuo pointed out that we can just recommend that users add a service account to a different project. (IMO Google as well as library devs should recommend this.)

[tseaver]

Like Danny, I've run out of steam to argue.

The reason that I suggested starting a DX document is that I think it will either make the concerns I have obvious, or else make it clear that they can be addressed cleanly.

At the moment, the only way I can see to cleanly implement clients without re-welding them in to every class we just cleaned up, is to have the clients work at a layer "above" the classes we have now, with proxies for the "real" objects that pass in the connection from the client to every call that takes one. Maybe we can come up with a cleaner implementation than that, if we have a specific outline of how the client objects get used.