PubSub ValueError: Project from client (A) should agree with project from resource(B).

[tarzanjw]

Say that I have 2 projects, A and B.

From project A, I have topics: projects/A/topics/events.
From project B, I add a subscription to above topic, subscription name is: projects/B/subscriptions/on-A-events.

Note: I created the subscription via Graph API Explorer, not via the Cloud Console (just can't do it with Cloud Console)

So, these code will raise an Exception:

  from gcloud import pubsub

  c = pubsub.Client(project='A')
  t = c.topic('events')
  subs = t.list_subscriptions()  # Error

  Traceback (most recent call last):
    File "<input>", line 1, in <module>
    File "<input>", line 1, in <listcomp>
    File ".../site-packages/gcloud-0.14.0-py3.5.egg/gcloud/pubsub/topic.py", line 311, in list_subscriptions
      sub_name = subscription_name_from_path(sub_path, self.project)
    File "../site-packages/gcloud-0.14.0-py3.5.egg/gcloud/pubsub/_helpers.py", line 73, in subscription_name_from_path
      return _name_from_project_path(path, project, _SUBSCRIPTION_TEMPLATE)
    File ".../site-packages/gcloud-0.14.0-py3.5.egg/gcloud/_helpers.py", line 464, in _name_from_project_path
      'project from resource(%s).' % (project, found_project))
  ValueError: Project from client (A) should agree with project from resource(B).

I take a dive into the source codes, and I see that you guys are assuming that every subscriptions that returned have to be in same project with the client, but it's not in my case.

Is it a bug or my misunderstanding?

[dhermes]

@tmatsuo Can you take a peek (sanity check our project "verification")?

[tarzanjw]

On the other hand, this error happens again

  client = pubsub.Client(project='B')
  client.list_subscriptions()  # same error

[dhermes]

What is the stack trace in that case?

[tarzanjw]

I ran it in python console, I also hide some information about path.

Do you need something more?

[dhermes]

I'm curious what's actually triggering the error, so providing a stack trace like you did above is helpful. (I need to reproduce the bug.)

[tarzanjw]

This is the full log

Python log

Python 3.5.1 (default, Jan 22 2016, 08:54:32) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
>>> from gcloud import pubsub
>>> client = pubsub.Client(project='vnpapis')
>>> topic = client.topic('binlog2-vatgia-orders-insert')
>>> topic.list_subscriptions()
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/Users/tarzan/sources/vnpapis/vnpapi-etl/lib/python3.5/site-packages/gcloud-0.14.0-py3.5.egg/gcloud/pubsub/topic.py", line 311, in list_subscriptions
    sub_name = subscription_name_from_path(sub_path, self.project)
  File "/Users/tarzan/sources/vnpapis/vnpapi-etl/lib/python3.5/site-packages/gcloud-0.14.0-py3.5.egg/gcloud/pubsub/_helpers.py", line 73, in subscription_name_from_path
    return _name_from_project_path(path, project, _SUBSCRIPTION_TEMPLATE)
  File "/Users/tarzan/sources/vnpapis/vnpapi-etl/lib/python3.5/site-packages/gcloud-0.14.0-py3.5.egg/gcloud/_helpers.py", line 464, in _name_from_project_path
    'project from resource(%s).' % (project, found_project))
ValueError: Project from client (vnpapis) should agree with project from resource(vatgia-voicecall).

Graph API log

GET https://pubsub.googleapis.com/v1/projects/vatgia-voicecall/subscriptions?key={YOUR_API_KEY}

{
 "subscriptions": [
  {
   "name": "projects/vatgia-voicecall/subscriptions/binlog2-vatgia-voicecall-on-order-insert",
   "topic": "projects/vnpapis/topics/binlog2-vatgia-orders-insert",
   "pushConfig": {
    "pushEndpoint": "https://vatgia-voicecall.appspot.com/on_order_change"
   },
   "ackDeadlineSeconds": 10
  }
 ]
}

[tseaver]

FWIW, the reason we are verifying the project is because the subscription uses the client (which binds a single project) to perform API requests. If we want to support this usecase, we will have to detect the mismatch and somehow fabricate a new client on the fly for the "foreign" subscription.

[daspecster]

@tseaver @dhermes
GET https://pubsub.googleapis.com/v1/{project=projects/*}/subscriptions
https://cloud.google.com/pubsub/reference/rest/v1/projects.subscriptions/list

Would it make sense to allow the project to be passed to list_subscriptions(project='A')?

or maybe more intuitively, could topic.list_subscriptions() just return the subscriptions but have Subscription._project be the actual project ID instead of self._client.project? Would require some refactoring though.

[tseaver]

As I said, Subscription instances don't make API calls directly: they currently use their client instance bound to their topic. In the OP's case, that client instance has a different project ID than the subscription.

To make the "foreign subscription" case work, we would need to do the following:

• Have Subscription instances hold direct references to their client.
• When listing subscriptions for a topic (as opposed to for the client/project), be able to construct new Client instance(s) for those from foreign projects.

[daspecster]

@tseaver, couldn't the Subscriptions() track which project they're associated with? Maybe by having both the Subscription.project and Subscription._client.project. That could require more factoring of pubsub than I'm thinking though.

I think I see what you mean though, thanks for the clarification!