-
Notifications
You must be signed in to change notification settings - Fork 13
feat: Support GCP PubSub for publishmq & internalmq #327
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
internal/mqinfra/gcp.go
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nitpick: Shouldn't this file be called gcp_pubsub.go
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@alexbouchardd good call, I updated that along with a few other files to specify AWS SQS and GCP PubSub instead of just AWS / GCP.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great!
@alexluong - tested as internal MQ and as publish queue via GCP. Good to merge 👍 |
resolves #138 #325
This PR adds support for Google Cloud Pub/Sub as a message queue provider for both internal MQs and publish MQ.
Config
Config for internalmq along with the default values:
The only required config is
GCP_PUBSUB_PROJECT
.For publishmq, there's no default, so if you want to use it, you must set all config:
Local emulator
This PR added Google Cloud emulator to the dev Docker Compose (outpost-deps). To test locally, make sure you set the env
PUBSUB_EMULATOR_HOST="gcp:8085"
.Cloud env
There are 2 ways to authenticate with GCP Cloud. You can either rely on the automatic credentials mechanism, using
GOOGLE_APPLICATION_CREDENTIALS
env pointing to the credentials file, or use some sort of IAM / Workload Identity in production. The last resort is to use the "service account credentials" variable (GCP_PUBSUB_SERVICE_ACCOUNT_CREDENTIALS
for internalmq orPUBLISH_GCP_PUBSUB_SERVICE_ACCOUNT_CREDENTIALS
for publishmq). This env expects the escaped JSON string which you can get by doing something likejq tostring ./path/to/keys.json
.Publishmq local helper
We have
cmd/publish
server which is a helper dev server so you can test the publishmq locally easily. Once the server is runninggo run ./cmd/publish
on port 5555, you can use curl or Postman to interact with it like so:PublishMQ
For publishmq, the user MUST create and manage the topic & subscription themselves. All we do is subscribing to it and processing the incoming messages.
NOTE: for publishmq with GCP, technically we only care about the Subscription. In theory, we don't care about the Topic at all, but because of the way the mq logic is implemented, we do require topic to be correct. In the future, we can consider refactor the mq (as mentioned in another convo) so that we don't need to care about the publish topic anymore.