Skip to content

feat: Support GCP PubSub for publishmq & internalmq #327

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Apr 12, 2025
Merged

feat: Support GCP PubSub for publishmq & internalmq #327

merged 9 commits into from
Apr 12, 2025

Conversation

alexluong
Copy link
Collaborator

resolves #138 #325

This PR adds support for Google Cloud Pub/Sub as a message queue provider for both internal MQs and publish MQ.

Config

Config for internalmq along with the default values:

GCP_PUBSUB_PROJECT=""
GCP_PUBSUB_SERVICE_ACCOUNT_CREDENTIALS=""
GCP_PUBSUB_DELIVERY_TOPIC="outpost-delivery"
GCP_PUBSUB_DELIVERY_SUBSCRIPTION="outpost-delivery-sub"
GCP_PUBSUB_LOG_TOPIC="outpost-log"
GCP_PUBSUB_LOG_SUBSCRIPTION="outpost-log-sub"

The only required config is GCP_PUBSUB_PROJECT.

For publishmq, there's no default, so if you want to use it, you must set all config:

PUBLISH_GCP_PUBSUB_PROJECT=""
PUBLISH_GCP_PUBSUB_SERVICE_ACCOUNT_CREDENTIALS=""
PUBLISH_GCP_PUBSUB_TOPIC=""
PUBLISH_GCP_PUBSUB_SUBSCRIPTION=""

Local emulator

This PR added Google Cloud emulator to the dev Docker Compose (outpost-deps). To test locally, make sure you set the env PUBSUB_EMULATOR_HOST="gcp:8085".

Cloud env

There are 2 ways to authenticate with GCP Cloud. You can either rely on the automatic credentials mechanism, using GOOGLE_APPLICATION_CREDENTIALS env pointing to the credentials file, or use some sort of IAM / Workload Identity in production. The last resort is to use the "service account credentials" variable (GCP_PUBSUB_SERVICE_ACCOUNT_CREDENTIALS for internalmq or PUBLISH_GCP_PUBSUB_SERVICE_ACCOUNT_CREDENTIALS for publishmq). This env expects the escaped JSON string which you can get by doing something like jq tostring ./path/to/keys.json.

Publishmq local helper

We have cmd/publish server which is a helper dev server so you can test the publishmq locally easily. Once the server is running go run ./cmd/publish on port 5555, you can use curl or Postman to interact with it like so:

# provision the topic + subscription
curl --location --request POST 'localhost:5555/declare?method=gcp_pubsub'

# send an event to publishmq
curl --location 'localhost:5555/publish?method=gcp_pubsub' \
--header 'Content-Type: application/json' \
--data '{
    "tenant_id": "12345",
    "topic": "user.created",
    "eligible_for_retry": true,
    "metadata": {
        "meta": "data"
    },
    "data": {
        "user_id": "userid"
    }
}'

PublishMQ

For publishmq, the user MUST create and manage the topic & subscription themselves. All we do is subscribing to it and processing the incoming messages.

NOTE: for publishmq with GCP, technically we only care about the Subscription. In theory, we don't care about the Topic at all, but because of the way the mq logic is implemented, we do require topic to be correct. In the future, we can consider refactor the mq (as mentioned in another convo) so that we don't need to care about the publish topic anymore.

Copy link

vercel bot commented Apr 11, 2025

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
outpost-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Apr 11, 2025 3:42pm
outpost-website ✅ Ready (Inspect) Visit Preview 💬 Add feedback Apr 11, 2025 3:42pm

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: Shouldn't this file be called gcp_pubsub.go?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alexbouchardd good call, I updated that along with a few other files to specify AWS SQS and GCP PubSub instead of just AWS / GCP.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great!

@leggetter
Copy link
Collaborator

@alexluong - tested as internal MQ and as publish queue via GCP. Good to merge 👍

@leggetter leggetter moved this from Backlog to In Progress in Outpost Releases Apr 11, 2025
@alexluong alexluong merged commit 3093d9f into main Apr 12, 2025
3 checks passed
@alexluong alexluong deleted the gcp branch April 12, 2025 09:46
@leggetter leggetter moved this from In Progress to Done in Outpost Releases Apr 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

InternalMQ (Delivery & Log) - GCP Pub/Sub
3 participants