docs: add supabase guide #297

nicktrn · 2023-08-09T18:40:46Z

Adds a Supabase guide to the self-hosting section.

WIP for now. Will be adding connection pooling via PgBouncer to the guide.

This has first class support in both Supabase and Prisma, and should benefit not just self-hosters but the project in general. Can be easily self-hosted, and reduces the need for Data Proxy.

There is also a trimmed down compose file in the guide that may benefit from being included in triggerdotdev/docker

Fixes #279
/claim #279

changeset-bot · 2023-08-09T18:40:50Z

⚠️ No Changeset found

Latest commit: a1c8e25

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

ericallam · 2023-08-09T21:18:08Z

docs/documentation/guides/self-hosting/supabase.mdx

+## Requirements
+
+- [docker](https://docs.docker.com/engine/install/)
+- time


ericallam · 2023-08-09T21:21:28Z

This looks amazing! Great work 👍🏼. One requested tweak: could you specify a separate schema other than public for the Trigger.dev tables. Maybe just triggerdotdev?

nicktrn · 2023-08-09T21:40:43Z

Sure thing. Will change that when I get a chance to finish up. Just FYI, changing the graphile-worker schema is also possible, should the need ever arise.

ericallam · 2023-08-10T09:33:18Z

@nicktrn would it be possible to get this merged in before 3pm GMT today? We're announcing something at 4pm today in a blog post and would love to be able to link to it 👍

nicktrn · 2023-08-10T10:33:30Z

@ericallam Not a problem, ☕ solves (almost) anything

nicktrn · 2023-08-10T11:18:53Z

@ericallam I have added support for connection pools. Tests are passing. Will finish the writeup now. If you could have a look at this immediately, that would save time.

Worst case, will have to publish without the bonus section on connection pools.

ericallam · 2023-08-10T11:56:11Z

Oh shoot I didn't realize this had changes to the webapp to get it to work. Unfortunately, we probably won't be able to get it in under the deadline as I don't have time to pull it down and test. So I think I will link to this PR from the blog post instead. Sorry again and thanks for this! Will hopefully be able to get it merged tomorrow 👍

nicktrn · 2023-08-10T11:58:26Z

No worries, would you like me to take out the last section for now and revert the webapp changes?

That way you can publish the main part of the guide without any trouble and link to it directly.

nicktrn · 2023-08-10T13:03:35Z

@ericallam linked PR without app changes. Hope this helps!

docs: supabase guide without pooling section #299

nicktrn · 2023-08-10T17:23:44Z

Just finished this up. Quick summary of changes:

add DIRECT_URL env var and directUrl datasource field
symlink database .env to root .env
ensure graphile-worker is happy with pooling enabled

Small caveat: Prisma requires an update for Studio to play nicely with pooling. But I'm sure you'll be upgrading soon (hint hint 😉 )

edit:

DON'T review this yet!

Just ran the e2e tests again and getting timeouts with pooling enabled. Hope it's just a latency issue as the DB isn't local. Will ping you when it's ready.

nicktrn · 2023-08-11T08:12:44Z

@ericallam this is ready for review now.

End-result is that nothing changes unless DIRECT_URL is manually set.

The tests are passing fine manually with and without pooling. Managed to get it working via playwright, but quite flaky.

There are issues with the tests in general. Boils down to responses not being awaited before navigating, timeouts not considering latency (dev/prod, local/remote db), and setup/teardown being a mess. Ideally this should be done with fixtures or dependencies instead. Will have a proper look when I get a chance and sort things out.

Re the .env symlink, can avoid that for now if you'd like to carry on with the previous pattern. Just need to add a DIRECT_URL=${DATABASE_URL} line to set a default, just as I've done in .env.example

nicktrn · 2023-08-11T08:16:12Z

.env.example

+DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres?schema=public
+# This sets the URL used for direct connections to the database and should only be needed in limited circumstances
+# See: https://www.prisma.io/docs/reference/api-reference/prisma-schema-reference#fields:~:text=the%20shadow%20database.-,directUrl,-No
+DIRECT_URL=${DATABASE_URL}


This can't just be empty as we have to use an env var in schema.prisma

nicktrn · 2023-08-11T08:16:48Z

.github/workflows/publish.yml

@@ -122,7 +122,6 @@ jobs:
          # Setup environment variables
          cp ./.env.example ./.env
          cp ./examples/nextjs-test/.env.example ./examples/nextjs-test/.env.local
-          cp ./packages/database/.env.example ./packages/database/.env


not needed anymore due to symlink

nicktrn · 2023-08-11T08:21:12Z

CONTRIBUTING.md

-   cp .env.example .env && cp packages/database/.env.example packages/database/.env
+   cp .env.example .env


not needed anymore due to symlink

nicktrn · 2023-08-11T08:21:18Z

CONTRIBUTING.md

@@ -178,7 +178,6 @@ To run the end-to-end tests, follow the steps below:
 ```sh
 cp ./.env.example ./.env
 cp ./examples/nextjs-test/.env.example ./examples/nextjs-test/.env.local
-cp ./packages/database/.env.example ./packages/database/.env


not needed anymore due to symlink

nicktrn · 2023-08-11T08:22:00Z

apps/webapp/app/services/worker.server.ts

@@ -94,6 +94,7 @@ function getWorkerQueue() {
      connectionString: env.DATABASE_URL,
      concurrency: 5,
      pollInterval: 1000,
+      noPreparedStatements: env.DATABASE_URL !== env.DIRECT_URL,


No way around this with pooling

ericallam · 2023-08-11T09:07:44Z

I'm happy with the .env symlink thing (which I've seen done elsewhere as well). Can you help me understand the need for the DIRECT_URL env var? Is there any way to make it an optional env var or do we have to require it?

nicktrn · 2023-08-11T09:26:43Z

Sure. The Prisma CLI requires a direct connection for some commands, i.e. not via an external pool. Unless you hardcode this in the schema file, the only way is to use an env var.

Sadly, once we do this, that env var can't be unset / empty as Prisma doesn't allow us to set defaults and will complain otherwise:

# unset
error: Environment variable not found: DIRECT_URL.
  -->  schema.prisma:4

# empty
error: Error validating datasource `db`: You must provide a nonempty direct URL. The environment variable `DIRECT_URL` resolved to an empty string.
  -->  schema.prisma:4

ericallam · 2023-08-11T09:42:59Z

That makes sense. Let me know when this is ready for review 👍

nicktrn · 2023-08-11T09:52:56Z

Should be ready now!

ericallam · 2023-08-11T10:31:52Z

Okay, I just pulled this down and took a look at this and it all looks great. There is one issue though, our self-hosting guides will need to be updated:

And our docker repo:

https://github.com/triggerdotdev/docker

I'll update the docker repo but do you think you could update the self-hosting guides?

nicktrn · 2023-08-11T10:33:46Z

Yes, I did think that as well. I'll have a look and see what needs changing!

nicktrn · 2023-08-11T14:10:16Z

Fixed the Fly.io guide, got an error following the instructions however. Creating the db failed as the default instance couldn't run Postgres:

Select configuration: Development - Single node, 1x shared CPU, 256MB RAM, 1GB disk
Failed creating the Postgres cluster triggerdotdev-db: failed to launch VM: aborted: could not reserve resource for machine: insufficient memory available to fulfill request

The next tier up was shared-cpu-2x - CPU Kind: Shared, vCPUs: 2, Memory: 4GB which worked. Why the jump in specs I wonder..

The Render guide is a different matter. Don't know what to do there. It doesn't seem to have a DB attached and I couldn't set up an instance following those steps. My guess is it could all be configured in render.yml directly, including the DB and pooling (!). See here.

Made a quick change to the docker repo as well and small README change to fly.io to follow shortly.

ericallam · 2023-08-11T14:51:50Z

This looks good, I'm going to merge this and the other PRs now and create a new latest release of the docker image.

nicktrn · 2023-08-11T15:17:01Z

Excellent, let me know if you run into any issues due to my changes.

nicktrn · 2023-08-15T08:57:47Z

Tracking this here in case Postgres Extensions are ever added to the Prisma schema:

Drift detected: Your database schema is not in sync with your migration history - every migration on Supabase prisma/prisma#19100

algora-pbc bot mentioned this pull request Aug 9, 2023

[TRI-949] Create a self-hosting guide powered by a Supabase database #279

Closed

ericallam reviewed Aug 9, 2023

View reviewed changes

ericallam marked this pull request as ready for review August 10, 2023 11:56

nicktrn added 7 commits August 10, 2023 13:39

docs: add supabase guide

b1005ac

feat: add support for external connection poolers

dc0fda8

chore: use .env symlink in database package

93fca00

chore: remove database .env.example

8e78631

docs: finish supabase pooling section

0b69825

docs: fix small typo

5881514

docs: replace prisma with app wording

af6e575

nicktrn force-pushed the docs-supabase-guide branch from 1b8edff to af6e575 Compare August 10, 2023 17:13

docs: fix up supabase pooling section

0437d66

nicktrn added 2 commits August 11, 2023 08:20

fix: deleted too much in contributing docs

62599cb

docs: change pooling heading

0a10946

nicktrn commented Aug 11, 2023

View reviewed changes

nicktrn requested a review from ericallam August 11, 2023 09:53

nicktrn added 2 commits August 11, 2023 13:56

docs: reduce linux shaming

4cbea05

docs: include direct connection url in fly.io

a1c8e25

ericallam merged commit 603ffbc into triggerdotdev:main Aug 11, 2023

nicktrn deleted the docs-supabase-guide branch November 9, 2023 13:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: add supabase guide #297

docs: add supabase guide #297

nicktrn commented Aug 9, 2023

changeset-bot bot commented Aug 9, 2023 •

edited

Loading

ericallam Aug 9, 2023

ericallam commented Aug 9, 2023

nicktrn commented Aug 9, 2023

ericallam commented Aug 10, 2023

nicktrn commented Aug 10, 2023

nicktrn commented Aug 10, 2023

ericallam commented Aug 10, 2023

nicktrn commented Aug 10, 2023

nicktrn commented Aug 10, 2023

nicktrn commented Aug 10, 2023 •

edited

Loading

nicktrn commented Aug 11, 2023

nicktrn Aug 11, 2023

nicktrn Aug 11, 2023

nicktrn Aug 11, 2023

nicktrn Aug 11, 2023

nicktrn Aug 11, 2023

ericallam commented Aug 11, 2023

nicktrn commented Aug 11, 2023

ericallam commented Aug 11, 2023

nicktrn commented Aug 11, 2023

ericallam commented Aug 11, 2023 •

edited

Loading

nicktrn commented Aug 11, 2023

nicktrn commented Aug 11, 2023 •

edited

Loading

ericallam commented Aug 11, 2023

nicktrn commented Aug 11, 2023

nicktrn commented Aug 15, 2023

		cp .env.example .env && cp packages/database/.env.example packages/database/.env
		cp .env.example .env

docs: add supabase guide #297

docs: add supabase guide #297

Conversation

nicktrn commented Aug 9, 2023

changeset-bot bot commented Aug 9, 2023 • edited Loading

⚠️ No Changeset found

ericallam Aug 9, 2023

Choose a reason for hiding this comment

ericallam commented Aug 9, 2023

nicktrn commented Aug 9, 2023

ericallam commented Aug 10, 2023

nicktrn commented Aug 10, 2023

nicktrn commented Aug 10, 2023

ericallam commented Aug 10, 2023

nicktrn commented Aug 10, 2023

nicktrn commented Aug 10, 2023

nicktrn commented Aug 10, 2023 • edited Loading

nicktrn commented Aug 11, 2023

nicktrn Aug 11, 2023

Choose a reason for hiding this comment

nicktrn Aug 11, 2023

Choose a reason for hiding this comment

nicktrn Aug 11, 2023

Choose a reason for hiding this comment

nicktrn Aug 11, 2023

Choose a reason for hiding this comment

nicktrn Aug 11, 2023

Choose a reason for hiding this comment

ericallam commented Aug 11, 2023

nicktrn commented Aug 11, 2023

ericallam commented Aug 11, 2023

nicktrn commented Aug 11, 2023

ericallam commented Aug 11, 2023 • edited Loading

nicktrn commented Aug 11, 2023

nicktrn commented Aug 11, 2023 • edited Loading

ericallam commented Aug 11, 2023

nicktrn commented Aug 11, 2023

nicktrn commented Aug 15, 2023

changeset-bot bot commented Aug 9, 2023 •

edited

Loading

nicktrn commented Aug 10, 2023 •

edited

Loading

ericallam commented Aug 11, 2023 •

edited

Loading

nicktrn commented Aug 11, 2023 •

edited

Loading