AppSheet + Supabase: Since new Pooler, the connection is basically impossible to work with

[SkrOYC]

Bug report

Describe the bug

When using port "5432" things work with the obvious limits. AppSheet (a no-code platform owned by Google) opens a connection for each CRUD operation and this was solved by using the pooler on port "6543".
Everything worked perfect untill PgBouncer was left aside.
It justs seems using the pooler's port is not a transparent solution from the client's side.
Since this is a platform not managed by me I don't know the ins and outs of it, but I can say this was working perfectly before with PgBouncer.

To Reproduce

Use AppSheet and connect to the database using the Postgres connector with the 5432 port. Everything works perfect.
Change it to 6543 and you will face timeout issues. I can't add tables to my apps nor sync data on others apps already working with 5432.

Expected behavior

That the connection to the database under 6543 should work the same way 5432 does, being transparent on the client's side.

Additional info

This is a post in Appsheet's official community I made to promote Suabase and where users have been reporting the issue, which I was able to confirm
https://www.googlecloudcommunity.com/gc/Tips-Tricks/Updated-Supabase-mets-AppSheet/m-p/442135/highlight/true#M6685

[encima]

Thanks for opening!
To confirm, you have 3 options to connect with AppSheet:

1. Direct - 5432 - This is a direct connection to the database
2. Pooler - Transaction - 6543 - This is a connection to your database through the Pooler where the connection is opened only for the lifetime of a transaction
3. Pooler - Session - 5432 - This is a connection to your database through the Pooler where the connection is open until the client disconnects (so you are likely to hit connection limits here)

AppSheet may recommend connection types but I would typically go through the Pooler's transaction (and modify timeouts if needed) rather than going through session as the client may keep connections open longer than needed so they are not freed for others.

Read more here

Let us know if that answers your question!

[SkrOYC]

That's kinda what I already knew about it.
The issue in this case is that I need transaction mode and it just doesn't work at all

[encima]

"Doesn't work at all" - Do you have more info on this? What are the logs on your instance reporting for the timeouts? What is the instance size and pool size you have configured?

Your Expected behaviour section expects Transaction mode to work the same as a direct connection but this is not the case. Session mode works the same as a direct connection.

[SkrOYC]

What I mean to what I expect is that from the client side we should see things working since Transaction mode has benefits on top of what Session mode can offer. I can connect using Session mode but if I change to port 6543 to not have issues with open connections (since the platform I'm using doesn't close them) it doesn't do anything other than authenticating. I can't sync data.
I could do some troubleshooting if you tell me where I should find the logs for the sync attempts

[encima]

Noted, thanks for the extra info. I have a gut feeling that the outcome of this will be: "AppSheet does not support transaction mode when connecting to Postgres"

But, I am not sure so let's confirm this: you can check your Pooler logs in the dashboard under Logs by clicking Pooler.

[SkrOYC]

About the your first comment, my only knowledge about the way AppSheet works under the hood is that it was working wonderfully before with PgBouncer. I expect it to work as good as before with Supavisor but for some reason it doesn't.
I though Supavisor was supposed to be a drop in replacement for PgBouncer.
If there is a difference in the way Supavisor exposed the database in comparison to PgBouncer, that should be the issue.

I'll do some troubleshooting and post the logs here.

Thanks for your help!

[encima]

Hey @SkrOYC

That is a good point and I am not sure if the switch was called out as pgbouncer was set to transaction mode by default. When you say AppSheet worked with pgBouncer was that after changing to session mode as well?

[encima]

Closing due to inactivity

[SkrOYC]

Finding time to keep troubleshooting. This is still an issue.
In the meantime, I think we need to confirm from the people that was tied to the supavisor implementation if there is any difference in the way it behaves compared to pgbouncer that may make this not a 1:1 replacement to pgbouncer

[SkrOYC]

Anyone has any idea about why the new Supavisor integration was not a 1:1 replacement to pgBouncer?

[SkrOYC]

Btw @encima

When you say AppSheet worked with pgBouncer was that after changing to session mode as well?

AppSheet worked perfectly with pgBouncer in Transaction mode, CRUD operations were running quickly

[SkrOYC]

"ClientHandler: socket closed with reason {:shutdown, :socket_closed}"

Same message again and again in the logs

[SkrOYC]

I just found this message in the dashboard

Is it possible that the change to Supavisor is half done or it's expecting pgBouncer and the infrastructure is not there anymore?
@encima

[SkrOYC]

Also found some of these under the "Postgres" logs:

[encima]

@SkrOYC You likely need to upgrade your instance.
pgbouncer is not enabled on (almost) all instances but the user still exists in the DB.
"pending removal" means that the disabled pgbouncer will be removed completely from future instances but is not currently
To confirm, the switch to supavisor is complete but the phasing out of pgbouncer is not.
Supavisor is not a 1:1 replacement (both are open source so you are welcome to compare) but you can expect similar behaviours, if not the same. You can add "pgbouncer=true" to your Supavisor connection string to make the behaviour more similar which will disable prepared statements.

[SkrOYC]

You likely need to upgrade your instance

Any clue on how to deal with it?

Also, we are connecting through the AppSheet connector for it, which takes the host:port, database name, user, password and nothing else

[encima]

@SkrOYC you can see upgrades in the "Infrastructure" section in your project settings.

Ah, in that case you can use Supavisor transaction mode, session mode or a direct connection to the DB

[SkrOYC]

in that case you can use Supavisor transaction mode, session mode or a direct connection to the DB

That's what we are trying to do, and worked perfectly before under pgBouncer and now it doesn't

you can see upgrades in the "Infrastructure" section in your project settings.

I can't see any upgrade options btw @encima

[encima]

I can't see any upgrade options btw @encima
That should mean you are all good!

I see what you mean, now, sorry, I think I was confusing this with a different case.

Direct connections do work but I can reproduce your issue when using Supavisor and enforcing SSL

So, I can connect when using Supavisor, disabling SSL enforcement and not requiring SSL in AppSheet.

I will check with the pooler team if they can investigate more!

[SkrOYC]

I can reproduce your issue when using Supavisor and enforcing SSL

SSL doesn't work but that's not a big issue for the time being, it's an AppSheet thing.

I can connect when using Supavisor, disabling SSL enforcement and not requiring SSL in AppSheet.

So can I, but after that I cannot add any table to any app nor sync anything on apps that were already working

[SkrOYC]

I tried with a fresh database, so no "pgBouncer pending removal" issue. Transaction mode, new app inside AppSheet instead of an already working one and no luck. It doesn't work.

These are the logs when I tried to connect

[encima]

Transferring to the Supavisor repo so the @supabase/pooler team are aware

[SkrOYC]

Hi everyone. I have a client wanting to use Supabase but still laking a solution on this

[encima]

Checking in with the Pooler team, @SkrOYC, thank you for the ping!

[Khoanguyen0109]

could i know how long will it take to help resolve this issue?

[encima]

Difficult to estimate, we will update this issue with any updates so just be sure to subscribe for notifications!

[NguyenHoadev]

Difficult to estimate, we will update this issue with any updates so just be sure to subscribe for notifications!

Have you fixed this bug yet?

[NguyenHoadev]

??????????????????????????????

[SkrOYC]

Hi @encima.
Is there anyone in particular in the Supavisor team that could help us with testing? This has been left aside for months.
Thanks

[encima]

Hey @SkrOYC
The pooler team is aware of this issue and it is currently in their backlog

[SkrOYC]

@encima Do you have any words from the team? My fellow devs are ditching Supabase and I'm worried I'm not going to be able to advice it's usage any more

[encima]

@SkrOYC Thank you for chasing this and for staying on top of it. The last thing we want to see is you or anyone feel the need to leave Supabase because of this issue and we have dropped the ball here in terms of communications and fixes.

The issue here is with Supavisor and the SSL connections, though we haven't been able to capture enough information as to determine the root cause.

The docs should be clearer for integrations and I will ensure they are updated this week as we have some changes coming to the connection docs.

The first thing to note is that direct connections should be used in the case of integrations such as these. The docs will be updated to make this clear and specify that we recommend only the use of direct connections in these cases, notably when the tools do support IPv6. If they don't, the addon is charged hourly so it is worth enabling to check and confirm.

We will link the PR to this issue so you see the changes

[jppty]

Chiming in as I am also a long-time AppSheet developer/consultant, and I've steered many of my clients towards Supabase.

The pooler changes have been crippling.

I have had some success by using the ipv4 addon and the old connection string, however AppSheet persists the connections and quickly floods the allowed connections. This did not happen prior to the pooler changes, and it also does not happen with sql server or postgres instances outside of Supabase.

[encima]

Hello all,

I think there are 2 issues here (one is solved and the other needs documenting)

1. Supavisor connection issues when connecting with Appsheet

This is fixed now! The issue is due to how different clients handle/enforce SSL connections and there was a discrepancy due to SCRAM. Some of the more recent changes are outlined here
The fix is out in our Staging environment and we will ping here when it is available for you to use; thanks so much for your patience here!

2. Many connections opened

This is more of a documentation issue, I believe. Depending on how you configure the data source, the connection mode will be in transaction or session mode. We have recently updated our project settings page to recommend using direct connections to connect to apps in non-serverless (serverful) environments but, if you do use Supavisor, then we strongly recommend transaction mode. That said, that will create a new connection for each transaction needed to run and AppSheet ultimately handles this. The same is true for session mode but those connections will be kept open for longer and are typically not limited to a single transaction.

Let us know your thoughts on this and we can work it into the docs better or even include it as a guide?

[SkrOYC]

Thanks for your work @encima !

Regarding this:

That said, that will create a new connection for each transaction needed to run and AppSheet ultimately handles this

Do you have some confirmation whether pgBouncer was behaving the same as Supavisor does for this case? As far as I can tell, AppSheet opens a conection for each CRUD operation but it expects the pooler to handle the rest

[encima]

@SkrOYC we can run tests on this to compare the two.

Pooler handles the rest

That may be true but the pooler will only open a connection for each transaction request it gets. It will never open more than requested (unless configured to hold locked connections or similar)

[SkrOYC]

In that case, when the change is pushed to production it should solve the issue. Looking forward! BR. Oscar