Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

community[patch]: Allow using a pool for PostgresRecordManager #4409

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

mkesper
Copy link
Contributor

@mkesper mkesper commented Feb 14, 2024

Enable to use a predefined pool for PostgresRecordManager, allowing to use multiple indices per database. Without, each index will create a new pool, leading to Postgres connection exhaustion.

Fixes #4407

Enable to use a predefined pool for PostgresRecordManager, allowing to use multiple indices per database. Without, each index will create a new pool, leading to Postgres connection exhaustion.
Copy link

vercel bot commented Feb 14, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
langchainjs-api-refs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Feb 14, 2024 1:59pm
langchainjs-docs ✅ Ready (Inspect) Visit Preview Feb 14, 2024 1:59pm

@dosubot dosubot bot added size:S This PR changes 10-29 lines, ignoring generated files. auto:improvement Medium size change to existing code to handle new use-cases labels Feb 14, 2024
Copy link
Contributor

@MJDeligan MJDeligan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @mkesper , thanks for adding this, great thinking! Left a couple of comments (not a maintainer, wrote the original implementation)

this.namespace = namespace;
this.pool = new pg.Pool(postgresConnectionOptions);
if (pool instanceof Pool) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

violates eslint rule. checking that pool is not undefined should be fine

} else {
try {
this.pool = new pg.Pool(postgresConnectionOptions);
} catch (err: unknown) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pool constructor does not throw when config is undefined/invalid as far as I can tell. checking that pool and config aren't both undefined should work

@mkesper
Copy link
Contributor Author

mkesper commented Feb 15, 2024

@MJDeligan Thanks, you're right, that code could be improved. :) Since I posted I'm not anymore sure what's the best way to handle the problem that connections are held for a long time. Could also be a problem in the code calling the RecordManager not closing it. (A hint for that in the examples would be really nice).

@MJDeligan
Copy link
Contributor

@mkesper I think you're right that having multiple pools will lead to the creation of too many clients. Being able to pass your own pool is definitely good.

Not ending the pool(s) will of course worsen this. Sorry about the documentation, the long example is a little unwieldy. Maybe a very minimal example at the beginning would be good to add.

@jacoblee93
Copy link
Collaborator

Thank you for the review @MJDeligan!

I think technically the instanceof check is ok here (with a linter ignore) because it's Node only. Have seen instanceof checks be unreliable in environments that use bundlers like Next.js since you can wind up with multiple declarations in different chunks. But probably best to avoid yes.

@jacoblee93 jacoblee93 self-assigned this Feb 15, 2024
@jacoblee93 jacoblee93 added the close PRs that need one or two touch-ups to be ready label Feb 15, 2024
@jacoblee93 jacoblee93 changed the title Allow using a pool for PostgresRecordManager community[patch]: Allow using a pool for PostgresRecordManager Feb 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto:improvement Medium size change to existing code to handle new use-cases close PRs that need one or two touch-ups to be ready size:S This PR changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow using a pool for PostgresRecordManager
3 participants