✨ feat: support server db mode with Postgres / Drizzle ORM / tRPC #2556

arvinxx · 2024-05-18T16:34:09Z

💻 变更类型 | Change Type

🔀 变更说明 | Description of Change

整体进展

LobeChat 服务端 DB 版本功能实现，目前完成度为 100% 。

Note

由于之前并没有写过 SQL 和 Postgres ，所以可能整体服务端实现上仍然存在瑕疵，也可能存在 critical 的问题，欢迎大家点评指正。

技术栈

DB：Postgres ，默认采用 Neon；
ORM: Dirzzle ORM
Auth: Clerk
server router: tRPC

由于基于 Drizzle ORM，非常期待看到未来拓展到其他 Postgres DB、甚至 SQLite、MySQL 。以及有待 @cy948 帮忙实现 next-auth 相关的 db 鉴权方案。

待解决工程问题

工程链路上还有两个问题待解决：

1. 数据库无感 migration ： [RFC] 036 - LobeChat 服务端数据库 & ORM #1851 (comment) ；
2. 带有真实数据库的 ci/cd workflow ： [RFC] 036 - LobeChat 服务端数据库 & ORM #1851 (comment)

服务端相关的配置

# Postgres DB Url
DATABASE_URL=

# 指定 service mode 为 server 
NEXT_PUBLIC_SERVICE_MODE=server

# Clerk 相关配置
NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=
CLERK_SECRET_KEY=
CLERK_WEBHOOK_SECRET=

# S3 相关配置（上传图片必须）
NEXT_PUBLIC_S3_DOMAIN=

S3_ACCESS_KEY_ID=
S3_SECRET_ACCESS_KEY=
S3_ENDPOINT=
S3_BUCKET=

📝 补充信息 | Additional Information

[RFC] 036 - LobeChat 服务端数据库 & ORM #1851

vercel · 2024-05-18T16:34:12Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
lobe-chat-community	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 16, 2024 10:59am

lobehubbot · 2024-05-18T16:34:23Z

👍 @arvinxx

Thank you for raising your pull request and contributing to our Community
Please make sure you have followed our contributing guidelines. We will review it as soon as possible.
If you encounter any problems, please feel free to connect with us.
非常感谢您提出拉取请求并为我们的社区做出贡献，请确保您已经遵循了我们的贡献指南，我们会尽快审查它。
如果您遇到任何问题，请随时与我们联系。

Matinal111 · 2024-05-24T08:14:42Z

目前是可以配置服务端数据库了吗

lobehubbot · 2024-05-24T08:14:55Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

Is it possible to configure the server database now?

arvinxx · 2024-05-24T10:02:50Z

目前是可以配置服务端数据库了吗

@cfwl1111 等这个 PR 合并后，就可以了

lobehubbot · 2024-05-24T10:03:00Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

Is it possible to configure the server database now?

@cfwl1111 After this PR is merged, it will be fine

coderabbitai · 2024-05-24T11:58:43Z

Important

Review Skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

Matinal111 · 2024-05-30T03:41:05Z

目前是可以配置服务端数据库了吗

@cfwl1111 等这个 PR 合并后，就可以了

服务端 Postgres 准备好了，期待它的到来

lobehubbot · 2024-05-30T03:41:16Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

Is it possible to configure the server database now?

@cfwl1111 After this PR is merged, it will be fine

Server-side Postgres is ready and looking forward to its arrival

lee88688 · 2024-05-31T00:57:35Z

图片的保存可以考虑支持直接存储在PostgreSQL数据库中。自部署还需要单独起一个S3兼容的后端服务管理起来更复杂，特别是大部分自部署用户的使用强度不是那么高，使用数据库完全没问题。

lobehubbot · 2024-05-31T00:57:46Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

The saving of pictures can be considered to be directly stored in the PostgreSQL database. Self-deployment also requires a separate S3-compatible back-end service, which is more complicated to manage, especially since most self-deployment users are not so intensive in usage, and there is no problem using the database.

arvinxx · 2024-05-31T01:19:08Z

图片的保存可以考虑支持直接存储在PostgreSQL数据库中。自部署还需要单独起一个S3兼容的后端服务管理起来更复杂，特别是大部分自部署用户的使用强度不是那么高，使用数据库完全没问题。

不建议，后面还要支持各种文件上传的。如果不想承担这个复杂度，要么仍然可以用客户端数据库db的模式，不需要s3。或者用未来我们host的cloud 版

lobehubbot · 2024-05-31T01:19:19Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

The saving of images can be considered to be directly stored in the PostgreSQL database. Self-deployment also requires a separate S3-compatible back-end service, which is more complicated to manage, especially since most self-deployment users are not so intensive in usage, and there is no problem using the database.

Not recommended, it will support various file uploads later. If you don’t want to bear this complexity, you can either use the locally deployed version without s3, or use the cloud version of our host in the future.

### [Version 0.162.25](v0.162.24...v0.162.25) <sup>Released on **2024-06-16**</sup> #### 🐛 Bug Fixes - **misc**: Fix issues for client fetch. <br/> <details> <summary><kbd>Improvements and Fixes</kbd></summary> #### What's fixed * **misc**: Fix issues for client fetch, closes [#2753](#2753) ([6f5be5d](6f5be5d)) </details> <div align="right"> [![](https://img.shields.io/badge/-BACK_TO_TOP-151515?style=flat-square)](#readme-top) </div>

lobehubbot · 2024-06-16T11:02:07Z

❤️ Great PR @arvinxx ❤️

The growth of project is inseparable from user feedback and contribution, thanks for your contribution! If you are interesting with the lobehub developer community, please join our discord and then dm @arvinxx or @canisminor1990. They will invite you to our private developer channel. We are talking about the lobe-chat development or sharing ai newsletter around the world.
项目的成长离不开用户反馈和贡献，感谢您的贡献! 如果您对 LobeHub 开发者社区感兴趣，请加入我们的 discord，然后私信 @arvinxx 或 @canisminor1990。他们会邀请您加入我们的私密开发者频道。我们将会讨论关于 Lobe Chat 的开发，分享和讨论全球范围内的 AI 消息。

)

## [Version 0.163.0](v0.162.25...v0.163.0) <sup>Released on **2024-06-17**</sup> #### ✨ Features - **misc**: Support server db mode with Postgres / Drizzle ORM / tRPC. <br/> <details> <summary><kbd>Improvements and Fixes</kbd></summary> #### What's improved * **misc**: Support server db mode with Postgres / Drizzle ORM / tRPC, closes [#2556](#2556) ([b26afbf](b26afbf)) </details> <div align="right"> [![](https://img.shields.io/badge/-BACK_TO_TOP-151515?style=flat-square)](#readme-top) </div>

xingpingcn · 2024-06-17T07:42:53Z

这个新特性是不是在文档中没有更新？

客户端数据库db的模式

这个又应该如何配置？

lobehubbot · 2024-06-17T07:43:04Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

Is this new feature not updated in the documentation?

Schema of client database db

How should this be configured?

## [Version 1.43.0](v1.42.19...v1.43.0) <sup>Released on **2024-06-17**</sup> #### ✨ Features - **misc**: Release LobeChat 1.0, support server db mode with Postgres / Drizzle ORM / tRPC. #### 💄 Styles - **misc**: Remove r2 cdn url. #### 🔧 Chores - **misc**: Bump version. <br/> <details> <summary><kbd>Improvements and Fixes</kbd></summary> #### What's improved * **misc**: Release LobeChat 1.0, closes [lobehub#2897](https://github.com/bentwnghk/lobe-chat/issues/2897) ([97bb377](97bb377)) * **misc**: Support server db mode with Postgres / Drizzle ORM / tRPC, closes [lobehub#2556](https://github.com/bentwnghk/lobe-chat/issues/2556) ([b26afbf](b26afbf)) #### Styles * **misc**: Remove r2 cdn url, closes [lobehub#2878](https://github.com/bentwnghk/lobe-chat/issues/2878) ([19fd4d5](19fd4d5)) #### Chores * **misc**: Bump version, closes [lobehub#2896](https://github.com/bentwnghk/lobe-chat/issues/2896) ([6e2d03b](6e2d03b)) </details> #### 💥 BREAKING CHANGES * **misc**: Update LICENSE to Apache 2.0 * **misc**: update to 1.0.0 <div align="right"> [![](https://img.shields.io/badge/-BACK_TO_TOP-151515?style=flat-square)](#readme-top) </div>

ZeroDeng01 · 2024-06-17T11:40:08Z

CLERK_WEBHOOK_SECRET这个配置后是否需要去clerk配置Endpoint？我clerk的相关密钥已经做了配置，用户也可以登录，但是我查看数据库并没有新建clerk的用户信息，现在我的项目完全无法使用。

我查询日志，就是因为在数据库获取不到用户信息。应为users表为空：

@arvinxx

lobehubbot · 2024-06-17T11:40:21Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

After configuring CLERK_WEBHOOK_SECRET, do I need to go to clerk to configure the Endpoint? The relevant keys of my clerk have been configured and users can log in, but when I checked the database, there was no new user information for the clerk. Now my project is completely unusable.

I checked the log because I couldn't get the user information in the database. The users table should be empty:

@arvinxx

arvinxx · 2024-06-17T12:52:47Z

CLERK_WEBHOOK_SECRET这个配置后是否需要去clerk配置Endpoint？我clerk的相关密钥已经做了配置，用户也可以登录，但是我查看数据库并没有新建clerk的用户信息，现在我的项目完全无法使用。

需要的，类似这样：

@ZeroDeng01 @xingpingcn 目前文档还没完善，有问题可以在 Discord 里咨询，我们这几天会把部署指南补充完善

cmj2002 · 2024-06-19T07:40:59Z

以及有待 @cy948 帮忙实现 next-auth 相关的 db 鉴权方案。

该部分工作是否有相关联的 issue/discussion/pull request ?

lobehubbot · 2024-06-19T07:41:11Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

And waiting for @cy948 to help implement next-auth related db authentication scheme.

Is there an associated issue/discussion/pull request for this part of the work?

arvinxx · 2024-06-19T10:30:28Z

该部分工作是否有相关联的 issue/discussion/pull request ?

@cy948 感觉可以先写一篇 RFC 来讨论下？

lobehubbot · 2024-06-19T10:30:41Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

Is there an associated issue/discussion/pull request for this part of the work?

@cy948 I feel like I can write an RFC to discuss it first?

cy948 · 2024-06-19T11:28:21Z

@arvinxx 我试完第一个方案之后写个rfc描述下现在的情况吧

lobehubbot · 2024-06-19T11:28:33Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

@arvinxx After I try the first solution, I will write an RFC to describe the current situation.

gaarry · 2024-06-20T11:07:54Z

用上数据库了，朋友也注册我的网站，体验很棒～

lobehubbot · 2024-06-20T11:08:07Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

I have used the database and my friends have also registered on my website. The experience is great~

gaspire · 2024-06-25T14:00:01Z

用上数据库了，朋友也注册我的网站，体验很棒～

分享下教程，谢谢！

lobehubbot · 2024-06-25T14:00:14Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

I have used the database, and my friends have also registered on my website, and the experience is great ~

Share the tutorial, thank you!

…behub#2556)

## [Version 0.163.0](lobehub/lobe-chat@v0.162.25...v0.163.0) <sup>Released on **2024-06-17**</sup> #### ✨ Features - **misc**: Support server db mode with Postgres / Drizzle ORM / tRPC. <br/> <details> <summary><kbd>Improvements and Fixes</kbd></summary> #### What's improved * **misc**: Support server db mode with Postgres / Drizzle ORM / tRPC, closes [lobehub#2556](lobehub#2556) ([b26afbf](lobehub@b26afbf)) </details> <div align="right"> [![](https://img.shields.io/badge/-BACK_TO_TOP-151515?style=flat-square)](#readme-top) </div>

arvinxx changed the title ~~✨ feat: support server db with postgres~~ ✨ feat: support server db with postgres / Drizzle / trpc May 18, 2024

arvinxx changed the title ~~✨ feat: support server db with postgres / Drizzle / trpc~~ ✨ feat: support server db with postgres / Drizzle ORM / trpc May 18, 2024

arvinxx changed the title ~~✨ feat: support server db with postgres / Drizzle ORM / trpc~~ ✨ feat: support server db mode with postgres / Drizzle ORM / trpc May 18, 2024

arvinxx changed the title ~~✨ feat: support server db mode with postgres / Drizzle ORM / trpc~~ ✨ feat: support server db mode with postgres / Drizzle ORM / tRPC May 18, 2024

arvinxx changed the title ~~✨ feat: support server db mode with postgres / Drizzle ORM / tRPC~~ ✨ feat: support server db mode with Postgres / Drizzle ORM / tRPC May 19, 2024

cy948 mentioned this pull request May 21, 2024

[Request] kubernetes 部署指导 #2587

Open

arvinxx force-pushed the feat/server-db branch from 0f0cd14 to 4067946 Compare May 24, 2024 11:58

vercel bot deployed to Preview May 24, 2024 12:10 View deployment

arvinxx force-pushed the feat/server-db branch from 4067946 to d37928e Compare May 28, 2024 16:33

vercel bot deployed to Preview May 28, 2024 16:39 View deployment

arvinxx force-pushed the feat/server-db branch from d37928e to 1d4de28 Compare May 29, 2024 12:43

vercel bot deployed to Preview May 29, 2024 12:49 View deployment

lobehub deleted a comment from lobehubbot May 29, 2024

arvinxx force-pushed the feat/server-db branch from 1d4de28 to 1c682f8 Compare June 3, 2024 11:50

vercel bot deployed to Preview June 3, 2024 11:55 View deployment

arvinxx force-pushed the feat/server-db branch from 1c682f8 to 7051aed Compare June 3, 2024 18:19

vercel bot had a problem deploying to Preview June 3, 2024 18:22 Failure

arvinxx force-pushed the feat/server-db branch from 1399a6d to 7e53dfe Compare June 16, 2024 10:44

arvinxx changed the base branch from main to release/1.0 June 16, 2024 10:44

vercel bot deployed to Preview June 16, 2024 10:50 View deployment

✨ feat: support server db with Postgres and drizzle ORM

a418a5e

arvinxx force-pushed the feat/server-db branch from 7e53dfe to a418a5e Compare June 16, 2024 10:55

vercel bot deployed to Preview June 16, 2024 10:59 View deployment

arvinxx merged commit d8bb6f4 into release/1.0 Jun 16, 2024
8 checks passed

arvinxx deleted the feat/server-db branch June 16, 2024 11:01

arvinxx added a commit that referenced this pull request Jun 17, 2024

✨ feat: support server db mode with Postgres / Drizzle ORM / tRPC (#2556

b26afbf

)

ipoly pushed a commit to ipoly/lobe-chat that referenced this pull request Jul 29, 2024

✨ feat: support server db mode with Postgres / Drizzle ORM / tRPC (lo…

e25ce2f

…behub#2556)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ feat: support server db mode with Postgres / Drizzle ORM / tRPC #2556

✨ feat: support server db mode with Postgres / Drizzle ORM / tRPC #2556

arvinxx commented May 18, 2024 •

edited

Loading

vercel bot commented May 18, 2024 •

edited

Loading

lobehubbot commented May 18, 2024

Matinal111 commented May 24, 2024

lobehubbot commented May 24, 2024

arvinxx commented May 24, 2024

lobehubbot commented May 24, 2024

coderabbitai bot commented May 24, 2024

Review Skipped

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (`.coderabbit.yaml`)

Documentation and Community

Matinal111 commented May 30, 2024

lobehubbot commented May 30, 2024

lee88688 commented May 31, 2024

lobehubbot commented May 31, 2024

arvinxx commented May 31, 2024 •

edited

Loading

lobehubbot commented May 31, 2024

lobehubbot commented Jun 16, 2024

xingpingcn commented Jun 17, 2024

lobehubbot commented Jun 17, 2024

ZeroDeng01 commented Jun 17, 2024

lobehubbot commented Jun 17, 2024

arvinxx commented Jun 17, 2024 •

edited

Loading

cmj2002 commented Jun 19, 2024

lobehubbot commented Jun 19, 2024

arvinxx commented Jun 19, 2024

lobehubbot commented Jun 19, 2024

cy948 commented Jun 19, 2024

lobehubbot commented Jun 19, 2024

gaarry commented Jun 20, 2024

lobehubbot commented Jun 20, 2024

gaspire commented Jun 25, 2024

lobehubbot commented Jun 25, 2024

✨ feat: support server db mode with Postgres / Drizzle ORM / tRPC #2556

✨ feat: support server db mode with Postgres / Drizzle ORM / tRPC #2556

Conversation

arvinxx commented May 18, 2024 • edited Loading

💻 变更类型 | Change Type

🔀 变更说明 | Description of Change

整体进展

技术栈

待解决工程问题

服务端相关的配置

📝 补充信息 | Additional Information

vercel bot commented May 18, 2024 • edited Loading

lobehubbot commented May 18, 2024

Matinal111 commented May 24, 2024

lobehubbot commented May 24, 2024

arvinxx commented May 24, 2024

lobehubbot commented May 24, 2024

coderabbitai bot commented May 24, 2024

Review Skipped

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (.coderabbit.yaml)

Documentation and Community

Matinal111 commented May 30, 2024

lobehubbot commented May 30, 2024

lee88688 commented May 31, 2024

lobehubbot commented May 31, 2024

arvinxx commented May 31, 2024 • edited Loading

lobehubbot commented May 31, 2024

lobehubbot commented Jun 16, 2024

xingpingcn commented Jun 17, 2024

lobehubbot commented Jun 17, 2024

ZeroDeng01 commented Jun 17, 2024

lobehubbot commented Jun 17, 2024

arvinxx commented Jun 17, 2024 • edited Loading

cmj2002 commented Jun 19, 2024

lobehubbot commented Jun 19, 2024

arvinxx commented Jun 19, 2024

lobehubbot commented Jun 19, 2024

cy948 commented Jun 19, 2024

lobehubbot commented Jun 19, 2024

gaarry commented Jun 20, 2024

lobehubbot commented Jun 20, 2024

gaspire commented Jun 25, 2024

lobehubbot commented Jun 25, 2024

arvinxx commented May 18, 2024 •

edited

Loading

vercel bot commented May 18, 2024 •

edited

Loading

CodeRabbit Configration File (`.coderabbit.yaml`)

arvinxx commented May 31, 2024 •

edited

Loading

arvinxx commented Jun 17, 2024 •

edited

Loading