Skip to content

Commit

Permalink
♻️ refactor: refactor s3 env and support path-style for minio (lobehu…
Browse files Browse the repository at this point in the history
…b#3559)

* ♻️ refactor: refactor the `NEXT_PUBLIC_S3_DOMAIN` to `S3_PUBLIC_DOMAIN`

* 👷 build: support path-style for s3
  • Loading branch information
arvinxx authored Aug 22, 2024
1 parent 2ecd5e3 commit 1658403
Show file tree
Hide file tree
Showing 15 changed files with 45 additions and 26 deletions.
2 changes: 1 addition & 1 deletion .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ OPENAI_API_KEY=sk-xxxxxxxxx
#S3_ENDPOINT=https://0b33a03b5c993fd2f453379dc36558e5.r2.cloudflarestorage.com

# Public access domain for the bucket
#NEXT_PUBLIC_S3_DOMAIN=https://s3-for-lobechat.your-domain.com
#S3_PUBLIC_DOMAIN=https://s3-for-lobechat.your-domain.com

# Bucket region, such as us-west-1, generally not needed to add
# but some service providers may require configuration
Expand Down
1 change: 1 addition & 0 deletions Dockerfile.database
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,7 @@ ENV NEXT_AUTH_SECRET="" \

# S3
ENV NEXT_PUBLIC_S3_DOMAIN="" \
S3_PUBLIC_DOMAIN="" \
S3_ACCESS_KEY_ID="" \
S3_BUCKET="" \
S3_ENDPOINT="" \
Expand Down
2 changes: 1 addition & 1 deletion docs/self-hosting/advanced/s3.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ In this documentation, S3 refers to a compatible S3 storage solution, which supp

Whether to set the ACL to `public-read` when uploading files. This option is enabled by default. If the service provider does not support setting individual ACLs for files (i.e., all files inherit the ACL of the storage bucket), enabling this option may cause request errors. Set `S3_SET_ACL` to `0` to disable it.

### `NEXT_PUBLIC_S3_DOMAIN`
### `S3_PUBLIC_DOMAIN`

The public access domain of the storage bucket, used to access files in the storage bucket. This address needs to be **publicly readable**. The reason is that when OpenAI's gpt-4o and other vision models recognize images, OpenAI will try to download this image link on their servers. Therefore, this link must be publicly accessible. If it is a private link, OpenAI will not be able to access the image and thus will not be able to recognize the image content properly.
Expand Down
2 changes: 1 addition & 1 deletion docs/self-hosting/advanced/s3.zh-CN.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ S3_ENDPOINT=https://0b33a03b5c993fd2f453379dc36558e5.r2.cloudflarestorage.com

是否在上传文件时设置 ACL 为 `public-read`。该选项默认启用。如果服务商不支持为文件设置单独的 ACL(即所有文件继承存储桶的 ACL),启用此选项可能会导致请求错误,将 `S3_SET_ACL` 设置为 `0` 即可关闭。

### `NEXT_PUBLIC_S3_DOMAIN`
### `S3_PUBLIC_DOMAIN`

存储桶对外的访问域名,用于访问存储桶中的文件,这个地址需要**允许互联网可读**。 原因是 OpenAI 的 gpt-4o 等视觉模型识别图片时,OpenAI 会尝试在他们的服务器中下载这个图片链接,因此这个链接必须是公开可访问的,如果是私有的链接,OpenAI 将无法访问到这个图片,进而无法正常识别到图片内容。

Expand Down
4 changes: 2 additions & 2 deletions docs/self-hosting/advanced/s3/cloudflare-r2.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ S3_BUCKET=lobechat
# Request endpoint of the bucket (note that the path in this link includes the bucket name, which must be removed, or use the link provided on the page for applying S3 API token)
S3_ENDPOINT=https://0b33a03b5c993fd2f453379dc36558e5.r2.cloudflarestorage.com
# Access domain of the bucket
NEXT_PUBLIC_S3_DOMAIN=https://s3-for-lobechat.your-domain.com
S3_PUBLIC_DOMAIN=https://s3-for-lobechat.your-domain.com
```

<Callout type={'warning'}>`S3_ENDPOINT` must have its path removed, otherwise uploaded files cannot be accessed.</Callout>
Expand Down Expand Up @@ -138,7 +138,7 @@ S3_BUCKET=lobechat
# Bucket Request Endpoint
S3_ENDPOINT=https://0b33a03b5c993fd2f453379dc36558e5.r2.cloudflarestorage.com
# Public Access Domain for the Bucket
NEXT_PUBLIC_S3_DOMAIN=https://s3-dev.your-domain.com
S3_PUBLIC_DOMAIN=https://s3-dev.your-domain.com

# Bucket Region, such as us-west-1. Generally not required, but some service providers may need it.
# S3_REGION=us-west-1
Expand Down
4 changes: 2 additions & 2 deletions docs/self-hosting/advanced/s3/cloudflare-r2.zh-CN.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ S3_BUCKET=lobechat
# 存储桶的请求端点(注意此处链接的路径带存储桶名称,必须删除该路径,或使用申请 S3 API token 页面所提供的链接)
S3_ENDPOINT=https://0b33a03b5c993fd2f453379dc36558e5.r2.cloudflarestorage.com
# 存储桶对外的访问域名
NEXT_PUBLIC_S3_DOMAIN=https://s3-for-lobechat.your-domain.com
S3_PUBLIC_DOMAIN=https://s3-for-lobechat.your-domain.com
```

<Callout type={'warning'}>`S3_ENDPOINT`必须删除其路径,否则会无法访问所上传文件</Callout>
Expand Down Expand Up @@ -139,7 +139,7 @@ S3_BUCKET=lobechat
# 存储桶的请求端点
S3_ENDPOINT=https://0b33a03b5c993fd2f453379dc36558e5.r2.cloudflarestorage.com
# 存储桶对外的访问域名
NEXT_PUBLIC_S3_DOMAIN=https://s3-dev.your-domain.com
S3_PUBLIC_DOMAIN=https://s3-dev.your-domain.com

# 桶的区域,如 us-west-1,一般来说不需要添加,但某些服务商则需要配置
# S3_REGION=us-west-1
Expand Down
4 changes: 2 additions & 2 deletions docs/self-hosting/server-database/docker.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ S3_ACCESS_KEY_ID=xxxxxxxxxx
S3_SECRET_ACCESS_KEY=xxxxxxxxxx
S3_ENDPOINT=https://xxxxxxxxxx.r2.cloudflarestorage.com
S3_BUCKET=lobechat
NEXT_PUBLIC_S3_DOMAIN=https://s3-for-lobechat.your-domain.com
S3_PUBLIC_DOMAIN=https://s3-for-lobechat.your-domain.com
```

### Start the lobe-chat-database Docker image
Expand Down Expand Up @@ -125,7 +125,7 @@ $ docker run -it -d --name lobe-chat-database -p 3210:3210 \
-e S3_SECRET_ACCESS_KEY=xxxxxxxxxx \
-e S3_ENDPOINT=https://xxxxxxxxxx.r2.cloudflarestorage.com \
-e S3_BUCKET=lobechat \
-e NEXT_PUBLIC_S3_DOMAIN=https://s3-for-lobechat.your-domain.com \
-e S3_PUBLIC_DOMAIN=https://s3-for-lobechat.your-domain.com \
lobehub/lobe-chat-database
```

Expand Down
4 changes: 2 additions & 2 deletions docs/self-hosting/server-database/docker.zh-CN.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ S3_ACCESS_KEY_ID=xxxxxxxxxx
S3_SECRET_ACCESS_KEY=xxxxxxxxxx
S3_ENDPOINT=https://xxxxxxxxxx.r2.cloudflarestorage.com
S3_BUCKET=lobechat
NEXT_PUBLIC_S3_DOMAIN=https://s3-for-lobechat.your-domain.com
S3_PUBLIC_DOMAIN=https://s3-for-lobechat.your-domain.com
```

### 启动 lobe-chat-database docker 镜像
Expand Down Expand Up @@ -130,7 +130,7 @@ $ docker run -it -d --name lobe-chat-database -p 3210:3210 \
-e S3_SECRET_ACCESS_KEY=xxxxxxxxxx \
-e S3_ENDPOINT=https://xxxxxxxxxx.r2.cloudflarestorage.com \
-e S3_BUCKET=lobechat \
-e NEXT_PUBLIC_S3_DOMAIN=https://s3-for-lobechat.your-domain.com \
-e S3_PUBLIC_DOMAIN=https://s3-for-lobechat.your-domain.com \
lobehub/lobe-chat-database
```

Expand Down
8 changes: 4 additions & 4 deletions docs/self-hosting/server-database/vercel.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -250,7 +250,7 @@ S3_BUCKET=lobechat
# Storage bucket request endpoint (note that the path in this link includes the bucket name, which must be removed, or use the link provided on the S3 API token application page)
S3_ENDPOINT=https://0b33a03b5c993fd2f453379dc36558e5.r2.cloudflarestorage.com
# Public access domain for the storage bucket
NEXT_PUBLIC_S3_DOMAIN=https://s3-for-lobechat.your-domain.com
S3_PUBLIC_DOMAIN=https://s3-for-lobechat.your-domain.com
```

<Callout type={'warning'}>
Expand Down Expand Up @@ -312,7 +312,7 @@ S3_BUCKET=lobechat
# Bucket request endpoint
S3_ENDPOINT=https://0b33a03b5c993fd2f453379dc36558e5.r2.cloudflarestorage.com
# Public domain for bucket access
NEXT_PUBLIC_S3_DOMAIN=https://s3-dev.your-domain.com
S3_PUBLIC_DOMAIN=https://s3-dev.your-domain.com

# Bucket region, such as us-west-1, generally not required, but some providers may need to configure
# S3_REGION=us-west-1
Expand Down Expand Up @@ -384,7 +384,7 @@ src={'https://github.com/lobehub/lobe-chat/assets/28616219/da84edc3-46f7-4e2b-a0
<Image
alt={'Login successful state'}
src={'https://github.com/lobehub/lobe-chat/assets/28616219/9cb5150d-6e1e-4c59-9a18-4e418dce1a5d'}/>

</Steps>

## Appendix
Expand Down Expand Up @@ -416,7 +416,7 @@ S3_BUCKET=lobechat
# Bucket request endpoint
S3_ENDPOINT=https://0b33a03b5c993fd2f453379dc36558e5.r2.cloudflarestorage.com
# Public access domain for the bucket
NEXT_PUBLIC_S3_DOMAIN=https://s3-for-lobechat.your-domain.com
S3_PUBLIC_DOMAIN=https://s3-for-lobechat.your-domain.com
# Bucket region, such as us-west-1, generally not needed to add, but some service providers may require configuration
# S3_REGION=us-west-1
```
12 changes: 6 additions & 6 deletions docs/self-hosting/server-database/vercel.zh-CN.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ tags:

<Callout type={'warning'}>
进行后续操作前,请务必确认以下事项:
- 导出所有数据,部署服务端数据库后,原有用户数据无法自动迁移,只能提前备份后进行手动导入!
- 环境变量中的`ACCESS_CODE`未设置或已清除!
- 导出所有数据,部署服务端数据库后,原有用户数据无法自动迁移,只能提前备份后进行手动导入!
- 环境变量中的`ACCESS_CODE`未设置或已清除!
- 配置服务端数据库所需要的环境变量时,需全部填入后再进行部署,否则可能遭遇数据库迁移问题!
</Callout>

Expand All @@ -46,7 +46,7 @@ tags:

<Callout type={'warning'}>
请确认您的供应商所提供的 `Postgres` 类型,若为 `Node Postgres`,请切换到 `Node Postgres` Tab 。

</Callout>

Serverless Postgres 需要填写的变量如下:
Expand Down Expand Up @@ -233,7 +233,7 @@ S3_BUCKET=lobechat
# 存储桶的请求端点(注意此处链接的路径带存储桶名称,必须删除该路径,或使用申请 S3 API token 页面所提供的链接)
S3_ENDPOINT=https://0b33a03b5c993fd2f453379dc36558e5.r2.cloudflarestorage.com
# 存储桶对外的访问域名
NEXT_PUBLIC_S3_DOMAIN=https://s3-for-lobechat.your-domain.com
S3_PUBLIC_DOMAIN=https://s3-for-lobechat.your-domain.com
```

<Callout type={'warning'}>`S3_ENDPOINT`必须删除其路径,否则会无法访问所上传文件</Callout>
Expand Down Expand Up @@ -291,7 +291,7 @@ S3_BUCKET=lobechat
# 存储桶的请求端点
S3_ENDPOINT=https://0b33a03b5c993fd2f453379dc36558e5.r2.cloudflarestorage.com
# 存储桶对外的访问域名
NEXT_PUBLIC_S3_DOMAIN=https://s3-dev.your-domain.com
S3_PUBLIC_DOMAIN=https://s3-dev.your-domain.com
# 桶的区域,如 us-west-1,一般来说不需要添加,但某些服务商则需要配置
# S3_REGION=us-west-1
Expand Down Expand Up @@ -397,7 +397,7 @@ S3_BUCKET=lobechat
# 存储桶的请求端点
S3_ENDPOINT=https://0b33a03b5c993fd2f453379dc36558e5.r2.cloudflarestorage.com
# 存储桶对外的访问域名
NEXT_PUBLIC_S3_DOMAIN=https://s3-for-lobechat.your-domain.com
S3_PUBLIC_DOMAIN=https://s3-for-lobechat.your-domain.com
# 桶的区域,如 us-west-1,一般来说不需要添加,但某些服务商则需要配置
# S3_REGION=us-west-1
```
17 changes: 16 additions & 1 deletion src/config/file.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,19 @@ import { z } from 'zod';
const DEFAULT_S3_FILE_PATH = 'files';

export const getFileConfig = () => {
if (!!process.env.NEXT_PUBLIC_S3_DOMAIN) {
console.warn(
'⚠️ `NEXT_PUBLIC_S3_DOMAIN` will be de deprecated in the next major version, please replace it with `S3_PUBLIC_DOMAIN` in your env',
);
}

const S3_PUBLIC_DOMAIN = process.env.S3_PUBLIC_DOMAIN || process.env.NEXT_PUBLIC_S3_DOMAIN;

return createEnv({
client: {
/**
* @deprecated
*/
NEXT_PUBLIC_S3_DOMAIN: z.string().url().optional(),
NEXT_PUBLIC_S3_FILE_PATH: z.string().optional(),
},
Expand All @@ -18,7 +29,9 @@ export const getFileConfig = () => {

S3_ACCESS_KEY_ID: process.env.S3_ACCESS_KEY_ID,
S3_BUCKET: process.env.S3_BUCKET,
S3_ENABLE_PATH_STYLE: process.env.S3_ENABLE_PATH_STYLE === '1',
S3_ENDPOINT: process.env.S3_ENDPOINT,
S3_PUBLIC_DOMAIN,
S3_REGION: process.env.S3_REGION,
S3_SECRET_ACCESS_KEY: process.env.S3_SECRET_ACCESS_KEY,
S3_SET_ACL: process.env.S3_SET_ACL !== '0',
Expand All @@ -30,8 +43,10 @@ export const getFileConfig = () => {
// S3
S3_ACCESS_KEY_ID: z.string().optional(),
S3_BUCKET: z.string().optional(),
S3_ENDPOINT: z.string().url().optional(),
S3_ENABLE_PATH_STYLE: z.boolean(),

S3_ENDPOINT: z.string().url().optional(),
S3_PUBLIC_DOMAIN: z.string().url().optional(),
S3_REGION: z.string().optional(),
S3_SECRET_ACCESS_KEY: z.string().optional(),
S3_SET_ACL: z.boolean(),
Expand Down
1 change: 1 addition & 0 deletions src/server/modules/S3/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ export class S3 {
secretAccessKey: fileEnv.S3_SECRET_ACCESS_KEY,
},
endpoint: fileEnv.S3_ENDPOINT,
forcePathStyle: fileEnv.S3_ENABLE_PATH_STYLE,
region: fileEnv.S3_REGION || DEFAULT_S3_REGION,
});
}
Expand Down
4 changes: 1 addition & 3 deletions src/server/routers/lambda/file.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
import { TRPCError } from '@trpc/server';
import urlJoin from 'url-join';
import { z } from 'zod';

import { fileEnv } from '@/config/file';
import { AsyncTaskModel } from '@/database/server/models/asyncTask';
import { ChunkModel } from '@/database/server/models/chunk';
import { FileModel } from '@/database/server/models/file';
Expand Down Expand Up @@ -137,7 +135,7 @@ export const fileRouter = router({
embeddingError: embeddingTask?.error ?? null,
embeddingStatus: embeddingTask?.status as AsyncTaskStatus,
finishEmbedding: embeddingTask?.status === AsyncTaskStatus.Success,
url: urlJoin(fileEnv.NEXT_PUBLIC_S3_DOMAIN!, item.url!),
url: getFullFileUrl(item.url!),
};
});
}),
Expand Down
2 changes: 1 addition & 1 deletion src/server/utils/files.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,5 @@ import { fileEnv } from '@/config/file';
export const getFullFileUrl = (url?: string | null) => {
if (!url) return '';

return urlJoin(fileEnv.NEXT_PUBLIC_S3_DOMAIN!, url);
return urlJoin(fileEnv.S3_PUBLIC_DOMAIN!, url);
};
4 changes: 4 additions & 0 deletions src/services/file/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,10 @@ export class ServerService implements IFileService {
return lambdaClient.file.createFile.mutate({ ...params, knowledgeBaseId } as CreateFileParams);
}

/**
* @deprecated
* @param id
*/
async getFile(id: string): Promise<FilePreview> {
if (!fileEnv.NEXT_PUBLIC_S3_DOMAIN) {
throw new Error('fileEnv.NEXT_PUBLIC_S3_DOMAIN is not set while enable server upload');
Expand Down

0 comments on commit 1658403

Please sign in to comment.