Skip to content

Pre merge feat builtin knowledge #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
hllshiro wants to merge 125 commits into from
Closed

Pre merge feat builtin knowledge #3

hllshiro wants to merge 125 commits into from

Conversation

@hllshiro
Copy link
Owner

@hllshiro hllshiro commented Jul 15, 2025

Pull Request Description (中文)

你的功能请求是否与某个问题有关？请描述一下。
请对问题进行清晰扼要的描述。

  • 增加了内置知识库实现,允许用户在本地创建、管理和检索知识，为 DeepChat 带来了离线环境下的 RAG (Retrieval-Augmented Generation) 能力。
  • 优化了部分ollama相关的逻辑。
  • 将FilePresenter的getContent设置为public，用于文本原始内容提取。
  • 修改了大量i18n相关内容。

桌面应用程序的 UI/UX 更改
如果此 PR 引入了 UI/UX 更改，请详细描述它们。
image
image
image
image

平台兼容性注意事项
如果此 PR 具有特定的平台兼容性考虑因素（Windows、macOS、Linux），请在此处描述。

  • 是否有任何平台特定的行为或代码调整？duckdb 的平台兼容性问题
  • 你是否已在所有相关平台上进行过测试？否，仅测试windows平台

新增依赖

  • @duckdb/node-api: 用于本地向量数据库。
  • @langchain/core & @langchain/textsplitters: 用于文本处理和分块。
  • dayjs: 用于日期时间格式化。
  • scripts/postinstall.js: 增加了 duckdb 扩展的安装脚本，在安装依赖时将 vss 扩展拷贝到 runtime 目录下，构建时作为额外资源打包，避免运行时重复下载。

已知问题

  • “重排序模型“ 目前未实现相关逻辑，选项暂时隐藏
  • 文本切分方式只有一种，后续会扩展
  • 上传文件后，文档处理过程中关闭应用，重启后无法恢复任务，后续会优化
  • 目前的相似度算法可能不适用于所有向量模型，导致出现远大于1的相似度值，后续会优化

Summary by CodeRabbit

  • New Features

    • Introduced a built-in knowledge base system for local storage, retrieval, and management of knowledge files.
    • Added a user interface for configuring, uploading, searching, and managing built-in knowledge bases and files.
    • Enabled similarity search within uploaded knowledge files using local embedding models and vector database.
    • Provided auto-detection of embedding dimensions and normalization options.
    • Integrated support for multiple languages in the user interface and settings.

  • Enhancements

    • Added advanced configuration options for chunking, overlap, and fragment numbers in knowledge base settings.
    • Improved file status tracking and real-time updates during upload and processing.
    • Centralized MIME type icon mapping for file display consistency.

  • Bug Fixes

    • Corrected filtering logic in model selection and improved error handling for file operations.

  • Documentation

    • Added comprehensive documentation for the built-in knowledge base architecture, design, and workflows.

  • Localization

    • Extended localization support for built-in knowledge base features across multiple languages.

hllshiro and others added 30 commits June 18, 2025 15:46
@hllshiro
(WIP) feat: add Builtin Knowledge Server and settings integration
32061c8
@hllshiro
Merge branch 'dev' into feat-builtin-knowledge
fb14b0d
@hllshiro
feat: add multiple languages translation
f2aa990
@hllshiro
feat: enhance BuiltinKnowledgeSettings with model selection and updat…
e6abcb6 
…e translations
@hllshiro
feat: update BuiltinKnowledgeSettings with enhanced configuration opt…
a85db7e 
…ions and translations
@hllshiro
feat: update knowledge base settings to use 'builtinKnowledge' and en…
bb80a7e 
…hance BuiltinKnowledgeSettings with URL query parameter handling
@hllshiro
feat: enhance BuiltinKnowledgeSettings with model selection and error…
8976ede 
… handling for missing models
@hllshiro
feat: add confirmation dialog and error messages for removing built-i…
dd28483 
…n knowledge configurations
@hllshiro
Merge branch 'dev' into feat-builtin-knowledge
c9c7752
@sqsyli
props
b4e1226
@hllshiro
[WIP] feat: implement KnowledgePresenter and related embedding functi…
226aaaf 
…onality
@hllshiro
[WIP] feat: add KnowledgeConfHelper for managing knowledge base confi…
638bb7c 
…gurations
@hllshiro
[WIP] feat: log new knowledge config additions in KnowledgePresenter
ba9a31f
@hllshiro
[WIP] feat: enhance knowledge base settings and descriptions across c…
a46611d 
…omponents
@hllshiro
[WIP] feat: enhance Built-in Knowledge settings and descriptions, add…
71ffa14 
… advanced options and tooltips
@hllshiro
Merge branch 'feat-builtin-knowledge' of github.com:hllshiro/deepchat…
3d42d48 
… into feat-builtin-knowledge
@hllshiro
[WIP] feat: add dimensionsHelper to settings for better user guidance…
54610c1 
… on embedding dimensions
@hllshiro
[WIP] feat: add getDimensions method and update embedding handling ac…
16c5e44 
…ross providers
@hllshiro
[wip] feat: enhance embedding handling by adding error handling and r…
c18a71b 
…esetting model selection in settings
@hllshiro
[WIP] feat: refactor embedding handling to use modelId and providerId…
a056121 
…, enhance KnowledgePresenter integration
@hllshiro
[WIP] feat: update KnowledgePresenter and LLMProviderPresenter to imp…
4e2b5bb 
…rove embedding handling and error logging
@hllshiro
[WIP] feat: enhance BuiltinKnowledgeSettings with additional paramete…
dd0e3cf 
…rs and loading logic for better user experience
@hllshiro
[WIP] feat: enhance KnowledgePresenter to handle deleted configs and …
394b0cf 
…improve reset logic
@hllshiro
[WIP] feat: update LLMProviderPresenter and OllamaProvider to enhance…
3eaf865 
… model listing with additional configuration properties
@hllshiro
[WIP] feat: enhance Ollama model integration by updating local models…
0b6089b 
… to include dynamic configuration retrieval
@hllshiro
[WIP] fix: update getRagApplication to include baseURL in Embeddings …
258f6ef 
…instantiation
@hllshiro
[WIP] feat: update getDimensions method to return structured response…
3116785 
… with error handling
@hllshiro
[WIP] feat: enhance BuiltinKnowledgeSettings with dynamic dimension d…
42c1af3 
…etection and loading state
@hllshiro
feat: add duration to toast notifications for improved user feedback
02f8047
@sqsyli
feat: add BuiltinKnowledge file upload box
a7c6370
hllshiro added 28 commits July 11, 2025 09:06
@hllshiro
feat: add crash reporter initialization for error tracking
3becdb4
@hllshiro
fix: enhance logging and error handling in DuckDBPresenter methods
76e657b
@hllshiro
fix: move crash reporter initialization inside logging check
555e62c
@hllshiro
Merge remote-tracking branch 'origin/dev' into pre-merge-feat-builtin…
be1df0d 
…-knowledge

# Conflicts:
#	package.json
#	src/main/index.ts
#	src/main/presenter/index.ts
#	src/main/presenter/llmProviderPresenter/index.ts
#	src/renderer/src/events.ts
@hllshiro
feat: add toast messages for model status and L2 normalization suppor…
34aaf7f 
…t in multiple languages
@hllshiro
refactor: simplify fileTask method by removing unnecessary promise wr…
c70da9d 
…apping and adding comments
@hllshiro
refactor: update model handling by removing unnecessary ModelConfig r…
af9163d 
…eferences and enhancing model info structure
@hllshiro
fix: update company name format in crash reporter configuration
5c86c50
@hllshiro
fix: fix embedding model default settings and revert ModelConfigItem …
dc1b1e6 
…changed
@hllshiro
fix: cancel crash report
cb43391
@hllshiro
fix: fix pulling model type not assignable problem
88c3915
@hllshiro
fix: remove unneccessary files
a883e9f
@hllshiro
fix: remove unnecessary files
3cfb171
@hllshiro
fix: block option rerank model (not implemented yet)
374bf45
@hllshiro
fix: dynamically decide whether to show model customization configura…
564a985 
…tion button
@hllshiro
Merge branch 'dev' into pre-merge-feat-builtin-knowledge
dc849d7
@hllshiro
fix: remove useless i18n translations
9b5f8eb
@hllshiro
Merge remote-tracking branch 'origin/pre-merge-feat-builtin-knowledge…
bbaa9b2 
…' into pre-merge-feat-builtin-knowledge
@hllshiro
fix: remove useless dependencies and improve definitions
54403d5
@hllshiro
perf: imporve knowledgePresenter resource release
dee1b7e
@hllshiro
perf: convert to async function for better error handling
aa2eaea
@hllshiro
perf: convert to async function for better error handling
85162bb
@hllshiro
perf: imporve vector utils
8df6adc
@hllshiro
fix: fix error words
014eb4b
@hllshiro
perf: mark the interrupted task as a user-cancelled task when app sta…
c514d99 
…rtup
@hllshiro
perf: add try-catch to enhance program stability
7166690
@hllshiro
fix: declared but never read error
fac2eb4
@hllshiro
fix: missing attr file_id when insert vector(s)
25ef9cb
@hllshiro hllshiro closed this Jul 22, 2025
@hllshiro hllshiro deleted the pre-merge-feat-builtin-knowledge branch July 25, 2025 02:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@hllshiro @sqsyli