merge from main #19
Open
merge from main #19
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* initial commit * fix: 修复检索模块功能并更新依赖 * fix: 更新检索模块部分功能实现 * test: 增加bm25使用配置注入后的测试 * perf: bm25检索器、索引器、base增加配置文件注入 * chore: 更新requirement.txt * feat: 对encapsulation/llm中增加了配置文件注入 * feat: 增加配置注入 * chore: 微调变量名、注释等 * feat: 增加配置注入 * chore: 修改注释 * fix: 修复了配置注入的问题 * test: 增加llm相关模块测试 * fix: 修复配置类 * fix: 修复检索相关的配置类 * feat:检索辅助类 * chore * chore * chore * fix: 配置注入 * chore: 修改配置文件 * test:所有检索器的测试 * feat: 引入@yuxuan 的配置注入代码 * chore * chore: 删除没有的文件 * fix: 配置注入修复 * perf: 增加配置注入并优化提取性能 * feat: 增加配置注入，修改为使用tsv格式输出 * test: LLM信息抽取测试 * feat: 增加配置注入 * test: neo4j测试 * test: 共享模块测试 * chore: graph extrator prompt * chore: 增加多路融合策略和检索辅助类 * chore: 删除多余的代码 * style: 修改retriever和index的初始化逻辑 * test: 所有检索的测试 * feat: 增加共享模块装饰器 * feat: 检索接口 * feat: 增加ID管理，避免重复添加文档等 * chore * test: 检索pipeline * feat: 增加了@Harver-zhou的数据库相关代码 * feat: 支持IndexManager,Index管理统一由IndexManager管理 * chore: 修复了配置文件 * fix: 由于shared_module带来self.logger无法正确初始化的问题 * chore: 删除多余import * chore * chore: 去除绝对路径模型 * feat: Graph相关代码及测试 * feat: Graph相关代码及测试 * chore * chore * chore: 迁移数据模型位置 * chore: 完善neo4j，增加向量化 * fix: 修复了huggingface模型下载不稳定的问题 * feat: graph retrieval * fix: 整合代码 * add core layer module and pipeline test * remove api key * feat: graph检索及构图 * chore: 调整配置文件路径 * chore: 修改导入路径等 * chore * add encapsulation layer module * chore: clean up * chore: rename * chore: rename * fix: 修复配置文件错误而无法启动 * fix: graph无法检索的问题 * fix: test_llm * chore * fix: 修复OOM问题，增加相应提示 * feat: 增加使用NetWorkX的本地Graph存储 * chore: clean * feat: indexing 输入json文件路径、文件路径列表，输出构建是否成功 * multiple refactor and gather config * api key load from env * fix: 导入路径，换源 * fix: 修复误删的graph_db配置文件 * fix: 修复无法正确并发的问题 * test: 测试索引 * test: 增加测试文件、index_manager的测试代码 * feat: index_manager,输入file_id，自动从数据库中获取数据，然后自动解析、切分 * chore * chore * Create .env * Refactor: encapsulation llm implementation * Refactor: parser implementation * Refactor: file storage implementation * update requirement.txt * fix: 修复导入路径(embeding rerank chat..) * chore * refactor: Document->Chunk, 增加配置文件注解 * Refactor: query rewrite and rerank * feat: add graph indexing * Refactor: file storage * fix minor issues; support env var in json config; support rag_inference in mcp * fix: index manager因file storage拆分后不能使用的问题 * test: graph 的index manager测试 * feat: index_manager增加删除整个file，其中graph为硬删除、faiss、bm25为软删除 * feat: index删除时，增加parsed contente的删除 * feat: listwise_reranker * fix:delete_file从async -> sync * fix: none处理，storage写在config中 * fix: bm25索引加载问题,delete file删除失败问题 * Merge dev/main into feat/local-dev - Resolved merge conflict in core/file_management/index_manager.py - Integrated upstream changes to index manager configuration and implementation - Preserved local changes for knowledge management and API enhancements * fix: bm25删除未成功 * Add application and api layer support. * fix: 删除失败的问题 * Delete data/unified_faiss_index directory * feat: 用户管理、消息管理、会话管理、redis * feat: 使用gpt的OCR * feat: 索引隔离，根据metadata做过滤 * Add auth support; enable websocket * update readme.md * update architecture.png * update architecture.png * add performence pic * Enhance README with performance comparison visuals Added performance comparison section with multiple images. * Add files via upload * Enhance README-CN with performance comparison section Added performance comparison images and section to README-CN. * chore: update readme.md * fix: minor bug * fix: minor bug * Add support for /knowledge/list_files. And return chunks for stream chat * feat: pruned hipporag2 * feat: 增加图谱可视化 * fix: 修复chat接口返回内容个数不正确的问题 * Fix total files content logic * fix: 增加snapshot_download，避免dots.ocr直接下载失败的问题 * chore: 调整构图、检索代码 * fix: 使用snapshot下载能够正确使用本地路径加载模型 * chore: 增加docker相关 * Update repository URL in README for cloning * 1. bug fix: config injection from .env should be str only. 2. support file bulk indexing * chore: 增加指定实体类型，导出子图样式变化 * fix: clean ocr output base64 * fix: bm25增加并发锁，定期构建，batch构建 * 1. Add support for file bulk indexing 2. optimize async logic * fix: 修复bm25锁触发方式 * feat: 新增完善的权限管理、审计、用户管理等 * update schema design * Add support for file permission control * Guard _index_multiple_files_background under the same semaphore as _index_file_background * update mcp logic * update README.md * Bug fix * update readme.md * Update MD files and start script * replace pip with uv * Bug fix; change mirror sorce for uv install in China --------- Co-authored-by: Mizzle <131973837+Mi221e@users.noreply.github.com> Co-authored-by: sun254667 <sun254667307a@163.com> Co-authored-by: 陈明祯 <chenmingzhen@idea.edu.cn> Co-authored-by: Harvey-zhou <cs.hzhou@outlook.com> Co-authored-by: Mizzle <q1326635912@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.