using the cfs to the updated extract#1145
Open
jack-wang-176 wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
描述 (Description)
本 PR 主要为项目引入了基于 Redis 的多租户公平调度器(模拟linux内核对进程分配cpu资源),解决了高频提交任务可能导致的 Asynq 底层队列饿死其他租户的问题。同时将在
container中创建的context改为在main中创建并传递,对于desktop的入口在Wails OnShutdown hook协程中添加cancel产生信号,对于server的入口在协程外创建context并通过调整进程间通信来保留原来针对多次ctrl+c的强制退出和单次的等待资源清理后退出核心变更内容:
TenantFairScheduler接口,基于 Redis ZSET (cfs:vruntime) 和 List (cfs:tenant:{tenantID}) 实现 vruntime 算力追踪,保障租户公平性。interfaces.CfsTaskOptions和interfaces.CfsTaskWrapper,解耦业务层与 Asynq 指针,确保TaskID、MaxRetry、Queue等动态配置在经过 Redis 序列化/反序列化后能被精准还原。knowledge.go与new_knowledge.go中所有直连asynq.NewTask的逻辑重构为调用 CFS 调度器,并封装了相应的Cost算力预估函数(如estimateFAQImportTaskCost)。BuildContainer中将 CFS 调度器注册到dig容器,并关联其后台轮询大循环。main.go中的全局Context,严格规范关闭顺序(释放 Listener -> 等待 HTTP 排空 -> 关闭后台 CFS 大循环 -> 执行resourceCleaner清理数据库/Redis 连接),防止产生僵尸进程或断连 Panic。变更类型 (Type of Change)
影响范围 (Scope)
测试 (Testing)
测试步骤 (Test Steps)
cfs:vruntimeZSET 分数变动以及cfs:tenant:{tenantID}List,验证新租户进入时的 vruntime 补偿机制是否生效,调度器是否按预期公平拉取任务。TaskID的场景下,验证前端或进度查询 API 是否能正常获取基于指定 ID 的任务进度(证明 Option 透传成功)。SIGINT(Ctrl+C) 信号,观察终端日志,验证 HTTP 服务器是否先停止接收请求,后台 CFS 调度器是否跳出循环,最后再执行 DB/Redis 资源清理,确保全程无connection refused相关的 Panic。检查清单 (Checklist)
相关 Issue
无
截图/录屏 (Screenshots/Recordings)
数据库迁移 (Database Migration)
配置变更 (Configuration Changes)
无新增配置项。复用现有的
REDIS_ADDR判断,当存在 Redis 环境时自动启用 CFS 调度器,否则降级为同步执行器。其他信息 (Additional Information)
目前对于文件类型和文件大小的权重计算只是通过个人经验,并没有相关数据支撑,后续可以针对这一点进行优化