-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
【New feature】新增黑客松小助手 #37
Changes from 3 commits
3223813
e655be8
7bb4f21
c75d620
99df700
b9e39a1
583de9f
4f6a748
069f0d7
cf3978a
ea64705
b899fcf
e1653cd
a9ea5fd
f7b6737
f9dbfcc
2f13a30
cc812c5
3fb01cb
d3883c0
da23a93
7d11255
be6ddeb
2784c76
d8ebbc9
6b055da
e6aa11b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
__pycache__ |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,88 @@ | ||
# 黑客松小助手RFC | ||
|
||
## 📄 背景 | ||
|
||
💡💡💡 `Hackathon5`开赛在即,本次大赛会有上百个赛题等着大家攻克,但统计这些赛题的报名信息和当前进度是一项繁重的工作 (感谢 @花花!🍻)。 | ||
|
||
如果有一个小助手💡可以帮助我们自动统计和展示赛题进度,那自然是极好的,所以`黑客松小助手`它带着期盼走来了!🎉🎉🎉 | ||
|
||
## 🚩 功能 | ||
|
||
`黑客松小助手`目前的主要功能如下,欢迎各位大佬前来补充: | ||
|
||
* 根据issue回复自动填写报名信息,完成任务认领。 | ||
* 根据提交的PR状态,自动更新issue中表单信息,完成状态变更。 | ||
* 汇总所有的赛题,形成总体看板。 | ||
|
||
## 🚩 实现方案 | ||
|
||
### 🚀 状态 | ||
|
||
在比赛期间,我们为参加比赛的大佬们设置了`五种状态`(如下),整体状态的变更顺序为:`报名状态` -> `提交RFC` -> `完成设计文档` -> `提交PR` -> `完成任务`。 | ||
|
||
| 状态名称 | 状态标志 | 状态说明 | | ||
| :--------------: | :----------------------------------------------------------: | :-------------------------: | | ||
| 报名状态 | <img src="https://img.shields.io/badge/状态-报名-2ECC71" /> | 表明通过issue评论区进行报名 | | ||
| 提交RFC状态 | <img src="https://img.shields.io/badge/状态-提交RFC-F1C40F" /> | 表明已经提交设计方案RFC | | ||
| 完成设计文档状态 | <img src="https://img.shields.io/badge/状态-完成设计文档-3498DB" /> | 表明已经完成设计文档 | | ||
| 提交PR状态 | <img src="https://img.shields.io/badge/状态-提交PR-F39C12" /> | 表明已经提交PR | | ||
| 完成任务状态 | <img src="https://img.shields.io/badge/状态-完成任务-9B59B6" /> | 表明已经完成任务 | | ||
|
||
### 🚀 榜单设计 | ||
|
||
为了更好地展示赛题进度,我们设计了榜单。整体的榜单设计如下,比赛进行期间,小助手会自动更新`Github ID/状态/PR`栏和`完成队伍`栏信息。 | ||
|
||
![image-20230729121046527](./images/1.png) | ||
|
||
`Github ID/状态/PR`栏内容的格式为`Github ID + 状态 + Pr`,其中`PR`可以有多个,`赛题参加人数`也可以有多个,以回车`<br>`分隔。 | ||
|
||
### 🚀 报名信息监控 | ||
|
||
考虑到报名赛题不会产生任何`PR`,所以通过`评论`来实现此功能,其他状态均可以通过监控`PR`实现。 | ||
|
||
#### 实现逻辑 | ||
|
||
通过监控`issue`下的评论完成报名信息监控,实现逻辑如下: | ||
|
||
* 获取`issue`下评论。 | ||
* 从评论中抽取报名信息,比如`Github ID`、`报名赛题编号`等信息。 | ||
* 更新`issue`中个人报名信息,将个人状态更新为`报名状态`。 | ||
|
||
#### 报名格式 | ||
|
||
为了自动填写报名信息,需要在`issue`下回复报名信息,格式如下: | ||
|
||
``` | ||
【状态】: 报名 | ||
【序号】:2、3 | ||
``` | ||
|
||
> 其中`【状态】: 报名`表示当前评论是赛题`报名`,`序号`表示报名的赛题序号,多个赛题之间需要用`中文顿号、`分隔。 | ||
|
||
### 🚀 其他状态变更监控 | ||
|
||
除`报名状态`外,剩下`四种状态`的变更可以通过监控`PR`的状态来完成, 具体的实现逻辑如下: | ||
|
||
* 获取`paddle`仓库下黑客松`开始之后`标题中包含`Hackathon No.`字样的所有`PR`。 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 这里有两点问题需要考虑:
建议是维护一个数据库,每天爬一次全量pr即可,黑客松的pr打标签,比如paddle下我们使用 PaddlePaddle Hackathon 标签来管理黑客松的pr,可以要求其他repo也使用同样的标签。 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. OpenVINO不是PaddlePaddle组织下的repo,是intel的项目,所以我们是不是考虑设置一个repo范围(比如一个list),并且可以灵活增减 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 好的,可以设置一个repo范围,这样就可以动态操作监控的repo |
||
* 如果`PR`是`PaddlePaddle/community`仓库下的,说明该PR与设计文档有关。继续判断是否`merge`,如果未`merge`,说明状态为`提交RFC`;如果已经`merge`,说明状态为`完成设计文档`。 | ||
* 如果`PR`不是`PaddlePaddle/community`仓库下的,说明该`PR`与提交代码有关。继续判断是否`merge`,如果未`merge`,说明状态为`提交PR`;如果已经`merge`,说明状态为`完成任务`。 | ||
|
||
#### PR格式 | ||
|
||
为了完成状态变更,只需要在`PR`的标题中以`【Hackathon No.xxx】`开头即可,程序会自动提取赛题编号并更新榜单。 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 这里需要考虑的特殊情况:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 所以这个定时任务每次跑都是:
行倒是也行,就是感觉有点暴力,如果榜单能做到增量更新最好,这样也留有人工修改issue的空间 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 目前的逻辑确实是这样的,每次都是在重写榜单,如果想要增量更新,可以更新后记录更新时间,下次更新时只改变上一次更新时间之后的回复和PR。 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 我觉得不是修改查询时间来解决的,这个时间肯定是需要有overlap的,或者是全量的去查也没有问题,但是需要记录上一次的状态,和这一次做对比,然后拿到diff,更新到表单上 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 根据讨论可以先这样: 假设我们每天定时任务更新2次,0-12h更新一次,12-24h更新一次 然后计算一下API的调用频次,@gouzil 表示一个token每天最多调用60次,可能需要估算一下需要多少token 以及记得保存日志和异常情况报警 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 是不是可以考虑一下仓库的 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
|
||
## 🚩 代码结构 | ||
|
||
整体的代码文件分为两个: | ||
|
||
* `utils.py`:负责拉取评论和PR、根据评论更新状态、根据PR更新状态。 | ||
* `bot.py`:小助手整体运行逻辑。 | ||
|
||
## 🚩 未来规划 | ||
|
||
* **预警功能**:自动提醒每个赛题的状态,针对长时间没有变更状态的赛题进行预警。 | ||
huajiao-hjyp marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
* **看板功能**:后期可以扩展看板功能,进行数据分析,更好的量化开发者的贡献,类似于 [openGauss 贡献看板](https://datastat.opengauss.org/zh/overview) [一款开源的开源社区贡献看板](https://ost.51cto.com/posts/14589)。 | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 应该需要一个仓库做一下每天,或者每次执行前的备份,防止封号 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 可以,现在是执行的时候保存在一个 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 可以尝试提交到一个另一个分支上,可能会有不同的人来找备份,这部分最好还是公开一下。 可以参考: There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 好的~ |
||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,76 @@ | ||
import utils | ||
|
||
|
||
def update_issue_automatically(): | ||
# 1. 获取issue表格 | ||
response = utils.request_get_issue(issue_url) | ||
with open('C:/Users/25942/Desktop/test.txt', mode='w', encoding='utf-8') as f: | ||
f.write(response['body']) | ||
|
||
# 从issue中提取题目列表 | ||
task_list = utils.process_issue(response['body'], task_num) | ||
|
||
# 2. 根据评论更新表格 | ||
comment_url = issue_url + '/comments' | ||
comments = utils.request_get_multi(comment_url) | ||
for comment in comments: | ||
utils.update_status_by_comment(task_list, comment) | ||
|
||
with open('./test.txt', mode='w', encoding='utf-8') as f: | ||
f.write(response['body']) | ||
|
||
|
||
# 3. 根据PR更新表格 | ||
# - 根据提出的PR更新状态为已提交 | ||
# - 根据close的PR更新状态为已完成 | ||
repo_urls = ['https://api.github.com/repos/Tomoko-hjf/paddleviz/pulls'] | ||
|
||
for repo_url in repo_urls: | ||
params = { | ||
"state": "open" | ||
} | ||
pulls = utils.request_get_multi(repo_url, params) | ||
for pull in pulls: | ||
utils.update_status_by_pull(task_list, pull) | ||
|
||
params = { | ||
"state": "closed" | ||
} | ||
pulls = utils.request_get_multi(repo_url, params) | ||
for pull in pulls: | ||
if pull['merged_at']: | ||
utils.update_status_by_pull(task_list, pull) | ||
|
||
|
||
# 4. 更新榜单内容 | ||
updated_issue = response['body'] | ||
for task in task_list: | ||
num = int(task['num'].strip(' ')) | ||
start = updated_issue.find('| {} |'.format(num)) | ||
end = start + 1 | ||
while end < len(updated_issue) and updated_issue[end] != '\r' and updated_issue[end] != '\n': | ||
end += 1 | ||
|
||
# TODO:这里后期需要定制化表头 | ||
row = '| {} | {} | {} | {} | {} |'.format(num, task['difficulty'], task['issue'], task['status'], task['team']) | ||
updated_issue = f'{updated_issue[:start]}{row}{updated_issue[end:]}' | ||
|
||
|
||
# 处理换行符 | ||
updated_issue = updated_issue.replace('\r', '') | ||
with open('./test1.txt', mode='w', encoding='utf-8') as f: | ||
f.write(updated_issue) | ||
|
||
# 5. 更新 issue | ||
res = utils.request_update_issue(issue_url, updated_issue) | ||
|
||
|
||
if __name__ == '__main__': | ||
# TODO: 替换为自己的GitHub访问令牌和要获取评论的issue的信息 | ||
owner = 'PaddlePaddle' | ||
repo = 'Paddle' | ||
issue_number = '43938' | ||
task_num = 200 | ||
issue_url = 'https://api.github.com/repos/Tomoko-hjf/paddleviz/issues/1' | ||
|
||
update_issue_automatically() |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
| 序号 | 难度 | 任务 ISSUE | 队伍名称/状态/PR | 完成队伍 | | ||
| :--: | :--: |:--: | :--: |:--: | | ||
| 1 | ⭐ | [新增星星任务](https://github.com/Tomoko-hjf/paddleviz/issues/1) | @Tomoko-hjf <img src="https://img.shields.io/badge/状态-提交PR-F39C12" /> [#2](https://github.com/Tomoko-hjf/paddleviz/pull/2) <br> | | | ||
| 2 | ⭐⭐ | [保护星星](https://github.com/Tomoko-hjf/paddleviz/issues/1) | @Tomoko-hjf <img src="https://img.shields.io/badge/状态-报名-2ECC71" /> <br> | | | ||
| 3 | ⭐⭐⭐ | [收集星星](https://github.com/Tomoko-hjf/paddleviz/issues/1) | @Tomoko-hjf <img src="https://img.shields.io/badge/状态-报名-2ECC71" /> <br> | | | ||
| 4 | ⭐⭐⭐⭐ | [飞桨之星](https://github.com/Tomoko-hjf/paddleviz/issues/1) | | | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
需要考虑对不符合格式规范的回复如何处理?提供两条思路:
:
:
(btw你提供的格式里也没有区分中英文的冒号哦There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
这个确实会有风险,这里我加一个判断赛题题号的逻辑吧,如果数字大于赛题数,那么就在评论区提示用户
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
建议我们在实现的啥时候,支持非中文的[]:、,符号,以及对空格的兼容,这样可能就没有找第一个数字字符的问题存在。例如代码中可以通过以下方式实现
s = '【序号】: 2, 3'
s = s.replace(' ').replace(':',':').replace(',',‘、’).replace('[','【').replace(']','】')
这样可以获取到较为纯净的字符,格式可能也较为统一
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
定时任务更新表单和一些不规范指令识别的逻辑肯定是要分开的,定时任务一天跑两次,但是报名信息格式校验应该是实时的,或者至少一小时一次,自动格式化好像也不是不行,可以调研一下技术路径,飞桨项目有一个机器人paddle-bot用于自动回复,可以看看能不能接入
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
我们最开始的时候是不是考虑过这个机器人,但是可能漏包?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
漏包的问题现在有人在看,可以先假设这个问题修复了?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
漏包的话可以用定时兜底
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
确认了排期,漏包的问题9.8可以完成修复