Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

更新细胞词库 #16

Merged
merged 157 commits into from
Jul 27, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
157 commits
Select commit Hold shift + click to select a range
4452964
修改注音
gaboolic Jul 2, 2024
d738f47
分词
gaboolic Jul 2, 2024
6dde1ec
计算方法修改
gaboolic Jul 2, 2024
2d7b373
分词
gaboolic Jul 3, 2024
fcade0c
1
gaboolic Jul 3, 2024
af3e4db
分词
gaboolic Jul 3, 2024
022003a
更新日志
gaboolic Jul 4, 2024
e81791a
多音字
gaboolic Jul 4, 2024
b5832b6
多音字
gaboolic Jul 4, 2024
ddd2222
生成字频词频
gaboolic Jul 4, 2024
b89fac0
使用知乎频率重做词频
gaboolic Jul 4, 2024
495b4b3
1
gaboolic Jul 4, 2024
8b68cf0
1
gaboolic Jul 4, 2024
e101e5f
词频
gaboolic Jul 5, 2024
a392991
1
gaboolic Jul 5, 2024
af1cf1c
1
gaboolic Jul 5, 2024
1e58860
# todo 按音节、按频率排序
gaboolic Jul 5, 2024
d4dacf3
维基词频
gaboolic Jul 6, 2024
1e916d7
维基词频
gaboolic Jul 6, 2024
665a2e1
修词频
gaboolic Jul 7, 2024
bad5f5d
字表排序
gaboolic Jul 7, 2024
e0480f0
删词
gaboolic Jul 7, 2024
1bfec54
更新多音字
gaboolic Jul 8, 2024
2b0d7e0
删词加词
gaboolic Jul 8, 2024
bde8713
加词
gaboolic Jul 8, 2024
9bd3382
词频745396750字
gaboolic Jul 8, 2024
c0e8c56
词频745396750字
gaboolic Jul 8, 2024
9f4ad8c
字频
gaboolic Jul 8, 2024
42218f1
更新readme
gaboolic Jul 8, 2024
80d4d88
readme
gaboolic Jul 8, 2024
963e23a
棒杀
gaboolic Jul 8, 2024
4b2dd95
删词
gaboolic Jul 8, 2024
7739366
更新readme
gaboolic Jul 8, 2024
4b473d7
1
gaboolic Jul 8, 2024
d98874e
更新readme
gaboolic Jul 10, 2024
8f2afd0
Merge branch 'master' of https://github.com/gaboolic/rime-frost into …
gaboolic Jul 10, 2024
9af426b
处理arpa
gaboolic Jul 11, 2024
7474ad6
1
gaboolic Jul 11, 2024
b34bff2
readme
gaboolic Jul 12, 2024
d95d9b4
更新readme
gaboolic Jul 12, 2024
2d1464e
Merge branch 'master' of https://github.com/gaboolic/rime-frost into …
gaboolic Jul 12, 2024
814ba21
修改配方
gaboolic Jul 12, 2024
fc5fc49
修改词库名称
gaboolic Jul 12, 2024
89e670f
词频
gaboolic Jul 13, 2024
9066e70
词频、象棋词库
gaboolic Jul 13, 2024
4d42e45
1
gaboolic Jul 13, 2024
8e4a426
1
gaboolic Jul 13, 2024
dfdc261
1
gaboolic Jul 13, 2024
be61de9
1
gaboolic Jul 13, 2024
e970a38
Merge branch 'master' of https://github.com/gaboolic/rime-frost into …
gaboolic Jul 13, 2024
0e19016
加细胞词库
gaboolic Jul 14, 2024
d8078c4
词库
gaboolic Jul 14, 2024
4156915
name
gaboolic Jul 14, 2024
af517f5
五子棋
gaboolic Jul 14, 2024
3134965
拼音测试
gaboolic Jul 14, 2024
045cbfc
数理化
gaboolic Jul 14, 2024
90f22f4
地理
gaboolic Jul 14, 2024
d573fce
1
gaboolic Jul 14, 2024
29692c2
细胞词库
gaboolic Jul 14, 2024
19850a2
细胞词库
gaboolic Jul 14, 2024
ebd0da9
多音字
gaboolic Jul 14, 2024
24eceb4
多音字处理
gaboolic Jul 14, 2024
dadb9c6
去重
gaboolic Jul 15, 2024
7558afe
薰字频率
gaboolic Jul 15, 2024
a61dda9
Merge branch 'zhi-freq-dev' of https://github.com/gaboolic/rime-frost…
gaboolic Jul 15, 2024
edc1e24
细胞词库去重
gaboolic Jul 15, 2024
5182b08
去重
gaboolic Jul 15, 2024
cb79e4f
注音
gaboolic Jul 15, 2024
66033bc
细胞词库去重
gaboolic Jul 15, 2024
29892c9
降频词
gaboolic Jul 15, 2024
28fd6a3
细胞词库
gaboolic Jul 15, 2024
7c38eb5
腾讯词库注音
gaboolic Jul 15, 2024
3b4742b
腾讯词库注音
gaboolic Jul 15, 2024
0e7e240
腾讯词库注音
gaboolic Jul 15, 2024
34a7f80
腾讯词库去重
gaboolic Jul 15, 2024
e8cb7d0
腾讯词库去重
gaboolic Jul 15, 2024
114ee50
细胞词库去重
gaboolic Jul 15, 2024
14bdcc8
皮肤设置
gaboolic Jul 15, 2024
15c32b3
去掉超级简拼
gaboolic Jul 15, 2024
46ed867
- lua_filter@*is_in_user_dict # 用户词典的词加上一个*
gaboolic Jul 15, 2024
2152a11
1
gaboolic Jul 15, 2024
d6fd9bc
1
gaboolic Jul 15, 2024
c530736
1
gaboolic Jul 15, 2024
d787916
1
gaboolic Jul 15, 2024
091244a
1
gaboolic Jul 15, 2024
5a3e326
+词
gaboolic Jul 16, 2024
2e7cc64
1
gaboolic Jul 16, 2024
de5c490
词频
gaboolic Jul 16, 2024
dbe37a0
词频
gaboolic Jul 16, 2024
a3d1633
分词词库构建
gaboolic Jul 16, 2024
334eb17
细胞词库
gaboolic Jul 16, 2024
812d586
地名
gaboolic Jul 16, 2024
43d7424
频率统计
gaboolic Jul 16, 2024
13ff308
1
gaboolic Jul 17, 2024
2c814d8
词频修改
gaboolic Jul 17, 2024
96eebd8
1
gaboolic Jul 17, 2024
e9d772f
1
gaboolic Jul 17, 2024
d79271b
细胞词库词频
gaboolic Jul 17, 2024
3775530
1
gaboolic Jul 17, 2024
0a75862
Merge branch 'master' of https://github.com/gaboolic/rime-frost into …
gaboolic Jul 17, 2024
44e34b2
1
gaboolic Jul 17, 2024
b4a15bf
1
gaboolic Jul 17, 2024
f4a7a12
清洗数据
gaboolic Jul 17, 2024
102fffb
1
gaboolic Jul 17, 2024
b046b02
整理细胞词库
gaboolic Jul 17, 2024
4b336c1
todo 统计中文语料中的英文词频
gaboolic Jul 17, 2024
fc22629
Merge branch 'master' of https://github.com/gaboolic/rime-frost into …
gaboolic Jul 17, 2024
bdb3d9c
1
gaboolic Jul 17, 2024
66adea1
1
gaboolic Jul 17, 2024
9184e18
1
gaboolic Jul 17, 2024
784ea77
1
gaboolic Jul 17, 2024
c2ebe74
1
gaboolic Jul 17, 2024
00f8032
1
gaboolic Jul 18, 2024
45dace9
细胞词库
gaboolic Jul 18, 2024
f09d921
1
gaboolic Jul 18, 2024
b78c35f
1
gaboolic Jul 18, 2024
908153f
1
gaboolic Jul 18, 2024
8d419fc
1
gaboolic Jul 19, 2024
a3497e7
1
gaboolic Jul 19, 2024
7fc93cb
1
gaboolic Jul 19, 2024
104dfc1
1
gaboolic Jul 19, 2024
c0f1a8c
1
gaboolic Jul 19, 2024
6b843bf
修改细胞词库
gaboolic Jul 19, 2024
567d2b3
1
gaboolic Jul 19, 2024
1dd008a
1
gaboolic Jul 19, 2024
2d49a1f
词频
gaboolic Jul 19, 2024
1713dfc
1
gaboolic Jul 19, 2024
2185f38
1
gaboolic Jul 19, 2024
97a9274
Merge branch 'master' of https://github.com/gaboolic/rime-frost into …
gaboolic Jul 19, 2024
d78c6b0
1
gaboolic Jul 19, 2024
c002ea3
词频修改
gaboolic Jul 19, 2024
50fd753
词频修改
gaboolic Jul 19, 2024
87a86cc
1
gaboolic Jul 19, 2024
c07f82b
1
gaboolic Jul 19, 2024
caca877
1
gaboolic Jul 19, 2024
08d41b3
1
gaboolic Jul 19, 2024
8de9782
加入运动词库
gaboolic Jul 23, 2024
b43f2d6
去重
gaboolic Jul 23, 2024
a0bab38
add todo
gaboolic Jul 23, 2024
6845d8a
1
gaboolic Jul 23, 2024
bfe6708
1
gaboolic Jul 25, 2024
81ad4c0
动物
gaboolic Jul 25, 2024
23f0e62
词频修改
gaboolic Jul 25, 2024
754ab3d
1
gaboolic Jul 25, 2024
3412282
1
gaboolic Jul 25, 2024
0c6ee0a
1
gaboolic Jul 25, 2024
8ad4e24
1
gaboolic Jul 25, 2024
30b0f5c
1
gaboolic Jul 25, 2024
6198025
1
gaboolic Jul 25, 2024
309c50a
1
gaboolic Jul 25, 2024
c15ba8a
词频修改
gaboolic Jul 25, 2024
02fa91b
词频修改
gaboolic Jul 25, 2024
cace339
1
gaboolic Jul 25, 2024
e7e3353
.zip忽略
gaboolic Jul 27, 2024
12c5e40
Merge branch 'master' of https://github.com/gaboolic/rime-frost into …
gaboolic Jul 27, 2024
bfdf518
分词
gaboolic Jul 27, 2024
bbed37e
细胞词库
gaboolic Jul 27, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
1
  • Loading branch information
gaboolic committed Jul 17, 2024
commit 96eebd8b9cc19fb05eacc240657e74b731e5b7c8
2 changes: 1 addition & 1 deletion cn_dicts/8105.dict.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7100,7 +7100,6 @@ sort: by_weight
茓 xue 0
峃 xue 0
寻 xun 1139
薰 xun 794
熏 xun 732
训 xun 712
逊 xun 487
Expand All @@ -7113,6 +7112,7 @@ sort: by_weight
荀 xun 181
旬 xun 146
殉 xun 139
薰 xun 79
询 xun 72
迅 xun 71
洵 xun 55
Expand Down
18 changes: 9 additions & 9 deletions cn_dicts/base.dict.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2317,7 +2317,7 @@ sort: by_weight
阿扑吗啡 a pu ma fei 2
阿普唑仑 a pu zuo lun 65
阿普唑仑片 a pu zuo lun pian 2
阿七 a qi 72
阿七 a qi 7
阿奇 a qi 72
阿奇霉素分散片 a qi mei su fen san pian 1
阿强 a qiang 252
Expand Down Expand Up @@ -84627,7 +84627,7 @@ sort: by_weight
盖屋 gai wu 8
该系 gai xi 49
改下 gai xia 152
垓下 gai xia 212
垓下 gai xia 21
盖下 gai xia 21
改下去 gai xia qu 8
该县 gai xian 70
Expand Down Expand Up @@ -127232,7 +127232,7 @@ sort: by_weight
加林查 jia lin cha 71
佳林路 jia lin lu 1
驾龄 jia ling 171
贾玲 jia ling 628
贾玲 jia ling 62
假令 jia ling 10
驾凌 jia ling 2
假领 jia ling 3
Expand Down Expand Up @@ -143970,7 +143970,7 @@ sort: by_weight
九眼 jiu yan 5
久演不衰 jiu yan bu shuai 4
九眼桥 jiu yan qiao 150
九阳 jiu yang 1007
九阳 jiu yang 100
久仰 jiu yang 39
久仰大名 jiu yang da ming 30
九阳股份有限公司 jiu yang gu fen you xian gong si 1
Expand Down Expand Up @@ -150231,7 +150231,7 @@ sort: by_weight
烤鸭 kao ya 924
烤鸭店 kao ya dian 34
考验 kao yan 9003
考研 kao yan 50552
考研 kao yan 5055
烤烟 kao yan 20
考研班 kao yan ban 134
考研报名 kao yan bao ming 131
Expand Down Expand Up @@ -154394,7 +154394,7 @@ sort: by_weight
酷冷至尊 ku leng zhi zun 16
苦力 ku li 709
酷吏 ku li 263
库里 ku li 6655
库里 ku li 665
库利克 ku li ke 6
库里斯 ku li si 3
苦练 ku lian 1025
Expand Down Expand Up @@ -207469,7 +207469,7 @@ sort: by_weight
千张 qian zhang 88
欠账 qian zhang 119
千丈 qian zhang 44
前掌 qian zhang 219
前掌 qian zhang 21
# 前张 qian zhang 0
牵张反射 qian zhang fan she 29
钱掌柜 qian zhang gui 31
Expand Down Expand Up @@ -245877,7 +245877,7 @@ sort: by_weight
爽片 shuang pian 195
双片 shuang pian 14
双拼 shuang pin 285
双频 shuang pin 321
双频 shuang pin 32
双屏 shuang ping 190
双平 shuang ping 3
爽气 shuang qi 17
Expand Down Expand Up @@ -362304,7 +362304,7 @@ sort: by_weight
周树人 zhou shu ren 84
周顺昌 zhou shun chang 1
周四 zhou si 1056
宙斯 zhou si 1531
宙斯 zhou si 153
宙斯盾 zhou si dun 32
周四前 zhou si qian 3
宙斯神 zhou si shen 18
Expand Down
7 changes: 6 additions & 1 deletion others/program/reduce_freq_base_to_zhifreq.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ def find_english_words(string):
with open(os.path.join('others', '降频词.txt'), 'r', encoding='utf-8') as dict_file:
for line in dict_file:
line = line.strip()
if not '\t' in line or line.startswith("#"):
if line.startswith("#"):
continue
need_to_reduce_freq_word_map[line] = ''

Expand Down Expand Up @@ -68,9 +68,14 @@ def find_english_words(string):
encoding = params[1]
freq = 0
if character in word_freq_map:
if character == '薰':
print(int(word_freq_map[character]))
freq = int(word_freq_map[character])
if character in need_to_reduce_freq_word_map or character +'\t' + encoding in need_to_reduce_freq_word_map:
freq = freq//10
if character == '薰':
print(freq)
print(character in need_to_reduce_freq_word_map)

if character+encoding in char_yin_freq_map:
if character == '她':
Expand Down