Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Llm #431

Merged
merged 50 commits into from
Nov 7, 2023
Merged

Llm #431

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
816ed87
update kenlm model.
shibing624 Oct 29, 2023
bed879d
add llm model for correction.
shibing624 Oct 30, 2023
2387a1c
update demo.
shibing624 Oct 30, 2023
9bcc996
update readme.
shibing624 Oct 30, 2023
5996abe
update tokenizer.
shibing624 Oct 30, 2023
d800970
update args
Oct 30, 2023
ebc63c1
update demo.
shibing624 Oct 30, 2023
1e92a24
update infer example.
shibing624 Oct 30, 2023
3579c3d
update infer example.
shibing624 Oct 30, 2023
06dbe3a
update do sample.
shibing624 Oct 31, 2023
ddb068b
update args.
shibing624 Oct 31, 2023
b70dd4d
update chatglm3.
shibing624 Oct 31, 2023
822df79
update chatglm3 output layer.
shibing624 Oct 31, 2023
4f86027
update optim saved.
shibing624 Nov 1, 2023
06fb49c
update gpt training with eval.
shibing624 Nov 2, 2023
fc0af76
update template.
shibing624 Nov 2, 2023
39d2a24
update test case.
shibing624 Nov 2, 2023
4fd620a
update infer demo.
shibing624 Nov 2, 2023
d498a27
update infer demo.
shibing624 Nov 2, 2023
4245ab9
update infer demo.
shibing624 Nov 2, 2023
99e16a7
update origin model.
shibing624 Nov 2, 2023
f5b6f80
update origin model for predict.
shibing624 Nov 2, 2023
c6e9257
update origin model for predict.
shibing624 Nov 2, 2023
3f87b6b
update chat result.
shibing624 Nov 2, 2023
4f38e20
update chat result.
shibing624 Nov 2, 2023
a807da4
update merge.
shibing624 Nov 3, 2023
8e83647
update t5 and conv seq2seq model.
shibing624 Nov 3, 2023
38706e3
update conv seq2seq.
shibing624 Nov 5, 2023
bbf8d44
update correct result format.
shibing624 Nov 6, 2023
8c345e9
update model predict demo.
shibing624 Nov 6, 2023
aa92f5c
update model predict demo.
shibing624 Nov 6, 2023
a80fd7b
update model eval.
shibing624 Nov 6, 2023
524262b
update model eval.
shibing624 Nov 6, 2023
3ee744d
update model test.
shibing624 Nov 6, 2023
1628ee8
update eval test.
shibing624 Nov 6, 2023
9b7b80c
update eval.
shibing624 Nov 6, 2023
1ebc3e6
update mkdir for deep context lm.
shibing624 Nov 6, 2023
a96cbc4
update eval
shibing624 Nov 6, 2023
2023381
update deep context lm train.
shibing624 Nov 6, 2023
c90133e
update vocab message.
shibing624 Nov 6, 2023
46f89e4
update train demo.
shibing624 Nov 6, 2023
17554f8
update device
shibing624 Nov 6, 2023
cf1e750
update t5 batch predict.
shibing624 Nov 6, 2023
66c66e9
update t5 batch predict.
shibing624 Nov 6, 2023
1283dee
update get errors.
shibing624 Nov 6, 2023
f31ac48
update get errors.
shibing624 Nov 6, 2023
529c410
update readme.
shibing624 Nov 6, 2023
da78d04
update lm max length.
shibing624 Nov 6, 2023
0201ade
update readme.
shibing624 Nov 6, 2023
0ba3b5d
update shibing624/chatglm3-6b-csc-chinese-lora model.
shibing624 Nov 7, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update correct result format.
  • Loading branch information
shibing624 committed Nov 6, 2023
commit bbf8d447ba2098a65b19eb5017021fc32b76df64
27 changes: 0 additions & 27 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -554,33 +554,6 @@ query:我的家乡是有明的渔米之乡 => 我的家乡是有名的渔米之

```

#### 使用PaddleNLP库调用纠错:

可以使用PaddleNLP提供的Taskflow工具来对输入的文本进行一键纠错,具体使用方法如下:

```python

from paddlenlp import Taskflow

text_correction = Taskflow("text_correction")
text_correction('遇到逆竟时,我们必须勇于面对,而且要愈挫愈勇,这样我们才能朝著成功之路前进。')
text_correction('人生就是如此,经过磨练才能让自己更加拙壮,才能使自己更加乐观。')

```

output:

```shell

[{'source': '遇到逆竟时,我们必须勇于面对,而且要愈挫愈勇,这样我们才能朝著成功之路前进。',
'target': '遇到逆境时,我们必须勇于面对,而且要愈挫愈勇,这样我们才能朝著成功之路前进。',
'errors': [{'position': 3, 'correction': {'竟': '境'}}]}]

[{'source': '人生就是如此,经过磨练才能让自己更加拙壮,才能使自己更加乐观。',
'target': '人生就是如此,经过磨练才能让自己更加茁壮,才能使自己更加乐观。',
'errors': [{'position': 18, 'correction': {'拙': '茁'}}]}]

```

### Bart模型

Expand Down
3,011 changes: 0 additions & 3,011 deletions examples/data/CGED/CGED16_HSK_Test_Input.txt

This file was deleted.

Loading