Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add (M)Luke model training for Token Classification in the examples #14880

Merged
merged 24 commits into from
Jan 31, 2022
Merged

Add (M)Luke model training for Token Classification in the examples #14880

Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
062e6f4
Add Luke training
jplu Dec 20, 2021
1ebb675
Fix true label tags
jplu Dec 20, 2021
96216d1
Fix true label tags
jplu Dec 20, 2021
3a09d36
Fix true label tags
jplu Dec 20, 2021
9f352e5
Update the data collator for Luke
jplu Dec 20, 2021
d637a82
Some training refactor for Luke
jplu Dec 20, 2021
87c21c1
Improve data collator for Luke
jplu Dec 21, 2021
3e64bf8
Fix import
jplu Dec 21, 2021
8540152
Fix datasets concatenation
jplu Dec 21, 2021
446a044
Add the --max_entity_length argument for Luke models
jplu Dec 21, 2021
3cf7926
Remove unused code
jplu Dec 22, 2021
909b6b7
Fix style issues
jplu Dec 22, 2021
17cf050
Fix style issues
jplu Dec 22, 2021
11bd52a
Move the Luke training into a separate folder
jplu Dec 23, 2021
631f8a9
Fix style
jplu Dec 23, 2021
6f1b0b1
Fix naming
jplu Dec 23, 2021
77e27c4
Fix filtering
jplu Dec 23, 2021
736bd66
Fix filtering
jplu Dec 23, 2021
80c8d04
Fix filter
jplu Dec 23, 2021
5784c9c
Update some preprocessing
jplu Jan 26, 2022
74823d0
Move luke to research_projects
jplu Jan 26, 2022
0ac462d
Checkstyle
jplu Jan 26, 2022
b15c101
Address comments
jplu Jan 31, 2022
0e19986
Fix style
jplu Jan 31, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix true label tags
  • Loading branch information
@jplu
jplu committed Jan 26, 2022
commit 3a09d367cee9101b20f8b908583754adb271de08
2 changes: 1 addition & 1 deletion examples/pytorch/token-classification/run_ner_no_trainer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -706,7 +706,7 @@ def get_luke_labels(outputs, ner_tags, original_entity_spans):
predicted_sequence[span[0] + 1 : span[1]] = [label] * (span[1] - span[0] - 1)

true_predictions.append(predicted_sequence)
true_labels.append([label_list[tag_id] for tag_id in ner_tags])
true_labels.append([label_list[tag_id] for tag_id in tags])

return true_predictions, true_labels

Expand Down