Skip to content

Conversation

@ytivy
Copy link
Contributor

@ytivy ytivy commented Aug 5, 2024

v3 model用 megatron -> huggingface コンバーターを追加

@ytivy ytivy mentioned this pull request Aug 5, 2024
3 tasks
@ytivy

This comment was marked as outdated.

ytivy and others added 3 commits August 7, 2024 19:27
deal wirh relative path setting on source checkpoint
@ytivy ytivy marked this pull request as ready for review August 8, 2024 07:46
@ytivy ytivy self-assigned this Aug 8, 2024
@ytivy ytivy requested review from Taka008, hkiyomaru and odashi August 8, 2024 07:47
@ytivy
Copy link
Contributor Author

ytivy commented Aug 8, 2024

#12 への対応前提のため、動作確認時には environment/src/llm-jp-tokenizer で v3.0b2 ブランチに事前に切り替えてください

git fetch
git checkout v3.0b2

@ytivy
Copy link
Contributor Author

ytivy commented Aug 8, 2024

#12 merge完了

@ytivy

This comment was marked as resolved.

@ytivy ytivy requested a review from odashi August 13, 2024 15:08
ytivy and others added 2 commits August 14, 2024 08:44
Co-authored-by: Yusuke Oda <yusuke.oda@predicate.jp>
Co-authored-by: Yusuke Oda <yusuke.oda@predicate.jp>
@ytivy
Copy link
Contributor Author

ytivy commented Aug 13, 2024

1ノード占有は勿体無い気もしますが処理時間が短い(13Bで5分以内, 172Bで約3時間)ので問題ないかと思いsuggestionに変更しました

@ytivy ytivy requested a review from odashi August 13, 2024 23:55
@ytivy ytivy merged commit d2866fd into main Aug 14, 2024
@ytivy ytivy deleted the add-v3-converter branch August 14, 2024 09:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants