Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create data composition #25

Open
RaymondLi0 opened this issue Feb 7, 2023 · 1 comment
Open

Create data composition #25

RaymondLi0 opened this issue Feb 7, 2023 · 1 comment
Milestone
Train final model

Comments

@RaymondLi0
Copy link
Collaborator

RaymondLi0 commented Feb 7, 2023 •
edited
Loading

Once the stack v1.2 is done #24 , attribute weight to each data source (PL and NL).

Some PLs like HTML and CSS should probably be down-sampled.
On the other hand, we might want to upsample some low-resource PLs.

For the final model the goal would be to have a limit of 5 epochs maximum for example, and check that each of the data sources stays under that limit. An estimated 600B training tokens can be used to check this limit.
@RaymondLi0 RaymondLi0 added this to the Train final model milestone Feb 7, 2023
@RaymondLi0
Copy link
Collaborator Author

This is addressed by https://github.com/bigcode-project/bigcode-data-mix and #32 .
It would be good to have a summary of the number of epochs done on each data-source for the final training

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
Train final model
Development

No branches or pull requests
1 participant
@RaymondLi0