Code complete?

[zhoumengbo]

I'd like to know if the code in this repository is complete. Has anyone tried pre-training this model from scratch?

[mdingemanse]

for pre-training from scratch you'd need the pretraining data to be specified and available, which it isn't; therefore the code is by definition not complete.

[zhoumengbo]

for pre-training from scratch you'd need the pretraining data to be specified and available, which it isn't; therefore the code is by definition not complete.

Yes, you are right. Because I want to train Mistral 7B from scratch using a different dataset, I'm eager to know the integrity of the model code. If you are familiar with the code and can inform me about it, I would be immensely grateful.

[mdingemanse]

I'm not connected to Mistral and have the same questions as you. From what I can see the repo is set up to share some code for running released versions of Mistral (and if you trawl the issues you see that it is not complete for even that, eg people cannot get the instruct version to run).

It does not look like this repo contains the code you would need to pretrain from scratch.