FileNotFoundError(path): R0 checkpoint_and_raise - error=wiki_data

[SarahGaga0822]

Hi,
I encountered an issue when I tried to train locally by running this command:
CUDA_VISIBLE_DEVICES=0,1 torchrun --standalone --nnodes=1 --nproc-per-node=2 \ -m lcm.train launcher=standalone \ +pretrain=mse \ ++trainer.data_loading_config.max_tokens=1000 \ ++trainer.output_dir="checkpoints/mse_lcm" \ +trainer.use_submitit=false

This is the error:

Could you please help me find out the reason?
Do I need to add a local parquet_path to the datacard? and how? (I do not have a SLURM cluster.)

[hasanyazarr]

Before the training process, make sure you have downloaded the parquet dataset with code:

# run the data preparation script
!python large_concept_model/scripts/prepare_wikipedia.py /content/large_concept_model/sample_data

Then configure your large_concept_model/lcm/datacards/datacards.yaml file. You should update the path, as an example:

# FIXME
name: "pretraining_data_train"
parquet_path:
  s3: /content/large_concept_model/sample_data/train_data.parquet # update here with downloaded dataset path 
source_column: "text_sentences_sonar_emb"
source_text_column: "text_sentences"
# partition columns:
# "split" (train, validation)

Then training process will start I guess.

[SarahGaga0822]

Thanks for your response!
After preparing the wiki data, I get a parquet file like this:

And I modified the datacards to:

Unfortunately, I got another error:

What's the reason?

[hasanyazarr]

I'm not sure why this error occurs, but it was also mentioned in #15. Did you uncomment the partition_columns line in datacards.yaml? That might be causing the error. I didn't encounter this issue when I commented out the partition_columns (lines 8 and 9 in your file).

[SarahGaga0822]

'partition_columns' is also commented out in my datacards.yaml.
I followed your instructions in #9 (comment), and thankfully, this issue is solved!
But sadly, another error occurs:

Is there any possible solution?

[hasanyazarr]

I encountered these errors too but I think this is not the correct way to continue, if you solve this error, you will get another one as I remember and it stucks in somewhere. I think I should delete that solution since it is outdated. So, I run this codes in Colab:

# clone the repo
!git clone https://github.com/facebookresearch/large_concept_model.git

cd large_concept_model

# Install GPU-enabled Fairseq2 compatible with PyTorch 2.5.1
!pip install fairseq2==v0.3.0rc1 --pre --extra-index-url  https://fair.pkg.atmeta.com/fairseq2/whl/rc/pt2.5.1/cu121

# Install other dependencies required for the project
!pip install -e ".[data,eval]" -f /content/large_concept_model

# create directory for prep_wiki data
!mkdir -p /content/large_concept_model/sample_data

# run the data preparation script
!python large_concept_model/scripts/prepare_wikipedia.py /content/large_concept_model/sample_data

# create directory for normalizer.pt file
!mkdir -p /content/large_concept_model/normalizer

# create normalizer.pt file
!python /content/large_concept_model/scripts/fit_embedding_normalizer.py --ds pretraining_data_train --save_path "/content/large_concept_model/normalizer/normalizer_train.pt" --max_nb_samples 1000000

# training script for MSE_LCM
!CUDA_VISIBLE_DEVICES=0 torchrun --standalone --nnodes=1 --nproc-per-node=1 -m large_concept_model.lcm.train \
    launcher=standalone +pretrain=mse \
    ++trainer.data_loading_config.max_tokens=512 \
    ++trainer.output_dir="/content/drive/MyDrive/LCM/checkpoints/mse_lcm" \
    +trainer.use_submitit=false

You need to do 2 things:

First update the datacards.yaml file after you downloaded wikipedia data as you did.

Then, go to the large_concept_models/recipes/train/pretrain/mse.yaml file and at the end of file, there are variable names like pretraining_data=train and pretraining_data=validation. In your datacards.yaml file, your dataset is named as pretraining_data, so update 'pretraining_data=train' to 'pretraining_data'. If you have validation data, then you can add it to your datacards.yaml file like:

# FIXME
name: "pretraining_data_train"
parquet_path:
  s3: /content/large_concept_model/sample_data/train_data.parquet
source_column: "text_sentences_sonar_emb"
source_text_column: "text_sentences"
---
# FIXME
name: "pretraining_data_val"
parquet_path:
  s3: /content/large_concept_model/sample_data/val_data.parquet
source_column: "text_sentences_sonar_emb"
source_text_column: "text_sentences"

After these changes, training process should start. I deleted my comment since it is outdated.

[SarahGaga0822]

Thanks a lot! I would try this later~

[shraditya]

@hasanyazarr how much time its take to for the pre-training and gpu you are using

[hasanyazarr]

It took ~4 hours for 10.000 steps in single A100 40 GB GPU.

[xymou]

Hi @hasanyazarr . Thanks for the comments before. I have downloaded and processed the sample data and make sure the dataset name in datacards.yaml and mse.yaml file are both "pretraining_data", but I still encounter such issue. Do you know how to solve it? Thanks a lot!

  0%|                                                                                                                                                               | 0/1 [00:01<?, ?it/s]libmkl_rt.so: cannot open shared object file: No such file or directory
> /root/miniconda3/envs/lcm/lib/python3.10/ctypes/__init__.py(374)__init__()
-> self._handle = _dlopen(self._name, mode)
(Pdb) libmkl_rt.so: cannot open shared object file: No such file or directory
> /root/miniconda3/envs/lcm/lib/python3.10/ctypes/__init__.py(374)__init__()
-> self._handle = _dlopen(self._name, mode)

[hasanyazarr]

Hi @xymou, unfortunately I don’t remember the details of the environment and code I used for this project, and I haven’t checked this repo since February. But if you share your email, I can send you the last notebook I worked with, but I can’t guarantee it will run since it depends on many libraries.

[xymou]

Hi @xymou, unfortunately I don’t remember the details of the environment and code I used for this project, and I haven’t checked this repo since February. But if you share your email, I can send you the last notebook I worked with, but I can’t guarantee it will run since it depends on many libraries.

Thank you very much! I've solved it!