OOV Train Error

[b1ackwater3]

Traceback (most recent call last):
  File "/volume/demo/lliao/ReCall/src/verl/trainer/main_ppo.py", line 59, in main
    run_ppo(config)
  File "/volume/demo/lliao/ReCall/src/verl/trainer/main_ppo.py", line 77, in run_ppo
    ray.get(runner.run.remote(config))
  File "/miniconda3/envs/re-call/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
    return fn(*args, **kwargs)
  File "/miniconda3/envs/re-call/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/miniconda3/envs/re-call/lib/python3.10/site-packages/ray/_private/worker.py", line 2822, in get
    values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
  File "/miniconda3/envs/re-call/lib/python3.10/site-packages/ray/_private/worker.py", line 930, in get_objects
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(ValueError): �[36mray::TaskRunner.run()�[39m (pid=2676523, ip=10.45.24.116, actor_id=07794c1f27cfaa2e2e21c95801000000, repr=<main_ppo.TaskRunner object at 0x1445a5fb7970>)
  File "/volume/demo/lliao/ReCall/src/verl/trainer/main_ppo.py", line 205, in run
    trainer.fit()
  File "/volume/demo/lliao/ReCall/src/verl/trainer/ppo/ray_trainer.py", line 863, in fit
    gen_batch_output = self.actor_rollout_wg.generate_sequences(gen_batch)
  File "/volume/demo/lliao/ReCall/src/verl/single_controller/ray/base.py", line 43, in func
    output = ray.get(output)
ray.exceptions.RayTaskError(ValueError): �[36mray::WorkerDict.actor_rollout_generate_sequences()�[39m (pid=2688731, ip=10.45.24.116, actor_id=15cd82bc253fb887a3d5134001000000, repr=<verl.single_controller.ray.base.WorkerDict object at 0x143b5cd39120>)
  File "/volume/demo/lliao/ReCall/src/verl/single_controller/ray/base.py", line 439, in func
    return getattr(self.worker_dict[key], name)(*args, **kwargs)
  File "/volume/demo/lliao/ReCall/src/verl/single_controller/base/decorator.py", line 409, in inner
    return func(*args, **kwargs)
  File "/volume/demo/lliao/ReCall/src/verl/workers/fsdp_workers.py", line 528, in generate_sequences
    output = self.rollout.generate_sequences(prompts=prompts)
  File "/miniconda3/envs/re-call/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/volume/demo/lliao/ReCall/src/verl/workers/rollout/vllm_rollout/vllm_rollout_spmd.py", line 492, in generate_sequences
    outputs = self.inference_engine.generate(
  File "/miniconda3/envs/re-call/lib/python3.10/site-packages/vllm/utils.py", line 1134, in inner
    return fn(*args, **kwargs)
  File "/miniconda3/envs/re-call/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 462, in generate
    self._validate_and_add_requests(
  File "/miniconda3/envs/re-call/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 1342, in _validate_and_add_requests
    self._add_request(
  File "/miniconda3/envs/re-call/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 1360, in _add_request
    self.llm_engine.add_request(
  File "/miniconda3/envs/re-call/lib/python3.10/site-packages/vllm/v1/engine/llm_engine.py", line 186, in add_request
    request = self.processor.process_inputs(request_id, prompt, params,
  File "/miniconda3/envs/re-call/lib/python3.10/site-packages/vllm/v1/engine/processor.py", line 236, in process_inputs
    self._validate_model_inputs(processed_inputs, lora_request)
  File "/miniconda3/envs/re-call/lib/python3.10/site-packages/vllm/v1/engine/processor.py", line 328, in _validate_model_inputs
    self._validate_model_input(decoder_inputs,
  File "/miniconda3/envs/re-call/lib/python3.10/site-packages/vllm/v1/engine/processor.py", line 351, in _validate_model_input
    raise ValueError(f"Token id {max_input_id} is out of vocabulary")
ValueError: Token id 151750 is out of vocabulary

The Start Command:

bash train.sh \
    --train_batch_size 8 \
    --ppo_mini_batch_size 4 \
    --use_re_call True \
    --prompt_template_name re_call_template_sys \
    --actor_model_path qwen2-5-7b-ins \
    --search_url {your-hosted-retriever-url} \
    --sandbox_url {your-hosted-sandbox-url} \
    --project_name {wandb-project-name} \
    --experiment_name {wandb-experiment-name} \
    --nnodes 1 \
    --n_gpus_per_node 8 \
    --save_freq 5 \
    --test_freq 5 \
    --total_epochs 2 \
    --wandb_api_key {your-wandb-api-key} \
    --save_path {path/to/save} \
    --train_files "['train1.parquet', 'train2.parquet']" \
    --test_files "['test1.parquet', 'test2.parquet']"

Have you encountered the error?
@AnselCmy

[AnselCmy]

Yes, this issue may be encountered occasionally. We haven't yet identified the specific cause. This problem was more frequent with vllm 0.8.2, but after upgrading to version 0.8.4, it occurs much less frequently. We are continuously investigating this issue. In the meantime, the current workaround is to continue training from the existing checkpoint.

[YqjMartin]

        if self.model_hf_config.vocab_size > self.tokenizer.vocab_size:
            print(f"use fix_oov_logits_processor, {self.model_hf_config.vocab_size=} is out of {self.tokenizer.vocab_size=}")
            max_token_id = max(self.tokenizer.get_vocab().values())
            kwargs["allowed_token_ids"] = list(range(max_token_id + 1))

@AnselCmy @b1ackwater3 I think this may helpful?

[BaiMeiyingxue]

kwargs["allowed_token_ids"]

Hi, where should add the code? we want to try