使用在线量化的baichuan 13b chat 报错 LookupError: model.layers.5.mlp.down_proj.weight

[allendred]

运行命令

python -m fastedit.editor     --data data/fastedit.json     --model  ../../weights/Baichuan-13B-Chat/    --config llama-7b     --template baichuan

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /opt/conda/lib/python3.10/runpy.py:196 in _run_module_as_main                                    │
│                                                                                                  │
│   193 │   main_globals = sys.modules["__main__"].__dict__                                        │
│   194 │   if alter_argv:                                                                         │
│   195 │   │   sys.argv[0] = mod_spec.origin                                                      │
│ ❱ 196 │   return _run_code(code, main_globals, None,                                             │
│   197 │   │   │   │   │    "__main__", mod_spec)                                                 │
│   198                                                                                            │
│   199 def run_module(mod_name, init_globals=None,                                                │
│                                                                                                  │
│ /opt/conda/lib/python3.10/runpy.py:86 in _run_code                                               │
│                                                                                                  │
│    83 │   │   │   │   │      __loader__ = loader,                                                │
│    84 │   │   │   │   │      __package__ = pkg_name,                                             │
│    85 │   │   │   │   │      __spec__ = mod_spec)                                                │
│ ❱  86 │   exec(code, run_globals)                                                                │
│    87 │   return run_globals                                                                     │
│    88                                                                                            │
│    89 def _run_module_code(code, init_globals=None,                                              │
│                                                                                                  │
│ /workspace/llm/baichuan/code/FastEdit/fastedit/editor.py:79 in <module>                          │
│                                                                                                  │
│   76                                                                                             │
│   77                                                                                             │
│   78 if __name__ == "__main__":                                                                  │
│ ❱ 79 │   fire.Fire(test_rome)                                                                    │
│   80                                                                                             │
│                                                                                                  │
│ /opt/conda/lib/python3.10/site-packages/fire/core.py:141 in Fire                                 │
│                                                                                                  │
│   138 │   context.update(caller_globals)                                                         │
│   139 │   context.update(caller_locals)                                                          │
│   140                                                                                            │
│ ❱ 141   component_trace = _Fire(component, args, parsed_flag_args, context, name)                │
│   142                                                                                            │
│   143   if component_trace.HasError():                                                           │
│   144 │   _DisplayError(component_trace)                                                         │
│                                                                                                  │
│ /opt/conda/lib/python3.10/site-packages/fire/core.py:475 in _Fire                                │
│                                                                                                  │
│   472 │     is_class = inspect.isclass(component)                                                │
│   473 │                                                                                          │
│   474 │     try:                                                                                 │
│ ❱ 475 │   │   component, remaining_args = _CallAndUpdateTrace(                                   │
│   476 │   │   │   component,                                                                     │
│   477 │   │   │   remaining_args,                                                                │
│   478 │   │   │   component_trace,                                                               │
│                                                                                                  │
│ /opt/conda/lib/python3.10/site-packages/fire/core.py:691 in _CallAndUpdateTrace                  │
│                                                                                                  │
│   688 │   loop = asyncio.get_event_loop()                                                        │
│   689 │   component = loop.run_until_complete(fn(*varargs, **kwargs))                            │
│   690   else:                                                                                    │
│ ❱ 691 │   component = fn(*varargs, **kwargs)                                                     │
│   692                                                                                            │
│   693   if treatment == 'class':                                                                 │
│   694 │   action = trace.INSTANTIATED_CLASS                                                      │
│                                                                                                  │
│ /workspace/llm/baichuan/code/FastEdit/fastedit/editor.py:55 in test_rome                         │
│                                                                                                  │
│   52 │   │   print("\n\n".join([queries[i] + " " + pre_update_text[i] for i in range(len(quer    │
│   53 │                                                                                           │
│   54 │   print_loud(f"Applying rome to model")                                                   │
│ ❱ 55 │   model_new, _ = apply_rome_to_model(                                                     │
│   56 │   │   model_old,                                                                          │
│   57 │   │   tokenizer,                                                                          │
│   58 │   │   requests,                                                                           │
│                                                                                                  │
│ /workspace/llm/baichuan/code/FastEdit/fastedit/rome/rome_main.py:56 in apply_rome_to_model       │
│                                                                                                  │
│    53 │   weights_diff = {}                                                                      │
│    54 │                                                                                          │
│    55 │   for request in requests:                                                               │
│ ❱  56 │   │   deltas = execute_rome(model, tokenizer, request, hparams, batch_first)             │
│    57 │   │                                                                                      │
│    58 │   │   with torch.no_grad():                                                              │
│    59 │   │   │   for w_name, (delta_u, delta_v) in deltas.items():                              │
│                                                                                                  │
│ /workspace/llm/baichuan/code/FastEdit/fastedit/rome/rome_main.py:97 in execute_rome              │
│                                                                                                  │
│    94 │   start_time = time.time()                                                               │
│    95 │                                                                                          │
│    96 │   # Retrieve weights that user desires to change                                         │
│ ❱  97 │   weights = {f"{hparams.rewrite_module_tmp.format(layer)}.weight":                       │
│    98 │   │   │      nethook.get_parameter(model, f"{hparams.rewrite_module_tmp.format(layer)}   │
│    99 │   │   │      for layer in hparams.layers}                                                │
│   100                                                                                            │
│                                                                                                  │
│ /workspace/llm/baichuan/code/FastEdit/fastedit/rome/rome_main.py:98 in <dictcomp>                │
│                                                                                                  │
│    95 │                                                                                          │
│    96 │   # Retrieve weights that user desires to change                                         │
│    97 │   weights = {f"{hparams.rewrite_module_tmp.format(layer)}.weight":                       │
│ ❱  98 │   │   │      nethook.get_parameter(model, f"{hparams.rewrite_module_tmp.format(layer)}   │
│    99 │   │   │      for layer in hparams.layers}                                                │
│   100 │                                                                                          │
│   101 │   # Save old weights for future restoration                                              │
│                                                                                                  │
│ /workspace/llm/baichuan/code/FastEdit/fastedit/utils/nethook.py:372 in get_parameter             │
│                                                                                                  │
│   369 │   for n, p in model.named_parameters():                                                  │
│   370 │   │   if n == name:                                                                      │
│   371 │   │   │   return p                                                                       │
│ ❱ 372 │   raise LookupError(name)                                                                │
│   373                                                                                            │
│   374                                                                                            │
│   375 def replace_module(model, name, new_module):                                               │
╰──────────────────────────────────────────────────

[hiyouga]

目前还不能量化，我们会在之后支持该功能

[allendred]

不量化13b的模型在a100单卡上会报显存,使用多卡又会出现tensor在不同的显卡上

[tczbzb]

支持 model(...torch_dtype=torch.float16...) 吗？