Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bugs in core.py file and others #1859

Closed
vin136 opened this issue Feb 28, 2024 · 2 comments
Closed

Bugs in core.py file and others #1859

vin136 opened this issue Feb 28, 2024 · 2 comments
Labels
bug Confirmed bugs

Comments

@vin136
Copy link

vin136 commented Feb 28, 2024 •
edited by Kartik14
Loading

🐛 Bug

I'm trying to follow this tutorial to get Zephyr-7B(mistral variant) using the latest master branch. First, there was a minor bug here,where the parenthesis is not closed. I've fixed it but there still seems to be some more.

To Reproduce

Followed the steps in the tutorial to install mlc and then got the latest master git clone --recursive https://github.com/mlc-ai/mlc-llm.git.

Steps to reproduce the behavior:

import mlc_llm
import mlc_chat
import tvm

hf_path = 'HuggingFaceH4/zephyr-7b-beta'

build_args = mlc_llm.BuildArgs(
    hf_path=hf_path,
    quantization="q0f16",
    target="cuda"
)

if __name__ == '__main__':
    lib_path, model_path, chat_config_path = mlc_llm.build_model(build_args)
    print(f"lib path:{lib_path},model path:{model_path},chat_config_path:{chat_config_path}")
Traceback (most recent call last):
  File "/home/vinay/Projects/LLM/scratch/compile.py", line 14, in <module>
    lib_path, model_path, chat_config_path = mlc_llm.build_model(build_args)
                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vinay/Projects/LLM/scratch/mlc-llm/mlc_llm/core.py", line 986, in build_model
    build_model_from_args(args)
  File "/home/vinay/Projects/LLM/scratch/mlc-llm/mlc_llm/core.py", line 836, in build_model_from_args
    mod, param_manager, params, model_config = model_generators[args.model_category].get_model(
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vinay/Projects/LLM/scratch/mlc-llm/mlc_llm/relax_model/mistral.py", line 1016, in get_model
    create_encoding_func(bb, param_manager, config, args.quantization, sep_embed)
  File "/home/vinay/Projects/LLM/scratch/mlc-llm/mlc_llm/relax_model/mistral.py", line 866, in create_encoding_func
    logits, key_value_cache = model(
                              ^^^^^^
  File "/home/vinay/miniconda3/envs/mlc-prebuilt/lib/python3.11/site-packages/tvm/relax/testing/nn.py", line 263, in __call__
    return self.forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vinay/miniconda3/envs/mlc-prebuilt/lib/python3.11/site-packages/tvm/relax/frontend/nn/subroutine.py", line 87, in new_forward
    return old_forward(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vinay/Projects/LLM/scratch/mlc-llm/mlc_llm/relax_model/mistral.py", line 766, in forward
    hidden_states, key_value_cache = self.model(
                                     ^^^^^^^^^^^
  File "/home/vinay/miniconda3/envs/mlc-prebuilt/lib/python3.11/site-packages/tvm/relax/testing/nn.py", line 263, in __call__
    return self.forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vinay/miniconda3/envs/mlc-prebuilt/lib/python3.11/site-packages/tvm/relax/frontend/nn/subroutine.py", line 87, in new_forward
    return old_forward(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vinay/Projects/LLM/scratch/mlc-llm/mlc_llm/relax_model/mistral.py", line 720, in forward
    hidden_states, key_value_cache = decoder_layer(
                                     ^^^^^^^^^^^^^^
  File "/home/vinay/miniconda3/envs/mlc-prebuilt/lib/python3.11/site-packages/tvm/relax/testing/nn.py", line 263, in __call__
    return self.forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vinay/miniconda3/envs/mlc-prebuilt/lib/python3.11/site-packages/tvm/relax/frontend/nn/subroutine.py", line 87, in new_forward
    return old_forward(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vinay/Projects/LLM/scratch/mlc-llm/mlc_llm/relax_model/mistral.py", line 581, in forward
    hidden_states, present_key_value = self.self_attn(
                                       ^^^^^^^^^^^^^^^
  File "/home/vinay/miniconda3/envs/mlc-prebuilt/lib/python3.11/site-packages/tvm/relax/testing/nn.py", line 263, in __call__
    return self.forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vinay/miniconda3/envs/mlc-prebuilt/lib/python3.11/site-packages/tvm/relax/frontend/nn/subroutine.py", line 87, in new_forward
    return old_forward(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vinay/Projects/LLM/scratch/mlc-llm/mlc_llm/relax_model/mistral.py", line 489, in forward
    key, value, updated_key_value = self.interleave_kv(
                                    ^^^^^^^^^^^^^^^^^^^
  File "/home/vinay/Projects/LLM/scratch/mlc-llm/mlc_llm/relax_model/mistral.py", line 346, in interleave_kv
    relax.call_pure_packed(
  File "/home/vinay/miniconda3/envs/mlc-prebuilt/lib/python3.11/site-packages/tvm/relax/utils.py", line 173, in wrapper
    bound = sig.bind(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vinay/miniconda3/envs/mlc-prebuilt/lib/python3.11/inspect.py", line 3212, in bind
    return self._bind(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vinay/miniconda3/envs/mlc-prebuilt/lib/python3.11/inspect.py", line 3201, in _bind
    raise TypeError(
TypeError: got an unexpected keyword argument 'args'
[20:14:31] /workspace/tvm/src/relax/ir/block_builder.cc:65: Warning: BlockBuilder destroyed with remaining blocks!

Expected behavior

This should run without error.

Environment

TVM Info:
USE_NVTX: OFF
USE_GTEST: AUTO
SUMMARIZE: OFF
USE_IOS_RPC: OFF
USE_MSC: OFF
USE_ETHOSU:
CUDA_VERSION: 12.1
USE_LIBBACKTRACE: AUTO
DLPACK_PATH: 3rdparty/dlpack/include
USE_TENSORRT_CODEGEN: OFF
USE_THRUST: OFF
USE_TARGET_ONNX: OFF
USE_AOT_EXECUTOR: ON
BUILD_DUMMY_LIBTVM: OFF
USE_CUDNN: OFF
USE_TENSORRT_RUNTIME: OFF
USE_ARM_COMPUTE_LIB_GRAPH_EXECUTOR: OFF
USE_CCACHE: AUTO
USE_ARM_COMPUTE_LIB: OFF
USE_CPP_RTVM:
USE_OPENCL_GTEST: /path/to/opencl/gtest
USE_MKL: OFF
USE_PT_TVMDSOOP: OFF
MLIR_VERSION: NOT-FOUND
USE_CLML: OFF
USE_STACKVM_RUNTIME: OFF
USE_GRAPH_EXECUTOR_CUDA_GRAPH: OFF
ROCM_PATH: /opt/rocm
USE_DNNL: OFF
USE_VITIS_AI: OFF
USE_MLIR: OFF
USE_RCCL: OFF
USE_LLVM: llvm-config --ignore-libllvm --link-static
USE_VERILATOR: OFF
USE_TF_TVMDSOOP: OFF
USE_THREADS: ON
USE_MSVC_MT: OFF
BACKTRACE_ON_SEGFAULT: OFF
USE_GRAPH_EXECUTOR: ON
USE_NCCL: ON
USE_ROCBLAS: OFF
GIT_COMMIT_HASH: 2c1ce3ab467f9367c14afd9579ed1388aaae0b90
USE_VULKAN: ON
USE_RUST_EXT: OFF
USE_CUTLASS: ON
USE_CPP_RPC: OFF
USE_HEXAGON: OFF
USE_CUSTOM_LOGGING: OFF
USE_UMA: OFF
USE_FALLBACK_STL_MAP: OFF
USE_SORT: ON
USE_RTTI: ON
GIT_COMMIT_TIME: 2024-02-26 17:23:17 -0500
USE_HEXAGON_SDK: /path/to/sdk
USE_BLAS: none
USE_ETHOSN: OFF
USE_LIBTORCH: OFF
USE_RANDOM: ON
USE_CUDA: ON
USE_COREML: OFF
USE_AMX: OFF
BUILD_STATIC_RUNTIME: OFF
USE_CMSISNN: OFF
USE_KHRONOS_SPIRV: OFF
USE_CLML_GRAPH_EXECUTOR: OFF
USE_TFLITE: OFF
USE_HEXAGON_GTEST: /path/to/hexagon/gtest
PICOJSON_PATH: 3rdparty/picojson
USE_OPENCL_ENABLE_HOST_PTR: OFF
INSTALL_DEV: OFF
USE_PROFILER: ON
USE_NNPACK: OFF
LLVM_VERSION: 15.0.7
USE_MRVL: OFF
USE_OPENCL: OFF
COMPILER_RT_PATH: 3rdparty/compiler-rt
RANG_PATH: 3rdparty/rang/include
USE_SPIRV_KHR_INTEGER_DOT_PRODUCT: OFF
USE_OPENMP: OFF
USE_BNNS: OFF
USE_CUBLAS: ON
USE_METAL: OFF
USE_MICRO_STANDALONE_RUNTIME: OFF
USE_HEXAGON_EXTERNAL_LIBS: OFF
USE_ALTERNATIVE_LINKER: AUTO
USE_BYODT_POSIT: OFF
USE_HEXAGON_RPC: OFF
USE_MICRO: OFF
DMLC_PATH: 3rdparty/dmlc-core/include
INDEX_DEFAULT_I64: ON
USE_RELAY_DEBUG: OFF
USE_RPC: ON
USE_TENSORFLOW_PATH: none
TVM_CLML_VERSION:
USE_MIOPEN: OFF
USE_ROCM: OFF
USE_PAPI: OFF
USE_CURAND: OFF
TVM_CXX_COMPILER_PATH: /opt/rh/gcc-toolset-11/root/usr/bin/c++
HIDE_PRIVATE_SYMBOLS: ON

Additional context
@vin136 vin136 added the bug Confirmed bugs label Feb 28, 2024
@vin136 vin136 mentioned this issue Feb 28, 2024
Closed
@MasterJH5574
Copy link
Member

Hi @vin136, thank you for reporting! On one hand, I just sent a PR to fix the args issue in #1874.

On the other hand, we have switched to use the new model compilation flow (named “SLM”) and therefore the notebook you referred is outdated (we should have marked this notebook as outdated, sorry about it). So you may want to follow the docs at https://llm.mlc.ai/docs/compilation/convert_weights.html and https://llm.mlc.ai/docs/compilation/compile_models.html to compile the Zephyr model. I didn't get time to try it on my own, so please let me know if you run into any issues and need help.

@tqchen
Copy link
Contributor

tqchen commented Mar 4, 2024

closing as it is fixed in #1874, also see #1886

@tqchen tqchen closed this as completed Mar 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Confirmed bugs
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tqchen @vin136 @MasterJH5574