Compatibility issues with Chinese and slow response speed.

[B1gM8c]

There seem to be some compatibility issues with ChatBot, as an error occurs when inputting Chinese characters.

    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 69, in app
    await response(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/sse_starlette/sse.py", line 227, in __call__
    async with anyio.create_task_group() as task_group:
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
    raise exceptions[0]
  File "/usr/local/lib/python3.10/site-packages/sse_starlette/sse.py", line 230, in wrap
    await func()
  File "/usr/local/lib/python3.10/site-packages/sse_starlette/sse.py", line 219, in stream_response
    async for data in self.body_iterator:
  File "/www/transcribe/ai-language-model/llama-cpp-python/examples/high_level_api/fastapi_server.py", line 215, in server_sent_events
    for chat_chunk in chat_chunks:
  File "/usr/local/lib/python3.10/site-packages/llama_cpp/llama.py", line 720, in _convert_text_completion_chunks_to_chat
    for i, chunk in enumerate(chunks):
  File "/usr/local/lib/python3.10/site-packages/llama_cpp/llama.py", line 482, in _create_completion
    "text": text[start:].decode("utf-8"),
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 0: unexpected end of data

In addition to that, I'm also experiencing some slow speed using the ggml-vicuna-13b-1.1-q4_1.bin model. Although I noticed that the GPU memory usage is not very high, so I'm not sure what could be causing this.

Nonetheless, I would like to express my gratitude to the author for developing such a great interface that mimics OpenAI. If only the response speed could be faster.

[jmtatsch]

To use GPU you would need to set LLAMA_CUBLAS=1 before pip installing. For the Chinese characters there is a pull request open

[B1gM8c]

I tried the following command:

export LLAMA_CUBLAS=1
pip3.10 install llama-cpp-python

Then I launched the project with the environment variable LLAMA_CUBLAS=1, but it seems to still be running on CPU. The current speed is as follows:

llama_print_timings:        load time =  1281.94 ms
llama_print_timings:      sample time =     5.45 ms /     8 runs   (    0.68 ms per run)
llama_print_timings: prompt eval time =  4111.80 ms /    27 tokens (  152.29 ms per token)
llama_print_timings:        eval time =  1556.38 ms /     7 runs   (  222.34 ms per run)
llama_print_timings:       total time =  5675.18 ms

I think I still haven't successfully used the GPU.

[B1gM8c]

I think I understand what you mean. I generated a library in a C++ project by running this command:

LLAMA_CUBLAS=1 make libllama.so

Then I packaged libllama.so and put it in the llama_cpp path. I also put fastapi_server.py in the root directory and ran it. However, I still don't feel like there is a significant improvement in speed.

[jmtatsch]

Can you pip install with verbose to see if it builds correctly? Or replace the lib this package built with your manually built one? Speed increase should be substantial.

[B1gM8c]

Can you pip install with verbose to see if it builds correctly? Or replace the lib this package built with your manually built one? Speed increase should be substantial.

You seem to be right.

LLAMA_OPENBLAS=1 pip3.10 install llama-cpp-python

Here, an error occurred during the installation, apparently due to the lack of the openblas library.

      FAILED: /tmp/pip-install-_1721mma/llama-cpp-python_832569dd8b6b4df08f2392ea79b93b0f/vendor/llama.cpp/libllama.so
      cd /tmp/pip-install-_1721mma/llama-cpp-python_832569dd8b6b4df08f2392ea79b93b0f/vendor/llama.cpp && make libllama.so
      I llama.cpp build info:
      I UNAME_S:  Linux
      I UNAME_P:  x86_64
      I UNAME_M:  x86_64
      I CFLAGS:   -I.              -O3 -DNDEBUG -std=c11   -fPIC -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -pthread -march=native -mtune=native -DGGML_USE_OPENBLAS -I/usr/local/include/openblas
      I CXXFLAGS: -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native
      I LDFLAGS:  -lopenblas
      I CC:       cc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
      I CXX:      g++ (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0

      g++ -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -c llama.cpp -o llama.o
      cc  -I.              -O3 -DNDEBUG -std=c11   -fPIC -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -pthread -march=native -mtune=native -DGGML_USE_OPENBLAS -I/usr/local/include/openblas   -c ggml.c -o ggml.o
      ggml.c:1082:13: warning: ‘quantize_row_q4_2_reference’ defined but not used [-Wunused-function]
       1082 | static void quantize_row_q4_2_reference(const float * restrict x, block_q4_2 * restrict y, int k) {
            |             ^~~~~~~~~~~~~~~~~~~~~~~~~~~
      g++ -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -shared -fPIC -o libllama.so llama.o ggml.o -lopenblas
      /usr/bin/ld: cannot find -lopenblas: No such file or directory
      collect2: error: ld returned 1 exit status
      make: *** [Makefile:186: libllama.so] Error 1
      ninja: build stopped: subcommand failed.
      Traceback (most recent call last):
        File "/tmp/pip-build-env-pfrcg0br/overlay/lib/python3.10/site-packages/skbuild/setuptools_wrap.py", line 674, in setup
          cmkr.make(make_args, install_target=cmake_install_target, env=env)
        File "/tmp/pip-build-env-pfrcg0br/overlay/lib/python3.10/site-packages/skbuild/cmaker.py", line 691, in make
          self.make_impl(clargs=clargs, config=config, source_dir=source_dir, install_target=install_target, env=env)
        File "/tmp/pip-build-env-pfrcg0br/overlay/lib/python3.10/site-packages/skbuild/cmaker.py", line 736, in make_impl
          raise SKBuildError(msg)

      An error occurred while building with CMake.
        Command:
          /tmp/pip-build-env-pfrcg0br/overlay/lib/python3.10/site-packages/cmake/data/bin/cmake --build . --target install --config Release --
        Install target:
          install
        Source directory:
          /tmp/pip-install-_1721mma/llama-cpp-python_832569dd8b6b4df08f2392ea79b93b0f
        Working directory:
          /tmp/pip-install-_1721mma/llama-cpp-python_832569dd8b6b4df08f2392ea79b93b0f/_skbuild/linux-x86_64-3.10/cmake-build
      Please check the install target is valid and see CMake's output for more information.

      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for llama-cpp-python
Failed to build llama-cpp-python
ERROR: Could not build wheels for llama-cpp-python, which is required to install pyproject.toml-based projects

I tried to install the related library by executing the following command in the terminal:

sudo apt-get install libopenblas-dev

Then it looks like there were no errors, and the following command LLAMA_OPENBLAS=1 pip3.10 install llama-cpp-python was executed successfully:

Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting llama-cpp-python
  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/1b/ea/3f2aff10fd7195c6bc8c52375d9ff027a551151569c50e0d47581b14b7c1/llama_cpp_python-0.1.36.tar.gz (1.1 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: typing-extensions>=4.5.0 in /usr/local/lib/python3.10/site-packages (from llama-cpp-python) (4.5.0)
Building wheels for collected packages: llama-cpp-python
  Building wheel for llama-cpp-python (pyproject.toml) ... done
  Created wheel for llama-cpp-python: filename=llama_cpp_python-0.1.36-cp310-cp310-linux_x86_64.whl size=159345 sha256=87365f722e83f66dc40bc9d2391f4f3c0b2e23796ba0fb8f03adc1cc8defc759
  Stored in directory: /root/.cache/pip/wheels/d0/72/98/5a896af92a0ecb8c5c943ed4d5f8ed1c6d3a2e38c32d0eae46
Successfully built llama-cpp-python
Installing collected packages: llama-cpp-python
Successfully installed llama-cpp-python-0.1.36