dependency failed to start: container vllm-service exited (1)

[roshanjoebenny]

I was trying the example from Deploy the ChatQnA Solution.
https://opea-project.github.io/latest/getting-started/README.html

when I was executing the command - docker compose -f compose.yaml up -d.
I was getting the error as dependency failed to start: container vllm-service exited (1).

please guide how to solve the issue

[yinghu5]

Hi @roshanjoebenny

Could you please tell some informations like

1. Hareware and Software platform, cloud or local machine?
2. GenExample branch:  Are you on the latest branch or use one fix version, i.e git checkout v1.3?

If latest branch, I guess, as there are updates, like vllm version updates etc,
please follow the guide in
https://opea-project.github.io/latest/GenAIExamples/ChatQnA/README.html

=> Deploy table to select the suitable platform
Basically, please check the below environment
'''
export host_ip="External_Public_IP" #ip address of the node
export HF_TOKEN="Your_Huggingface_API_Token"
export http_proxy="Your_HTTP_Proxy" #http proxy if any
export https_proxy="Your_HTTPs_Proxy" #https proxy if any
export no_proxy=localhost,127.0.0.1,$host_ip #additional no proxies if needed
export no_proxy=$no_proxy,chatqna-xeon-ui-server,chatqna-xeon-backend-server,dataprep-redis-service,tei-embedding-service,retriever,tei-reranking-service,tgi-service,vllm-service,llm-faqgen
source ./set_env.sh
'''

[roshanjoebenny]

Hi @yinghu5 ,
I was using azure vm - ubundu. The version I was using was v1.3.

[roshanjoebenny]

I tried - docker logs vllm-service | grep Complete to check for the logs.
Error I got was

Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:08<00:24, 8.10s/it]
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:43<00:48, 24.07s/it]
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 1121, in
uvloop.run(run_server(args))
File "/opt/venv/lib/python3.12/site-packages/uvloop/init.py", line 109, in run
return __asyncio.run(
^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 195, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/opt/venv/lib/python3.12/site-packages/uvloop/init.py", line 61, in wrapper
return await main
^^^^^^^^^^
File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 1069, in run_server
async with build_async_engine_client(args) as engine_client:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 269, in build_async_engine_client_from_engine_args
raise RuntimeError(
RuntimeError: Engine process failed to start. See stack trace for the root cause.
/root/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

[yinghu5]

Hi @roshanjoebenny

Thank you for the log info. From there, the error happened when load checkpoints 50%. If possible, could you try the small model:
like vllm-project/vllm#13765 =>google/gemma-2-2b-it
You can change it from set_env.sh=>LLM_MODEL_ID

What is VM's CPU core, RAM and Disk Space?

Thanks

[xiguiw]

@roshanjoebenny
Could show the complete log? Thanks!

docker logs vllm-service

[roshanjoebenny]

Hi @yinghu5 @xiguiw
cpu core - 4

RAM

           total        used        free      shared  buff/cache   available

Mem: 15Gi 1.2Gi 10Gi 4.0Mi 3.9Gi 14Gi

Disk Space

Filesystem Size Used Avail Use% Mounted on
/dev/root 61G 33G 28G 55% /

I have changed the LLM_MODEL_ID to google/gemma-2-2b-it

set_env.sh

pushd "../../../../../" > /dev/null
source .set_env.sh
popd > /dev/null

export EMBEDDING_MODEL_ID="BAAI/bge-base-en-v1.5"
export RERANK_MODEL_ID="BAAI/bge-reranker-base"
export LLM_MODEL_ID="google/gemma-2-2b-it"
export INDEX_NAME="rag-redis"

Set it as a non-null string, such as true, if you want to enable logging facility,

otherwise, keep it as "" to disable it.

export LOGFLAG="true"

Set OpenTelemetry Tracing Endpoint

export JAEGER_IP=$(ip route get 8.8.8.8 | grep -oP 'src \K[^ ]+')
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=grpc://$JAEGER_IP:4317
export TELEMETRY_ENDPOINT=http://$JAEGER_IP:4318/v1/traces

Set no proxy

export no_proxy="$no_proxy,chatqna-xeon-ui-server,chatqna-xeon-backend-server,dataprep-redis-service,tei-embedding-service,retriever,tei-reranking-service,tgi-service,vllm-service,jaeger,prometheus,grafana,node-exporter,$JAEGER_IP"

export LLM_ENDPOINT_PORT=8010
export LLM_SERVER_PORT=9000
export CHATQNA_BACKEND_PORT=8888
export CHATQNA_REDIS_VECTOR_PORT=6379
export CHATQNA_REDIS_VECTOR_INSIGHT_PORT=8001
export CHATQNA_FRONTEND_SERVICE_PORT=5173
export NGINX_PORT=80
export FAQGen_COMPONENT_NAME="OpeaFaqGenvLLM"
export LLM_ENDPOINT="http://${host_ip}:${LLM_ENDPOINT_PORT}"
pushd "grafana/dashboards" > /dev/null
source download_opea_dashboard.sh
popd > /dev/null

compose.yaml

services:
redis-vector-db:
image: redis/redis-stack:7.2.0-v9
container_name: redis-vector-db
ports:
- "6379:6379"
- "8001:8001"
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 3s
retries: 10
dataprep-redis-service:
image: ${REGISTRY:-opea}/dataprep:${TAG:-latest}
container_name: dataprep-redis-server
depends_on:
redis-vector-db:
condition: service_healthy
tei-embedding-service:
condition: service_started
ports:
- "6007:5000"
environment:
no_proxy: ${no_proxy}
http_proxy: ${http_proxy}
https_proxy: ${https_proxy}
REDIS_URL: redis://redis-vector-db:6379
REDIS_HOST: redis-vector-db
INDEX_NAME: ${INDEX_NAME}
TEI_ENDPOINT: http://tei-embedding-service:80
HUGGINGFACEHUB_API_TOKEN: ${HF_TOKEN}
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:5000/v1/health_check || exit 1"]
interval: 10s
timeout: 5s
retries: 50
restart: unless-stopped
tei-embedding-service:
image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.5
container_name: tei-embedding-server
ports:
- "6006:80"
volumes:
- "${MODEL_CACHE:-./data}:/data"
shm_size: 1g
environment:
no_proxy: ${no_proxy}
http_proxy: ${http_proxy}
https_proxy: ${https_proxy}
command: --model-id ${EMBEDDING_MODEL_ID} --auto-truncate
retriever:
image: ${REGISTRY:-opea}/retriever:${TAG:-latest}
container_name: retriever-redis-server
depends_on:
- redis-vector-db
ports:
- "7000:7000"
ipc: host
environment:
no_proxy: ${no_proxy}
http_proxy: ${http_proxy}
https_proxy: ${https_proxy}
REDIS_URL: redis://redis-vector-db:6379
REDIS_HOST: redis-vector-db
INDEX_NAME: ${INDEX_NAME}
TEI_EMBEDDING_ENDPOINT: http://tei-embedding-service:80
HUGGINGFACEHUB_API_TOKEN: ${HF_TOKEN}
LOGFLAG: ${LOGFLAG}
RETRIEVER_COMPONENT_NAME: "OPEA_RETRIEVER_REDIS"
restart: unless-stopped
tei-reranking-service:
image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.5
container_name: tei-reranking-server
ports:
- "8808:80"
volumes:
- "${MODEL_CACHE:-./data}:/data"
shm_size: 1g
environment:
no_proxy: ${no_proxy}
http_proxy: ${http_proxy}
https_proxy: ${https_proxy}
HUGGINGFACEHUB_API_TOKEN: ${HF_TOKEN}
HF_HUB_DISABLE_PROGRESS_BARS: 1
HF_HUB_ENABLE_HF_TRANSFER: 0
command: --model-id ${RERANK_MODEL_ID} --auto-truncate
vllm-service:
image: ${REGISTRY:-opea}/vllm:${TAG:-latest}
container_name: vllm-service
ports:
- "9009:80"
volumes:
- "${MODEL_CACHE:-./data}:/root/.cache/huggingface/hub"
shm_size: 4g
environment:
no_proxy: ${no_proxy}
http_proxy: ${http_proxy}
https_proxy: ${https_proxy}
HF_TOKEN: ${HF_TOKEN}
LLM_MODEL_ID: ${LLM_MODEL_ID}
VLLM_TORCH_PROFILER_DIR: "/mnt"
VLLM_CPU_KVCACHE_SPACE: 4
healthcheck:
test: ["CMD-SHELL", "curl -f http://$host_ip:9009/health || exit 1"]
interval: 10s
timeout: 10s
retries: 100
command: --model $LLM_MODEL_ID --host 0.0.0.0 --port 80
chatqna-xeon-backend-server:
image: ${REGISTRY:-opea}/chatqna:${TAG:-latest}
container_name: chatqna-xeon-backend-server
depends_on:
redis-vector-db:
condition: service_started
dataprep-redis-service:
condition: service_healthy
tei-embedding-service:
condition: service_started
retriever:
condition: service_started
tei-reranking-service:
condition: service_started
vllm-service:
condition: service_healthy
ports:
- "8888:8888"
environment:
- no_proxy=${no_proxy}
- https_proxy=${https_proxy}
- http_proxy=${http_proxy}
- MEGA_SERVICE_HOST_IP=chatqna-xeon-backend-server
- EMBEDDING_SERVER_HOST_IP=tei-embedding-service
- EMBEDDING_SERVER_PORT=${EMBEDDING_SERVER_PORT:-80}
- RETRIEVER_SERVICE_HOST_IP=retriever
- RERANK_SERVER_HOST_IP=tei-reranking-service
- RERANK_SERVER_PORT=${RERANK_SERVER_PORT:-80}
- LLM_SERVER_HOST_IP=vllm-service
- LLM_SERVER_PORT=80
- LLM_MODEL=${LLM_MODEL_ID}
- LOGFLAG=${LOGFLAG}
ipc: host
restart: always
chatqna-xeon-ui-server:
image: ${REGISTRY:-opea}/chatqna-ui:${TAG:-latest}
container_name: chatqna-xeon-ui-server
depends_on:
- chatqna-xeon-backend-server
ports:
- "5173:5173"
environment:
- no_proxy=${no_proxy}
- https_proxy=${https_proxy}
- http_proxy=${http_proxy}
ipc: host
restart: always
chatqna-xeon-nginx-server:
image: ${REGISTRY:-opea}/nginx:${TAG:-latest}
container_name: chatqna-xeon-nginx-server
depends_on:
- chatqna-xeon-backend-server
- chatqna-xeon-ui-server
ports:
- "${NGINX_PORT:-80}:80"
environment:
- no_proxy=${no_proxy}
- https_proxy=${https_proxy}
- http_proxy=${http_proxy}
- FRONTEND_SERVICE_IP=chatqna-xeon-ui-server
- FRONTEND_SERVICE_PORT=5173
- BACKEND_SERVICE_NAME=chatqna
- BACKEND_SERVICE_IP=chatqna-xeon-backend-server
- BACKEND_SERVICE_PORT=8888

  - DATAPREP_SERVICE_IP=dataprep-redis-service
  - DATAPREP_SERVICE_PORT=5000
ipc: host
restart: always

networks:
default:
driver: bridge

I have only changed shm_size: 4g and VLLM_CPU_KVCACHE_SPACE: 4 in vllm service.

But still I am getting error.

For docker compose -f compose.yaml up -d
I am getting error as dependency failed to start: container vllm-service exited (1)

For docker logs vllm-service | grep Complete
I am getting error as
docker logs vllm-service | grep Complete
Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:01<00:01, 1.53s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:37<00:00, 21.52s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:37<00:00, 18.53s/it]

Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 1121, in
uvloop.run(run_server(args))
File "/opt/venv/lib/python3.12/site-packages/uvloop/init.py", line 109, in run
return __asyncio.run(
^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 195, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/opt/venv/lib/python3.12/site-packages/uvloop/init.py", line 61, in wrapper
return await main
^^^^^^^^^^
File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 1069, in run_server
async with build_async_engine_client(args) as engine_client:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 269, in build_async_engine_client_from_engine_args
raise RuntimeError(
RuntimeError: Engine process failed to start. See stack trace for the root cause.
/root/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

For docker logs vllm-service
I am getting error as

docker logs vllm-service
INFO 06-16 05:29:57 [init.py:239] Automatically detected platform cpu.
WARNING 06-16 05:30:01 [_logger.py:72] Torch Profiler is enabled in the API server. This should ONLY be used for local development!
INFO 06-16 05:30:01 [api_server.py:1034] vLLM API server version 0.8.3
INFO 06-16 05:30:01 [api_server.py:1035] args: Namespace(host='0.0.0.0', port=80, uvicorn_log_level='info', disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=[''], allowed_methods=[''], allowed_headers=[''], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='google/gemma-2-2b-it', task='auto', tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=None, guided_decoding_backend='xgrammar', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, prefix_caching_hash_algo='builtin', disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=None, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, use_tqdm_on_load=True, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_config=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, additional_config=None, enable_reasoning=False, reasoning_parser=None, disable_cascade_attn=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False)
INFO 06-16 05:30:10 [config.py:600] This model supports multiple tasks: {'classify', 'score', 'generate', 'reward', 'embed'}. Defaulting to 'generate'.
WARNING 06-16 05:30:10 [_logger.py:72] device type=cpu is not supported by the V1 Engine. Falling back to V0.
INFO 06-16 05:30:10 [config.py:1634] Disabled the custom all-reduce kernel because it is not supported on current platform.
WARNING 06-16 05:30:10 [_logger.py:72] uni is not supported on CPU, fallback to mp distributed executor backend.
INFO 06-16 05:30:10 [api_server.py:246] Started engine process with PID 60
INFO 06-16 05:30:16 [init.py:239] Automatically detected platform cpu.
WARNING 06-16 05:30:17 [_logger.py:72] Torch Profiler is enabled in the API server. This should ONLY be used for local development!
INFO 06-16 05:30:17 [llm_engine.py:242] Initializing a V0 LLM engine (v0.8.3) with config: model='google/gemma-2-2b-it', speculative_config=None, tokenizer='google/gemma-2-2b-it', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=google/gemma-2-2b-it, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=None, chunked_prefill_enabled=False, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True,
INFO 06-16 05:30:18 [cpu.py:45] Using Torch SDPA backend.
INFO 06-16 05:30:19 [importing.py:16] Triton not installed or not compatible; certain GPU-related functions will not be available.
INFO 06-16 05:30:19 [cpu_worker.py:196] Profiling enabled. Traces will be saved to: /mnt
INFO 06-16 05:30:19 [parallel_state.py:957] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0
WARNING 06-16 05:30:19 [logger.py:63] Torch SPDA does not support logits soft cap. Outputs may be slightly off.
INFO 06-16 05:30:19 [weight_utils.py:265] Using model weights format ['.safetensors']
Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:01<00:01, 1.53s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:37<00:00, 21.52s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:37<00:00, 18.53s/it]

INFO 06-16 05:30:56 [loader.py:447] Loading weights took 37.16 seconds
INFO 06-16 05:30:56 [executor_base.py:112] # cpu blocks: 787, # CPU blocks: 0
INFO 06-16 05:30:56 [executor_base.py:117] Maximum concurrency for 8192 tokens per request: 12.30x
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 1121, in
uvloop.run(run_server(args))
File "/opt/venv/lib/python3.12/site-packages/uvloop/init.py", line 109, in run
return __asyncio.run(
^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 195, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/opt/venv/lib/python3.12/site-packages/uvloop/init.py", line 61, in wrapper
return await main
^^^^^^^^^^
File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 1069, in run_server
async with build_async_engine_client(args) as engine_client:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 269, in build_async_engine_client_from_engine_args
raise RuntimeError(
RuntimeError: Engine process failed to start. See stack trace for the root cause.
/root/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

[roshanjoebenny]

Hi @yinghu5 @xiguiw , please guide me how to solve this issue.

[xiguiw]

[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True, INFO 06-16 05:30:18 [cpu.py:45] Using Torch SDPA backend. INFO 06-16 05:30:19 [importing.py:16] Triton not installed or not compatible; certain GPU-related functions will not be available. INFO 06-16 05:30:19 [cpu_worker.py:196] Profiling enabled. Traces will be saved to: /mnt INFO 06-16 05:30:19 [parallel_state.py:957] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0 WARNING 06-16 05:30:19 [logger.py:63] Torch SPDA does not support logits soft cap. Outputs may be slightly off. INFO 06-16 05:30:19 [weight_utils.py:265] Using model weights format ['_.safetensors'] Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:01<00:01, 1.53s/it] Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:37<00:00, 21.52s/it] Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:37<00:00, 18.53s/it]

INFO 06-16 05:30:56 [loader.py:447] Loading weights took 37.16 seconds INFO 06-16 05:30:56 [executor_base.py:112] # cpu blocks: 787, # CPU blocks: 0 INFO 06-16 05:30:56 [executor_base.py:117] Maximum concurrency for 8192 tokens per request: 12.30x

Here the model weights is loaded completely.

Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 1121, in uvloop.run(run_server(args)) File "/opt/venv/lib/python3.12/site-packages/uvloop/init.py", line 109, in run return __asyncio.run( ^^^^^^^^^^^^^^ File "/root/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 195, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File "/root/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete File "/opt/venv/lib/python3.12/site-packages/uvloop/init.py", line 61, in wrapper return await main ^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 1069, in run_server async with build_async_engine_client(args) as engine_client: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 210, in aenter return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client async with build_async_engine_client_from_engine_args( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 210, in aenter return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 269, in build_async_engine_client_from_engine_args raise RuntimeError( RuntimeError: Engine process failed to start. See stack trace for the root cause. /root/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '

@roshanjoebenny

build_async_engine_client_from_engine_args raise RuntimeError( RuntimeError: Engine process failed to start.

build_async_engine_client_from_engine_args raise RuntimeError( RuntimeError: Engine process failed to start. See stack trace for the root cause. /root/.local/share/uv/python/cpython-3.12.10-linux-x86_64-

[xiguiw]

@roshanjoebenny

I cannot reproduce your issue with the model google/gemma-2-2b-it

The log is incomplete, it ends here. It indicates that it's blocked by something, there maybe some issue of the system level.

UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '

Not quite sure it's the leaked semaphore, other resource limitation of CPU or memory etc. （samiliar issue in vllm: https://github.com/vllm-project/vllm/issues/17372）

Would you please try to start vLLM only (so it consumes lesss resource) with the following command
1. set all environments (the same as you start ChatQnA before)
2. start vllm-service only by

docker docker rmi $(docker images -a -q)  
docker compose up vllm-service

1. Do you use 4-VCPU?
   Could you try 8-VCPU or 16-VCPU? For ChatQna, at least 16-VCPU (8 physical core are needed.
2. The memory is 15G? Even for AIPC, we recommend 32G memory. and the disk space: select OS disk size as 128GiB.
   The recommended configurations are Amazon EC2 C7i.8xlarge and C7i.16xlarge instance types. These instances are Intel® Xeon® with AMX, to leverage 4th Generation Intel® Xeon® Scalable processors that are optimized for demanding workloads.

Here is my log:

docker compose up vllm-se                           rviceble Watch
WARN[0000] Found orphan containers ([codegen-xeon-ui-server codegen-xeon-backend-server dataprep-redis-server retr                           iever-redis llm-codegen-vllm-server tei-embedding-server llm-textgen-server tei-embedding-serving]) for this proje                           ct. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphan                           s flag to clean it up.
[+] Running 1/1
 ✔ Container vllm-service  Created                                                                           0.0s
Attaching to vllm-service
vllm-service  | INFO 06-16 09:47:59 [__init__.py:239] Automatically detected platform cpu.
vllm-service  | WARNING 06-16 09:48:00 [_logger.py:72] Torch Profiler is enabled in the API server. This should ON                           LY be used for local development!
vllm-service  | INFO 06-16 09:48:00 [api_server.py:1034] vLLM API server version 0.8.3
vllm-service  | INFO 06-16 09:48:00 [api_server.py:1035] args: Namespace(host='0.0.0.0', port=80, uvicorn_log_leve                           l='info', disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'],                            allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_c                           ontent_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ss                           l_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_fronten                           d_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, to                           ol_parser_plugin='', model='google/gemma-2-2b-it', task='auto', tokenizer=None, hf_config_path=None, skip_tokenize                           r_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code                           =False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'a                           uto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=None, guided_decoding_backend='xgrammar', logits_process                           or_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_s                           ize=1, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsig                           ht=False, block_size=None, enable_prefix_caching=None, prefix_caching_hash_algo='builtin', disable_sliding_window=                           False, use_v2_block_manager=True, num_lookahead_slots=0, seed=None, swap_space=4, cpu_offload_gb=0, gpu_memory_uti                           lization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_partial_prefills=1, max_long_part                           ial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quant                           ization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8                           192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_confi                           g=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False                           , enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_s                           caling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapte                           rs=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, use_tqdm_on_load=True, multi_step_stream_o                           utputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_config=None, model_loader_extra_                           config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, sh                           ow_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_                           proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None,                            override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_extension                           _cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=F                           alse, additional_config=None, enable_reasoning=False, reasoning_parser=None, disable_cascade_attn=False, disable_l                           og_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server                           _load_tracking=False)
vllm-service  | INFO 06-16 09:48:14 [config.py:600] This model supports multiple tasks: {'embed', 'classify', 'rew                           ard', 'score', 'generate'}. Defaulting to 'generate'.
vllm-service  | WARNING 06-16 09:48:14 [_logger.py:72] device type=cpu is not supported by the V1 Engine. Falling                            back to V0.
vllm-service  | INFO 06-16 09:48:14 [config.py:1634] Disabled the custom all-reduce kernel because it is not suppo                           rted on current platform.
vllm-service  | WARNING 06-16 09:48:14 [_logger.py:72] uni is not supported on CPU, fallback to mp distributed exe                           cutor backend.
vllm-service  | INFO 06-16 09:48:14 [api_server.py:246] Started engine process with PID 280
vllm-service  | INFO 06-16 09:48:22 [__init__.py:239] Automatically detected platform cpu.
vllm-service  | WARNING 06-16 09:48:24 [_logger.py:72] Torch Profiler is enabled in the API server. This should ON                           LY be used for local development!
vllm-service  | INFO 06-16 09:48:24 [llm_engine.py:242] Initializing a V0 LLM engine (v0.8.3) with config: model='                           google/gemma-2-2b-it', speculative_config=None, tokenizer='google/gemma-2-2b-it', skip_tokenizer_init=False, token                           izer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtyp                           e=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeli                           ne_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=True, kv_cache_dtype=auto,  d                           evice_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar', reasoning_backend=None), obse                           rvability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_t                           ime=False, collect_model_execute_time=False), seed=None, served_model_name=google/gemma-2-2b-it, num_scheduler_ste                           ps=1, multi_step_stream_outputs=True, enable_prefix_caching=None, chunked_prefill_enabled=False, use_async_output_                           proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config=                           {"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,                           160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outp                           uts=True,
vllm-service  | INFO 06-16 09:48:28 [cpu.py:45] Using Torch SDPA backend.
vllm-service  | INFO 06-16 09:48:28 [importing.py:16] Triton not installed or not compatible; certain GPU-related                            functions will not be available.
vllm-service  | INFO 06-16 09:48:28 [cpu_worker.py:196] Profiling enabled. Traces will be saved to: /mnt
vllm-service  | INFO 06-16 09:48:28 [parallel_state.py:957] rank 0 in world size 1 is assigned as DP rank 0, PP ra                           nk 0, TP rank 0
vllm-service  | WARNING 06-16 09:48:28 [logger.py:63] Torch SPDA does not support logits soft cap. Outputs may be                            slightly off.
vllm-service  | INFO 06-16 09:48:29 [weight_utils.py:265] Using model weights format ['*.safetensors']

vllm-service  | INFO 06-16 09:54:52 [weight_utils.py:281] Time spent downloading weights for google/gemma-2-2b-it: 382.841347 seconds
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:00<00:00,  8.84it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:02<00:00,  1.17s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:02<00:00,  1.01s/it]
vllm-service  |
vllm-service  | INFO 06-16 09:54:57 [loader.py:447] Loading weights took 2.46 seconds
vllm-service  | INFO 06-16 09:54:57 [executor_base.py:112] # cpu blocks: 3150, # CPU blocks: 0
vllm-service  | INFO 06-16 09:54:57 [executor_base.py:117] Maximum concurrency for 8192 tokens per request: 49.22x
vllm-service  | INFO 06-16 09:55:11 [llm_engine.py:448] init engine (profile, create kv cache, warmup model) took 14.46 seconds
vllm-service  | INFO 06-16 09:55:14 [api_server.py:1081] Starting vLLM API server on http://0.0.0.0:80
vllm-service  | INFO 06-16 09:55:14 [launcher.py:26] Available routes are:
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /openapi.json, Methods: GET, HEAD
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /docs, Methods: GET, HEAD
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /docs/oauth2-redirect, Methods: GET, HEAD
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /redoc, Methods: GET, HEAD
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /health, Methods: GET
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /load, Methods: GET
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /ping, Methods: GET, POST
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /tokenize, Methods: POST
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /detokenize, Methods: POST
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /v1/models, Methods: GET
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /version, Methods: GET
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /v1/chat/completions, Methods: POST
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /v1/completions, Methods: POST
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /v1/embeddings, Methods: POST
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /pooling, Methods: POST
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /score, Methods: POST
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /v1/score, Methods: POST
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /v1/audio/transcriptions, Methods: POST
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /rerank, Methods: POST
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /v1/rerank, Methods: POST
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /v2/rerank, Methods: POST
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /invocations, Methods: POST
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /start_profile, Methods: POST
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /stop_profile, Methods: POST
vllm-service  | INFO 06-16 09:55:14 [launcher.py:34] Route: /metrics, Methods: GET
vllm-service  | INFO:     Started server process [1]
vllm-service  | INFO:     Waiting for application startup.
vllm-service  | INFO:     Application startup complete.
vllm-service  | INFO:     172.22.0.1:45796 - "GET /health HTTP/1.1" 200 OK
vllm-service  | INFO:     172.22.0.1:52968 - "GET /health HTTP/1.1" 200 OK
vllm-service  | INFO:     172.22.0.1:46554 - "GET /health HTTP/1.1" 200 OK
vllm-service  | INFO:     172.22.0.1:41200 - "GET /health HTTP/1.1" 200 OK
vllm-service  | INFO:     172.22.0.1:53756 - "GET /health HTTP/1.1" 200 OK
vllm-service  | INFO:     172.22.0.1:55924 - "GET /health HTTP/1.1" 200 OK

[roshanjoebenny]

Hi @xiguiw ,
I tried out docker compose up vllm-service command and I was getting the response shown above.
But when I try again with docker compose -f compose.yaml up -d command and I am getting the dependency failed to start: container vllm-service exited (1) error.

[xiguiw]

Hi @xiguiw , I tried out docker compose up vllm-service command and I was getting the response shown above. But when I try again with docker compose -f compose.yaml up -d command and I am getting the dependency failed to start: container vllm-service exited (1) error.

@roshanjoebenny

I suspect the resource limitation.
So even just start vllm-container only, you get the same error.

1. Did you stop and remove all the containers and start vllm-only?

docker compose -f compose.yaml down
docker compose -f compose.yaml up vllm-service

2. Have you tried Amazon EC2 C7i.8xlarge and C7i.16xlarge instance types?