Cannot start Phi-3-mini-128k-instruct from Docker

[annadmitrieva]

System Info

Using two official Docker images (latest and main).

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

After running
docker run --gpus all -p 8080:80 -v /home/ubuntu/lorax_weight_cache ghcr.io/predibase/lorax:latest --model-id microsoft/Phi-3-mini-128k-instruct --trust-remote-code

I get:

  File "/opt/conda/bin/lorax-server", line 8, in <module>
    sys.exit(app())

  File "/opt/conda/lib/python3.10/site-packages/lorax_server/cli.py", line 83, in serve
    server.serve(

  File "/opt/conda/lib/python3.10/site-packages/lorax_server/server.py", line 309, in serve
    asyncio.run(

  File "/opt/conda/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)

  File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()

  File "/opt/conda/lib/python3.10/site-packages/lorax_server/server.py", line 243, in serve_inner
    model = get_model(

  File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/__init__.py", line 251, in get_model
    return FlashPhi3(

  File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/flash_phi3.py", line 88, in __init__
    model = FlashPhi3ForCausalLM(config, weights)

  File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/custom_modeling/flash_phi3_modeling.py", line 482, in __init__
    self.model = FlashPhi3Model(config, weights)

  File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/custom_modeling/flash_phi3_modeling.py", line 422, in __init__
    [

  File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/custom_modeling/flash_phi3_modeling.py", line 423, in <listcomp>
    FlashPhi3Layer(

  File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/custom_modeling/flash_phi3_modeling.py", line 360, in __init__
    self.self_attn = FlashPhi3Attention(

  File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/custom_modeling/flash_phi3_modeling.py", line 190, in __init__
    self.rotary_emb = PositionRotaryEmbedding.static(

  File "/opt/conda/lib/python3.10/site-packages/lorax_server/utils/layers.py", line 879, in static
    scaling_factor = rope_scaling["factor"]

KeyError: 'factor'

After running
docker run --gpus all -p 8080:80 -v /home/ubuntu/lorax_weight_cache ghcr.io/predibase/lorax:main --model-id microsoft/Phi-3-mini-128k-instruct --trust-remote-code=True

I get:

  File "/opt/conda/bin/lorax-server", line 8, in <module>
    sys.exit(app())
  File "/opt/conda/lib/python3.10/site-packages/typer/main.py", line 311, in __call__
    return get_command(self)(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/typer/core.py", line 778, in main
    return _main(
  File "/opt/conda/lib/python3.10/site-packages/typer/core.py", line 216, in _main
    rv = self.invoke(ctx)
  File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/typer/main.py", line 683, in wrapper
    return callback(**use_params)  # type: ignore
  File "/opt/conda/lib/python3.10/site-packages/lorax_server/cli.py", line 87, in serve
    server.serve(
  File "/opt/conda/lib/python3.10/site-packages/lorax_server/server.py", line 371, in serve
    asyncio.run(
  File "/opt/conda/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 636, in run_until_complete
    self.run_forever()
  File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
    self._run_once()
  File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
    handle._run()
  File "/opt/conda/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
> File "/opt/conda/lib/python3.10/site-packages/lorax_server/server.py", line 251, in serve_inner
    model = get_model(
  File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/__init__.py", line 275, in get_model
    return FlashPhi3(
  File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/flash_phi3.py", line 88, in __init__
    model = FlashPhi3ForCausalLM(config, weights)
  File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/custom_modeling/flash_phi3_modeling.py", line 482, in __init__
    self.model = FlashPhi3Model(config, weights)
  File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/custom_modeling/flash_phi3_modeling.py", line 422, in __init__
    [
  File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/custom_modeling/flash_phi3_modeling.py", line 423, in <listcomp>
    FlashPhi3Layer(
  File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/custom_modeling/flash_phi3_modeling.py", line 360, in __init__
    self.self_attn = FlashPhi3Attention(
  File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/custom_modeling/flash_phi3_modeling.py", line 190, in __init__
    self.rotary_emb = PositionRotaryEmbedding.static(
  File "/opt/conda/lib/python3.10/site-packages/lorax_server/utils/layers.py", line 479, in static
    raise NotImplementedError(f"rope scaling type {rope_type} is not implemented or invalid")
NotImplementedError: rope scaling type longrope is not implemented or invalid

The --trust-remote-code flag is not an issue, running the same command without it outputs the same errors.

Expected behavior

Server should start successfully.

[prd-tuong-nguyen]

Same problem
The reason might be that Microsoft just updated their weight and changed rope_scaling from "type": "su" to "type": "longrope".

[prd-tuong-nguyen]

@tgaddair Can you help me with this? I really need to solve this issue for my project.

[prd-tuong-nguyen]

@tgaddair transformers seems to fixed this issue, can you upgrade it? huggingface/transformers#31436

[huytuong010101]

@tgaddair can you help me check this PR? #576

[annadmitrieva]

Dear all, I'm sorry, but the issue still persists. I ran the same command two times:

model=microsoft/Phi-3-mini-128k-instruct
volume=$PWD/data

docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/predibase/lorax:main --model-id $model

The first time it suggested that I upgrade to Ubuntu>=12.4, which hasn't happened before:

Status: Downloaded newer image for ghcr.io/predibase/lorax:main
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: requirement error: unsatisfied condition: cuda>=12.4, please update your driver to a newer version, or use an earlier cuda container: unknown.

After I switched to a machine with Ubuntu 12.4, this happened:

2024-08-20T13:55:44.120455Z ERROR shard-manager: lorax_launcher: Shard complete standard error output:

/opt/conda/lib/python3.10/site-packages/torch/cuda/__init__.py:128: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination (Triggered internally at /opt/conda/conda-bld/pytorch_1720538438429/work/c10/cuda/CUDAFunctions.cpp:108.)
  return torch._C._cuda_getDeviceCount() > 0
The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
Traceback (most recent call last):

  File "/opt/conda/bin/lorax-server", line 8, in <module>
    sys.exit(app())

  File "/opt/conda/lib/python3.10/site-packages/lorax_server/cli.py", line 87, in serve
    server.serve(

  File "/opt/conda/lib/python3.10/site-packages/lorax_server/server.py", line 400, in serve
    asyncio.run(

  File "/opt/conda/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)

  File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()

  File "/opt/conda/lib/python3.10/site-packages/lorax_server/server.py", line 266, in serve_inner
    model = get_model(

  File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/__init__.py", line 275, in get_model
    from lorax_server.models.flash_phi3 import FlashPhi3

  File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/flash_phi3.py", line 9, in <module>
    from lorax_server.models.custom_modeling.flash_phi3_modeling import (

  File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/custom_modeling/flash_phi3_modeling.py", line 32, in <module>
    from lorax_server.utils import flash_attn, paged_attention

  File "/opt/conda/lib/python3.10/site-packages/lorax_server/utils/flash_attn.py", line 277, in <module>
    raise NotImplementedError("flash attention is not installed")

NotImplementedError: flash attention is not installed
 rank=0
Error: ShardCannotStart
2024-08-20T13:55:44.219106Z ERROR lorax_launcher: Shard 0 failed to start
2024-08-20T13:55:44.219138Z  INFO lorax_launcher: Shutting down shards

When I tried to run lorax:latest, I got the same error as in my original message. @huytuong010101 perhaps you have succeeded in running lorax with Phi3 after your pr had been merged? Could you share your environment specifications?

[prd-tuong-nguyen]

@annadmitrieva You can try this tag: https://github.com/predibase/lorax/pkgs/container/lorax/259352704?tag=bb2cdd6 instead of main tag. It works from my side.
I run on Ubuntu 20.04 and GPU A10G

[annadmitrieva]

Just leaving this here for future reference: I was able to successfully run the tag https://github.com/predibase/lorax/pkgs/container/lorax/259352704?tag=bb2cdd6 on Ubuntu 22.04, but not on Ubuntu 24. Might have something to do with the pre-installed software on the machines rather than the OS itself though.