[Serve] Flagscale serve supports automatically composition of multiple models #732
Conversation
|
I'm not sure, maybe it's a better choice to implement it at a higher level (adding a new compose-mode with no changes in serve-mode) ? |
| deploy: | ||
| port: 6701 | ||
| use_fs_serve: true | ||
| enable_omposition: true |
There was a problem hiding this comment.
what does this parameter means?
There was a problem hiding this comment.
enable_composition means using composition of multiple models.
|
|
||
| from dag_utils import check_and_get_port | ||
| from fastapi import FastAPI, HTTPException, Request | ||
| from pydantic import create_model | ||
| from ray import workflow | ||
| from ray import serve | ||
| from ray.serve.handle import DeploymentHandle |
There was a problem hiding this comment.
need add ray version restrictions？
There was a problem hiding this comment.
Nice idea, more ray versions will be tested later
flagscale/serve/engine.py
Outdated
|
|
||
| from flagscale.logger import logger | ||
|
|
||
| RequestData = create_model( |
There was a problem hiding this comment.
how to deal with multimodal input
There was a problem hiding this comment.
set config as follows:
deploy:
request:
args:
- prompt
- num
types:
- str
- int
As discussed, what you mentioned refers to combining different independent services, while this scenario focuses on combining multiple different models. Your idea will be taken into consideration |
There was a problem hiding this comment.
uses: ./.github/workflows/functional-tests.yml should be changed to uses: ./.github/workflows/functional-tests-nvidia.yml
cyber-pioneer
changed the title
cyber-pioneer
force-pushed
the
submodules
branch
from
9f273a5 to
d4e58d0
Compare
Description
Flagscale serve supports automatically composition of multiple models
python run.py --config-path ./examples/qwen2_5/conf --config-name serve_multiple_models action=run