Skip to content

Update the "InterleavedTextMedia" type #635

Merged
ashwinb merged 12 commits intomainfrom
no_urls_in_models
Dec 17, 2024
Merged

Update the "InterleavedTextMedia" type #635
ashwinb merged 12 commits intomainfrom
no_urls_in_models

Conversation

@ashwinb
Copy link
Contributor

@ashwinb ashwinb commented Dec 16, 2024

What does this PR do?

This is a long-pending change and particularly important to get done now.

Specifically:

  • we cannot "localize" (aka download) any URLs from media attachments anywhere near our modeling code. it must be done within llama-stack.
  • PIL.Image is infesting all our APIs via ImageMedia -> InterleavedTextMedia and that cannot be right at all. Anything in the API surface must be "naturally serializable". We need a standard { type: "image", image_url: "<...>" } which is more extensible
  • UserMessage, SystemMessage, etc. are moved completely to llama-stack from the llama-models repository.

See meta-llama/llama-models#244 for the corresponding PR in llama-models.

Test Plan

cd llama_stack/providers/tests

pytest -s -v -k "fireworks or ollama or together" inference/test_vision_inference.py
pytest -s -v -k "(fireworks or ollama or together) and llama_3b" inference/test_text_inference.py
pytest -s -v -k chroma memory/test_memory.py \
  --env EMBEDDING_DIMENSION=384 --env CHROMA_DB_PATH=/tmp/foobar

pytest -s -v -k fireworks agents/test_agents.py  \
   --safety-shield=meta-llama/Llama-Guard-3-8B \
   --inference-model=meta-llama/Llama-3.1-8B-Instruct

Updated the client sdk (see PR ...), installed the SDK in the same environment and then ran the SDK tests:

cd tests/client-sdk
LLAMA_STACK_CONFIG=together pytest -s -v agents/test_agents.py
LLAMA_STACK_CONFIG=ollama pytest -s -v memory/test_memory.py

# this one needed a bit of hacking in the run.yaml to ensure I could register the vision model correctly
INFERENCE_MODEL=llama3.2-vision:latest LLAMA_STACK_CONFIG=ollama pytest -s -v inference/test_inference.py

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Dec 16, 2024
@ashwinb ashwinb changed the title (WIP) Update the "InterleavedTextMedia" type Update the "InterleavedTextMedia" type Dec 17, 2024
@ashwinb ashwinb merged commit 8de8eb0 into main Dec 17, 2024
2 checks passed
@ashwinb ashwinb deleted the no_urls_in_models branch December 17, 2024 19:18
ashwinb added a commit to llamastack/llama-stack-client-python that referenced this pull request Dec 17, 2024
ashwinb added a commit to llamastack/llama-stack-apps that referenced this pull request Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants