Skip to content

Multimodal support #35

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
ScReameer wants to merge 27 commits into
base: main
Choose a base branch
from
Open

Multimodal support #35

ScReameer wants to merge 27 commits into from

Conversation

@ScReameer
Copy link

@ScReameer ScReameer commented Dec 1, 2025

Key changes

  • Enable multimodal embeddings: text stays on embed, image requests now call image_embed, and an audio path is stubbed with NotImplementedError for future support.
  • Update infinity-emb to 0.0.77.
  • Pin optimum==1.24.0 so the Docker image no longer crashes with ModuleNotFoundError.
  • Extend handler.py to read an optional modality field (text default; accepts text, image, audio) for both native and OpenAI-compatible payloads.
  • Updated RunPod tests .runpod/tests.json to cover text and image scenarios
  • Add a repository-level httpx.AsyncClient with configurable timeout, pooling, and User-Agent to stabilise remote image downloads.
  • Add a local testing with pytest tests -v

@ScReameer
update infinity-emb version to 0.0.77 and add optimum dependency
8d07ffb
@ScReameer
added clip-based model
9acf3f0
@ScReameer
multimodal tests
2c130ca
@ScReameer
added multimodal routing (images and/or text)
17fee84
@ScReameer
support mixed text and image inputs, improve error handling and logging
3b6a67e
@ScReameer
update README to reflect multimodal support for text and image inputs
ed7b1f8
@ScReameer
local test modality ignore
267e959
@ScReameer
add httpx dependency
6aa6a3f
@ScReameer
updated tests
6b66b7d
@ScReameer
refactor: enhance embedding service with HTTP client and improved mod…
d86b8ac 
…ality validation
@ScReameer
fix: add modality extraction for OpenAI embeddings handling
9fe3f61
@ScReameer
feat: implement http client with configurable settings and user agent
802f465
@ScReameer
feat: enhance image handling with RGB conversion, URL downloading, an…
c96b657 
…d base64 support
@ScReameer
remove extract_modality function and simplify modality extraction in …
9db8204 
…handler
@ScReameer
fix: update CMD syntax in Dockerfile
3ba1f15
@ScReameer
docs: update README to clarify multimodal support and explicit modali…
f61a9cc 
…ty selection
@ScReameer
fix: remove test_modality.py from .gitignore
d8c8292
@ScReameer
docs: update README to reflect explicit modality selection
96a9afb
@ScReameer
test: add integration tests for explicit modality API using pytest
2e965d7
@ScReameer
docs: reorder sections in README for improved clarity
abde653
@ScReameer
docs: update section on modality selection to validation and image fe…
a84f8cf 
…tching defaults
@ScReameer
fix: streamline modality specification in test cases by removing extr…
6101244 
…a_body
@ScReameer
refactor: centralize HTTP client configuration using HttpClientConfig
bcfcc18
@ScReameer
fix: enhance error handling for modality in embedding routes
108ee4d
@ScReameer ScReameer marked this pull request as ready for review December 1, 2025 14:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ScReameer