You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Proposal: when the bot's active model doesn't support image input, automatically describe the image with a user-configurable auxiliary vision model and forward the description (plus the original caption) to the main model as text.
Why: today, if the model has no vision support, the photo is discarded and the user only gets "model doesn't support image input" (plus the caption at best). The visual content is completely lost. This feature:
eliminates that information loss, regardless of which model is running;
lets users pair a cheap/fast vision model (for describing) with a more capable non-vision model (for reasoning), instead of being forced to use an expensive vision model for everything.
Design principles: opt-in and backward compatible — if the fallback model isn't configured, current behavior is unchanged.
Detailed request: the full proposal (requirements, prior art from Hermes Agent and OpenClaw) is tracked in issue #151.
Status: the behavior described above is still present in main. The codebase has since matured exactly where this feature needs it — model-capabilities-service (getModelCapabilities/supportsInput), model-selection-service (getStoredModel), and the runtime /settings command (#171) — so it could now be implemented in a fairly surgical way.
Question to @grinev: is this a direction you'd be willing to accept? If so, I'm happy to contribute in small reviewable steps (e.g. starting with photo-handler only, no new commands), or to provide whatever input you'd find useful. Thanks!
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Proposal: when the bot's active model doesn't support image input, automatically describe the image with a user-configurable auxiliary vision model and forward the description (plus the original caption) to the main model as text.
Why: today, if the model has no vision support, the photo is discarded and the user only gets "model doesn't support image input" (plus the caption at best). The visual content is completely lost. This feature:
Design principles: opt-in and backward compatible — if the fallback model isn't configured, current behavior is unchanged.
Detailed request: the full proposal (requirements, prior art from Hermes Agent and OpenClaw) is tracked in issue #151.
Status: the behavior described above is still present in
main. The codebase has since matured exactly where this feature needs it —model-capabilities-service(getModelCapabilities/supportsInput),model-selection-service(getStoredModel), and the runtime/settingscommand (#171) — so it could now be implemented in a fairly surgical way.Question to @grinev: is this a direction you'd be willing to accept? If so, I'm happy to contribute in small reviewable steps (e.g. starting with
photo-handleronly, no new commands), or to provide whatever input you'd find useful. Thanks!All reactions