feat(server): send sampled video frames as ordered images

docs/environment-variables.md

@@ -415,6 +415,8 @@ both be on `PATH` and must be **ffmpeg 5.0 (2022) or newer**. See
 | `MLXCEL_VIDEO_MAX_PIXELS` | unsigned integer | `16777216` | Rejects source videos whose `width × height` exceeds the cap. |
 | `MLXCEL_VIDEO_MAX_DURATION_SEC` | float seconds | `600` | Rejects source videos longer than the cap. |
 | `MLXCEL_VIDEO_MAX_PNG_FRAME_BYTES` | unsigned integer bytes | `268435456` | Per-frame cap for the ffmpeg PNG stream splitter. |
+| `MLXCEL_VIDEO_MAX_FRAMES` | unsigned integer (minimum 2) | `16` | Frames kept when a clip is served as ordered still images, because the loaded checkpoint has no native video path. Values below 2 are raised to 2, so the first and the last sampled frame always survive. The flag spelling is `--video-max-frames` on `mlxcel serve`, `mlxcel-server` and `mlxcel generate`. Frames spend the per-request image budget (`--max-images`), so raising this past that limit is refused. A family with a native video path (Gemma 4, Inkling, Kimi-VL, Qwen-VL) samples through its own processor and ignores this. |
+| `MLXCEL_VIDEO_FPS` | float > 0 | `2.0` | Sampling rate the video-to-images fallback decodes a clip at when the request's own `video_url.fps` is absent. Server-side only (`--video-fps`); the CLI keeps `--fps` for the same dial. Read only on the fallback path, for the same reason as the row above. |
 
 ## Hardware and kernel diagnostic variables
 

docs/llama-server-compat.md

@@ -468,6 +468,18 @@ Capability comes from the same `config.json`-only model-type probe that already
 
 One arm of the gate is not a capability gap and does not take that shape. A checkpoint that consumes video alone and audio alone but has no merge path for the pair refuses a request carrying `video_url` and `input_audio` together with `Combined video and audio inputs are not supported`, typed `invalid_request_error` at HTTP 400 rather than 501, because dropping either part leaves a request the checkpoint can serve (issue #1349). `gemma4_unified` merges both and is admitted. The per-modality checks run first, so a text-only checkpoint sent the pair still hears which modality it has no tower for.
 
+### `video_url` on a checkpoint with no native video path
+
+A `video_url` content block used to be refused by every family except the handful with a temporal processor, although each of the refusing ones could read a sequence of still images perfectly well. Since issue #1322 a checkpoint with a vision tower and no native video path serves the block instead: the clip is decoded at the request's `video_url.fps` (falling back to `--video-fps`, default 2.0), evenly subsampled to at most `--video-max-frames` frames with the first and the last always kept, and rewritten into that many `image_url` parts in chronological order, preceded by a text part reading `Here is a video as a sequence of N frames in chronological order.` The rewrite happens on the request, after the capability gate and before the template renders, so the prompt carries one image placeholder per frame and everything downstream (soft-token budgets, the per-request image limit, the prompt-cache multimodal digest) treats the frames as the images they now are.
+
+```text
+model gemma-3-4b-it-4bit has no native video path; sending 8 of 8 sampled frames from /srv/clips/demo.mp4 as ordered images
+```
+
+That `info` line is the only signal that the substitution happened; `GET /props` reports `video: true` either way, because either way the block is accepted. The families with a native path (Gemma 4 VL, Gemma 4 Unified, Inkling, Kimi-VL / Kimi-VL 2.5, and the Qwen-VL generations) are untouched by this and never log it. Muse Glimmer is excluded on purpose: the CLI refuses `--video` for that family by name, and admitting the clip on the HTTP boundary alone would leave the two fronts disagreeing about one checkpoint. A checkpoint with no vision tower keeps its 501.
+
+Two limits are worth knowing. The frames spend the ordinary per-request image budget, so `--video-max-frames` above `--max-images` (default 16 for both) is refused with a message naming the frame count, the caller's own image count and the limit rather than with a bare image-count error. And the substitution carries no temporal information: the model receives ordered pictures and the sentence saying they are one clip, not frame timestamps or motion, so a reply can describe what changes across the frames but cannot be asked about speed or duration. A local `file://` path still has to sit inside a directory named by `MLXCEL_VIDEO_DIR_ALLOWLIST`, which is fail-closed and unchanged.
+
 ### `--image-min-tokens` / `--image-max-tokens` move real pixel bounds
 
 Upstream converts a token budget into pixel bounds in `clip_hparams::set_limit_image_tokens`:

docs/supported-models.md

@@ -214,6 +214,54 @@ model must also merge the two into one prompt, which today only `gemma4_unified`
 does. Video frame extraction uses the system `ffmpeg`/`ffprobe` binaries at
 runtime.
 
+#### Video on a model with no native video path
+
+Two different things answer a clip, and the difference is worth knowing before
+reading a reply.
+
+**Native video** means the family consumes the clip as a clip: a 3D patch grid,
+adjacent-frame temporal planes, or per-frame scatter into video placeholders.
+Gemma 4 VL, Gemma 4 Unified, Inkling, Kimi-VL / Kimi-VL 2.5 and the Qwen-VL
+families (Qwen2-VL, Qwen2.5-VL, Qwen3-VL, Qwen3-VL MoE, Qwen3.5/3.6/3.8-VL) all
+do this, each through its own processor, and none of them is affected by the
+paragraph below.
+
+**The frames fallback** is what every other checkpoint with a vision tower does
+(issue #1322). The clip is decoded at the requested fps, evenly subsampled to at
+most `--video-max-frames` frames with the first and last always kept,
+PNG-encoded, and sent to the model as that many ordinary images in chronological
+order, preceded by the sentence `Here is a video as a sequence of N frames in
+chronological order.` so the model reads them as one clip. Nothing about the
+image pipeline changes: the template emits one image placeholder per frame, the
+vision tower sees stills, and the prompt-cache multimodal digest hashes the
+frame bytes, so two requests for the same clip at the same fps and frame cap
+share a prefix and a different clip does not.
+
+What this costs and what it does not buy:
+
+- The prompt grows by the model's per-image token cost times the kept frame
+  count, which for a 16-frame default on a family that spends 256 tokens per
+  image is about 4,096 prompt tokens.
+- The frames spend the per-request image budget (`--max-images`, default 16), so
+  a clip plus the caller's own pictures can exceed it. The refusal names the
+  frame count, the caller's own image count and the limit.
+- There is no temporal encoding. The model sees an ordered set of pictures and
+  the sentence saying they are one video; it does not receive frame timestamps
+  or motion between frames. Expect it to describe what changes across the
+  frames, not to reason about speed or duration.
+
+The fallback is on wherever the checkpoint has a vision tower and no native path
+(Gemma 3, LLaVA, LLaVA-NeXT, Pixtral, SmolVLM / Idefics3, LFM2-VL, InternVL,
+MiniCPM-o, and so on), on the CLI (`--video`) and on the server (`video_url`).
+Muse Glimmer is the one exception: it refuses `--video` and `video_url` by name
+on both fronts until the family is qualified for multi-image prompts. A
+checkpoint with no vision tower at all keeps refusing video, as before.
+
+`GET /props` reports `video: true` for both kinds, because both accept a
+`video_url` block. There is no wire-level flag distinguishing them; the server
+log line `model <id> has no native video path; sending K of N sampled frames
+from <file> as ordered images` is what says the substitution happened.
+
 **Supported ffmpeg range: 5.0 (2022) or newer**, on both the CLI (`--video`)
 and the server (`video_url`). Both binaries must be on `PATH`; neither is a
 build-time dependency, and a missing one produces a named error rather than a

src/bin/mlx_server.rs

@@ -1470,6 +1470,34 @@ struct ServerArgs {
     #[arg(long = "vision-cache-size", default_value_t = 20, value_name = "N")]
     vision_cache_size: usize,
 
+    /// Maximum sampled frames kept when a `video_url` block is served as
+    /// ordered still images.
+    ///
+    /// Applies only when the loaded checkpoint has no native video path; a
+    /// native video family samples through its own processor and ignores this.
+    /// Values below 2 are raised to 2, so the first and last sampled frame are
+    /// always kept. Also reads `MLXCEL_VIDEO_MAX_FRAMES`.
+    #[arg(
+        long = "video-max-frames",
+        env = "MLXCEL_VIDEO_MAX_FRAMES",
+        default_value_t = mlxcel::multimodal::video::DEFAULT_FALLBACK_MAX_FRAMES,
+        value_name = "N"
+    )]
+    video_max_frames: usize,
+
+    /// Frames-per-second the video-to-images fallback decodes a clip at when
+    /// the request carries no per-`video_url` `fps` of its own.
+    ///
+    /// Applies only when the loaded checkpoint has no native video path. Also
+    /// reads `MLXCEL_VIDEO_FPS`.
+    #[arg(
+        long = "video-fps",
+        env = "MLXCEL_VIDEO_FPS",
+        default_value_t = mlxcel::multimodal::video::DEFAULT_FPS,
+        value_name = "FLOAT"
+    )]
+    video_fps: f64,
+
     /// Maximum encoded bytes accepted for each image input.
     ///
     /// Also reads `LLAMA_ARG_MAX_IMAGE_PAYLOAD_SIZE`.
@@ -2590,6 +2618,8 @@ fn build_startup_input(mut args: ServerArgs) -> anyhow::Result<ServerStartupInpu
         tp_embedding_mode: args.tp_embedding_mode,
         tp_lm_head_mode: args.tp_lm_head_mode,
         vision_cache_size: args.vision_cache_size,
+        video_max_frames: args.video_max_frames,
+        video_fps: args.video_fps,
         max_image_payload_size: args.max_image_payload_size,
         max_images_per_request: args.max_images_per_request,
         max_image_width: args.max_image_width,

src/commands/generate.rs

@@ -864,6 +864,104 @@ fn cli_video_content_part_count(model_path: &Path, num_videos: usize) -> usize {
     }
 }
 
+/// Frames a `--video` clip was replaced with, plus the guards that delete the
+/// temporary PNGs when the run ends (issue #1322).
+pub(crate) struct CliVideoFrames {
+    /// Temporary PNG paths, in chronological order, to append to `--image`.
+    pub(crate) frame_paths: Vec<std::path::PathBuf>,
+    /// Sentence telling the model the images are one clip.
+    pub(crate) lead_text: String,
+    /// RAII guards. Held for the life of the run; dropping them unlinks the
+    /// PNGs, so they must outlive the vision-tower read.
+    pub(crate) _guards: Vec<mlxcel::video::TempFile>,
+}
+
+/// Decode `--video` into ordered still images when the checkpoint has no
+/// native video path (issue #1322).
+///
+/// `Ok(None)` for a family that consumes the clip itself, and for a request
+/// with no `--video` at all, so the native paths in
+/// `generate_vlm::compute_vlm_embeddings` keep seeing their video list. A
+/// checkpoint with no vision tower also returns `Ok(None)`: there is nowhere to
+/// send frames, and the refusal it already produces names that.
+///
+/// The clip is decoded at `target_fps`, evenly subsampled to `max_frames`
+/// (first and last always kept), PNG-encoded, and written to the system temp
+/// directory. Server-side the equivalent rewrite happens in
+/// `server::chat_request::expand_video_parts_to_frames`.
+pub(crate) fn expand_cli_videos_to_frames(
+    model_path: &Path,
+    video_paths: &[std::path::PathBuf],
+    target_fps: f64,
+    max_frames: usize,
+) -> Result<Option<CliVideoFrames>> {
+    if video_paths.is_empty() {
+        return Ok(None);
+    }
+    let Ok(model_type) = mlxcel::models::get_model_type(model_path) else {
+        return Ok(None);
+    };
+    if mlxcel::models::model_type_has_native_video(model_type)
+        || !mlxcel::models::model_type_is_vision_capable(model_type)
+        || matches!(model_type, mlxcel::models::ModelType::MuseGlimmerVLM)
+    {
+        return Ok(None);
+    }
+    ensure!(
+        mlxcel::video::ffmpeg_available(),
+        "--video requires `ffmpeg` on PATH. Install ffmpeg (e.g. `brew install ffmpeg` on macOS \
+         or `apt install ffmpeg` on Linux) and retry."
+    );
+
+    let max_frames = max_frames.max(mlxcel::video::MIN_FALLBACK_MAX_FRAMES);
+    let temp_dir = std::env::temp_dir();
+    let mut frame_paths = Vec::new();
+    let mut guards = Vec::new();
+    let mut total_frames = 0usize;
+    for path in video_paths {
+        // The bounded decode the server front uses: reading the clip at
+        // `target_fps` alone would hold up to `FPS_MAX_FRAMES` full-resolution
+        // frames to keep `max_frames` of them. Sharing the helper also keeps
+        // the two fronts choosing the same frames out of the same clip.
+        let source = mlxcel::video::VideoSource::from_path(path.clone());
+        let (frames, sampled) =
+            mlxcel::video::load_video_source_frames_fallback(&source, target_fps, max_frames)
+                .map_err(|err| anyhow!("Failed to load video {}: {err}", path.display()))?;
+        let kept = mlxcel::video::subsample_evenly(frames, max_frames);
+        let encoded = mlxcel::video::frames_to_png(&kept)
+            .map_err(|err| anyhow!("Failed to encode frames of {}: {err}", path.display()))?;
+        println!(
+            "model_type={:?} has no native video path; sending {} of {} sampled frames from {} \
+             as ordered images",
+            model_type,
+            encoded.len(),
+            sampled,
+            path.display()
+        );
+        total_frames += encoded.len();
+        for png in encoded {
+            let frame_path =
+                temp_dir.join(format!("mlxcel-video-frame-{}.png", uuid::Uuid::new_v4()));
+            std::fs::write(&frame_path, png).map_err(|err| {
+                anyhow!(
+                    "Failed to write video frame {}: {err}",
+                    frame_path.display()
+                )
+            })?;
+            guards.push(mlxcel::video::TempFile::new(frame_path.clone()));
+            frame_paths.push(frame_path);
+        }
+    }
+
+    Ok(Some(CliVideoFrames {
+        frame_paths,
+        lead_text: format!(
+            "Here is a video as a sequence of {total_frames} frames in chronological order."
+        ),
+        _guards: guards,
+    }))
+}
+
 fn tokenize_prompt(
     tokenizer: &mlxcel::tokenizer::MlxcelTokenizer,
     prompt: &str,
@@ -2188,7 +2286,7 @@ pub(crate) fn run_generate(mut args: GenerateArgs) -> Result<()> {
 /// One-shot (`-p`-supplied) text generation: the historical `generate` flow.
 fn run_generate_once(mut args: GenerateArgs) -> Result<()> {
     // Safe: the only caller (`run_generate`) guarantees `prompt` is `Some`.
-    let user_prompt = args
+    let mut user_prompt = args
         .generation
         .prompt
         .clone()
@@ -2333,6 +2431,29 @@ fn run_generate_once(mut args: GenerateArgs) -> Result<()> {
         None => None,
     };
 
+    // Video-to-frames fallback (issue #1322). Runs after every validator that
+    // reads `--video` (pipeline parallelism, `--output-audio`,
+    // `--layout-detections`, the Muse Glimmer guard) so none of them changes
+    // meaning, and before the prompt is rendered so the template emits one
+    // image placeholder per frame. On the fallback path
+    // `compute_vlm_embeddings` never sees a video: the clip is already an
+    // ordered run of `--image` inputs by then. The guards live until this
+    // function returns, which is after the vision tower has read the PNGs.
+    let _video_frame_guards = match expand_cli_videos_to_frames(
+        &args.model.model,
+        &args.generation.video,
+        args.generation.fps,
+        args.generation.video_max_frames,
+    )? {
+        Some(expansion) => {
+            args.generation.image.extend(expansion.frame_paths);
+            user_prompt = format!("{}\n\n{}", expansion.lead_text, user_prompt);
+            args.generation.video.clear();
+            Some(expansion._guards)
+        }
+        None => None,
+    };
+
     let tokenizer = load_tokenizer(&args.model.model)?;
     let prompt = load_cli_prompt(
         &args.model.model,