Skip to content

fix(gemma3): make value_states contiguous before the KV cache append - #3902

Open
sidd-27 wants to merge 1 commit into
huggingface:mainfrom
sidd-27:fix/gemma3-contiguous-value-states
Open

sidd-27 wants to merge 1 commit into
huggingface:mainfrom
sidd-27:fix/gemma3-contiguous-value-states

Conversation

@sidd-27

@sidd-27 sidd-27 commented Aug 17, 2026

Copy link
Copy Markdown

gemma3::Model::forward fails for every input with slice-set only supports contiguous tensors.

value_states is transposed and then handed straight to KvCache::append, which slice_sets it into the cache buffer and requires a contiguous source. key_states gets away with it only incidentally — apply_rotary_emb_qkv calls .contiguous() on it first (gemma3.rs:109-110) — and value_states never goes through that path.

Reproduced on 0.10.2 and on current main. No downloads needed:

use candle_core::{DType, Device, Tensor};
use candle_nn::{VarBuilder, VarMap};
use candle_transformers::models::gemma3::{Config, Model};

fn cfg() -> Config {
    Config {
        attention_bias: false, head_dim: 8,
        hidden_activation: candle_nn::Activation::GeluPytorchTanh,
        hidden_size: 32, intermediate_size: 64,
        num_attention_heads: 4, num_hidden_layers: 2, num_key_value_heads: 2,
        rms_norm_eps: 1e-6, rope_theta: 10000.0, rope_local_base_freq: 10000.0,
        vocab_size: 64, final_logit_softcapping: None, attn_logit_softcapping: None,
        query_pre_attn_scalar: 8, sliding_window: 16, sliding_window_pattern: 2,
        max_position_embeddings: 128,
    }
}

fn main() {
    let device = Device::Cpu;
    let varmap = VarMap::new();
    let vb = VarBuilder::from_varmap(&varmap, DType::F32, &device);
    let mut model = Model::new(false, &cfg(), vb).unwrap();
    let ids = Tensor::new(&[[3u32, 9, 14, 2]], &device).unwrap();
    match model.forward(&ids, 0) {
        Ok(t) => println!("forward OK, shape {:?}", t.dims()),
        Err(e) => println!("forward FAILED: {e}"),
    }
}

Before:

forward FAILED: slice-set only supports contiguous tensors

After:

forward OK, shape [1, 1, 64]

It fails with sliding_window_pattern: 1 too, so both the Normal and Rotating cache paths are affected.

I suspect this has gone unnoticed because the recent Gemma 3 activity (#3745, #3747, #3326, #3299) is all against quantized_gemma3.rs, which has its own attention path.

Model::forward fails for every input with "slice-set only supports
contiguous tensors". value_states is transposed and passed straight to
KvCache::append, which slice_sets it into the cache buffer and requires a
contiguous source. key_states avoids this only incidentally, because
apply_rotary_emb_qkv calls .contiguous() on it first.

Reproduced on 0.10.2 and on main.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant