Add support for new KV Cache Offloading API

[abetlen]

Source ggerganov/llama.cpp#4309

[Ph0rk0z]

I force on the ctx param to offload kv_cache. The logs print that it's using F16 cache. 81 layers offloaded. memory sizes for K and V. assume KV "should" be. Speeds on 70b are now half and prompt processing is less than 1/4 with no context. Womp Womp.

[Ph0rk0z]

Ok, I got it working when I moved the kv to the proper place in the struct.

    _fields_ = [
        ("seed", c_uint32),
        ("n_ctx", c_uint32),
        ("n_batch", c_uint32),
        ("n_threads", c_uint32),
        ("n_threads_batch", c_uint32),
        ("rope_scaling_type", c_int8),
        ("rope_freq_base", c_float),
        ("rope_freq_scale", c_float),
        ("yarn_ext_factor", c_float),
        ("yarn_attn_factor", c_float),
        ("yarn_beta_fast", c_float),
        ("yarn_beta_slow", c_float),
        ("yarn_orig_ctx", c_uint32),
        ("type_k", c_int),
        ("type_v", c_int),
        ("mul_mat_q", c_bool),
        ("logits_all", c_bool),
        ("embedding", c_bool),
        ("offload_kqv", c_bool),
    ]