Feature Request: Support Zyphra/Zamba2-2.7B

[tomasmcm]

Feature Description

Zamba2-2.7B is a hybrid model composed of state-space and transformer blocks

https://huggingface.co/Zyphra/Zamba2-2.7B

Motivation

Quality and Size comparable to Phi2-2.7B but twice as fast and using half the memory.

Possible Implementation

https://github.com/Zyphra/Zamba2

An extension to #7727?

[mseri]

They released also a 1.7B model and very recently a 7B one: https://www.zyphra.com/post/zamba2-7b

[hg0428]

Please add support for this. Zamba2 is a state-of-the-art model and we would like to use it.