mmBERT-base encoder instead of mdeberta-v3-base

[uasan]

Hi, we see that the multilingual version of Gliner (mdeberta-v3) is significantly less accurate even in English compared to the Gliner model with the deberta-v3 encoder (not multilingual).

Has anyone experimented with using mmBERT-base as an encoder for Gliner?

[uasan]

I found that understanding languages ​​is indeed better, but it requires even more domain tuning.
VAGOsolutions/SauerkrautLM-GLiNER

[Ingvarstep]

Hi @uasan , from my experiments, encoders with positional bias work the best for GLiNER, and mmBERT is RoPE-based. You can also try mT5-based model such like that: https://huggingface.co/knowledgator/gliner-x-large

[uasan]

I tried it, but it's noticeably slower, despite the fact that its maximum context size is 1024, in SauerkrautLM-GLiNER 2048