Skip to content

resolved packing params not propagating; added support for other backbones#301

Merged
Ingvarstep merged 3 commits intourchade:mainfrom
vivekkalyanarangan30:vk/mt5-backbone-support
Oct 21, 2025
Merged

resolved packing params not propagating; added support for other backbones#301
Ingvarstep merged 3 commits intourchade:mainfrom
vivekkalyanarangan30:vk/mt5-backbone-support

Conversation

@vivekkalyanarangan30
Copy link
Contributor

@vivekkalyanarangan30 vivekkalyanarangan30 commented Oct 14, 2025

Summary

  • Ensure span and token forward paths keep packing_config/pair_attention_mask alive all the way to the encoder.
  • Overhaul Transformer.forward so packed block-diagonal masks are translated into what each backbone expects (generic additive mask, plus dedicated paths for DeBERTa, ModernBERT, and T5/MT5).
  • Extend the packing code path (including the packed shortcut) to pass both standard and pairwise masks, and clean up T5 embedding/attention handling so packed inference works for MT5-based checkpoints.

Testing

PYTHONPATH=. python bench/bench_gliner_e2e.py \
       --model knowledgator/gliner-pii-edge-v1.0

PYTHONPATH=. python bench/bench_gliner_e2e.py \
       --model knowledgator/gliner-pii-small-v1.0

PYTHONPATH=. python bench/bench_gliner_e2e.py \
       --model knowledgator/gliner-x-small

PYTHONPATH=. python bench/bench_gliner_e2e.py \
       --model urchade/gliner_small-v2.1

PYTHONPATH=. python bench/bench_gliner_e2e.py \
       --model urchade/gliner_medium-v2.1

PYTHONPATH=. python bench/bench_gliner_e2e.py \
       --model knowledgator/gliner-bi-edge-v2.0

https://www.diffchecker.com/2kR276Om/
All differences in output is because of 6th place of decimal precision

Revised benchmarks

Model Without packing (s) With packing (s) Δ time saved (s) Speedup
knowledgator/gliner-pii-edge-v1.0 3.268 1.746 1.522 1.87×
knowledgator/gliner-pii-small-v1.0 6.392 2.682 3.710 2.38×
knowledgator/gliner-x-small 10.454 5.135 5.319 2.04×
urchade/gliner_small-v2.1 3.795 1.288 2.507 2.95×
urchade/gliner_medium-v2.1 6.891 1.626 5.265 4.24×
knowledgator/gliner-bi-edge-v2.0 3.039 1.183 1.856 2.57×

@vivekkalyanarangan30 vivekkalyanarangan30 marked this pull request as draft October 14, 2025 19:50
@vivekkalyanarangan30 vivekkalyanarangan30 marked this pull request as ready for review October 14, 2025 19:51
@vivekkalyanarangan30 vivekkalyanarangan30 marked this pull request as draft October 14, 2025 19:55
…iner-pii-edge-v1.0

PYTHONPATH=. python bench/bench_gliner_e2e.py        --model knowledgator/gliner-pii-small-v1.0
PYTHONPATH=. python bench/bench_gliner_e2e.py        --model knowledgator/gliner-x-small
PYTHONPATH=. python bench/bench_gliner_e2e.py        --model urchade/gliner_small-v2.1
PYTHONPATH=. python bench/bench_gliner_e2e.py        --model urchade/gliner_medium-v2.1
PYTHONPATH=. python bench/bench_gliner_e2e.py        --model knowledgator/gliner-bi-edge-v2.0
@vivekkalyanarangan30 vivekkalyanarangan30 marked this pull request as ready for review October 14, 2025 20:17
@vivekkalyanarangan30
Copy link
Contributor Author

@Ingvarstep couple of steps were missing for propagating the packing config from Gliner through base to encoder, fixed it
Checked benchmarks, now the gains are visible. Added support for mt5 backbone as well.

@Ingvarstep Ingvarstep merged commit 4e09780 into urchade:main Oct 21, 2025
@Ingvarstep
Copy link
Collaborator

Awesome, thank you @vivekkalyanarangan30 once again for your contributions.

@vivekkalyanarangan30 vivekkalyanarangan30 deleted the vk/mt5-backbone-support branch October 22, 2025 18:42
@vivekkalyanarangan30 vivekkalyanarangan30 restored the vk/mt5-backbone-support branch October 22, 2025 18:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants