Skip to content

Tags: openvinotoolkit/openvino_tokenizers

Tags

2025.2.1.0

Toggle 2025.2.1.0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Bump product version 2025.2.1 (#517)

2025.2.0.0

Toggle 2025.2.0.0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
[GGUF] Create tokenizers factory for GGUF support in OpenVINO GenAI (#…

…494)

* [GGUF] Create tokenizers factory method

Needed for GGUF tokenizers support

Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>

* Clean-up implementation

Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>

* Update src/tokenizers_factory.cpp

* Update src/tokenizers_factory.cpp

---------

Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>

2025.1.0.0

Toggle 2025.1.0.0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
[MERGE] Ported compile flags fixes (#446)

* ICU compile flags (#440)

* get the ov:runtime compile defenitions properly

* use common cmake flags

* Revert "get the ov:runtime compile defenitions properly"

This reverts commit 90740ec.

* Revert "use common cmake flags"

This reverts commit f8fc68d.

* set ICU_LINKER_FLAGS

* try AdditionalLinkerOptions

* try AdditionalOptions

* set CL and LINK env

* patch icu

* Update cmake/external/icu.cmake

* get the ov:runtime compile defenitions properly (#438)

2025.0.1.0

Toggle 2025.0.1.0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Bump product version 2025.0.1 (#426)

2025.0.0.0

Toggle 2025.0.0.0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Replace openvino.runtime imports with openvino (#379)

2024.6.0.0

Toggle 2024.6.0.0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Use RC1 (#337)

* Use RC1

* Update windows.yml

* Update pyproject.toml

* Update linux.yml

---------

Co-authored-by: Alina Kladieva <alina.kladieva@intel.com>

2024.5.0.0

Toggle 2024.5.0.0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Port Fixes from Master to Release (#315)

* Fix Split Operation For Merging Modes (#311)

* Fix Split Operation For Merging Modes

* Format Tests

* Update Stats

* Update Stats

(cherry picked from commit 45f441a)

* Add max_length Option to CLI Convert Tool (#309)

* Add max_length Option to CLI Convert Tool

* Add max_length Option to CLI Convert Tool

* Add max_length Option to CLI Convert Tool

* Add max_length Option to CLI Convert Tool

(cherry picked from commit 5c61c1a)

* Update Regex For Clean Tokenization Spaces

* Fix Replace Pattern Rewrite

* Update Pass Rate

* fixed sdl issue (#313)

(cherry picked from commit c30086c)

* Del Unused Regex

---------

Co-authored-by: Mikhail Ryzhov <mikhail.ryzhov@intel.com>

2024.4.1.0

Toggle 2024.4.1.0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Bump OV version to 2024.4.1 (#266)

2024.4.0.0

Toggle 2024.4.0.0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
fix memory issues (#257)

2024.3.0.0

Toggle 2024.3.0.0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Гse PCRE2 fallback for RegexNormalization'

* add pcre2

* use string with length instead of ZERO_TERMINATION

* add Licence to thirt party

* set m_search_pattern_re to nullptr if PCRE2 is used

* remove unnecessary const char*

* fix typo in m_replace_pattern

* Update Del Control Chars Regex

* Update Pass Rate

* fix win build

* revert added_tests in conftest.py

* revert Readme & pass rates

* update README & pass_rates

* do not log errors

* add comment on malloc

---------

Co-authored-by: Artur Paniukov <chgk1101@gmail.com>