Tags: openvinotoolkit/openvino_tokenizers
Tags
[GGUF] Create tokenizers factory for GGUF support in OpenVINO GenAI (#… …494) * [GGUF] Create tokenizers factory method Needed for GGUF tokenizers support Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com> * Clean-up implementation Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com> * Update src/tokenizers_factory.cpp * Update src/tokenizers_factory.cpp --------- Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
[MERGE] Ported compile flags fixes (#446) * ICU compile flags (#440) * get the ov:runtime compile defenitions properly * use common cmake flags * Revert "get the ov:runtime compile defenitions properly" This reverts commit 90740ec. * Revert "use common cmake flags" This reverts commit f8fc68d. * set ICU_LINKER_FLAGS * try AdditionalLinkerOptions * try AdditionalOptions * set CL and LINK env * patch icu * Update cmake/external/icu.cmake * get the ov:runtime compile defenitions properly (#438)
Port Fixes from Master to Release (#315) * Fix Split Operation For Merging Modes (#311) * Fix Split Operation For Merging Modes * Format Tests * Update Stats * Update Stats (cherry picked from commit 45f441a) * Add max_length Option to CLI Convert Tool (#309) * Add max_length Option to CLI Convert Tool * Add max_length Option to CLI Convert Tool * Add max_length Option to CLI Convert Tool * Add max_length Option to CLI Convert Tool (cherry picked from commit 5c61c1a) * Update Regex For Clean Tokenization Spaces * Fix Replace Pattern Rewrite * Update Pass Rate * fixed sdl issue (#313) (cherry picked from commit c30086c) * Del Unused Regex --------- Co-authored-by: Mikhail Ryzhov <mikhail.ryzhov@intel.com>
Гse PCRE2 fallback for RegexNormalization' * add pcre2 * use string with length instead of ZERO_TERMINATION * add Licence to thirt party * set m_search_pattern_re to nullptr if PCRE2 is used * remove unnecessary const char* * fix typo in m_replace_pattern * Update Del Control Chars Regex * Update Pass Rate * fix win build * revert added_tests in conftest.py * revert Readme & pass rates * update README & pass_rates * do not log errors * add comment on malloc --------- Co-authored-by: Artur Paniukov <chgk1101@gmail.com>
PreviousNext