Port Fixes from Master to Release (#315)

* Fix Split Operation For Merging Modes (#311)

* Fix Split Operation For Merging Modes

* Format Tests

* Update Stats

* Update Stats

(cherry picked from commit 45f441a)

* Add max_length Option to CLI Convert Tool (#309)

* Add max_length Option to CLI Convert Tool

* Add max_length Option to CLI Convert Tool

* Add max_length Option to CLI Convert Tool

* Add max_length Option to CLI Convert Tool

(cherry picked from commit 5c61c1a)

* Update Regex For Clean Tokenization Spaces

* Fix Replace Pattern Rewrite

* Update Pass Rate

* fixed sdl issue (#313)

(cherry picked from commit c30086c)

* Del Unused Regex

---------

Co-authored-by: Mikhail Ryzhov <[email protected]>

Bump OV version to 2024.4.1 (#266)

fix memory issues (#257)

Гse PCRE2 fallback for RegexNormalization'

* add pcre2

* use string with length instead of ZERO_TERMINATION

* add Licence to thirt party

* set m_search_pattern_re to nullptr if PCRE2 is used

* remove unnecessary const char*

* fix typo in m_replace_pattern

* Update Del Control Chars Regex

* Update Pass Rate

* fix win build

* revert added_tests in conftest.py

* revert Readme & pass rates

* update README & pass_rates

* do not log errors

* add comment on malloc

---------

Co-authored-by: Artur Paniukov <[email protected]>

Cherry Pick For Release (#174)

* Fix Tokenization of Special Tokens in Sentencepiece

(cherry picked from commit 6093bd1)

* Add Left Padding and Padding to Max Length

(cherry picked from commit 128f7fc)

Merge pull request #143 from apaniukov/releases/2024/1

Update Version To 2024.1.0.2

Merge pull request #139 from mryzhov/ver_2024.1.0.1

Version 2024.1.0.1

Cherry Pick For 24.1 Release (#118)

* Add Example RWKV to Readme

(cherry picked from commit fb4142d)

* Update Example

(cherry picked from commit 6441bef)

* Add Cache to VocabEncoder

(cherry picked from commit fd32aa5)

* Install torch CPU

(cherry picked from commit dcc5fcd)

* Parse QWEN EOD_TOKEN_ID

(cherry picked from commit 029a341)

* Fix Parse

(cherry picked from commit 0298fea)

* Fix Pass-Rate

---------

Co-authored-by: Ilya Lavrenov <[email protected]>

[TESTS] updated pass rate (#77) (#81)

fixed pass-rate node path