Adding bert_tokenizer library as a dependency
Run this command in a terminal, in your project's directory:
cargo add bert_tokenizer
To add it manually, edit your project's Cargo.toml file and add to the [dependencies] section:
bert_tokenizer = "0.1.3"
The bert_tokenizer library will be automatically available globally.
Read the bert_tokenizer library documentation.