maintain todos

saber5433 · Feb 18, 2024 · 9ab2500 · 9ab2500
1 parent 7843c96
commit 9ab2500
Showing 1 changed file with 5 additions and 2 deletions.
diff --git a/README.md b/README.md
@@ -62,8 +62,11 @@ to run the tests.
 
 ## todos
 
-- write more optimized versions, both in Python and/or C/Rust?
-- handle special tokens? think through...
+- write a more optimized Python version that could run over large files and big vocabs
+- write an even more optimized C or Rust version (think through)
+- rename GPT4Tokenizer to GPTTokenizer and support GPT-2 as well?
+- write a LlamaTokenizer similar to GPT4Tokenizer (i.e. attempt sentencepiece equivalent)
+- handle special tokens
 - video coming soon ;)
 
 ## License