Commits

Branch selector

 master

User selector

All users

Datepicker

All time

Commit History

Commits on Feb 19, 2024

• create exercise to rewrite minbpe for learners

  karpathy

  committed
  a98e0a5

• add small amount of code to write out the gpt4 vocab in the same format as the base class, taking into account the byte shuffle

  karpathy

  committed
  1cc9d06

• Merge pull request karpathy#18 from ahmedivy/master

  Show description for 872c086
  karpathy

  authored
  872c086

• add link to another relevant issue here

  karpathy

  committed
  f7122fe

• create requirements.txt

  ahmedivy

  committed
  fa474de

• fix linter error

  ahmedivy

  committed
  29bee94

• Merge pull request karpathy#17 from eltociear/patch-1

  Show description for aac75c8
  karpathy

  authored
  aac75c8

• change the handling of special tokens inside encode of RegexTokenizer, make the default behavior consistent with tiktoken, i.e. the person has to explicitly declare intent to use or not use special…

  Show description for ade909f
  karpathy

  committed
  ade909f

• refactor the readme file and adjust the API on how special tokens are handled

  karpathy

  committed
  4ccf644

• Update regex.py

  Show description for a5ecf46
  eltociear

  authored
  a5ecf46

• fix pytest -v, it was printing all of taylor swift text

  karpathy

  committed
  37b63c2

• meaty change: adding special tokens handling, so now we have full parity with the GPT-4 Tokenizer

  karpathy

  committed
  e82c123

Commits on Feb 18, 2024

• add small warnings about gpt4 tokenizer just being a pretrained tokenizer

  karpathy

  committed
  e0ed1bc

• maintain todos

  karpathy

  committed
  9ab2500

• optimization: allow get_stats to update an existing counts dict. train.py runtime goes from 32s to 22s doing this. ty @gklab for original suggestion in a PR

  karpathy

  committed
  7843c96

• Merge pull request karpathy#13 from gklab/spelling_error

  Show description for ff20c92
  karpathy

  authored
  ff20c92

• decompose long line here

  karpathy

  committed
  b52bf9b

• let's add a version line. i learned this from llama2c, where not doing this caused a ton of chaos and suffering

  karpathy

  committed
  bbf845d

• Adjust comments & block commit of the models folder

  gklab

  committed
  cf120a4

• gitignore add

  karpathy

  committed
  a0a05f8

• add caution around save/load of gpt4 tokenizer

  karpathy

  committed
  f9e19ea

• let's not clutter the main directory

  karpathy

  committed
  a04b85a

• major refactor to make this a sensible package with nice tests etc. we almost look legitimate now

  karpathy

  committed
  ac3ca85

• Merge branch 'ViswanathaReddyGajjala-master'

  karpathy

  committed
  d278867

• simplify pytest a lot

  karpathy

  committed
  45cb371

Commits on Feb 17, 2024

• Refactor testing code: Consolidated tests into a single file

  ViswanathaReddyGajjala

  committed
  743b3a8

• Update README.md with pytest installation instructions

  ViswanathaReddyGajjala

  authored
  c86aece

• added unit tests for all the tokenizers

  ViswanathaReddyGajjala

  authored
  665b782

• remove educational as constraint. i think we might be able to make this repo have teeth and have some more optimized versions. but we'd keep the simple reference code

  karpathy

  committed
  3e68413

• major changes and refactoring, creating base class, joining the utility functions, and adding the save/load functionality in the base class

  karpathy

  committed
  2c521cb

• Merge pull request karpathy#1 from cyrilzakka/master

  Show description for 02e9e20
  karpathy

  authored
  02e9e20

• Fixes imports in train.py

  cyrilzakka

  authored
  1abcd54

• mild improve docs

  karpathy

  committed
  cc7fb34

Commits on Feb 16, 2024

• bunch of changes that just make sense

  karpathy

  committed
  43ffc2f

• add note of caution on parity

  karpathy

  committed
  1c0520f

Pagination

PreviousNext