Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: karpathy/minbpe
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: master
Choose a base ref
...
head repository: alexandermorgan/BatchBPE
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: master
Choose a head ref
Checking mergeability… Don’t worry, you can still create the pull request.
  • 13 commits
  • 14 files changed
  • 1 contributor

Commits on Jun 12, 2024

  1. Add optimized batch tokenizer 

    Includes added test, and updates to the README and train.py to
include batch.py in the descriptions.
    @alexandermorgan
    alexandermorgan committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    fd3eeca View commit details
    Browse the repository at this point in the history

  2. Merge branch 'karpathy:master' into master

    @alexandermorgan
    alexandermorgan authored Jun 12, 2024
    Configuration menu
    Copy the full SHA
    13164ad View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2024

  1. Add max_batch_size param to Batch tokenizer 

    Raises an error if below 1. Includes tests for the setting.
    @alexandermorgan
    alexandermorgan committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    23a638f View commit details
    Browse the repository at this point in the history

Commits on Jul 22, 2024

  1. WIP: change from minbpe to batchbpe

    @alexandermorgan
    alexandermorgan committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    88d85d7 View commit details
    Browse the repository at this point in the history

  2. Finish conversion to BatchBPE

    @alexandermorgan
    alexandermorgan committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    8fd06f2 View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2024

  1. WIP: begin README rewrite

    @alexandermorgan
    alexandermorgan committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    37f7055 View commit details
    Browse the repository at this point in the history

  2. Add: stop_list_size and freq_cutoff tokenizer params

    @alexandermorgan
    alexandermorgan committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    ed0ac45 View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2024

  1. WIP: add more to readme 

    Still needs details in Special Features section, and specific files
to load.
    @alexandermorgan
    alexandermorgan committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    2de137a View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2024

  1. Add QuickTokenizer

    @alexandermorgan
    alexandermorgan committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    5995d40 View commit details
    Browse the repository at this point in the history

  2. Update base, batch, and README

    @alexandermorgan
    alexandermorgan committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    cbe1926 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2024

  1. Add link to HF dataset and path to json fil

    @alexandermorgan
    alexandermorgan committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    93cc4bf View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2024

  1. Store/read csv instead of json, and clean up train.py

    @alexandermorgan
    alexandermorgan committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    c3fe367 View commit details
    Browse the repository at this point in the history

  2. Update README to reflect move to csv

    @alexandermorgan
    alexandermorgan committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    52cbf5d View commit details
    Browse the repository at this point in the history
Loading