Commits

Breadcrumbs

History for

on1eaceae

User selector

All users

Datepicker

All time

Commit History

Commits on Jun 14, 2023

• Merge pull request karpathy#274 from apivovarov/gelu

  Show description for f08abb4
  karpathy

  authored
  f08abb4

  Copy full SHA for f08abb4

Commits on May 17, 2023

• Remove pos unsqueeze(0)

  apivovarov

  committed
  39ae397

  Copy full SHA for 39ae397

• Use nn.GELU

  apivovarov

  committed
  594068e

  Copy full SHA for 594068e

Commits on May 6, 2023

• simplify configure_optimizers by a lot

  karpathy

  committed
  7fe4a09

  Copy full SHA for 7fe4a09

Commits on Apr 13, 2023

• Merge pull request karpathy#240 from YassineYousfi/master

  Show description for 01e48ec
  karpathy

  authored
  01e48ec

  Copy full SHA for 01e48ec

• Merge pull request karpathy#142 from kovkev/patch-1

  Show description for ad62003
  karpathy

  authored
  ad62003

  Copy full SHA for ad62003

Commits on Apr 11, 2023

• dont always dropout!

  YassineYousfi

  committed
  7399dfe

  Copy full SHA for 7399dfe

Commits on Mar 24, 2023

• Fix GPT.crop_block_size when flash attention is available

  python273

  authored
  c3f2548

  Copy full SHA for c3f2548

Commits on Mar 5, 2023

• enable sdpa for nonzero dropout

  drisspg

  committed
  6170531

  Copy full SHA for 6170531

Commits on Feb 12, 2023

• Fix the position of a comma

  kovkev

  authored
  c253115

  Copy full SHA for c253115

Commits on Feb 6, 2023

• small tweaks, make default WD be 0.1 as is often cited, and remove spurious init of LayerNorm, which is already initialized at 1,0

  karpathy

  committed
  8b1e432

  Copy full SHA for 8b1e432

Commits on Feb 5, 2023

• add the estimation of model flops utilization (MFU), a very commonly looked at metric that estimates the token throughput in units of A100 bfloat16 peak flops (312 TFLOPS). this gives us a sense of…

  Show description for ab0718a
  karpathy

  committed
  ab0718a

  Copy full SHA for ab0718a

Commits on Feb 4, 2023

• make more accurate the way in which we count parameters. previous count incorrectly included the positional encoding params, when typically only the number of weight parameters is reported for thes…

  Show description for 34720df
  karpathy

  committed
  34720df

  Copy full SHA for 34720df

• mildly dramatic refactor for handing all these usage cases across all possible supported and unsupported devices for all the possible switches and flags

  karpathy

  committed
  25d95db

  Copy full SHA for 25d95db

• padding 50257 -> 50304 vocab_size, the nerest multiple of 64. the biggest deal smallest optimization i've made in recent past, about 25% faster. this is because the last layer is a major latency bo…

  Show description for 77e7e04
  karpathy

  committed
  77e7e04

  Copy full SHA for 77e7e04

Commits on Feb 3, 2023

• use the new fused AdamW from pytorch nightly, if available

  karpathy

  committed
  e170e40

  Copy full SHA for e170e40

Commits on Feb 2, 2023

• if dropout > 0.0 disable Flash until pytorch fix. don't assert fail sigh

  karpathy

  committed
  1e87509

  Copy full SHA for 1e87509

Commits on Feb 1, 2023

• fix bug with loading GPT-2 parameters, assert gets incorrectly tripped due to .bias missing since it is now optionally present depending on flash or not

  karpathy

  committed
  d995c22

  Copy full SHA for d995c22

Commits on Jan 30, 2023

• add flash attention support, resolving last few issues but for now seems to work ok

  karpathy

  committed
  ae06d0b

  Copy full SHA for ae06d0b

Commits on Jan 27, 2023

• bunch of plumbing of bias all around. measuring bias=False to be about 6% faster

  karpathy

  committed
  e808a67

  Copy full SHA for e808a67

• add the bias option to config, default it to True for now

  karpathy

  committed
  cc5444e

  Copy full SHA for cc5444e

• rewrite model class so layernorm has an optional bias= parameter

  karpathy

  committed
  2bf07a3

  Copy full SHA for 2bf07a3

• attempt a non-biased model, per few papers that cite this as working well

  karpathy

  committed
  2892858

  Copy full SHA for 2892858

• try bring back mingpt init

  karpathy

  committed
  23a0bfa

  Copy full SHA for 23a0bfa

Commits on Jan 14, 2023

• add note of caution for the produced warning, investigate later

  karpathy

  committed
  89da79e

  Copy full SHA for 89da79e

• fix bug... if topk > vocab_size, torch.topk will throw error

  karpathy

  committed
  91d0251

  Copy full SHA for 91d0251

• reverse the order, making sure that the final layer init is preserved, and becomes the token embedding instead of the other way around. otherwise the loss can be all messed up from a bad init

  karpathy

  committed
  43b37fd

  Copy full SHA for 43b37fd

• tie the weights of lm_head.weight and transformer.wte.weight, i.e. the last linear layer of decoder and the token embeddings.

  karpathy

  committed
  7c82885

  Copy full SHA for 7c82885

Commits on Jan 12, 2023

• inference time mini-optimization low-hanging fruit ty @jxtps for raising: when we are running inference we can apply lm_head on only the very last token

  karpathy

  committed
  8f85b83

  Copy full SHA for 8f85b83

Commits on Jan 2, 2023

• disabling torch.jit.script here for massive performance boost when using torch.compile, our default. see issue karpathy#11. thanks @vgoklani for flagging

  karpathy

  committed
  177d5f7

  Copy full SHA for 177d5f7

Commits on Jan 1, 2023

• candidate changes to apis, have to think through more

  karpathy

  committed
  2febf44

  Copy full SHA for 2febf44

Commits on Dec 29, 2022

• Frozen GPTConfig

  ankandrew

  committed
  7f0e6d9

  Copy full SHA for 7f0e6d9

Commits on Dec 28, 2022

• first very bad commit

  karpathy

  committed
  fe80428

  Copy full SHA for fe80428

Loading