Part 1: Build GPT2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build GPT2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dataclasses import dataclass\n",
    "import math\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.nn import functional as F"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overall tranformer structure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We'll be implementing the right hand side (the decoder) that composes GPT-2: each unit on the right will be a block in our transformer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Transformer Architecture from \"Attention Is All You Need\" by Vaswani et al.](./assets/transformer.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The configuration below is the configuration for the entire tranformer, with each layer *h* pertaining to one of the blocks. We want to replicate the following structure from a GPT-2 model in Huggingface Transformers:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![HF Transformer](./assets/hf_transformer.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The code below is the skeleton on GPT2 config and main module that will allow us to replicate that structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@dataclass\n",
    "class GPTConfig:\n",
    "    block_size: int = 256\n",
    "    vocab_size: int = 65\n",
    "    n_layer: int = 6\n",
    "    n_head: int = 8\n",
    "    n_embd: int = 384\n",
    "\n",
    "class GPT(nn.Module):\n",
    "    def __init__(self,config):\n",
    "        super().__init__()\n",
    "        self.config = config\n",
    "        # With nn.ModuleDict() index into submodules just like a dictionary\n",
    "        self.tranformer = nn.ModuleDict(dict(\n",
    "            wte = nn.Embedding(config.vocab_size, config.n_embd),\n",
    "            wpe = nn.Embedding(config.block_size, config.n_embd),\n",
    "            h = nn.ModuleList([Block(config) for _ in range(config.n_layer)]),\n",
    "            ln_f = nn.LayerNorm(config.n_embd)\n",
    "            )\n",
    "        )\n",
    "        self.lm_head = nn.Linear(config.n_embd, config.vocab_size, bias=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- nn.ModuleDict allows you to index into submodules using keys, just like a dictionary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- nn.ModuleList allows us to index into each individual layer using an index, just like with a list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transformer Block"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Now let's implement the Block,..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Unlike the original GPT2 paper, establish a clean residual pathway by taking the layer norm of x and applying attention/multilayer perceptron layer to it *then* adding it to the input *x*.  Since addition allows for an adulterated gradient flow during backpropagation, this pre-layer norm configuration is the better than the post-layer norm configuration where the norm is applied after the addition. More formally, Xiong et al. (2020) have shown that if post-layer norm is used, a warm-up stage is needed to avoid training instability whereas if pre-layer norm is used, the gradients are well-behaved at initialization. See the difference between original (Post-LN) GPT-2 implementation and the 'corrected' pre-LN implementation used here:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Source: \"On Layer Normalization in the Transformer Architecture\" by Xiong et al. 2020](./assets/pre_vs_post_layer_norm.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Finally, onto the Block:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Block(nn.Module):\n",
    "    def __init__(self,config):\n",
    "        super().__init__()\n",
    "        self.config = config    \n",
    "        self.ln_1 = nn.LayerNorm(config.n_embd)\n",
    "        self.attn = CausalSelfAttention(config)\n",
    "        self.ln_2 = nn.LayerNorm(config.n_embd)\n",
    "        self.mlp = MLP(config)\n",
    "    def forward(self, x):\n",
    "        x = x + self.attn(self.ln_1(x))\n",
    "        x = x + self.mlp(self.ln_2(x))\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Note again how the layer norm is applied *before* the addition to the residual stream."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Andrej notes that attention is a communication operation, where tokens communicate with each other and aggregate information.   Thus attention can be thought of as a pooling function/weighted sum function/reduce operation.  On the other hand, the multilayer perceptron (MLP) is applied to each token individually, with no information exchanged between the tokens.  Thus attention is a *reduce* and MLP is the *map* operation and a transformer is a repeated application of MapReduce. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multilayer Perceptron"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Briefly summarizing from Andrej's previous video (Let's build GPT: from scratch, in code, spelled out.), multilayer perceptron is implemented using a standard \"bottleneck architecture\" where the dimensions are first expanded to learn more complex representations, nonlinearity is applied to help the model learn more complex patterns, and finally the data is projected down again to keep the computational complexity in check.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MLP(nn.Module):\n",
    "    def __init__(self,config):\n",
    "        super().__init__()\n",
    "        self.c_fc = nn.Linear(config.n_embd, config.n_embd*4)\n",
    "        self.gelu = nn.GELU(approximate='tanh')\n",
    "        self.c_proj = nn.Linear(config.n_embd*4, config.n_embd)\n",
    "    def forward(self,x):\n",
    "        x = self.c_fc(x)\n",
    "        x = self.gelu(x)\n",
    "        x = self.c_proj(x)\n",
    "        return x    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- GPT-2 used an approximate version of GeLU because at the time of GPT-2's creation, the erf function  was very slow in TensorFlow and GPT-2 and the approximate version was used.  Today there's no reason to use the approximate version but Andrej is using the tanh approximation for veracity.  \n",
    "- Also, GeLU is better than ReLU due to dead neuron problem since a local gradient is always present as seen below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Source: https://pytorch.org/docs/stable/generated/torch.nn.GELU.html](./assets/gelu.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Causal Self Attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Andrej's attention implementation is a more efficient implementation of the following simple one from \"Lets build GPT: from scratch, in code, spelled out\":"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Source: https://github.com/karpathy/ng-video-lecture/blob/master/gpt.py](./assets/attention_simple.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Notice how in the implemenation above it is clear that the heads are parallel streams whose outputs are concatenated.\n",
    "- The idea of the more efficient implementation is to make another batch dimension with *nh* so that PyTorch effectively makes batches of dimension *(B,nh)* and applies all the operations on both *B* and *nh* in parallel.  \n",
    "- Each token emits query, key, and value.  Queries and keys first multiply each other to deterimine \"how interesting they find each other\".  \n",
    "- Next, we apply an autoregressive mask to make sure the tokens only attend to tokens before them.  \n",
    "- The softmax normalizes the attention so it sums to 1.\n",
    "- The matrix multiply of attention with the values is a way, at every single token, to do a weigthed sum of the tokens each token finds intersting.  \n",
    "- Transpose, contiguous, and view reassembles everything in memory and performs what is equivalent of a concatenation operation.\n",
    "    - y.transpose(1,2): This line swaps the second and third dimensions of y. So the shape of y changes from (B, nh, T, hs) to (B, T, nh, hs).\n",
    "\n",
    "    - .contiguous(): This is used to ensure that the tensor is stored in a contiguous block of memory, which is required for some operations in PyTorch, including view.\n",
    "\n",
    "    - .view(B, T, C): This line reshapes the tensor y to have dimensions (B, T, C). Here, C is equal to nh*hs, which means that the last two dimensions of y (nh and hs) are flattened into a single dimension. This effectively concatenates the outputs of all the attention heads side by side.\n",
    "- Finally, the output projection doesn't change the dimension of *y*, but does introduce another learnable transformation so that the output can be projected in a way that is most useful for downstream tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CausalSelfAttention(nn.Module):\n",
    "    def __init__(self, config) -> None:\n",
    "        super().__init__()\n",
    "        assert config.n_embd % config.n_head == 0\n",
    "        #key, qury, value projections for all heads in a batch\n",
    "        self.c_attn = nn.Linear(config.n_embd, config.n_embd*3)\n",
    "        # output projection\n",
    "        self.c_proj = nn.Linear(config.n_embd, config.n_embd)\n",
    "        \n",
    "        self.n_head = config.n_head\n",
    "        self.n_embd = config.n_embd\n",
    "        # mask to prevent attention to future tokens\n",
    "        self.register_buffer(\"bias\", torch.tril(torch.ones(config.block_size, config.block_size))\n",
    "                             .view(1, 1, config.block_size, config.block_size)) # becomes available as self.bias\n",
    "    def forward(self, x):\n",
    "        B, T, C = x.size()\n",
    "        # calculate query, key, value for all heads in batch\n",
    "        qkv = self.c_attn(x) # (B,T, self.n_embd) x (self.n_embd,self.n_embd*3) = (B,T,self.n_embd*3)\n",
    "        q, k, v  = qkv.split(self.n_embd, dim=2) # (B,T,self.n_embd) x 3; make each split size self.n_embd by splitting dim 2\n",
    "        q = q.view(B, T, self.n_head, C//self.n_head).transpose(1,2) # (B, nh, T, hs)\n",
    "        k = k.view(B, T, self.n_head, C//self.n_head).transpose(1,2)\n",
    "        v = v.view(B, T, self.n_head, C//self.n_head).transpose(1,2)\n",
    "        # attention materializes a large (T,T) matrix fo each query and key\n",
    "        att = (q @ k.transpose(-2,-1))*(1.0/math.sqrt(k.size(-1))) # (B, nh, T, hs) x (B, nh, hs, T) = (B, nh, T, T)\n",
    "        att = att.masked_fill(self.bias[:,:,:T,:T] == 0, float('-inf'))\n",
    "        att = F.softmax(att, dim=-1)\n",
    "        y = att @ v # (B, nh, T, T) x (B, nh, T, hs) = (B, nh, T, hs)\n",
    "        # Change (B, nh, T, hs) to (B, T, nh, hs) with transpose, reassemle in memory, (B,T,C) makes nh*hs = n_embd (C)\n",
    "        y = y.transpose(1,2).contiguous().view(B, T, C) \n",
    "        # output projection: additional learnable transformation\n",
    "        y = self.c_proj(y) # (B, T, C)@(C, C) = (B, T, C)\n",
    "        return y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Copy over the Hugging Face GPT-2 model parameters into our model "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Ignore the attention mask buffers (these are not parameters)\n",
    "- The weights in Hugging Face version are transposed (as they are in the original TensorFlow implementation) from what PyTorch needs because they use Conv1D module.  Since we want to use plan nn.Linear, we hardcode these and transpose them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GPT(nn.Module):\n",
    "    ...\n",
    "    @classmethod\n",
    "    def from_pretrained(cls, model_type):\n",
    "        \"\"\"Loads pretrained GPT-2 model weights from huggingface\"\"\"\n",
    "        assert model_type in {'gpt2', 'gpt2-medium', 'gpt2-large', 'gpt2-xl'}\n",
    "        from transformers import GPT2LMHeadModel\n",
    "        print(\"loading weights from pretrained gpt: %s\" % model_type)\n",
    "\n",
    "        # n_layer, n_head and n_embd are determined from model_type\n",
    "        config_args = {\n",
    "            'gpt2':         dict(n_layer=12, n_head=12, n_embd=768),  # 124M params\n",
    "            'gpt2-medium':  dict(n_layer=24, n_head=16, n_embd=1024), # 350M params\n",
    "            'gpt2-large':   dict(n_layer=36, n_head=20, n_embd=1280), # 774M params\n",
    "            'gpt2-xl':      dict(n_layer=48, n_head=25, n_embd=1600), # 1558M params\n",
    "        }[model_type]\n",
    "        config_args['vocab_size'] = 50257 # always 50257 for GPT model checkpoints\n",
    "        config_args['block_size'] = 1024 # always 1024 for GPT model checkpoints\n",
    "        # create a from-scratch initialized minGPT model\n",
    "        config = GPTConfig(**config_args)\n",
    "        model = GPT(config)\n",
    "        sd = model.state_dict()\n",
    "        sd_keys = sd.keys()\n",
    "        sd_keys = [k for k in sd_keys if not k.endswith('.attn.bias')] # discard this mask / buffer, not a param\n",
    "\n",
    "        # init a huggingface/transformers model\n",
    "        model_hf = GPT2LMHeadModel.from_pretrained(model_type) # HF GPT2LMHeadModel has .from_pretrained method, just like ours\n",
    "        sd_hf = model_hf.state_dict()\n",
    "\n",
    "        # copy while ensuring all of the parameters are aligned and match in names and shapes\n",
    "        sd_keys_hf = sd_hf.keys()\n",
    "        sd_keys_hf = [k for k in sd_keys_hf if not k.endswith('.attn.masked_bias')] # ignore these, just a buffer\n",
    "        sd_keys_hf = [k for k in sd_keys_hf if not k.endswith('.attn.bias')] # same, just the mask (buffer)\n",
    "        transposed = ['attn.c_attn.weight', 'attn.c_proj.weight', 'mlp.c_fc.weight', 'mlp.c_proj.weight']\n",
    "        # basically the openai checkpoints use a \"Conv1D\" module, but we only want to use a vanilla Linear\n",
    "        # this means that we have to transpose these weights when we import them\n",
    "        assert len(sd_keys_hf) == len(sd_keys), f\"mismatched keys: {len(sd_keys_hf)} != {len(sd_keys)}\"\n",
    "        for k in sd_keys_hf:\n",
    "            if any(k.endswith(w) for w in transposed):\n",
    "                # special treatment for the Conv1D weights we need to transpose\n",
    "                assert sd_hf[k].shape[::-1] == sd[k].shape\n",
    "                with torch.no_grad():\n",
    "                    sd[k].copy_(sd_hf[k].t())\n",
    "            else:\n",
    "                # vanilla copy over the other parameters\n",
    "                assert sd_hf[k].shape == sd[k].shape\n",
    "                with torch.no_grad():\n",
    "                    sd[k].copy_(sd_hf[k])\n",
    "\n",
    "        return model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Forward"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The `nn.Embedding` layer in PyTorch takes an input tensor of arbitrary shape with values in the range `[0, vocab_size)` and maps each integer in that range to a dense vector of size `C` (the embedding dimension). In our case, we have an input tensor `idx` of shape `(B, T)`, where `B` is the batch size and `T` is the sequence length, then applying the embedding layer to `idx` will yield a tensor of shape `(B, T, C)`.  This is because each integer in `idx` is replaced with its corresponding embedding vector. Since the embedding vectors have `C` elements, this adds an extra dimension of size `C` to the output. So for every batch and every sequence position, an embedding of size `C` is constructed, resulting in an output of shape `(B, T, C)`.\n",
    "- Also note that the position embedding is broadcast to the token embedding and the same position embedding vector is learned at (T,C) for every element in B.  This works because the position embeddings are independent of the specific sequence it's in, so it can be shared across all sequences in the batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GPT2(nn.Module):\n",
    "    ...    \n",
    "    def forward(self, idx):\n",
    "        # input indices are always of shape (B, T) where B is batch size and T is block size\n",
    "        B, T = idx.size()\n",
    "        assert T <= self.config.block_size, \"Cannot forward, model block size is exhausted.\"\n",
    "        # forward the token and position embeddings\n",
    "        pos = torch.arange(0, T , dtype=torch.long, device=idx.device) # shape (T,)\n",
    "        tok_emb = self.transformer.wte(idx) # (B,T)-> (B, T, C)\n",
    "        pos_emb = self.transformer.wpe(pos) # (T,)->     (T, C) \n",
    "        x = tok_emb + pos_emb               # (B, T, C) + (T, C) -> (B, T, C) via broadcasting\n",
    "        # forward the blocks of the transformer\n",
    "        for block in self.transformer.h:\n",
    "            x = block(x)\n",
    "        # forward the final layer norm and the classifier head\n",
    "        x = self.transformer.ln_f(x)\n",
    "        logits = self.lm_head(x) # shape (B, T, vocab_size)\n",
    "        return logits\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate next sequence predictions with weights from pretrained GPT2 model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The goal is to get close to generations from the Hugging Face pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import pipeline, set_seed\n",
    "generator = pipeline('text-generation', model='gpt2')\n",
    "set_seed(42)\n",
    "generator(\"Hello, I'm a language model,\", max_length=30, num_return_sequences=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- To generate next token predictions, get the next top 50 tokens so that the model does not deviate too much from likely tokens, and sample one token from this distribution.\n",
    "- Concatenate the token obtained from sampling with input (or input plus previously sampled tokens) at each step.\n",
    "- The sampling will not match the Hugging Face generations exactly since there's likely a parameter hiding in the pipeline that's different, but will be sensible English."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_return_sequences = 5\n",
    "max_length = 30\n",
    "model = GPT.from_pretrained('gpt2')\n",
    "model.eval()\n",
    "model.to('cuda')\n",
    "\n",
    "# prefix tokens\n",
    "import tiktoken\n",
    "enc = tiktoken.get_encoding('gpt2')\n",
    "tokens = enc.encode(\"Hello I'm a language model, \")\n",
    "x = torch.tensor(tokens, dtype=torch.long).unsqueeze(0).repeat(num_return_sequences,1).to('cuda') # (5,8) since sent. tokenized to 8 tokens\n",
    "\n",
    "# generate: with each loop iteration, generate one more token\n",
    "torch.manual_seed(42)\n",
    "torch.cuda.manual_seed(42)\n",
    "while x.size(1) < max_length:\n",
    "    with torch.no_grad():\n",
    "        logits = model(x) # (B,T,vocab_size)\n",
    "        logits = logits[:, -1, :]  # take the logits at the last position\n",
    "        probs = F.softmax(logits, dim=-1) # get the probabilities\n",
    "        topk_probs, topk_indices = torch.topk(probs, 50, dim=-1) # get the top-50 tokens\n",
    "        ix = torch.multinomial(topk_probs, num_samples=1) # sample from the top 50\n",
    "        xcol = torch.gather(topk_indices, -1, ix) # select the indices of the sampled tokens\n",
    "        x = torch.cat((x, xcol), dim=1) # append the sampled token to the sequence\n",
    "\n",
    "for i in range(num_return_sequences):\n",
    "    tokens = x[i,:max_length].tolist()\n",
    "    decoded = enc.decode(tokens)\n",
    "    print('>',decoded)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize a random model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- To do this, simply replace the GPT model initialization as below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#model = GPT.from_pretrained('gpt2')\n",
    "model = GPT(GPTConfig())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Autodetect device\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Detect the most powerful device available and use it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = 'cpu'\n",
    "if torch.cuda.is_available():\n",
    "    device = 'cuda'\n",
    "    print(\"using GPU\")\n",
    "elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():\n",
    "    device = 'mps'\n",
    "    print(\"using MPS\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Note that we guarded against device mismatch by initializing pos on the correct device. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pos = torch.arange(0, T , dtype=torch.long, device=idx.device) # shape (T,)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the Tiny Shakespeare data set for quick debugging and load a batch of data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Andrej's favorite debugging dataset is Tiny Shakespeare which can be loaded and previewed as below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First Citizen:\n",
      "Before we proceed any further, hear me speak.\n",
      "\n",
      "All:\n",
      "Speak, speak.\n",
      "\n",
      "First Citizen:\n",
      "You\n"
     ]
    }
   ],
   "source": [
    "#!wget https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt\n",
    "with open(\"input.txt\", \"r\") as f:\n",
    "    text = f.read()\n",
    "data = text[:1000]\n",
    "print(data[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- To get the corresponding GPT-2 tokens, load the tokenizer via tiktoken's get_encoding method and encode the text data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[5962, 22307, 25, 198, 8421, 356, 5120, 597, 2252, 11, 3285, 502, 2740, 13, 198, 198, 3237, 25, 198, 5248, 461, 11, 2740, 13]\n"
     ]
    }
   ],
   "source": [
    "import tiktoken\n",
    "enc = tiktoken.get_encoding(\"gpt2\")\n",
    "tokens = enc.encode(data)\n",
    "print(tokens[:24])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Andrej's favorite way to create a batch is to use view, which returns a new tensor with same data but different shape.  It's a view because the returned tensor shares the same underlying data with the original tensor and a change in one will affect the other.\n",
    "- Since the desired outputs for every token in the (B,T) batch are just to the right of that token, extend the buffer by one element, take all but the last token as inputs and the first token onwards as outputs: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 5962, 22307,    25,   198,  8421,   356],\n",
      "        [ 5120,   597,  2252,    11,  3285,   502],\n",
      "        [ 2740,    13,   198,   198,  3237,    25],\n",
      "        [  198,  5248,   461,    11,  2740,    13]])\n",
      "tensor([[22307,    25,   198,  8421,   356,  5120],\n",
      "        [  597,  2252,    11,  3285,   502,  2740],\n",
      "        [   13,   198,   198,  3237,    25,   198],\n",
      "        [ 5248,   461,    11,  2740,    13,   198]])\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "buf = torch.tensor(tokens[:24+1])\n",
    "x = buf[:-1].view(4,6)\n",
    "y = buf[1:].view(4,6)\n",
    "print(x)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Update the forward pass to calculate the loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Pass in the optional targets and calculate cross entropy loss.  Cross entropy loss in PyTorch expects (B*T,vocab_size) logits and (B*T,) targets, so reshape it with view."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GPT(nn.Module):\n",
    "    ...\n",
    "\n",
    "    def forward(self, idx, targets = None):\n",
    "        # input indices are always of shape (B, T) where B is batch size and T is block size\n",
    "        B, T = idx.size()\n",
    "        assert T <= self.config.block_size, \"Cannot forward, model block size is exhausted.\"\n",
    "        # forward the token and position embeddings\n",
    "        pos = torch.arange(0, T , dtype=torch.long, device=idx.device) # shape (T,)\n",
    "        tok_emb = self.transformer.wte(idx) # (B,T)-> (B, T, C)\n",
    "        pos_emb = self.transformer.wpe(pos) # (T,)->     (T, C) \n",
    "        x = tok_emb + pos_emb               # (B, T, C) + (T, C) -> (B, T, C) via broadcasting\n",
    "        # forward the blocks of the transformer\n",
    "        for block in self.transformer.h:\n",
    "            x = block(x)\n",
    "        # forward the final layer norm and the classifier head\n",
    "        x = self.transformer.ln_f(x)\n",
    "        # --- Added code ---\n",
    "        loss = None\n",
    "        logits = self.lm_head(x) # shape (B, T, vocab_size)\n",
    "        if targets: \n",
    "            # F.cross_entropy expects (B, T, vocab_size)-> (B*T, vocab_size) shapes for logits\n",
    "            # and (B*T,) shape for targets. \n",
    "            loss = F.cross_entropy(logits.view(-1, logits.size(-1)), targets.view(-1))\n",
    "        return logits, loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimizer and training loop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- See the Andrew Ng's videos for a review of  [momentum](https://www.youtube.com/watch?v=k8fTYJPd3_I&list=PLkDaE6sCZn6Hn0vK8co82zjQtt3T2Nkqc&index=20) and [RMSProp](https://www.youtube.com/watch?v=_e-LFe_igno&list=PLkDaE6sCZn6Hn0vK8co82zjQtt3T2Nkqc&index=21) that compose the [Adam optimizer](https://www.youtube.com/watch?v=JXQT_vxqwIs&list=PLkDaE6sCZn6Hn0vK8co82zjQtt3T2Nkqc&index=22).\n",
    "- Make sure to zero the gradients since loss.backwards() always accumulates gradients.  \n",
    "- optimizer.step() will update the parameters to (ideally) decrease the loss.  \n",
    "- loss.item() will convert the loss to a float that's placed on CPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = torch.optim.AdamW(model.parameters(), lr=6e-4)\n",
    "for i in range(50):\n",
    "    optimizer.zero_grad()\n",
    "    logits, loss = model(x,y)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    print(f'iteration {i}, loss = {loss.item()}')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Running the training loop will yield the following output:"
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUkAAAM2CAYAAAB7YXGVAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AAAAtdEVYdENyZWF0aW9uIFRpbWUAU2F0IDA3IFNlcCAyMDI0IDA4OjQ2OjI5IEFNIEVEVFmgolMAACAASURBVHic7N0xTFtZ3vj97//VazemwA1+CqfAjdO4IY2nwA00jh4F/UdEzwo9I9CuHO2AJiKaiAgl2ogREVq0I6J5RHY0aEZEWUU7GjQrUPTQJA0U64Y0buLGFHEDjSlwYxfvW2DANr7nd66PbSD5faUtdi73nnPPvTjG2B/+z3/8x3/8f2iapmkt+38uewKapmlXuZ4/SAaSc/zz98ckA70eWdM0zX//b68HrB7kyO4EyFZ7PTIMJDPMTKSIR8MEywcUcpusrW6RL8Pwwu88uhWsfWWF41KR3Nt1nr18D4FhFv71iODqfzO/fXTyJYEkj399Qnj9jzzcOuz9yWia1pN6/iDJ/jbPn/d8VPqHH/P9o5scbK7zbKVApS9KfPgW8QHI79e+qPAb8yvvOCZMNDXBvbtP+Fv5Pg83ez9fTdOuRtYPkiMLv5MpL/KH5fcn/6H/Dn/7ZYz8/J9Yy5/8p+HpJaZScSJ9cHxQJL+5ytOtk42BoVl++W6UMEDl3yz+17OGZ5MjC7+TqWyyQZKxRIRQtcjOT4s83z1/lnbn8d+ZuHWDYPkDbzeLJCZivLt/n42iPP+JqS9gZ5GHa9naf9knn9tt/KJKidx+ESiyvz9POPGGe8kh2CzZLpOmaZ9YHXtNsj+9xIMU7Czf549/vM/86iaFugfB6vvnfPWf/8n/Xdzh2OMYfbduMbA9z1d/+JL5t5CamTp77XJo9hX3YkVeP/0z9xe36RtJcSPocaDmBidJRI7J7ez5OqdyteLr6zVN+/Tq2INkeKAfSgVevi9yeFhk//07Xm7nfR2jkt/mxfuT1/zy73YoBgeJR4HAMOlkiNzGKlu5IsX8O1Y3ctg+hAXCEcKUKB3UHrWHZnn15g1v3rzh5+lEy32iw7OMx4Mc5P2dg6Zpn1Yde01yf2ebfPoev/8cJ5cvUMjt8nI75+sYldLB+f85rlAhSDAIRGJEgiU+FI7ONpcLRQ4q4fYm+/4589/scGfuO27V//eb93jz5t7pbPj475/401oeAgPtjaNp2rXP+kHywrO25h9197eY/8MWieE7DN26RXJyiXRqlT/MbztP0rVq6YASQ4QjAdg/eTZZ3H/PYfOP04VN/rLyllK1yn6x8YXOSqX5lE/fw6Q/kmvap5z1j9vVChCoe5gIRwi3eE0wt7vFy+dPub/6b4gnGQl1YJYHBQ4qYW7E+s/+UygWJWL7muT+S3IHfSSSrX+0PqtywPv9/QsPkFTLlMsQCtUNGAwSpEK5XLachKZp1zHrB8lCvkDw5gjDtcepkfEUkbrtg+kMmfQQgwP99A/EGU3F6CsVedeJx5DqLtvZMvHxGdLxAQYGh8mMJy48mTX1ev3fMPqApclhEoODDA6Nc6vVo3zL3rP34ZhYOsOdoTiDgwlGMuMkKLCXu4Q3fGqa1rOsf9w+3HjI68TPzPzyT+4d7LO3/W8+3Dp/Ra98HCI28YClTJg+jjkoZFldXD/bPv7DG6Zi58d78q83ABR++zP3X8rv4Xm/tsj67CyTS7+QKX/g7eYOHydi4n6nHe0+Y54MmYl7LNwNw/EBhffrrL62e9109/lTwjMzjM19z72+CqVCjs3FFfR95Jr2afd/ri1wkZjl1UKY9a+edubZqqZpWouuD3AxOMyd4TjR/gCB/gTjE0kCuR19gNQ0rav1/mOJ7RaM8MXEPaYiYYKUKOxt8Oz5u8uelaZpn3jX98dtTdO0HnR9ftzWNE27hHw/SF53D/K6z791Q8y+OvmY5Zs3b1hKd+LNqZqmQRuvSRo9yP40S7+M83H+T7y4jI88W4x/mZ5lYnKB6dEEN0IVDvI7rK28INuRtxC95/lX/8lzEsy+Wmh4/+on02Ca2akR4rEYN8JBciv/xXzTb+2c1ncgSWZmkpHEDfo45mP+HS9X1lrun5j+maXbET789N883DqCUJqlX2e4+FGFCnsrX/H0XVk8/tDkAlOpONFIH8FKiY/5f/N6dY3dYtVqu7g+pvFt5i+sb+LOLBPpWwxGwvRRopB7y+vVl+frZ3H9PNcXIJRgfDbDWCJGuK9C6WOOnZerrLW4QC33d8j/L24uyYPsWJc0/4H0Eo/H+smuzvOsECY9M8vck0O+vL/R+8lcx4IBOMixvVMg9eD2hc2u6zv+ZI40b1n59il5oqSmHjD3pMKX9182fF1oaJbpm2U+1n8atbzNyjeFhk+gBRNTLExU2Nkr2x2/VGTn9TaFwxLVQIzhqUkeLcDRn1+Qq1psF9bHOL7F/KX1HfoiwsG71/xvochRcJA7mSmeLAT5z6/XrK6fcX2B5OxjpmI5Vp8u8u9SiMTEHA/m5jj488OG9yp77e+S9YOk0YMcGOeHX6Y4fWt34vs3nCxDjtX/nucU806MPyYzliAaDlI5yPNubaXhX4L49M8sxbZZ3okxcTdJLAyl3Gu+mt+gP5nhwUSSWDRMmAoHhSwbq2ts75etxpc8S6jzMIMVDgrv2VhdYbv2WW8b79JUKp2A7CLP3508xV1b2yH5fZrp+EbPnnWbzq9hewsP1GZ7V8tv8TwPBIa59eD2hU9bOa1v/x2SsQp7f11jd78KHLKxtknqhzTT8Zfn+4eSzMzEyC7/RnTpEfW8yuF+nvo7YWRikMreKu+O7I7/fmuN9+cny4dggtSTm8QjkCvK243rYzG+cf4W6/tyfr5u7xwfAgl+fZRgMgovpfmdZljfRLSPg70ttvMns9x9/Y67o+PcjAXYOqyK+7tk/SB54kE+J5Cc49Vc0wt6hxvc/88N44+7g+M/8HiszPbqU5YLZcLJSWbnnlD59j4v98+/LhgdZyLxmrVvVsgdVUkMDwHQFw5QfLvGRq7IQTVEYnyWme9mOPjjMu8txjfOH4hn/s6D1BGby9+yfdhHMjPHzHcPOPjzMu9rPxX03brFwOI8Xz07Ij75A0szU2Szy/KP7oFh4lHY3/5w/t8KOQqV28Ti/ZB3/5FASjq/Uw90c/k+20UIReOk6n5ul7ab6k8v8Y8Zj8/Nf9w8f7bRbq7rW/uOrdZfx0qFSjDCjVgI8ic3wPBshujOIiuFCHOm4w3cIZ2okl3M+jr+2en0DzI6kqCvtEfhgAtJ29s9P8/5t7G+wWAfVMoceuGxLTKt795+iXRimKH+HO+PYDB1i4FSjt/qPhZsfX181pv3SQaGuDs2QH79T7zMnlyQ4tYzNlO/kx4d5OVa3aMkOX57vkWudt1yuyf/fha3X1D/rfRufZt0aoxbMXjv/GQmwUjqBgfbz3j5/uQjklsrr/nilwzpoQDvd08uxAXvcmyEeBSy+54HPikUJhysUDquEs/8nYVknuffvqNchmg4DHT7QVI+vxMP9P3Zdg6L1J+WtN3U0fY83xTiLZ89VMo23+VCrut7uEX+4B4j6TRktwBITqSJAcWBMFBmYGSBe5Edvnq2DwHzvw6DY2lulnZ4+L5qfXw4+aXiqycp+oDKwQ4r86u8r3tgk7a7nJ9x/n7XN5RgcjxOaWfx7KdIKWl93y9/xdrjn3nyjzcn91Epx/pfltkt2+3vUm8eJCMxouE+Yg9+5c2Dxk3Hh2Go+3arHObPTry+wOAImXvjfBG/UffaSYmiH+XCq1CUG6EKxXzdN+xRkY+lIDdjEdg9eWDw9C6tq1ApHVEqlSn38hdHFucneaCuXuh+T/Di9tf39fI60bmpmid6zMedd2QPIkTLwECaB1MD7CwuygcKJBhLRSi8bSQCjcevVc0uM//NbwTDNxmZmCAzk+bd/Jb19rbPz2L+J9mtb3pujlT5NX94/t77i+qzWN/4+N+YiBVY/8siuVIfN8emmfxujoNvnrEb9HF92qh3n7ipHPC/Nr/1rrR+xXXqyQxfFNZZ/vM2ucPqyW/kXk10coIdPFZT5RKlSpC+vhD7G/N8vQEEhrkTgnKpV38/Rzg/yQN18EK7/uN2B9a3nN/g6Z82CPUPUD46hMAwj39NUyqVCMSGiIdvkPj+X4zV73TvH/ye+okvH54/UIWSd0iGCrzeLlofv779/X3Y3ydfCBD7xwRzyW2W617Pkba3c371tZy/j/UdefyKyfAOz+btfyEpru/TMhMTMQqrX7FVe+1rf3WNxKvvuDsaJVu0vz7t1NkHySrQ6tnVQYFiJczNxADk23jPS/8d4uES/97YIne6e2yQgSA0/LDmNb5UucjHch+JeASytZujP8qNcIWS1Ys+QtVd8sVHjCduwnbtdZ5YgljwgL0Wr+cMDPRzeNjOj+A5KhUIBJpec/VxfrndLXK7W7wefsyrB0lGQtsNn4+Xtreq6z9ud3B9y0cnN1h/apQERTZyZaqHz7j/5+j5FwWGmPn+HqHteZY3Cw37p27fIphb9dShWh3fuyAh4/t5pe3+x285f8v1HXn8M5nIDs/m1zCeVlPVrLC+gRFCweYXBWpfGgrI+zvW2QfJygEH5RDx4SHI1T3Vrr7nt+0i348/Yba0xkauRCAySGI4RSS7zJr0wkq5SLEc5mZy8OTvv4biTE6miFCl4Qc+r/HFcrzb+choeobJ3Crbh2GSkxMkylkWrV70kdvZzjGemWR2pMRGIUw6kyJcuPib18TsK5ZGj/ntm68bfqFlfSbFCqOpMRLvN8kVT29g+fwG0xlG2ePtXoESEb5IxegrZc8eAKXtUp34cTs6GCcUjBACiMaIx48pHxQpHlWt1/fk76vnG951ARAausN45JD3xTKhwREmJm5RzS6yUXuwKNZDzIEox0Dg4CPFw7oFiI6TjlfYW8nSnPH4oSSZ2QQHOzn2j0oQijE8PsHN4xwruaq83WJ9pPOT5i+t7/Dcz8zcLPJ6ZYdKNE4coFIhv39+E5vmZ17fLXKFKcbGZ0gfvmavFCJ+Z4pk3wFva6+RW12fNrN+kLTyIKvv2VjfYW7qCW9uB6l/C87+y/s8Lc8xOfGYHyJ9UDqgWNjjt0OLB6Hqe9ZXNnk0s8Q/01XKpQNyW+/IxVIXvs5rfGn++bWvWQksMTX3A3f7oFTI8tPiKtkOKUOH2/M8G1hgOrPEj8GTN+MuL1/8kSQUDEDliMM2fwrfXVvji7kMCz/eJVh3/tL5SR6otL3rDU7y+H/ucuP0/99d4vu78PG3b/j65b71+npVIUJifIKxSB+UPpJ791fur118sDAVT6eJlbOst7hpjMcvZzlgnPS9ESLhPoKVk/X96ela7S040nZ5fWzOzzR/4/oGknwxFCHYF2Hqu/q/GlXi7fxXPM/J85N6ubhMcGaSiYUfmQlWOP6Y5+1fl3nRA/RagYsr1vTPb7iVm+dPz/39ETVN07qTAhdXqeg4N0M5Ni21dE3Tup8+k9Q0TTOkzyQ1TdMM6YOkpmmaIfUkP4nUk9S0bqWeZK/y4en57xP3JC29Q2MW6z+QnGRmYpRENAyVA4o7a9x/cfI2GclLFL1HC6/SNL7N/p7eo836WXiNktdpmn+312do9hXfjTa6P6W3f+Er249GGlJPsldZenpaiyy8QzFh/QOJaZbmUpTevmR5rUA5MEA0fH5s0UsUvEfJk5TGl/Y3eo8W6yd5jZInKc2/2+sDUPnwG0/XsrUP4FaolBo/Gtpu6knW1U1P0srT63LX2ZOUvEMxYf1HJ0YJZhd5+OL0mUe+4dNckpdo9B6PZc/ROL6FByl5j9L6SV6jdHxp/bq6PqeVi+S6AKmoJ1mrq57kFeiT8iSbvUPXAkmGBit82Isw+7efSUZDlA9z7Kyv8vJ960dhk5d4wXus/RTo6TkWEubxJQ+yMOTPe2yxfkavUfIkCzd9rV/H1+f0miQy/PP3GYKVEsXcNmurG+Q6oBCqJwl03ZO89D4tT/KCd+haaIBwX5DYWJrs6+fMf4Cbd6fJzD3i8P48280/LHh4iZ7eo+Q5iuNL+/vzHlutn9Fr7BeOb7l+3VsfONzb5Ke9AvuHZQLRFHenplh4Al8+dP/zKOpJQg89yUvqU/Ikjd6hS0HKe+ss185pf+U1Q68eMZLsZ7vpj0l5eYkm71H2HM3j23mQFt6jx/oZvUar48vr1831Ke5ucPYKZD5PvhzmxydpMvEN1hxvPfUkzwfu4LGuYp+GJ+nlNTpVLlGuQKlekqnuclh+RDTS+ExM8hK9vEej52gxvrS/rffYcv1CI0avcXdTOL6P9evW+ly4pIUCBwwxEA5QMxTbrrNvJrfxJNupwZOsnXDNk7QaX6rmLUbj9S/CddCT9NnAQH+be5o9SZvzy+1u8fL5U+6v/hviSUaa3nIpbW/V0fY833z7Ld+2+N83z37zdYYn3uG2p9cIbaxfdZfCAYSjdesTSDIQgvLB+YOMfy/xovfY7DnmcmXr8U3754swmLh5/oU177HQ9Hpky/ULhAh5fM8EQgH5+D7m3+31OdscixGhzGHJ/SUZ9SSBXniSYPb0TlNP0pDBOzzNy4sE8/q/2/7A2FSG6b1VtvZhcHySW8Ec69mTgxi9RAvvUfIcpfGl/a08Ta/1O5K9Run4xvl3e30CQ0zO3aK0k6u9JjnE3akUoQ/rzj9qg3qSPfMkbT099SS9M3mHYsL6F7ceshheIHNvidthOP6YY3Nx5eQZl+glyt6j5Dkax7fY38bTNK2f5DVKxzfPv8vrE6hC3xB3Z9KE+4JUjg8ovP+Jp6tuf7bhNFWArljqSWra1UqBi6uUepKaduXSZ5KapmmG9JmkpmmaIX2Q1DRNM6Se5CeRepKa1q3Uk+xRkkfolnqSUjbr7+Q5WniVJo9R9BBdxxe2i16mD68xMf0zS7cjfPjpv3l4+pFOyasUtoseZRdTT7JHiR6h5l0HPElp/V09R8mrlDxGMHuIruNL223uTxuvMTQ0y/TNMh+bPgUreZXSdsmj7GbqSdbVTU9S8gh70efsSUrr7+o5Sl6l5DEC3h5iB8aXtlvdn5LXGEoyMxMju/wb0aVH1D/vlLxKabvRo+zy9496krV67UmaPMJupJ5kYw3rL3mSkueYF57NSh5j7fPVnh6i6/ht1Or+lLzG4dkM0Z1FVgoR5pqOZ/QqLbbXd8Gj7HLqSQI99yQ9PMLupZ5kQ83r3+/oOSI8SFl4j0YP0XV81/VB9hoHRha4F9nhq2f7ELj4r6fRq7TYDgaPssupJwk99yS9PMKupZ5kQ63XvxOeo5S3xyh5iJ0Z365W62OcXynNg6kBdhYXPY9p9CqP5O1g9ii7mXqS5wN38FjeSR5h91JPEjzW39VzlPLhPZ7t0uQhOo3vI9v7s35+gfAQ8fANEt//i7H6L7r3D35P/cSXT8tmr3I7bt6+cX5dvDzKbtbZB0kbTzLfxnteGjzJ2n+reZINP6w5epKJeASytQvSBU/S1iMcGOjn8LCdn8PNnqTN+eV2t8jtbvF6+DGvHiQZCW03cGjS9lZ18sftEw9xVfQkW62f5/pXdykcPGI0GoHTXw9Zeo4bNrBkdZd88RHjiZuwXXsdteYx7jX//Zlapx5ivslDbGt8y/x4mfXzq+afcf/P0bqNQ8x8f4/Q9jzLmwUIjBAKtn5RIBAKnHmWnts9u+hRdiP1JIFeeJJGj7Au9SQNOXiS0vq7eo5g9iqNHqOFh+g6vrTduD4W8yvWPwsPRDkGAgcfKR6WAcGrPNo3b7fwKLuZepK98CRFj/D8v6gn6V3bnqTF+rt6jpJXafYYZQ/RdXzj9tcR8/p8cPcaJa/SvF32KLuZKkBXLPUkNe1qpcDFVUo9SU27cukzSU3TNEP6TFLTNM2QPkhqmqYZUk/yk0g9SU3rVupJ9qjuenifuCeJP8/QKxfP0cajNB2/E16i8fiCF+nsQUrbJc9S8iQlL1MYvxP3h1fqSfaqS/TwPpVsPEOvXD1HyVuUju/qJYrzlzxJy/Xz8iDF7cL40vmLXqbF/FzuD1PqSdbVTU/yMj28066zJwnInqEhJ88R2VuUju/qJYrzlzxJm/UzeJDidmF84/lX0rKXaTM/h/vDlHqStXrpSfbaw4NPw5OUPEPPXD3HVnNp8Cjl4zt5iZbzl3LxIG22mzKef+0ldMnLlMZv+/4QUk8S6JUneVke3qfgSUqeoTFXz/HC8Zo9Svn4Tl6ixfFd10/yIKXtUsbzL8tepjS+0/0hpJ4k9MyTvCwP71PwJCVv0a72Pcf6vD1Q7+N3xkv0Pr6Ukwc5IHuRUtL5G71Mi/E7c3+0Tj3J84E7eCzvLsPDO+nT8CRPa/YWzV/s7jme5uVRGo8fGnHzEvf8z9/P+kke5H9tJMxe5EPhH3qL8zd5mYGY4FW2GN/X/SGkniT0zJO8WGsPTz1JOS9vEVqsX4c8R5NHaTy+q5fYxvyl/HiQ1cMtsxcpDmZ//q28zOqh4FUK5+eaepJA1z1JHx6eepItsvAMT/PyJF09R8mjNB5f8hQt7g/z8U/y9CLLCUcPUt5uHP9IOH9kL9M4vo/7o53Uk+yFJ1m29/DUk2yVu2fo5DlaeJTm47t7idLxzV6k+/qJCZ6l5EmKXqax7p6fKkBXLPUkNe1qpcDFVUo9SU27cukzSU3TNEP6TFLTNM2QPkhqmqYZUk/yk0g9SU3rVupJXkKS1+e/T9+TrK/d9TN5jJIXaeMVGr1HYCA5yczEKIloGCoHFHfWuP/i4ttc2vIcpeO7epYu+4fSLP06w8XPXFXYW/mKp+/KHfUmO516kj1O8vo0c+2un+QxSl4kmL1C6fiBxDRLcylKb1+yvFagHBggGr74Jtx2PUfp+K6epdP+5W1WvinUmQsQTEyxMFFhZ69sNT9bb7IbqSdZVzc9SUD28LrctfckHdZP8hglLxIweoXS8UcnRglmF3n44vSZZ54Lb/Ry8Byl47t6lq77H+7nqf9OGZkYpLK3evZm+Y54k11KPclavfAkXTw+1z4FT7Lt9WvDY2zwIk//m5dXKB2/cJOhwQof9iLM/u1nktEQ5cMcO+urvHx/PnbbnmMgKR7fybPswP4NDdwhnaiSXTx/qaET3mS3Uk8S6IUn6erxuXX9PUmn9bPwJBu/vsmLRPAKpeOHBgj3BYmNpcm+fs78B7h5d5rM3CMO78+zfejoOVoc38mzxH3/+gbH0tws7fCwbqOrN9nN1JOE7nuSHfD4nLrunmTH1s/OY2zlRRq9wrNlNR0/SHlvneXamu2vvGbo1SNGkv1sZ7/ogOdoOP7WkbNn2RkPEwgkGEtFKLxtJPacvMkup57k+cAdPFZj7Xh4ne/6epLO6+fDk2zpRbY6ZL1XWBCOXy5RrkCpXrKp7nJYfkQ0EhbPT/Qc583HJ3TLzbPMpdz2r3s9KpS8QzJU4PV23Vwdvclup54kdN2TrGb9eXjqSTbmvH6WHqOnF9miBq9QOn51l8LBI0ajETj99UUgyUAIygcl8fxEz7FaNh7f2bN03b+u1O1bBHOrJ78RPz+IkzfZ7dSTBLruSWLn8YF6kl7Zrl9bniSCF2nhFUrHf7f9gbGpDNN7q2ztw+D4JLeCOdazR1bnJ203Hl/yHCXP0nX/06LjpOMV9laa3hvaAW+ym6kn2QtP0kfqSXYno8coeZEfZK9Q8h6LWw9ZDC+QubfE7TAcf8yxubjS+IzKIen4rp6l6/4A8XSaWDnLeotvqu56k26pAnTFUk9S065WClxcpdST1LQrlz6T1DRNM6TPJDVN0wzpg6SmaZoh9SQ/idST1LRupZ5kj7LxCNvv0/ckRe9Q2l9Yf8mTBLMXKc5vMM3s1AjxWIwb4SC5lf9i3uNNpi29SMFTdB5f8hwRvEppfi4ep41H2cXUk+xhJo9QExK8Q5tM6y95kpIXKc4vGICDHNs7BVIPbnvO0cuLFD1Fx/Elz1HyKm28x7Y9TguPspupJ1lX1z1Jg0fYi66zJyl5hVYZ1l/yJCUvUpxffovneSAwzK0Ht1t+RNPTi+y/I3qKruNLXqTRq7SYn7T+0vpKHmU3U0+yVi88SU+PsAd9Cp7kaaJX6JGf9W/wJH16lO3Oz9OLrD2i2XqK7Yxv9Bwlr9Jyfm17nM3eZwuPspupJwn0wpM0eoRd7/p7kmDvFbbK1/o3e5L9dh6ly/yMXuShnafoMr7Rc+yXvEp5fk4eZ5P32cqj7GbqSUL3PUkEj7DbP7Fed0+ylpVX6JGf9W/lSZ5k9ijbnp+FF2njKbqsj9FzBCSvUpqfu8dZy8Oj7GbqSZ4P3MFjyTV4hPTiX8Tr60k2TFPwCm3zWv+WnqQPj7Kd+dl4mbaeYlvrI3mOm4JXyZFv79GXx1k/1VYeZZdTTxK67km2qsEjbEo9SdsueoVgt36t1t/Tk7T0KG3n15wfL9Ofp2g3vug5Ch5mO/Pz5XHW1dKj7HLqSQJd9yQtPMLT1JNska1XiIcnabH+Rk8SwYu0nF90ME4oGDn5u1bRGPH4MeWDIsWjquhFGj1F5/Flz1HyMI3z64DHeXICHh5ll1NPsieepOwRnqaeZIvKdl6hd8L6S55kTvAibeY3OMnj/7nLjdND313i+7vw8bdv+NriX0Ojp9iB8SXPUfIqzd6ju8cJZo+ym6kCdMVST1LTrlYKXFyl1JPUtCuXPpPUNE0zpM8kNU3TDOmDpKZpmiH1JD+J1JPUtG6lnmQPM3p8Tn36nqSN9+iZ5BFmU1ZeoZN3KHiNogcpeI3G+YHoSYrr6+hhip6kdH4W59+t7y/1JHuU5PFp5iTv0ZjkEVp4ha7eoeQ1Sh6k5DWK3qXgSYrr6+hhgtmTlM5P2t7N7y/1JOvqpidp9Ph61HX2JCXvUUryCKXtrt6h5DUaPchj2WuU5id5kuL6uniYp3l5kpJHeSCffze/v9STrNVVT1Ly+HrQp+RJQpP36DfJI2ze3gHv0Og1NnXBg6w92nh6jYUhf/OzqJ319fQwT4/p5UlKHmVJ2F5IdPX7Sz1JoOueZEjy+FznL/VpeJJnNXuPPpM8wgvbO+AdGr3GWp4epOQ1+pyfWBvra/QwETxP6fzE8+/u95d6ktATT1Ly+LraRmA3qQAAIABJREFUJ+JJnubtPVokeYTG7e17h0avsXb5TR6kjSdpNT+LfK+vhYcpeZ7S+cnn373vL/Ukzwfu4LGaKsseX/f7NDzJlt6jjySPsOV2H55ky/0lr3Hj/Gu9PEij1+hjflLtrK+Nh9lcs+cpeZTS+Xfz+0s9Sei+J+nD4wP1JL3y9B6bMq2f5BG23O7qHUpeo2cXPciWXmPb3mVjtuvbnB8P8+xLPDxVyaP0On8/319+U08S6LoniezxnaaeZOsk7/Hs61p5kqdJHqFhu5N3KHmNFh6k0Wu0nJ/Js7RZ37Y9TAtPUjo/abvt91c7qSfZE09S9vhOU0+yRRbeo02SR2ja7uodmr1G2YM0e40W8zN5kq8j8vo6eZiyJymdn7Td9vurnVQBumKpJ6lpVysFLq5S6klq2pVLn0lqmqYZ0meSmqZphvRBUtM0zZB6kp9E6klqWrdST7IX2XiDTqknKSV5jcbtNtfP5B1a7C+en8mjdJ1f81q38CBB8Codz9/P/HqdepK9yMIb1Mw5eZIgeo3G7RbXz+gdWuwvnZ/Zo3ScX11eHqTkVbqev+38LiP1JOvqpicpeYO96HP2JI1eY1Hebrx+kodo4U1K5yd5lK7zA4wepNGrtPAeOzK/S0o9yVpd9SSbkzzDLqSe5HkXvEaf2y9cP8lDzJfN+7eo+fz8eJTtzs/Tg5Q8Tcl7lM7f7/r1OPUkga57kk1JnmHnU08SDF6j5fbTLlw/yTtsoi3E69/i/Gw8Spf5GT1Iyat0PX+f+/c69SShR55kLckz7EbqSQJmr9FmO+B5/ey8R+/9pfOz8Sjbnp+FB3mSt1fpev7W+19C6kmeD9zBY3kneYbdSz1J8PYabbd7XT/JQ5T2N56fD4+ynfmJHuS87FW6nr/t/peRepLQfU+yLskzBPUkveqEJ9nYRa9R2i5dP8lDNO3veX4+PMp25lc9FDzIatnaq3Q5f5v9LyP1JIFeeJKA7BminqRXTp6k5DVaeI6A8fpJ3qG0v/H8JI+yA/MzepDIXqXr+Vvtf0mpJ9kjTxJkzxDUk2yZqydZlrxG2XME8/WTvEPj/hbnZ/YoOzM/U5JX6XT+HZhfN1MF6IqlnqSmXa0UuLhKqSepaVcufSapaZpmSJ9JapqmGdIHSU3TNEPqSX4SqSepad1KPcleZfIAnVNP0uoY7XqIyB5lwzjteIzC/dGJ8QeSk8xMjJKIhqFyQHFnjfsvam+zGUwzOzVCPBbjRjhIbuW/mD99E6ylh9r28aX5W4w/NPuK70bDDVtLb//CVz4/utoq9SR7lNkD7P18rluunqSThwiyR1mrXY9RvD8cxw8kplmaS1F6+5LltQLlwADRcN2DVDAABzm2dwqkHtxu3NnCg3Q6fl0t52/psVY+/MbTtWztA7gVKqXOfPRXPcm6uulJSh5gL/qcPUlXD1HyJoH2Pca8fH+4jj86MUowu8jDF+/PjtHwRrP8Fs/zQGCYWw9uX/gIqeRhuh5fmr+Vx1oukusgpHKaepK1uu1J+vIAu9Bn7Ul22EP08ibb9hjzR77uD//jJxkarPBhL8Ls334mGQ1RPsyxs77Ky/dtGAHNHmSHju85f2n8WsFEhn/+PkOwUqKY22ZtdYNcB1Br9SSBXniSfjzAzveZe5Id8hBN3qSTx8iR1f3R/vgDhPuCxMbSZF8/Z/4D3Lw7TWbuEYf359n2+XLPBQ+yA8c3zl8aHzjc2+SnvQL7h2UC0RR3p6ZYeAJfPmxPi6pPPUnoiSdp7QF2I/Uka7l5iJ7eZAc8Rpv7w238IOW9dZZr12x/5TVDrx4xkuxne8vHDejpYToc33r9vMcv7m5w9qpLPk++HObHJ2ky8Q3WHG899STPB+7gsZry4QF2r8/Ykyx3zkNs5U2u4Ogx+rg/2h2/XIFSvfRT3eWw/Iho5OSZrG0tPciy2/FFz/LhOXxs67GWCwUOGGIgHKBmKLadepLQfU/ShwcI6kl61bYnWd3tmIfY2Ik3Wd119Bh93h/tjF84eMRoNAKnv/4IJBkIQfnAHzfV0oOs7jodv5oV5i+N36JALEaEMvmS+2v+6kkCXfckbT1A1JP0ysmTxNFDtPAm3TxG4f7owPjvtj8wNpVhem+VrX0YHJ/kVjDHevZ8kaKDcULBCCGAaIx4/JjyQZHiUe17wOBBuh5fmr9x/MAQk3O3KO3kaq9JDnF3KkXow7rzj9qgnmTPPEkbDxDUk2yZqyeJo4coepTu45vvD/fxi1sPWQwvkLm3xO0wHH/Msbm4cv6MbHCSx/9zlxunO9xd4vu78PG3b/i69q+1yYPsxPGlvMevQt8Qd2fShPuCVI4PKLz/iaerWy2P4zdVgK5Y6klq2tVKgYurlHqSmnbl0meSmqZphvSZpKZpmiF9kNQ0TTOknuQnkXqSmtat1JPsVYJX6Nan70n68QhbJnme0vFN18/GW5SuvzR+N+eXTbltlzzJTqxPXV5epu12v6kn2aNEr1AzZ+kReiV6jcLxjdfPwjsUr790ft2cn+t2BE+yE+tTy8vLtN3eTupJ1tU1T7Jf9gp70XX2JK08QkOi52k6vsX1M3qHNtdfOr9uzg/37ZIn6bw+YPQmrba3mXqStbrqSdbuaFuvsBt9ap6k35w8T7/Xr9k77Pb1d51fc363+/Uk21wfyZu09ih9pp4k0HVP0tIr7F6fmCfZRk6ep8/rd8E77Pb1d51fU763+/Qk21kfyZv041H6TT1J6IknaeMVdq1PzJNsJ1fP0/r6eXiH3b7+rvNz327pSbazPpI36cejbCP1JM8H7uCxLmbrFXavT8OTbKsOeJ6218/LO+z29Xedn9N2H55kO+sjeZP/tZGw9ijbST1J6L4nWT+UhVeonqRbF9avba/xYtL1k7xDf16l/1zn19Z2H55kO+tTPTR7k9XDLWuPsp3UkwS67kkieIV1qSfpnegd4uFJWnqepuNbXT+Dt2izv3R+3Zyf63YbT9JlfSRv0sqjbDP1JHvkSRq9wrrUk/TI0SMUPU/h+DbXz+QtivtL59fl+bluFz1J1/W5xFQBumKpJ6lpVysFLq5S6klq2pVLn0lqmqYZ0meSmqZphvRBUtM0zZB6kp9F6k1qWrupJ9mrLDzExOQC06MJboRO/uTo2soL9SZtc/UmMXiIIHuUpv0tPMWhyQWmUnGikT6ClRIf8//m9eoau8WqfHwgcWeWifQtBiNh+ihRyL3l9erLs/tH2i6eP8L96ep1Wqzv2TxaeZGu4xtST7JXCR7gQHqJx2P9ZFfneVYIk56ZZe7JIV/e32hxMO1Cjt6k0UNE9ihdPUVKRXZeb1M4LFENxBiemuTRAhz9+QW5qjy/oS8iHLx7zf8WihwFB7mTmeLJQvDsI5/Sdun40v3p6nWK+9fy8iJdxzelnmRdXfMkQfQCU+kEZBd5/u7kKfDa2g7J79NMxzfUm7TJ0ZuUPETJo3TyFIH3W2ucf0Ysz4dggtSTm8QjkCvK83s5P1/3/3J8CCT49VGCySi8LMrbpeNL96eT12mxvoDRi3Qd35R6krW66klKBYaJR2F/+8P5fyvkKFRuE4v3Q96doJf6rL1JCw/R6FG6eorN0+kfZHQkQV9pj8KB3fyaCwb7oFLm8Lj1KTdsl45vcX86eZ3S+tYyeZGu45tSTxLouicpFQoTDlYoHVeJZ/7OQjLP82/fUS5DNNyoqHSnz9ybtPAQjR5lv6OnWCuQnOPVkxR9QOVgh5X5Vd5X/R+fUILJ8TilncXzz6+btkvHr8j3p5PXiex9Sl6k6/im1JOEnniSdlWolI4olcqUe/mLJfUmkTxEo0dpsf9ZBq+xml1m/pvfCIZvMjIxQWYmzbv5U+bL8vhAem6OVPk1f3jeGnlpvd1w/J3Tr/G+P129TuP+QdmLdB3flHqS5wN38Fg+K5coVYL09YXY35jn6w0gMMydEJTVm+z+j9uShxi6ZfYoN909xdP29/dhf598IUDsHxPMJbdZ3rM//sjjV0yGd3g23/oXfi23S+cv3Z+uXqewf7Zo9iS/fFp29kJNqScJPfUkW1bdJV98xHjiJmzXXqeKJYgFD9hr8XqkepNuXVg/yUOUPMoOeoqNBQkFLOZXa+Txz2QiOzybX6MVU+m5XTp+dd98f7p6ncL+1azZkyQw0jEvtFWd/cRNvedYX82TjI4/YXYkQXQgymBimDvTj8kMWZxEvScJdZ6k5fhiJ95iJD3D5FCUgWiCOzMn3uJ2hzxJOPEA47ETDzAQjRGPDxLtPzn/ne0cJCeZHYkTHUySyaQIF7YvPPNOzL7il1+WmBxsbw65YoVYaoxEtL/+v4rnP5jOkEkPMTjQT/9AnNFUjL5SscGbNG2X2s/nybf4337R/h8D0/qeNrzwO7/88oh0f+O+77Y/EEplmB6OE43GGZ458RB3skc1jxIS4zOkE1EGonGGMyceZa72Gqxx/7MJ1jzFt02/sAklyTzOcGc4SSIRJ5FMM/1kgpvHOXZqv3iQjj889zMzN4tsrO9QicaJx+PEB89vEmm7dHzj/WmxPsbrY7F/sVis+98hx0Dl1It0HV/IGrho9hhPa/AkgejILHNTKWLhRs8RIDE+x2R6iFi9J7n24vQlv9pbgDZbcuv9Q5M8mkkzGDr1JPNEMikOnn5FvSrmNb7N/OvfwlIqZPltdZWt2l9qG1n4nUx5kT8s117H6b/D334ZIz//J9ZsXk4bnOTv9R5grXoPMTG5wHQ6wY2g95vJk3P/5Elyn9U/zrd+UV4qOsLcXIZkrK/Bm5TOf2B4lgcTp29GPq69BWv17C1C0vauZ7G+4IHy1hqaXCAzmuBGzUPcXls5/+3xQJLMzCSpxA3CNY9y5/UqL3aLdvsD8czPfJ/K8Zc/Pqf53947j/9G+maUSLiPYOVk/TbX1s7W33j8QJK5V09I9TUvSom381/x/IOwPWc3f+P9Ka2PdH0s1vesQJLHvz4hvF73ZnLX8Q2pAnTNUm9S03qbAhfXKfUmNa3n6TNJTdM0Q/pMUtM0zZA+SGqaphlST/KzSD1JTWs39SR7leTZdcBD9E49Samh2Vd8Nxpu+G+lt3/hq9pH96TtYPYWe7G/yYMUvUpHj9HVs3RefxfvU0g9yV4leXaOHuJnXwfWr/LhN56uZWsf0KxQKRWtt9t4oN3cX/IgJa/S1WN09Sxd19/J+xRST7Kuy/QkXT3ETvQ5e5IAlIvkTNCGYbuVB9rF/SUPUvIqXT1GV89SOj9pu6v3aUo9yVqX6klegT5rT7JWMJHhn7/PEKyUKOa2WVvdIHdksd3SA+3e/v68yQteJZ33GH15ltL6WmzvqPfZlHqSwKV7kpfeZ+5JAod7m/y0V2D/sEwgmuLu1BQLT+DLhxvydgsPtLv723mTnl4lHfYY/XqWrusvzd+vx9mUepJwhTzJS0o9SYq7G5y9wpXPky+H+fFJmkx8g7W8sP1s2by9xe7vL3uTJq+ykx5jO56l0/rnhfljtz5eqSd5PnAHj3Ud+4w9yRaVCwUOGGIgHKBm8HlvL/j3QDu9v6032dKrzKU65jG25VlK5yetf0iYvw/vs1XqScLle5I+U0/SLZv1C8RiRCiTL7V+Ta5hu08PtBv723qWjdW8SlcPslbbnmWrcf2sfwe9z1Z19kGy3nPM1T3VrnmS348/Yba0xkauRCAySGI4RSS7zJpkNtZ7kvn9Ok+y2vgbKq/xxU48xdH0DJO5VbYPwyQnTzzFxQ57kqHgiWdHNEY8fkz5oEjxqGq1HU48yaXRY3775msE4alluWKF0dQYifeb5M6sRvn8B9MZRtnj7V6BEhG+SMXoK2UbPEnTdqlO/Lhts34tqbTAEJNztyjt5GqveQ1xdypF6MP6CYMnbefEWxzPTDI7UmKjECadSREu1H4z3YP9321/YGwqw/TeKlv7MDh+4kGuZ49OvMrZBAc7OfaPShCKMTx+4lWu5Ko1j3GKsfEZ0oev2SuFiN858RjfNnmMXut76lW+Xql5lQCVCvn9k5vUuN15/eX5G9dHSD3Jq+JJWnp36kl65OJJBhJMfjfN6GCEcF+QyvEBhfebvFyt/QJR2l7L01vs0f4mD1L0Kl08xtcRN8/yQwfWvwPep1eqAF2z1JPUtN6mwMV1Sj1JTet5+kxS0zTNkD6T1DRNM6QPkpqmaYbUk/wsUk9S09pNPcle5ejxuaWepJ8S0z+zdDvCh5/q/mSp4BUar18ozdKvM1z8TFGFvZWvePquLHuLJg/S4vjO91/tLTYjiRv0cczH/DterqxZe5Gd9FRbXR+b7x/1JG27op6kjbenGeqQxxkammX6ZpmPTZ/SlLxC4/Urb7PyTaHOHIBgYoqFiQo7eycPBOL1N3mQFsd3vf/Gn8yR5i0r3z4lT5TU1APmnlT48v5Lu/l3yFP1uj7S+OpJfgKepJW31+U+e08ylGRmJkZ2+TeiS4+od7Alr1C6fof7eervlJGJQSp7q7yrfW9I+0sepHR8p/vv+A7JWIW9v66xu18FDtlY2yT1Q5rp+Ete5C3u3054qobrI42vnuQn6ElKHl+nU08ShmczRHcWWSlEmGva5tdbNF6/gTukE1Wyi94/6pn2b+VB+j2+VMP4tUesav2pVipUghFuxEKQv/iMrBv3r+n6GMdXT/IT9CQlj6/jqSc5MLLAvcgOXz3bh8DFR3df3qJw/QbH0tws7fDQywXw2N/kQfo6vtSF8bfIH9xjJJ2G7MlHhpMTaWJAcSDMBVqiC/evdH2M46sn2YH5XTFPUvL4Ot7n7kkOpHkwNcDO4qLnl/jxFo3XL5BgLBWh8NabkPPa3+RB+jm+VKvxXy+vE52b4s2be8AxH3fekT2IEG3xvdrx+9fi+sjjqyfZiQl28FjtZ+vtdb7P15MMxIaIh2+Q+P5fjNVvuPcPfk/9xJdPy9beonT9Qsk7JEMFXm+3fqFZ2r+lB1n3eo90fCmv8cv5DZ7+aYNQ/wDlo0MIDPP41zSlJu+yG/eveH3qQJyW45fVk5THl7oinqStt6eepFvN61fNPuP+n6PnXxAYYub7e4S251neLEBgxMpbtLl+qdu3COZWT/4CYVN+vMWTah6k5fGlbMYvH50cuD81SoIiG3Vf6H/+donXRxpfPUmL8cUu35OUPL7T1JP0rm1PEijWP8sIRDkGAgcfKR6WsfEKra5fdJx0vMLeysVfqBj3lzxIi+NL6yPNPzR0h/HIIe+LZUKDI0xM3KKaXWTj0GL+ltfHtN18feTxXTxJ6wfJZo/xyb/eAE2eZPU9G+s7zE094c3tRs9x/+V9npbnmJx4zA/1nuShxYNQ9T3rK5s8mlnin+lTT/IduVjqwtd5jS/NP7/2NSuBJabmfuBuzVP8aXGVbKf+RRyc5HG9x3d3ie/vnnt8XwxFCPZFmPruVt1ONW+v7l+CUDAAlSMO5X8AW7a7tsYXcxkWfrzb4ElK518+DhGbeMBS5tyLXF1cPzuutL3rmda3nX9Nmnq5uExwZpKJhR+ZqXmFb/+6zIvaX+OzuX7xdJpYOct6800l7p/lgHHS90YaPMifnq6dv8XHdHxpfSzuvwoREuMTjEX6oPSR3Lu/cn8tazl/Yfyap9r29bMYv7j1kMXwApl7S9yueZKbiytWz7hVAbpmqSepab1NgYvrlHqSmtbz9JmkpmmaIX0mqWmaZkgfJDVN0wypJ/lZpJ6kprWbepK9SvDyjF6gc+pJ+qmVVyhdH9EzFDxKaXyQPUSX7UOzr/hutN7VgdLbv/BV7aN94v3p4m124PjO8zeknmSvkrw8kxfY6wf061iXPUnp+kieoeRRSuNLHqLrdoDKh994upatfUC1QqVU9wZu4fydvM0OHN91/qbUk6zrMj1JyQvsRepJenuF0vWRPEPJo5TGlzxE1+0AlIvkPD75JJ2/q7fpenzX+ZtST7LWVfIkRS+wC6knae8V2lyfZk/RxqP0HF/yEF23n845keGfv88QrJQo5rZZW90g1+JTe63Ov5PeZrvHd5m/KfUkgaviSdp6gZ1PPUkbr9D6+rTwFCWP0jh+SPAQK47bD+Fwb5Of9grsH5YJRFPcnZpi4Ql8+fBc0zGdfye8TZfju87flHqScGU8SSsvsBupJ2nlFdpen1aeodGjDNqMb/AQdxy3bx1R3N3g7KfOfJ58OcyPT9Jk4hus5eXz74S36XJ81/mbUk/yfOAOHqv9JC+we6knaeMVStenpWcYGjF6lNmiMP684CFKXmIbnmK5UOCAIQbCAWoGoff551Id8zZdj9/W/IXvL/Uk4cp4khe76AWCepKu+fYkPWu8Pp6eYSBk9CjF8atls4dY3Xfb3mpesRgRyuRLpgeQ2vkL5yeuT4eO3/b8hdSTBC7dk6zesvMCUU/SVNc8SQvP0egZHskepeQlSh6i0/bAEJNztyjt5Gqv6Q1xdypF6MP6yY+q0vlbnJ+Tlykd33X+QupJXgVP8qWdFwjqSXrWTU+yLFwfC8/Q6FFaJHmITtsDVegb4u5MmnBfkMrxAYX3P/F0dcvu/HH1Nl2P7z5/U6oAXbPUk9S03qbAxXVKPUlN63n6TFLTNM2QPpPUNE0zpA+SmqZphtST/CxST1LT2k09yV7lwzv08gTbTz1JKVfPEATPcSBJZmaSkcQN+jjmY/4dL1fWzj1FwUMUPVJpf2F+xv1DaZZ+neHiZ6Iq7K18xdOmdW51/4reprC/tH7G4/ucf3PqSfYqS+/Q0zPUzLl6ko6eoeQ1jj+ZI81bVr59Sp4oqakHzD2p8OX9l2dfY/QQLc7PtL+TJ1neZuWbQp2ZAMHEFAsTFXb2Go/hdf+KnqSwv7R+xuP7mH+r1JOs6zI9ScDoCfaiz9mTdPUMjV5j/x2SsQp7f11jd78KHLKxtknqhzTT8ZfnP/UYPESr8zPs7+pJHu7nqb/TRyYGqeytNr4Z23D/Sp6kcX+L9ZOObzV/j9STrHUVPElbz7AbqSd5nm/PUPIaa49o1fr7pFKhEoxwIxaC/MkNZusheuW5f4c9SQbukE5UyS5mG/6zn/u3lSfpub/l+knHl+bvlXqSwFXwJG08w+6lniQ4eIb9kte4Rf7gHiPpNGRPPiqXnEgTA4oDYaBs5SGaMu4veZSWnuRpg2NpbpZ2eFjnGvi6f1t4ksb9LdZPOr40f1PqScLle5KWnmHX+tw9yVpte4aA5DW+Xl4nOjfFmzf3gGM+7rwjexAhWrvXbTxEU8b9D+T5WY8fSDCWilB4W0fg+bx/L3iSFvtL62c8fn2t5i+knuT5wB08lr/8eIbd6/P1JOtryzPclL3Gcn6Dp3/aINQ/QPnoEALDPP41TanUWirx8hBta9i/0DlPMpS8QzJU4PV23S+FfNy/rTxJm/1t10/yKlvNX0o9Sbh0T9KvZ6iepFv262fpGVZ3rb3G8tHJ9v7UKAmKbHjAinYeoncN+/uYnzR+6vYtgrnVhr/waHv/enmSfu5/0/rZeJWt5i+lniRw6Z7kUVX0BE9TT9K7tj3JDniJkucYGrrDeOSQ98UyocERJiZuUc0usnGI7CFK51dOiPs7eZJnExgnHa+wt3LxFx7S/Wv0JC32N66fxfGl+ZtST/JKeJL2j3bqSXrksr6uXiKy51ghQmJ8grFIH5Q+knv3V+6vnX6zCh6idH6v5f2dPMla8XSaWDnLut9vCgtvU8q4fpbHb3f+qgBds9ST1LTepsDFdUo9SU3refpMUtM0zZA+k9Q0TTOkD5KapmmG1JP8LFJPUtPaTT3JXtUBD7D91JOUEj1JzB5jt71E8fwsvEvj+ML5ASQmF5geTXAjVOEgv8Payovz+TmOL97/0vqYjq+epM+usCdp9AQ1c132JCWPsdteonR+kncpjS+d30B6icdj/WRX53lWCJOemWXuySFf3t/oyPhgvv9tPE7P46sn+Ql5kiZPsAepJ3l2sAuepOQxdttLlM5P8i6N4yOfXyqdgOwiz9+dXM+1tR2S36eZjm/wIu8+PuB9/9t6nIbjqyf5iXiSrp6gS+pJnnfBk7T0GOvrtpfYnNG7lMaXzi8wTDwK+9sfzvcp5ChUbhOL90P+yG3802Xwuv8t18fas1RPsp0u35N09QTdUk8SDJ6k6EU2HajbXmKLjN6lNL7kTVbChIMVSsdV4pm/s5DM8/zbd5TLEA2fKEJO4yPc/xbr48ezVE+ynS7bk8TdE3RKPUnA7ElKHmN93fYSW2X0LoM23qPh/HZOv6ZCpXREqVSm3PT44jq+dP8b18ePZ6mepNMEO3gs91w9Qf+pJwkenuSevcfYbS+xZaERo3eZLQrjzwvnVy5RqgTp6wuxvzHP1xtAYJg7ISiXSu7jt/BSm+9/0/r48SzVk7ymnmSrTJ6gepJu+fYkLT3GbnuJngnepTh+tWw+v+o++eIjxhM3Ybv2Ol4sQSx4wF7+yH38Vvt53P+t1qd6aH989SSvqydp4QGepp6kd13zJJG9yG57iebzk71LaXzp/Ha2c4xnJpkdKbFRCJPOpAgXNmo/GTqOb+FZSutj5bGqJ2ke/0p7khYe4GnqSXrUZU/S7DF22Uu0OD/Ju5SSPMzD7XmeDSwwnVnix+DJm8mXl89fUnAbX77/xfWxSD3JzyT1JDWttylwcZ1ST1LTep4+k9Q0TTOkzyQ1TdMM6YOkpmmaIfUkP4vUk9S0dlNPsldZeIeS59d+6klKSZ6kuF3yEH3Mr6X36OpJmjxGC2/Rxss0epOOnqroffrwLP2mnmSvEjxAyfPThLrsSYrbETxQy/l5eouOnqTRY7TwFiUvU/ImnT1VYf1tPct2Uk+yrsv0JCXPrxepJ3l2sAuepLQdMHugNvMzeYsunmQlLXqMkrcoeZmSN+nqqUrrb+VZtpl6krUu1ZNswyvsdOpJnnfBk7Tc7uqBWnuILTJ6jrWXoK29SgtvscHLtPAmbbJdv1brb+NZtpt6ksBXVoLsAAAgAElEQVSle5KS5+f+soqQepJg8CQttrt6oH48xFYZPceyP69S9Babvcx+2ZuUslk/0/pLnqVL6knClfAk/XiFHU89SUDyJM3bnTxQPx6iR0bP8ciHV2nhLV7wMs/y9ialbNbPtP7S+buknuT5wB08ls/K9l5h91JPEjw8ybrXU6Ttp/nxQP14iC0TPMfdjaK1Vyl5i628TNGbbCOv9Wu5/rmUeP4uqScJl+9JWnqFp6kn6ZZvT7KN7SYPtLl2vMXGwcyeY32SV2nyFr28TKq7Zm+yjezWr7b+Ps6/ndSTBC7dkzyqip7faepJetc1T1LabuEhSvOz8RBdPEkbr9LkLUpeptmbFOYvearS+lucv0vqSV4FT/Llvuj5naaepEdd9SSF7QELD9Rlfhb7S56jjcfo6S1aeJmSN+nkqVp4n66epilVgK5Z6klqWm9T4OI6pZ6kpvU8fSapaZpmSJ9JapqmGdIHSU3TNEPqSX4WqSepae2mnmSvMnl6Fp6fW+pJSkleoY2neFpLDxLBCzV5j5bnZzq+q9conX8nPE1bT7Xl+qon2cGuoidp4flpQl32JCVP8TQvD1LyQo3eo8X52XikLl6jzfm7eJq2nqrX+qon+Rl4kpLn14vUkzw72AWvUPIUAaMHafRC+++I3mNHPFIHr9Hq/B08Tav5G9ZXPUk+cU+yOQvPr9OpJ3me5ElCk6dYy9ODlLzQ2iOGtfd4YcJ2HqmL12hz/m17mpbzN3mb6knyiXuSTYmeX8dTTxJkT/KsZk8RwYMUvVB/3uPF+cgeqavXKJ2/k6dpMX/J21RPks/Bk6xl4fl1PPUkAdmTPO2Cp2jlQZq9UGvvsc3ju3qNxvPH0dOU5p/9Qlxf9ST5xD3JuiTPr3upJwmyF9nKUxQ9yHnZC7X1HlvWhkfqy2sUzr/llHx4mtL8xfV9WlZP8pP3JOsyeX6nqSfpVruepJenKHqQ1bK1Fyp5jy3z6ZGCT69ROP/2j283f3F9AyPqSX4OnuTJF3h7fqepJ+ld1zxJZE9R8iAlL9TGe2zbI5W8S9fz74CnKa2PeX3Vk/wsPEkweH51qSfpUTc9SQtPUUryQkXv0cUjlbxL1/P/4O5p2nqqXqknqZ2lnqSm9TYFLq5T6klqWs/TZ5KapmmG9JmkpmmaIX2Q1DRNM6Se5GeRepKa1m7qSfYqydProoennqSc5Cl2wgNNTC4wPZrgRujkT66urbw48xiNXqPF8UXPEcFrlO4/abvr+gjXT7w+pvk5eq3qSfYqwdPrpof3WdRlT9LVAx1IL/F4rJ/s6jzPCmHSM7PMPTnky/snH+8zeo2W3qjJc5S8Run+E+9PVy9Vun7C9THPz81rVU+yrsv0JLvp4dmmnuTZwS54iq4eaCqdgOwiz9+drMfa2g7J79NMxzd4kZe9Ritv1OA5Sl6jdP+J96erlyrsL10faX4uXqt6krUu25Pspodnk3qS59l4isaaPdDAMPEo7G9/OP+aQo5C5TaxeD/kL36ntvIaPY9/uo+X52jhNUr3X0fvT0cvtdX18TU/n+OrJwlcBU+ymx6enHqS4MNTFLrggYbChIMVSsdV4pm/s5DM8/zbd5TLEA23UHpaeI3G4yN4jhZeo3T/dfL+bNdLNV0fP/PzO756knAlPMluenhi6kkC9p6iMaMHWqFSOqJUKlM2fH+28hql4xs9xwOQvEnp/uvY/engpZquj/X82hhfPcnzgTt4LJ+FRrrq4dmlniTInqJUSw+0XKJUCdLXF2J/Y56vN4DAMHdCUG7yIiWv0dYbbfAcC4I3Gbplvv+24x27P1291JbXJ5eynl8746snCZfvSQZCvjw89STdateTtKmlB1rdJV98xHjiJmzXXgeLJYgFD9irez3Sxmu08UahyXOUvEnp/vN5f5qynb9dtevjY37tjN/ZT9zUe4711TzJ6PgTZkcSRAeiDCaGuTP9mMyQxSLXe5JQ50laji924ilG0jNMDkUZiCa4M3PiKW532JOMx048vUA0Rjw+SLQ/AEdb5AqQGJ8hnYgyEI0znDnx8HJNHl5i9hW//LLE5GB7c8gVK8RSYySi/fX/VTz/wXSGTHqIwYF++gfijKZi9JWKDZ6kabvUfj5PvsX/9ov2/xh4rm9dwwu/88svj0jXn34oSeZxhjvDSRKJOIlkmuknJ57iTt0L/+LxTz3Qtxd/IbCznYPkJLMjcaKDSTKZFOHC9tlPVqde48Z6zWuMx4kPNl1kr+MHhpg8nX88wdDIJN/NpAh92D7zHN9tfyCUyjA9HCcajTM8c+I17mSP5PvP8v50WR/j/tL1sf3+Ecb3yhq4aPYYT2vwJIHoyCxzUyli4UbPESAxPsdkeohYvSe59uL0Jb/aW4A2+fLhxdeB+ocmeTSTZjB06knmiWRSHDxt9Py8xreZf/1bWEqFLL+trrJV+0t1Iwu/kykv8ofl2utE/Xf42y9j5Of/ZPc3PAYn+Xu9p1frzDus/XH6VOIG4ZqHt/N6lRe7jQ+Sybl/8iS5z+of51u+qC8WHWFuLkMy1tfgSUrnPzA8y4OJ0zc7H9fegrV69hYhaXvXk9a3Vkt0F7jz+G+kb0YbPMXNtbWz87c5fjzzM9+ncvzlj89b/sInMbnAdDrBjWDTm8kDSeZePSHV17xHo1fpefxAgsnvphkdjNR5jpu8XN1qeEY6NLlAZjTBjZrXuL22cv7XCKX7T9ruuj7C/uL1sfj+ka6PV6oAXbPUk9S03qbAxXVKPUlN63n6TFLTNM2QPpPUNE0zpA+SmqZphtST/CxST1LT2k09yV4leYe1tzCMJG7QxzEf8+94ubJGRzhJ9STFRK8QswdZX2L6Z5ZuR/jw03/zcOvIyjM0epIW85P2B7Mn6epZ2nioTp6lsL/kadqsj1fqSfYqwcsbfzJHmresfPuUPFFSUw+Ye1Lhy/svL2Gy17Aue5KSB3laaGiW6ZtlPtZ/ytPCUzR6khbzk/aXPElXz1LyJl09S2l/MHua4voaUk+yrkvzJPvvkIxV2PvrGrv7VeCQjbVNUj+kmY6/7NmzcvUkzw52wSuUPEgAQklmZmJkl38juvSI+uc1kmcoeZLS/KT9JU/S1bOUPEdXz1LaHzB6mtL5mVJPstalepK17+hq/XEqFSrBCDdiIci34VH5TD3J8y54hZYe5PBshujOIiuFCHOmASw8Q5MnaeNdNuxv4Un6Gb/V/I2eo6tnaTl/T0/T7/k1pZ4kcOme5OEW+YN7jKTTkD35SGZyIk0MKA6Eaf3R/U6mniQYvMJ+2YMcGFngXmSHr57tQ8D8r4PoGXp4ktbeZfP+/bInaTO+af5Gz9FifNf9jZ6m5fp6pZ4kXAlP8vXyOtG5Kd68uQcc83HnHdmDCNFeoLvqSQI2nqSHBzmQ5sHUADuLi/IgFp6hlydp61223t/sSdqMb5q/0XO0GN91f6On2XRrGc+vRepJng/cwWP5r5zf4OmfNgj1D1A+OoTAMI9/TVNq8ga7l3qS4OEV7pk9yEBsiHj4Bonv/8VY/cHu/YPfUz81gC2SZyh5kpJ32XL/suBJ1snobXmWkoe66ehZSvs3y+40eZoI6yOkniRcvidZP5WjkxPsT42SoMhGC1hQPUm3fHuSggdZzT/j/p+j57sFhpj5/h6h7XmWNwsNRzR5hjaeZMv5SftLnqSP8VvOX/IcXT1Ly/k37Ffvafo4v1Z19kGy3nPM1T2VrXmS348/Yba0xkauRCAySGI4RSS7zJrkFtV7kvn9Ok+y2vgbLq/xxU48xdH0DJO5VbYPwyQnTzzFxQ57kqHgiZdHNEY8fkz5oEjxqEpo6A7jkUPeF8uEBkeYmLhFNbvIRtM3U2L2FUujx/z2zdcNv/CyLVesMJoaI/F+k9yZ1Sif/2A6wyh7vN0rUCLCF6kYfaVsgydp2i7ViR+3Tet7WksqLZQkM5vgYCfH/lEJQjGGx0+8wpWaJ7mznWM8M8nsSImNQph0JkW4cP6b7WL9s5xAlGMgcPCR4mHdApx6hisXf2Fz6km+Xql5kgCVCvn9fav5GffnxJMcm8owvbfK1j4Mjp94kuvZI6v9jfM/2iJXmGJsfIb04Wv2SiHid048x7e116iN47vuHxhicu4WpZ1c7TXJIe5OpQh9WD/7Udvq/DyyfpBs9hif/OsN0ORJVt+zsb7D3NQT3txu9Bz3X97naXmOyYnH/FDvSdr8udTqe9ZXNnk0s8Q/06ee5DtysdSFr/MaX5p/fu1rVgJLTM39wN2ap/jT4irZTr0mODjJ43ov7+4S39899/IqREiMTzAW6YPSR3Lv/sr9tYvfTKFgACpHHLb5U/ju2hpfzGVY+PFugycpnX/5OERs4gFLmXMvcnVx/ey40vauJ6yvsXKWA8ZJ3xtp8Ap/erp29haXw+15ng0sMJ1Z4seaB7m8bP8jG0A8nSZWzrLefFMFknwxFCHYF2Hqu1t1G049SWF+4v5Q3HrIYniBzL0lbtc8yc3FlZNnhBb7G+cPvFxcJjgzycTCj8zUPMe3f13mRe1B3Di+6/6BKvQNcXcmXedp/sTT1S3L9TVfN1WArlnqSWpab1Pg4jqlnqSm9Tx9JqlpmmZIn0lqmqYZ0gdJTdM0Q+pJfhapJ6lp7aaeZI+y8exsvUL/qScp1QlP0uglCl6ozfhn82j2KpE9RXF8aX+L9TWuj4UXadrf1dNUT9JPlzR/ybOz9Qo1jy7Zk5S8Q9ELFcY/raVXWcvkKdp4pab9pfWV1kfyIsX739HTVE/yGniSkmdn5RV2OfUkzw7m25M0eocWXqg0PmD0KgFvT9HWKzV4jNL6SusjeZHS/q6epnqS19CTbPT+7LzCbqae5Hm+PcnCTbN36NML9fIiJa/S01O0HN+Px9g4Yfn+NXuR/u5/355mG9vrU08S6LknecH7k73C7qaeJDh4kiHJO7TzQk1epORVGj1Fi/F9eYzNheT71+xF2t3/bXuafrc3pZ4k9NyT9PbsPLzCbqeeJODgSQKSd2jjhXqOb+FVSp6iNL4fj9E77/WRvUhpfV09TfvtzakneT5wB4/lnZf3Z/IKe5N6ktCeJ2njNdp6oa3GX8Heqzyt2VP065V6eYytv1hYHwtv0vb+b8vTrEs9ySvuSZq8P5NX2Jx6km512pP04x3aeKHN41d37b3Ksy9p4Sn6Gd9r/5ZJ62PhRfq5/8+z9DQtt3ulniTQC09S8uwkr/A09SS9u0xPUvIajV6oxfhGr9LCUzSOb7G/tL7m9ZG9SOP+HfA01ZOs+7or6UlaeHa2XqF6kh5dsicpeYlGL9RifHOCpyiNb7G/tL7S+khepHF/aX2k768P6kl+VqknqWm9TYGL65R6kprW8/SZpKZpmiF9JqlpmmZIHyQ1TdMMqSf5WaSepKa1m3qSPUr07Bw9RHPqSUqJ18fRYxS9SFfvsf5cWniTrp6lzfieHmQozdKvM1z8TFWFvZWvePquLK+P4FE6r78h9SR7lOjZuXqIn3uO6yddH2ePUfAQnY9fy8ub7IRnaRrf6EGWt1n5plBnLkAwMcXCRIWdvbLV+JJH6br+ptSTrOsyPUlXD7ETfc6epPH6HLt7jEYPsQPHB7y9yU55lobxJQ/ycD9P/XfKyMQglb3VszfLS+NLHqXT+qsnaTF/LtmTvAKpJ9lYw/XpsMd4wUMMd+b4nt5khzxLz/H9eqgDd0gnqmQXs7Sq1fhGj1Kan8XxTaknCVy6J3npqSfZ0IXr0xmP0dND7ID3aPQmO+BZGse38CTrGxxLc7O0w8MmF8E0vtGjlOZncXxT6knCFfIkLyn1JBtqdX064TGaPESn45dkb9LJs5TGP7ttLDzUQIKxVITC24uEnml8o0d55L7+ptSTPB+4g8fyrh3PrjepJwne16dTHqOXh+hy/EBY9iZdPMvlFq8nNZxfwd6DDCXvkAwVeL3d+oXAluPnUmaPcuPisfyuvyn1JOHyPUmfqSfpltf62VyfznqMjR5iu8ev5u29yXY8S/H8fHiQqdu3COZWz34jba42vuRRSvOTji+kniRwFTxJsPMQ1ZP0rm1PEvn6OHmMFh6iq/do9Cal40vzsxjfykONjpOOV9hbafqFjTT+keBRdmD9TakneUU8SVsPUT1Jj1w8SYvr4+QxWniRzt6jkJNnGZDHt/FQ4+k0sXKW9eZvKov1MXqU0vwcvU5VgK5Z6klqWm9T4OI6pZ6kpvU8fSapaZpmSJ9JapqmGdIHSU3TNEPqSX4WqSepae2mnmSPkrxC0TN0Sj1JKXH9Bc8QYCA5yczEKIloGCoHFHfWuP8ia7e/NH9HTxEM3mPzWrTwKEVv0nG7NL7kYYrr75B6kj1K8gpFb1Iz12VPUvIMA4lpluZSlN6+ZHmtQDkwQDR8/iAneojC/F09RaP3WJeXRyl6jK7bhfElD1Naf5fUk6zrMj1J0ZvsQepJnnZx/SXPcHRilGB2kYcvTp+55Rs+DSZ6iML8XT1FyXsEvD1KZI/RdbtxfAsPU1p/l9STrHXVPMlee5PqSTbWvP5GzzCQZGiwwoe9CLN/+5lkNET5MMfO+iova7SejYdoyslTtPQePT3KpiSPsd3tbXuYhYS4/i6pJwlcOU+y596kepINtVh/o2fYP0C4L0hsLE329XPmP8DNu9Nk5h5xeH+e7UPZQ5Ry8hQtvEejR1lL8hhdtjt5mCF5/V1STxKunCfZc29SPcmGWq2/0TMEIEh5b53l2jnvr7xm6NUjRpL9bG8diR6ilJOnKHmPA7JHCbLH2PZ2i/FlD9O8/i6pJ3k+cAeP5Z3kSV6eN6meJHisf2jE7BlulihXoFQv8VR3OSw/IhoJQ+iWbw+xIWl8yVMUvMdATPYoT5M8xna2r+DoYZaF9ecqPUiqJ2lM8gptvUn1JN3y7UlKnmF1l8LBI0ajETj9c1eBJAMhKB+U2vIQm77IzVMUvEc/HmVjksdot7266+hhSuvvmHqSwFXwJG28SVBP0lTXPEnJMwTebX9gbCrD9N4qW/swOD7JrWCO9eyR1f7m+Tt6isjeo9GjlDxG1+3S+AgeprT+jqkneRU8yQ8W3mQt9SQ96rInafQMgeLWQxbDC2TuLXE7DMcfc2wurpy9h1HaX5q/k6eInffomegxum6XM3ub8vq7pArQNUs9SU3rbQpcXKfUk9S0nqfPJDVN0wzpM0lN0zRD+iCpaZpmSD3JzyL1JDWt3dST7FGSV+jH2/OfepJSrp6kjedo8g7F6y95ihbnL3mLLh6mOH9pfqbzC6VZ+nWGi5/JqrC38hVPa8cxeZkuXqt6kj1K9CItvT3No0v2JMHsOYreoXD9JU9ROn9pfGcPU7p/hfkZz6+8zco3hTqzAYKJKRYmKuzsncxR8jJdvFb1JOu6TE/SytvrcupJnubfkwSMnqPkHRqv/7HsKUrnL43v6mGK969pfhZe5OF+nvrvtJGJQSp7q2dvRpe8TBevVT3JWlfJk5Q8vm6knmRjvjzJ0308PUfZm6zvwvWv/RTv6SnmhZcVpPE77GH6vn8lL7L5/AbukE5UyS7WXgqw9DIbhvThtaonCVwVT1Ly+LqXepIN+fUkkTxHO+/Q8/pLnmJL+qL+fITxK53xMNu+f32e3+D/3979x8aZ3wd+f3t3Z7zx8GqO0SNTd9SGTIrRAR0HoNyAbkBeEjK+zF4g9TZU3BDZkheHqi3CChe7oKBKsKotF8ISXnDhK+XtMl5IXVdndwk7IjYg0BObC4XERBsKd54/TgMkw7Qi7JC4eJiUk+7O7Dr9g0Nyhnye7+f7PN9nfrD6vIEgwD56vj+e55nxkCJfOpfldHGVl/cHt/AypftrSj1JaBtPUvL4GpZ6knUF9iStPEfZOzTdf9lTlDLMvyqvz8bDdHl+rfcXy3BusJvCfS+Cz8fLPFJQr1U9ycOJIxzLP8mLlDy+xqWeJIT0JC08R1vv0O/+Gz1FKclblI4H8DDDPr+2+0v0n6U/UeDucs1aS2Yvs7YwXqt6ktA2nuTxvD0+9STdityT9PrvRzzHcN7h8fvv6SlKSfNXNhrkYUqe5PGk/Q0+d4Z4br5e9xG8zP2Cv/72Uk8SaLknaeHt7aeepH8t8yQtPEejd2hx/yVPUdq/5C06eZiWz69pfTb7IzVCNl1mfe74v6UteZm2XqtX1sDFUY9xvzpPEkgNTTE9Pkhvst5zBMiMTDOW7aO31pNcuLX/Lb/qjwDdq+Pi9+vsG+PyZJaexL4nmad7YpCt6/Xeot/8Nuuv/RGWYmGNd+fnWar+zdrQje8yUZrhv56tvvl2nuVrb58jf+WLBy8E32L9TL9zjcGOowcOvcKzV79G9nSqztu7t7BwMP9+/dPf5lr/BvO/cyXcPxKWGmJ6eoL+3o46T1Laf9fAFC+O7v8w7m71R7DmD35ESDre8HrG+Eatx1jtqCfp+SZpcX/2f9h5MHOKZNVzXL07z60HmxDLMPbKRYZ7ums8x3vcmV+q+8TSN3aDieEMp6re4fLC3MHfHkv3P9Y3wc3Joepr5zG51btcX3gQaP+m+cXjpv1brF9an7g/ID3xTV4fzPHV33nD8y+EMmM3uJjNcCp+5IfJbe6vIVWATljqSWpac1Pg4iSlnqSmNT39JKlpmmZIP0lqmqYZ0jdJTdM0Q+pJPhGpJ6lpYVNPskkF8ewyF7/Jzee6efTWb/Pykvu/G6yepJyrJ2mcX/IQ1watvEQnD7IJxw+updfzK9wfV2/VxvM0rs+QepJNytazS/RNcfF0icfN+S3J///Uak/SNL/kIVp4ia4eZKOP7+f7/Ar3Jwpv1eR5iuszpJ5kTa30JAFI9DM52cva7Lukbl4m6TVQA1NPcr8QnqQwv+QhSsddPchGHwfMz69wfSLxVg2ep7g+Q+pJVmsHT3JgaoLU6gxzhW6mgw/plHqS9YXxJK076iFKx109yEYfrxbl8xvGW/X1PB3Xp54k0A6eZNfQDS50r/LCqxsQa/Z3DdWTrCuEJxmkYx6idNzVg2z08e2In98Q3qrR88RtfepJQus9ya4sL453sToz47AJh9STrCuMJ2md0UM0HXfwIBt9fO1zkT6/YbxVo+dZdHt9qSd5OHGEY/nn5dnFevtIJ0+Ref17nKv9wxe+xXcH3/IEP6JPPUmIzpP0y9NDlI67epANPh7l8xuVt1rrecaSbutTTxJa7klW1l7l0pdSh/8h1sfk6xdILF9h9l7h2DjqSbrVDE/SL08PUTru6kE2+HjQ59evqLxVqPc8K3m39UX7Gze1nmNtVU8yNXKNqaEMqa4UPZkBzl68ykSfxUNW60lCjSdpOb/YnqfYnZ1krC9FVyrD2ck9T3E5Yk9y8XbVs0unSff0HBzf3Nys+b9tdoHy1mM2t+uflszUO7z99k3GeghVbrNM7+A5MqnO2v8q7r8nO8FEto+erk46u9IMD/bSUdys8yRNx6U28nnyHv+3sWn/PwapnjTp3j2vMJbqJZ3uIdVZ/3wN3Pgub799mWxn/bnG+7OzRK4AmZFJspkUXak0AxN7nmLu4eGnK3H+fQ/xvs9f2BiOryw/IjE4wcWBNKlUmoHJPe9xtcaDbOVxm+fXdH2M1z/Rz8TVCc4O9JPJpMn0Z7l4bc+rXM1V9jzP/ePpDH1DY7wyOUji0fIBY2j7+vLK+pPkUY/x2vfeA454kpWHLN5eZXr8Gu89V+85bty5xPXSNGOjV/l6rSe5bfEmVHnI7bl7XJ68ybez+57kCrnewWN/zm9+af35hS8zF7vJ+PTXOV/1FN+amWctxDfmjxXr53N93cQ7uhl/5UzNATvPrrZEPAblHbYt1H6vHiws8LnpCW68eb7Ok5T2X9pN0Dv6IjcnDr3I+ZnbB+NKxxtezxhXa73C8zd5/fxxT9Izi/tzZ2aW+OQYozfeZLLqKd5/bZZb+3+7bTF/Opult7TGbZ+HynR8c+llZpI3mLhwk+eq3uO9mbmDT5ytPi5muj53u4Xrv8YWI2QvDNV5lW9dX9j7EalYBTr6OD+ZrfE83+L6fDTfplIF6ISlnqSmNTcFLk5S6klqWtPTT5KapmmG9JOkpmmaIX2T1DRNM6Se5BORepKaFjb1JJuU6OUF8PCCp56klLMnWTuW4BV6HTd6iZJHuVKS11/9J2GHMqfoYJfH+RXuzC3Y7S8iD9N3/xb7E++vsL8gnuvR1JNsUjaepI2Hp/nUak+ymuQV+h43eYkW3qS0/pFr02S5z9xL18mTYnD8RaavlXn+0h2L/bl7mMb925wv3F9pf7aeq1fqSdbUck9S8vAanHqS+4XwJEH2Cg3HJS9R8iaN6989S39vmfXXFniwUQG2WVy4x+DXs1xM3+FWXt6fq4cp7V8833R/O+X9Wb3+fFJPslo7eJKSh9fI1JOsL4wnKXmFtp6hn5d4kORRHl1/9R2lUvuclsuU492c6k1AvhTMywzqYQbcv83+6jcr7+/YKQav8mjqSQLt4ElKHl5jU0+yrhCepOQV2niGJi+xNsmjPL7+JfJbFxjKZmFt71f1+kez9AKbXUmgFMjLDOxhWu7fen9H25b3V5ePV+mXepLQek8SwcNr9Fes6knWFdiTjAteoaUXavISD5I8Sp/13529TWp6nPfeuwDs8nh1hbWtblIli/3VvpGE8TCDeKkW+/NK2l9tfl6lX+pJHk4c4Vj+SV7efrUeXtWAa3DqSUI4T3Jt0+wVfmExY+0ZSl6i5FH6PV+l/CLXv7hIorOL0s42xAa4+p0sxWIxkJcZxsMM4k1K4/tl3J/F9TGlniS03JP0qtbDO5p6km5F7UlKnmJleymkZ3jcSzR5lDbPV2ln78TOwWEybLKYKwXyMsN4mEG8SWl8Kc/9VQvuVe4V7ZtkreeYq/koW/UkXx+5xlRxgcVckVh3D5mBQbrXZlmQvvdQ60nmN2o8yUr9v9jmN7/Ynqc4nJ1kLDfP8qiNblAAACAASURBVHaS/rE9T3EmYk/y7lzVywMol8lvbOx5eNNnKK7mqt+T7OP8+CCJR7ePfamdmXqHm8O7vPuVLyMJYF7lNssMD54j8/AeuQOrUd5/T3aCYda5v16gSDefG+ylo7hW50majktF8eV2qidNIr7nFZLqJZ3epbS1yeZOzV+u3Pgul8/k637qAoT7s7NErjDOuZFJstt3WS8mSJ/d8yTvV78Hu1mrdsdS7AKxGq/QeDzRz8RUhq3VHBs7RUj0MjCy5yXO1f7Fyb43OXf8LzSM6wcSfWcZ6d7m4WaJRM8Qo6NnqKzNsLgNIO9Pml86Ll0fm/FN99e8P/n6mFJPsh08yUf2Hp56kj612pN0qSR4idV8vUmL9ZfpJjMyyrnuDig+JrfyGpcWDt+MbPbn4mHaZDxfuL/G/Tl6rqoAnbDUk9S05qbAxUlKPUlNa3r6SVLTNM2QfpLUNE0zpG+SmqZphtSTfCJST1LTwqaeZJOy8ey6+seYHB0mk0pCeYvN1QUu3bL8JX9j6klKGe9PFJ6ihUeZGbvBxeEMpxJltvKrLMzdsvcQJS+ydq8hvEvx+gr7M3qZFsfF9VnsP+zrSz3JJiV5drHMRW5OD1K8f4fZhQKlWBepZBQ/pPmE1EhPUvIOLTxEyaPsyt7k6rlO1uav8GohSXZyiulr2zx/aVFeH7KnuF9o71K4vqK3afIyKxbHhfVJ+3d5faknWVMrPcnh0WHiazO8fGv/N4XyNPsHfdST3O/4/XH1FCWvcTCbgbUZ3ljZu14LC6v0v57lYnpR9hAtvEjAybuUrq+0P8nLlI4b12fhSbq8vtSTrNZSTzLWT19PmUfr3Ux97Zv0pxKUtnOs3p7nzsPmgJLqSdZn9AZDeIpGrzE2QDoFG8uPDsco5CiUn6M33Qn5489AUC8SovMuvQriUUpept9x3/VJ+y9knF5f6kkCLfckO7tIdsTpPZdl7e4bXHkEp89fZGL6MtuXrrAc8pf97VNPsi7BGwzjKRq9xs4kyXiZ4m6F9MQ3uNGf542XViiVIJVMAkcWEcKLjMK7NGXjUUpepum4cX2SJ5lwe32pJwlt4UlCnNL6bWarBuPG3F363rnMUH8nyx7fXI809STrMnqDYTxFBK/x4E+VKRd3KBZLlAxfnQT2IiXPMYj36JONRyl5mb7HLdYne5LhX1/qSR5OHOFY/nl6dqUipTIUa6WUygO2S5dJdXt8kmhI6kmC7A2G8RRFr/FekWI5TkdHgo3FK3x5EYgNcDYBJUsP0eQpSp5jEO/Se9P2HqXkZXodn0P2KI2epOPrSz1JaL0nWXlAYesyw6lu2P/2f6yfrgSUto5zP+pJuhXYk6wpjKcoeo2VB+Q3LzOSOQ3L1e9j9mbojW+xXvP9yLBeZGW7Ud6l5f58O+5leh2vPLD3KD09yYCvr6OpJwm03JMEVpYfcW58govr8yxtQM/IGGfiOW6v1b+Y1ZP0r2Ge5MEEIT1FC49ydTnHyMQYU0NFFgtJshODJAuLB195uXmRjt6leH2F/UlepoWnKa1P2r/t68sr9STbwZPMwebSy8wkbzBx4SbPJWH3cY57M3PHPrGoJ+lTgz1JcPMUJa9xe/kKr3bd4OLETd6M7/0w+ezsovX6JC/SOeH6mvcneZl2nqYpaf+2ry+vVAE6YaknqWnNTYGLk5R6kprW9PSTpKZpmiH9JKlpmmZI3yQ1TdMMqSf5RKSepKaFTT3JJuXkFQZwEb1TT1LK9f6IXuLUO7wyXO/uFO9/lRdqfrXQ6B1K+xM8RZv5D65FGG9S8hwFb9LZy3S8/6bUk2xSTl6hJtdKTxKsPMTyo3e5vrBW/QXQMuXi4Q9Ii96hsD8bT9I0/35hvUlpfsmbdPYyHe+/KfUka2qlJyl5hM1IPcn9gt8fKw+xtEnO5zeLRO/QtD8LT1GaHwjvTVrML3mTzl6m4/03pZ5ktZZ6kkeTvMIGpJ5kfS73x89DjGcm+PZ3J4mXi2zmllmYXyS3g7snaulJ+s5fLbQ3aTF/EG8SwnmZjUo9SaDlnuSRJK8w+tSTrCvk/TF5iNvr93hrvcDGdolYapDz4+PcuAbPv7zo7B2KniIl8/zInqST50jJypv0v/7y+I1MPUloE0+ymuQVNiL1JOsKe39MXuLmg0UOvgOYz5MvJXnzWpaJ9CILW+DqiUqeonH+ors3Kc1v403uF9jLbHDqSR5OHOFY/rl6hY1LPUlwvz+Sl7hfqVBgiz66kjEouHuiRk/R68/XzB9LunuTxvkDeJNhvMxGp54ktN6TrEnyCkE9Sdca6UnW5+8lxnp76aZEvlhx9g5r8/QUhfkr+ei8Sc/5Lb3JsF5mo1NPEmgHTxKQvULUkzTVMk9S8hBjfYxNn6G4mqt+T7CP8+ODJB7dZqG6bRvv0LQ/o6doMb+rN2n2HGVP09XLtL3/YVJPsk08SZC9QlBP0rdWepIlwUOMVaCjj/OTWZIdccq7WxQevsX1+cN/FkH0DoX9mT1FeX7XJM/R6E1G4WW63H8hVYBOWOpJalpzU+DiJKWepKY1Pf0kqWmaZkg/SWqaphnSN0lN0zRD6kk+EaknqWlhU0+ySYlenuDtuaWepFTDPUnH4zbPR2bsBheHM5xK7P2TtAtzt+y9UslrrL1WHp6k5FVKz7/oXQa4v34eptHrNKSeZJOSvDzJ29OE2t2TdDwuPR9d2ZtcPdfJ2vwVXi0kyU5OMX1tm+cvLVqt38ajBLM3afIqpedfOt/2/vqtT/Q6DaknWVMrPUnJ22tG6knuF70n6Xpcej4GsxlYm+GNlb3rvbCwSv/rWS6mF7mVF9Zv61FK3qTBq5Sur3S+1f01rE/0Og2pJ1mt1Z5kUG8v6tSTrK8RnqTLcePzERsgnYKN5UeHgxRyFMrP0ZvuhPyR31M/un5Lr1HyJiWvsu7PelzfIOd75bs+R69TPUmgHTzJQN5e5KknWVcDPEnX48bnozNJMl6muFshPfENbvTneeOlFUolSCWPK0LH1m/hQUrepORVStc30PkeGdfn6HWqJwlt4UkG8fYiTz3JuhrhSboeNz4fByOUKRd3KBZLlPy+APFZv9FrtPAkjV7lkVvrdX2DnH8si/W5eJ3qSR5OHOFY/nl6eQG8vcalniQ03pMMdTw3aH4+7hUpluN0dCTYWLzClxeB2ABnE1A64i36rd/kNcZ6zd7k8y8fhzLqvEwO9yddX+l8r8T1XXHzOtWThNZ7kpbe3n7qSbrVDp5koOPS81F5QH7zMiOZ07Bc/T5jb4be+BbrR74fKa3fy2usbJu9Sa/qvMxqNtfXdL5flTVhfZWSk9epniTQck9yR/b29lNP0r+29SRdj1s8H6vLOUYmxpgaKrJYSJKdGCRZWKz/ys3glUpeo9GTtPAqjdfX4nzp/krepY3X6Zd6km3iSRq9vZrUk/SpnT1JXI/Lz8f28hVe7brBxYmbvBnf+2Hy2dn6L2lNXqnoNRoTvErp+j6y8C4dvUjR6zSkCtAJSz1JTWtuClycpNST1LSmp58kNU3TDOknSU3TNEP6JqlpmmZIPcknIvUkNS1s6kk2KdGTDOD5BU89SSlXTxIMnuPRuY56h5bj23qIYT3F0OvHwoOU5g/gqXruTzrf4flQT7JJSZ6ereen+dRiT9LoOdbk6R1ajG/rIYb1FJ3WX83kQUrz23qqfvOL5zs8H+pJ1tQyT3LX0vNrcOpJ7hfck5Q8R8DoHUrjW3mIDp6i6/oBowcpzW/lqRrmF893eD7Uk6zWUk/S0vNrZOpJ1hfIk7T0HCWP0X98Ow8xtKcY0fp9PUiL9dt4qqb5G+mxqicJtN6TlD2/xqaeZF1BPcmE7DlKHqN5fNlDdPIUy+7rN3qQFuuXPFVp/kZ6rOpJQlt4kkbPr9GpJ1lXWE/S13O08g6l8Q0e4trn3DzFVff1Gz3ILWH+pR2zlxmX52+kx6qe5OHEEY7ln5+nZ/L8mpN6khDSkyyZPccgHqPf+CYP0dlTjHD9B0uu9SALwvyJM0Yvc21TmP96qaEeq3qS0HpPsnYpHp7f0dSTdCtyT1LwHCt5e4/Rb3yTh+jsKVY2Ilv/wR+p9SCF9Utepri/2FAgjzVo0f7GTa3nWFvVk0yNXGNqKEOqK0VPZoCzF68y0WexiVpPEmo8Scv5xfY8xe7sJGN9KbpSGc5O7nmKyxF7kou3q55eOk26p+fgeKLvLGPZfjKZDP1np3hl4gyVtbsHnt9+mal3ePvtm4z1EKrcZpnewXNkUp21/1Xcf092golsHz1dnXR2pRke7KWjuFnnSZqOS23k8+Q9/m9j0/5/DFI9adK9e95gLNVLOt1DqrP++Rq48V3efvsy2c76c6X7szdB1WO8f/xnE1eXc9A/xtRQmlRPPxMTgyQLywdfOW1ubtb83za7QLnGO5TGX1l+RGJwgosDaVKpNAOTex7iatVDlMaXzndaf6yPsasTnB3oJ5PO0Dc0xiuTgyQeLR94kMb5d5bIFSAzMkk2k6IrlWZgYs/LzFW/x22c3+J82+fDK/Uk28STtPX81JP0qZWeJHaeo5RpfBcP0eZ8t/XLHqQ0v62n6pd4vsPzoQrQCUs9SU1rbgpcnKTUk9S0pqefJDVN0wzpJ0lN0zRD+iapaZpmSD3JJyL1JDUtbOpJNqm+sRuMD6ZJdXcQLxd5nP8+d+cXeLB5uBBbzy946klKNdqTNHqLNuNL+wvgkYb1LI3Pp2l+afy1QbfjKyXRk7R5/fmlnmSzKm6yeneZwnaRSqyXgfExLt+AnS/dIlex9/w0n06AJ+nrLVqML+3P1iMN61lK+zPOL43vehwLT1J4/ZlST7KmRnqSD5cWOOQS8jyKZxi8dpp0N+Q2LT2/Bqee5H4N8iQN3qI0vnF/nZYeqYNnadzfljy/NL7rccmTlF5/ptSTrNZMTzLW2cPwUIaO4jqFLaw9v0amnmR9jfAkfb1FaXxxsXv/T/JIw3uWwv6KdvNb7y/E8SCe5LHXn5B6kkCzPMlY/zTvXBukAyhvrTJ3ZZ6HFaBT9vwam3qSdTXAkzR6i9L4UtuyR+rmWQr7s5g/yP7CHLfxJH1ff0LqSULTPMnK2ixXvvIu8eRphkZHmZjMsnKllpny8fwanXqSdUXuSSJ4i7VbN47vn9EjjcSzNO/P2kOV9hfyuI0nKb/+vFNP8nDiCMfyb2NjAzY2yBdi9H5rlOn+ZWbXzZ5fc1JPEhrjSXpV5y1y+I7jOb5FJo80Cs9S2p+thyrtL9TxxJC1J+n5+hO+X6aeJDTNkzxenEQM0SM8mnqSbjXbk/SqzluUxg+Ql0da2Xb3LG33J3mo0v5CHRc8Sv+qrz+haN8kaz3HXM2XKlVP8vWRa0wVF1jMFYl195AZGKR7bZYF6RsDtZ5kfqPGk6zU/4txfvOL7XmKw9lJxnLzLG8n6R/b8xRnovAkE/1MTGXYWs2xsVOERC8DI6Oc3s0xV/3G8upyjpGJMaaGiiwWkmQnBkkWjv/NdmbqHW4O7/LuV76MJIB57nSzzPDgOTIP75E7sBrl/fdkJxhmnfvrBYp087nBXjqKa3WepOm4VBRfbqd60iTie14gqV7S6V1KW5ts7tT8Df2N73L5TL7upy7g0JO8O1f1JAHKZfIbNRd533uc8/Ykfe9frI+x6TMUV3PV70n2cX58kMSj2/VfahvGl/aX6DvLSPc2DzdLJHqGGB09Q2Vt5sAj3axVwWMpdoGYn2cZdH8gzm+zv9DHd5bIFcY5NzJJdvsu68UE6bN7nuT9h5tWrz9T6kk2w5MsrbHFCNkLQ3QnO4iX97zFt64vHP6Ig6Xnp56kT23tScreojS+tD9bj9SUi5dpM79xf47HzZ6k/PozpQrQCUs9SU1rbgpcnKTUk9S0pqefJDVN0wzpJ0lN0zRD+iapaZpmSD3JJyL1JDUtbOpJNinRs3P0EM2pJyll9CSrdfWPMTk6TCaVhPIWm6sLXLq1Zn38YK6jnuP+fxc8UVtv1Gt8o2cZwfrF6yd5j8L6nL1PYX5T6kk2K8mzc/QQn/ga6UkCscxFbk4PUrx/h9mFAqVYF6nk4ZuwdHw/T88R2Wu09Ub9xgeDZxnB+qXrJ3qPwvpcvU+b+f1ST7KmVnqSrh5iFKknud9xT3J4dJj42gwv39q/i/m63/aSjgNGz1HyKK28SsP4gNGzdF2/dP0k71Fan6v3aTW/T+pJVmupJ9kGqSdZX50nGeunr6fMo/Vupr72TfpTCUrbOVZvz3Pn4Y58vJqv5yh5jYWMlVcpeZG+nqXr+qXrh533aO1teoxfl6M3eTT1JIGWe5ItTz3Juo56kp1dJDvi9J7Lsnb3Da48gtPnLzIxfZntS1dYLgvHtzF7jpLXaOFVSl6k0bNMOK5fun7I3mMQbzOw92kxvyn1JKGNPMkWpZ5kXd6eZJzS+m1mq3vamLtL3zuXGervZHlVOL72OUvPUfJEfY5beJFGz3IrqvXv5XX9JO/R2tv0Gf8gB2/SL/UkDyeOcCz/wnh2zUk9SfDxJEtFSmUo1ko6lQdsly6T6k6Kx0XP8YrgNQqeYxAv8mBLtZ5lwXH9NeN7Xr8A3qPn+mq8zVDeZ4j5a1NPElrvSQZMPUm3AnuSlQcUti4znOqG/b8eiPXTlYDSVhEqG8bjlTXBc6yUzF6j4DlW8vZe5MEfqfUshf2J65euXwjv0cvbDO19hvYm91JPEmgHTxLsPET1JP1rpCe5svyIc+MTXFyfZ2kDekbGOBPPcXttx+q45DlKXqN03Di+hWfpun7j9ZO8R4v1OXmf0vxC6km2iSdp6yGqJ+lTgz3JzaWXmUneYOLCTZ5Lwu7jHPdm5g4+sUjHpSSv0dYb9U72LJ3Wb3H9jN5jTFhfBN6n2Zs0pwrQCUs9SU1rbgpcnKTUk9S0pqefJDVN0wzpJ0lN0zRD+iapaZpmSD3JJyL1JDUtbOpJNinJk7TxDMOnnqSUqyfp6jWK3qjkIXb1MzE5xlDmFB3s8ji/wp25BTtv0sZjNJ1vsz5h/8bzpfWtDbqv35B6ks1K8CQlj08TarEnCY5eo/B8SB7iyLVpstxn7qXr5EkxOP4i09fKPH/pTt0aPT1IC4/ReD7y+qT9m88X1hfB+k2pJ1lTKz1JyctrRupJ7hfckwScvEbp+TB6iOUs/b1l1l9b4MFGBdhmceEeg1/PcjF9x8qblDxG6XzJa5T2L50vrc91/abUk6zWbp6k0ctrQOpJ1hfIk9w/x9Fr3M/r+TB6iNVvMVdqn8NymXK8m1O9CchXf3XQ1oP08Bil843rs9h/IO/RZ30u6zelniTQdp6k4OVFn3qSdQX1JLfdvUYwPx9GD7G0RH7rAkPZLKzt/Spf/2iWXmCzKwmUAnmQXh6jdL5xfRbXL4j36LU+1/WbUk8S2s6TNHp5jUg9yboCe5JLO25eY/UvD0zPh+Qh3p29TWp6nPfeuwDs8nh1hbWtblIlrLzJg7w8RovzjeuzuH7W3qOPF+m6flPqSR5OHOFY/kmepOTlNS71JCGkJ8nxj/tBvMba8z2fj9yg6CGW8otc/+Iiic4uSjvbEBvg6neyFAN6k14eo3j+9ZJ5ffeE/SfOWHuPnl5kTaHW7+Ft1qaeJLSNJ2nj5YF6kq5F7kl6FMRr9K/6fATwEEs7e+N3Dg6TYZPFXInKtr036eUxip5kbMi8Pmn/Afbn6UW6rl9IPUmgHTxJKy8P9SRNtcyTdPUapefDwkNM9J1lpHubh5slEj1DjI6eobI2w+I+5SZ4kHsX0MdjFM+X12fcv633aFif2/rNqSfZDp6kpZcH6kn61kpPUvIQkbxG2RuVPMQy3WRGRjnX3QHFx+RWXuPSgs+biU8mj1FKWp/kVdp4j9L6XNZvShWgE5Z6kprW3BS4OEmpJ6lpTU8/SWqaphnST5KapmmG9E1S0zTNkHqST0TqSWpa2NSTbFKSFyh6gk6pJynVDE/yYK6QHqPpfOP8Nl5kgOsXeH6L62fjYWbGbnBxOMOpxN4/qbswd+vwuHC+i9eqnmSzErxA8bhmrs09yf3CeozS+cb5bbxFy+sXan7k6yd5mF3Zm1w918na/BVeLSTJTk4xfW2b5y8tWp3v4rWqJ1lTKz1J6XgzUk9yv+g9ScDJY5TOl+YXvUWb6+cwv/H6dZ4VPczBbAbWZnhjZW/8hYVV+l/PcjG9yK0t+XwXr1U9yWrt5EnaeJNRp55kfZF6ktVCe4wW59vMf5DkMfoUen7p+lXfkX09zEIf6RRsLD86PF7IUSg/R2+6E4rC+fnj3zYI4rWqJwm0iydp7U1GnnqSdUXtSeLoMVqcL81fm+QxeuU0v+hpCh5mIkkyXqa4WyE98Q1u9Od546UVSiVIJZMgnX+UzgjotaonCW3jSdp6k5GnnmRdkXuSRUePMS6fb5y/9tJKHqNXFh6jq6dp9DAPKlMu7lAsligdeX+3O3+voF6repKHE0c4ln+SJykdb1zqSUJjPMlY0s1jXNsM7iHWeZYcPj+Sx+hVGI8xqKdp8jApFSmW43R0JNhYvMKXF4HYAGcTUCrufa1tPL+mMF6repLQNp6k7XH1JN1qtidZybt5jGE8xDrPsibJY/TKef4A18/Lw6TygPzmZUYyp2G5+n3U3gy98S3W8zvy+dVsvdajqScJtNyTtPAm91NP0r929iRdPUbj+Rbz710gs8doun6u8xuvH7KHubqcY2RijKmhIouFJNmJQZKFxYOvTKXzbb1Wr9STbAdP0sIT3E89SZ/a3JOUsvEU/bOb3+gtulw/i/klT1LyMLeXr/Bq1w0uTtzkzfjeD5PPzh5+yWw8P4DX6pUqQCcs9SQ1rbkpcHGSUk9S05qefpLUNE0zpJ8kNU3TDOmbpKZpmiH1JJ+I1JPUtLCpJ9mkgniRnt6gU+pJSrl6kuL5khcpeYqm/dl4kcL6bTzHg2vl83y6eI/O+zONb3l9/FJPsllZepEmr08z1GJPUjpf8iIlD9G4Pwsv0tVz3M/v+XT1Hl33ZxzfxtM0pJ5kTa30JAHZ62tw6knuF9yTlM43epHlrOghSvuTvEhXzxEwPp+u3qPT/izWL3qahtSTrNYOnqTk9TUy9STrC+NJ+p6P4EVWv0UcxEM0dtSLdPUcq/P7Pp+xgci9x0D7s1y/7/lC6kkC7eBJSl5fY1NPsq4QnqTxfAQvshTQQxQ65kW6eo6UzM9n1N5j0P0FHD+op6meJLTek7Tw+hqaepJ1hfEkpfONXuROMA/RmK8X6eA5Wj+f0XiPYfZnPX4IT1M9ycOJIxzLPy8vco7gXl/0qScJ7p6k5/mJIaMX+WBx09pDlPL0Ii3Wb5pf9CSvROc9htpfgPHDeJrqSULLPcnKg2Ben3qSbjXKk/Q9P5YwepG1mTxEmzy9SEfPsbItPJ+VUiTeY+j9BRg/jKepniTQck8SyRs8TD1J/1rmSUrn78hepOQhWu3P4EW6eo7S8+nqPbruz2Z8ydP0Sz3JtvAk7VNP0qeWepLy+ZIXKXmKNvszeZGunqOUk/cYwf5s1m/0NA2pAnTCUk9S05qbAhcnKfUkNa3p6SdJTdM0Q/pJUtM0zZC+SWqaphlST/KJSD1JTQubepJNSvIk+6be4ZXheveneP+rvPBGkJ/39Es9SSlXT1KcX/IkpfMD7M/Te3T1IoX1G6+fhefoej6YPUuX15d6ks3KwpMsP3qX6wtr1V8QLFMu2v/q1BNfiz1JaX7JkxTXb7k/P+/R1YuU1m+8fhaeo+v5kmcJ4V9f6knW1HJPsrRJroXQg3qS+wX3JKX5jZ7kdkVev83+/LzHCLxIaf3S9ZM8R9fzjZ7l/v5Cvr7Uk6zWDp5kPDPBt787SbxcZDO3zML8Irko/vUGi9STrM/Vkzya0ZOMKF/v0dWLDLH+o55mXRaeY6DzJc+y+vvjYV9f6kkC7eBJbq/f4631AhvbJWKpQc6Pj3PjGjz/8qJxzGhST7IuV0/SI6MnGUFG79HViwy6fg9PszbRcwx6vuRZsuP0+lJPElrvSQKbDxY5+A5JPk++lOTNa1km0ossNPorVvUk63L1JL2SPEmnLLxHVy8yyPq9r181C88x/Pn+nqXL60s9ycOJIxzLPy9Pctbj6/VSocAWfXQlY1QNuAanniS4e5KeWXiSLone48tLbl7k9ZL1+j2vX+2lEDzHUOeXZM/yaEFeX+pJQss9Sa9ivb10UyJfPH4D1ZN0q1GepG8BPMkwVdbsPdJQXmRsyGr9vtevJpPnGPr8ygNrz/Jg3YbX19HUkwRa7knG+hibPkNxNVf9nkkf58cHSTy6fexLAfUk/WulJ2meX/YkbdZvOi55j25epLx+6frtbcDfc3Q93+hZBnh9eaWeZDt4krEKdPRxfjJLsiNOeXeLwsO3uD5//J9tUE/Sp1Z6khbzS56kuH6X/eHuRRrXb3H9wOA5up6P5Fnav768UgXohKWepKY1NwUuTlLqSWpa09NPkpqmaYb0k6SmaZohfZPUNE0zpJ7kE5F6kpoWNvUkm5TkSYLgFTqlnqSU5EmKHqHkRVp4ji4eouhhSvML10+a33jcwoMUXx/C+mw80LCpJ9msBE9S9Ao1cw32JMHsEUreouQ5unqI0vpFT9Li+kkeo+9xCw9S9FaF9dncv7CpJ1lTKz1J0StsQupJ7nfcMwSMHqHRWyxnRc/R1UM0rn/XwpO0uX6Sx2g4LnmQorcqrM/q/oVMPclqLfUkI/AKXVNPsj4vz9DkERq9xeq3gH09x0Jf5B5i3fotPUnxmgjzW69P8CT9vNUgGT3KgKknCbTck4zAK3RLPcm6PDxDySM0eoslwXOM2kM8tn7Zk5SS5g+yPj9P0uStBkrwKIOmniS0hSfp6hU6pZ5kXV6eoeQRSt6i0XM8KBoP0Wv9dvP7J81vvT6DB2l+fdhn9ChDpJ7k4cQRjuWfpye57ugVRpJ6kiB7hvvVP2BL0QAAIABJREFUeYSJQdFbNHmOUXqIfus3zh8iyWP0Oy55krbeql+29y9I6klC6z3JgF6hepJuBfYkParzCAN4kV6eY1Qeos36PecPkeQx+h03eZLH8/dWvQpy/4KkniTQck8SO68Q1JM01TBPUvQIZW9R8hxdPUTJY5TmN16/UsY8v63X6OdBWrw+pPtr5VGGTD3JdvAksfAKq6kn6VMjPclHskcoeZGS5+jkIVp4jKInabp+d6X923mNvh6kxevDvL5uK48ybKoAnbDUk9S05qbAxUlKPUlNa3r6SVLTNM2QfpLUNE0zpG+SmqZphtSTfCJST1LTwqaeZJMyenkW3p5b6klKNdyTFI7beKMmb7JuLxe/yc3nunn01m/zcvVXWkVvUVq/o+dosz/f9Qd8fXjt3+X5UE+yWZm8PBtvTzPX5p6kdFzyFG28SYBE3xQXT5d4fOS3TKX9ietz9RwlL9K0/gCvD7/9uzwf6knW1EpPUvL2mpF6kvtF7EluV8Tj0vNh5U0m+pmc7GVt9l1SNy9T+7lX2p+0PlfPUfQihfVbvT4M57s8H+pJVmupJ3k0wdtrROpJ1hepJ2lxvLbj3uiAlTc5MDVBanWGuUI30wH3F2R9Ya+f7/6qWa/f5/URZP9BUk8SaLkneSQ/b69xqSdZV9SepMVxMHmjsjfZNXSDC92rvPDqBsSE/3Xy2J/N+lyun3F/EGj9Xq+PQPsPmHqS0CaeZDWDt9ew1JOsqxGepHQcbJ4PH2+yK8uL412szsyE3p/N+lyun3F/Qdbv9foIuP+gqSd5OHGEY/kneXmSt9e41JOEBnmSy2nRm9zPzxs1eZOx3j7SyVNkXv8e52oXeeFbfHfwLZ5/+fCN1nN/iSHr9bleP6/9zWG/fq/XR5D9h0k9SWi9J1mTjbennqRbTfckA3iT9R16oyZvspJ/lUtfStUM2sfk6xdILF9h9l5B3l/o9dUX3HPc21/lgd36wfv1UVmzPz9M6kkC7eBJAv7eXk3qSfrXtp7kzob5uMXzYfQmgc1a1T6WYheIbT1mc7sk729H9jCl62sc32J/0vr3FuD/+rA53+b58Eo9yTbxJMHg7dWknqRPbe5Jmo/Lz4fZmxSy8Cal9bt5jnbPv5TN68M3h+dDFaATlnqSmtbcFLg4SaknqWlNTz9JapqmGdJPkpqmaYb0TVLTNM2QepJPROpJalrY1JNsUqKnJ3l+TqknKeXkSVp6h139Y0yODpNJJaG8xebqApdurQVev6eXiOBNWoxvWp/kaUrPt7Nn2dXPxOQYQ5lTdLDL4/wKd+YW7M93SD3JZiV4eqLnp5lrpSdp4R3GMhe5OT1I8f4dZhcKlGJdpJI1b1KW6/fzEkVvUhhfXJ9p/yA+366e5ci1abLcZ+6l6+RJMTj+ItPXyjx/6Y7V+S6pJ1lTKz1J0fNrQupJ7hfck5S8w+HRYeJrM7x8a/8pyNf/tpjN+g1eouhNCuOL6xP2Lz3fTp5lOUt/b5n11xZ4sFEBtllcuMfg17NcTN/hVr6xrx/1JKu12pOM2vMLmnqS9QX1JOs66h3G+unrKfNovZupr32T/lSC0naO1dvz3Hlo/ysnvl6ipTfpm+X6bPcveqkE9Cyr30Kv1L4UymXK8W5O9SYgX2ro60c9SaAdPMlIPb/AqSdZVwhPsrZj3mGii2RHnN5zWdbuvsGVR3D6/EUmpi+zfekKyxZfjBi9xITsTZr3K6/PZv+2Xmpgz7K0RH7rAkPZLKzt/Spo/2iWXmCzKwmUGvr6UU8S2sKTjNLzC5x6knWF8SQP8vVA45TWbzNbvSYbc3fpe+cyQ/2dLC8JN9jaS/TxJq0yr89m/1ZeKuE8y7uzt0lNj/PeexeAXR6vrrC21U2qZHe+S+pJHk4c4Vj+eXqBOcEjDOD5hU89SQjpSSJ4oKUipTIUa6WaygO2S5dJdcuf9EQv8YrZmxQLsT6//UtealjPspRf5PoXF0l0dlHa2YbYAFe/k6VYLEbqYXqlniS03pMM6PmpJ+lW5J5kTZ4eaOUBha3LDKe6Yf+vd2L9dCWgtCW/iYleYqVk9CblCYKvz2//9dV7qVF4lqWdvfV1Dg6TYZPFXCkyD9Mv9SSBlnuSlp4fqCdpqnWe5P4C/L3DleVHnBuf4OL6PEsb0DMyxpl4jttrh4swrV/yEiVvUhrfuD5p/xZepKtnmeg7y0j3Ng83SyR6hhgdPUNlbYbFbRA9T8fUk2wTT1L0/KqpJ+lTiz1JMHuHm0svM5O8wcSFmzyXhN3HOe7NzB1+4nRZPxbepDC+cX0xYf/S8x2BZ1mmm8zIKOe6O6D4mNzKa1xaOPwfI9vXT5hUATphqSepac1NgYuTlHqSmtb09JOkpmmaIf0kqWmaZkjfJDVN0wypJ/lEpJ6kpoVNPckmJXqSkpfnlHqSUk6eZDVfj9HSmzxYi4cXKXqMpvOl+dcGndfn6lWKz79wXLo+Qa7f0dSTbFaCtyd5eZpQKz1JBI/Rwpvcz8+LtFmf7/nS/BGsz9WrlJ5/6bh0fWyvn1fqSdbUMk9y96zo5TUj9ST3C+5JSh6j5E0CRi/San2G86X5Xdfn5FV2Cs//lvz6kK6P1fXzST3Jai31JKtPm8nLa3TqSdYXyJMM6kUe9Sar+XqRluuzPt9n/ijXV5d0farvqL7Pf1E47vH68Lo+QY7Xpp4k0HJPclv28hqbepJ1BfUkA3qRx7xJBC/SYn1BzveaP9L1HVuvdH2E5z/o68Pj+gQ6fiT1JKEtPEnJy2to6knWFdiT3AJrL9LLm7T2In3WF+R8X+8yuvV5Z74+0vMf5PXhdf+CHD+aepKHE0c4ln9+3p7Ry2tK6klCSE+yYO8xenmTohf58iGk4bW+IOd7epc1ua7P+2LJ10d6/m1fH9L9s72/taknCa33JGuX4uXlHUk9Sbci9yQDeIxe3qToRQrrsz3fb/6o1udbgOsjPf+m49L9C3J/a1NPEmi5J4nk5R2mnqR/rfQkbbxIkzcpeZHG9VmcL83vuj5w8CqRn3/puHR9pOOm1JNsE09S8vL2U0/SpxZ7kqIXidmbdFqf5bd+pflDrw/cvErk5994XLx/btdPFaATlnqSmtbcFLg4SaknqWlNTz9JapqmGdJPkpqmaYb0TVLTNM2QepJPROpJalrY1JNsQZ4eH5AZu8HF4QynEnv/JOjC3C31JG1rsSdp402a7q/kjTp5lhbjk8gwMjXBuUwvyY4yxcc5Vu/MHypd0nFhf64epDh+7b30eX2FTT3JJufn8XVlb3L1XCdr81d4tZAkOznF9LVtnr9k/+tTT3Qt9iSl4+L9FbxRaXzJa5TG75+6ynhvjvnrM3y/mCAzOs2L09Nsfelllrbl49L+XD1I29eHr3fpkHqSNTXSkwSMHt9gNgNrM7yxsvcReGFhlf7Xs1xML6onaVOLPUnpuHR/jd5oBJ6lNH4m1cHW+hLL1V8bfnB3hfPDI5zujbG0XRGPG/cXgQdp9foweZcOqSdZrRmepK/HFxsgnYKN5UeH/62Qo1B+jt50J+Tdv2SQUk+yvkCepHQ84P095o2K4wfzLL3GX98oks0M0NeZ4+EO9AyeoauY493qx1jjcWl/rh6k5fUL7V0KqScJNMOTNHp8iSTJeJniboX0xDe40Z/njZdWKJUglaxXZBqTepJ1BfUkpeOW99fXGxXHt/MsTeM/nH2Bhavf5Nq33tu7D8Uct786e8AWGo93Cvtz9SCl8dlx8y6F1JOExnuS1h5fmXJxh2KxRKmZf7GknmRdgT3JvI03CdL9NXmjUXiWpvHTI19jtLfA7a/OkCt2cPrcRcZemWbrK6/yYEc4frAD//1F40H6jB+Jd+mfepKHE0c4Vn2ix3elSLEcp6MjwcbiFb68CMQGOJuAknqS7e9Jcvwd76g3aXt//bxRaXxbz9Jz/Nwgo6O9FOZfYKn6vaWN+QUy77zC+eEUD5bT5uP35P05eZAl8/jO3qWQepLQcE9S9PgqJfKblxnJnIblqmzSm6E3vsW6x/cj1ZN0K3JPUjpeeRDo/h523Bv1G9/Wa/QcP5YgEff+R0JiiZh8PMD+QnmQwviVvKN3KaSeJNBwTxLZ41tdzjEyMcbUUJHFQpLsxCDJwvG/2VZP0r+WeZIW3qTx/kreqKtnKY2/s0SuMM65kUmy23dZLyZInx2nv2OL+w83YWfDfFzaH+4epDS+lacZMvUkm+FJWrS9fIVXu25wceImb8b3flh2dvb4l3zqSfrUUk9S9iaN91fyRmOunqXsmd6ZmSU+OcbojTeZjJfZfZzn/muz3Kr+7bZ0XHp+nTzInP3roxGpAnTCUk9S05qbAhcnKfUkNa3p6SdJTdM0Q/pJUtM0zZC+SWqaphlST/KJSD1JTQubepItyNO7c/QQzaknKeXsSUpeIwYPMZHl5ncmOf47R2XW517g+krJan+St2jyJo3eo+v6bM6v3UdQb9VifBuv0i/1JJucr3fn6CE+8bXakxS8RqOHWFpm7iuFGpMA4plxboyWWV0vWe1P8hYlb9LoPbquz+b8aqG8VYvxbe6vX+pJ1tRKT9LVQ4wi9ST3C+5JSl6j5CFub+SpfZKGRnsor88f/LC3tD9pfKM32Sl7j67rE88HJ29VGt/q/vqknmS1lnqSbZB6kvWF8ST3O+Y1BvVCu86SzVRYm1mzW6w0fuG02ZusvqNZe49B13c0n/Mj81Yt1ud1f/1STxJouSfZ8tSTrCuEJwkGr9HCQ6yt51yW08VVXrZ1AySvUvQmg3mPgdd3JK/zo/RWxfV53F9T6klCG3mSLUo9ybrCeJJg9hr3svBCYxnODXZTuC8Tc8czjW/2Jq29R6f1+ZwfpbdqsT5/r9I79SQPJ45wrPoa7d1Fk3qS4O5JenqN6/aeZKL/LP2JAneXhW+U1S1GGL8ke5O23mOo9QnnR+mtSuuzvb+1qScJrfckA6aepFvN8CT3qnqNAbzFwefOEM/NV/Uey6TxA3iTJu8x9PqE86P0Vk3rC3J/a1NPEmgHTxLsPET1JP1rmScpeY1YeqGpEbLpMutz3n/hYNqfNL7Rm0T2Hl3XJ50fibdqGF/yKk2pJ9kmnqSth6iepE+t9CQlDxI7DzGdzdJbWuO210Mn7E8a3+xNCt5jBOsTzxdyun4WXqUpVYBOWOpJalpzU+DiJKWepKY1Pf0kqWmaZkg/SWqaphnSN0lN0zRD6kk+EaknqWlhU0+yBXl5eS7enZx6klKunqTN/TN5j6JX6Tp/IsPI1ATnMr0kO8oUH+dYvTO/p3BZeIw2XqbRq5TuT4D75/X6ka6PS+pJNjk/L8/Fu9NouScpnS95j9L4rvP3T11lvDfH/PUZvl9MkBmd5sXpaba+9DJL2xbeo+BlSl6leH8s75+vx2px/cKmnmRNrfQkXby7qFJPcr/gnqR0vuQhSuO7zp9JdbC1vsRy9deCH9xd4fzwCKd7YyxtV0SPUfIyjV4lyPfH5v6ZPFbh+riknmS1dvMkg3h3UaSeZH0unuSx8y09RGn80PMD6xtFspkB+jpzPNyBnsEzdBVzvJvzeHgFj/G4l9lv9iojSnr9BLk+QVJPEmg7TzKgd+eeepJ1hfQkfc+38CSl8Z3mBx7OvsDC1W9y7Vvv7V3nYo7bX531ZAn9PEZ/L1PyKsPdhtqk10+g6xMw9SSh7TzJoN6dc+pJ1hXWkzSdv5e/hyiN7zp/euRrjPYWuP3VGXLFDk6fu8jYK9NsfeVVHtT+D7HBYzR7mWav0imL10+Q6xM09SQPJ45wrPqCeJJhvLtoUk8S3D1Jz/Ml7zHA+KHmTwwxOtpLYf4FlqrfO9qYXyDzziucH07xYPHwm96Sx+jnZUpepUthPFbp+gVJPUloG0/S1rtTT9KtRnmSvucH8CRN44eeP5YgET/6jzDsH6p/FoJ5kYdepq1XGaYwHqud92mXepJAO3iStt6depL+tcyTlM5H8BCl8V3n31kiVxjn3Mgk2e27rBcTpM+O09+xxf2HNc+ln8do4WVKXqXN/TEdN75+LK6PS+pJtoMnGcC7U0/Sp1Z6khb3z+whCuNLxy3mvzMzS3xyjNEbbzIZL7P7OM/912a5VfO3274eo4WXKXmV4v1xuX/i9XNLFaATlnqSmtbcFLg4SaknqWlNTz9JapqmGdJPkpqmaYb0TVLTNM2QepJPROpJalrY1JNsQZ4enoXXFz71JKVcPUlw8yJt1m8av+7POXqLXucb1yd5lGuDoldp9C4t12/0LKX9GVJPssn5eniC16cJtdiTdPYihfXbjA/u3qLv+ab1lQSPUjqO5F3K6xc9S4vr45d6kjW10pOUvL5mpJ7kfsE9SWcvUli/1fiu3qLpfGF9kkcpHZe8S2n9omcp7c+QepLV2smTPOb1NSH1JOsL5ElG5EX6Zjm+q7cYxDs1JniUXsdtvEv/62/nWYbdn3qSQLt4kr5eX8NTT7KuoJ5kwt2LNK9HHt/VWwzknQr5eZSm45J3ab7+smfpsj/1JKFtPEmz19fA1JOsK7AneXDZwnuRdvmM7+otFoN5p8YMHqXpuORdytff4Fmufc5pf+pJHk4c4Vj1BfHwPL2+pvxVvHqSENKTLETvRdb/YfP4rt5iLBn8fL8kj9LzeADv0mv9FMyeZZjrU5t6ktA2nuTxql7fkdSTdCtyT7IBXmRdwviVvJu3GOZ8vySP0vN4AO/Sa/2SZxn+9beXepJAyz1JC69vP/Uk/WulJ+nkRVqs3zg+wvNlMb/knVpdXz+P8mAAn+OSd2mxfsmztNmfX+pJtoMnaeH17aeepE+t9CRx9SLl9ZvHl4rAW7S4vr4eZTXTcaN3GZPXL3qWDqkCdMJST1LTmpsCFycp9SQ1renpJ0lN0zRD+klS0zTNkL5JapqmGVJP8olIPUlNC5t6ki1I8uyCendy6klKSZ5k3Z/1uj9d/UxMjjGUOUUHuzzOr3BnbqHufJN3aPQSJa9xpSTu39mTRPAsJQ9S8FJdPc8g9y9o6kk2OcmzC+PdaTTFkwT/+zNybZos95l76Tp5UgyOv8j0tTLPX7oD2HmHvl6ihcdos38XT1LyLEUPUvBSXT1P2/sXJvUka2qlJ2l1vMGpJ7mfjyfpd386z9LfW2b9tQUebFSAbRYX7jH49SwX03e4lbf0Dg1eouQxWu3fwZOUPEvJg5S8VFfP0+r8kKknWa0dPMnIPL8QqSdZn5cn6Xt/qu9IldrnoFymHO/mVG8CChkr79Dam5S8Rr89hfUkLTxLGw/yYDgLLzWQ52l5ftjUkwTawZOM0vMLnnqSdXl4ksb7s71EfusCQ9ksrO39qlz/aJZeYLMraeUdBvEmJa/RKydP0sKzlDxICOClBvU8Lc53ST1JaL0naelNNiz1JOs65kla3J+7s7dJTY/z3nsXgF0er66wttVN6uBZNniHSzv23qTkNfoUjSfp72VKHiTYe6mBPc+8fL5L6kkeThzhWPVJnt0XFjOReX7hU08SvD1JG4+wlF/k+hcXSXR2UdrZhtgAV7+TpVgsQsnsHcLxjzt+3qTkNdoWyJO8IniZATxIyUsN5XkS/PwgqScJLfckK9tLgbw79STdCupJBvEISzt7D2jn4DAZNlnMlUTv0Cs/b1LyGm0L5ElWSmYvM4QHuVe9lxra8wxxfpDUkwRa7klaHN9PPUn/GuZJIt+fRN9ZRrq3ebhZItEzxOjoGSprMyxW38yM3qGlNyl5jb77L2WcPUmzZyl4kBZeqqvnKd0/l9STbAdPMkDqSfrUSE/S4lurZbrJjIxyrrsDio/JrbzGpYXDNzOjd2jhJYLgNZr2f9fdk5Q8S6MHieClunqeEdw/U6oAnbDUk9S05qbAxUlKPUlNa3r6SVLTNM2QfpLUNE0zpG+SmqZphtSTfCJST1LTwqaeZAvy8vqCeH/BU09SysmTtPEeJW/R4v4bPUfT/m3W5+JhWowvXV9nz1O4/y7epHqSTc7kRdp4f5pPrfQkLbxH0VvEfP8lz9G4f4v1OXmYFuNL19fV85Tuv4s3qZ5kTS33JCXvr8GpJ7lfQE8S2XuUvEXAeP8lz1Hav3F9EXiY0v6l6+t6/aX9u3iT6klWawdPMoiXF3XqSdYXyJM8mof3aOMt+t5/C88xUEfXF5GHadr/0STv0en6WxTEm1RPEmgHTzKQlxd56knWFdSTPJKX9yh5i8b7b+E5BunY+iLwMKX9S9dXOh6ptxrQm1RPElrvSRLMy4s89STrCuNJHuTjPUreovH+H9wWf8/ROp/1uXqY0vi1Sd6j0/W3KKg3qZ7k4cQRjlWfjUd4ND8vr3GpJwnhPcn9PL3HAN7ifnX3vyB4jgHy8yij8jAl71LyHl2vv1QYb1I9SWi5J+mVnycI6km61khP0tN7DOEt1t3/ygOz5xggyaN09TBN40veYxTX31RYb1I9SaDlnqStJ4h6kqZa6UnuLcDHe9wRvEWL+2/2HC33b/AonTxMaf8W1zeK62/av4s3qZ5kW3iS9t6fepI+tdiTBLP3aPQWLTxJyXO02b9pfU4eprR/0YuM4PobPc1up/FVATphqSepac1NgYuTlHqSmtb09JOkpmmaIf0kqWmaZkjfJDVN0wypJ/lEpJ6kpoVNPckW5OnhYfD6nFNPUsrZM0S4f8L6+sZuMD6YJtXdQbxc5HH++9ydX+DB5vEHNYxnafQqo/AmheM2Xqb0/Js8TWl8F69VPckm5+fhGb0+Ta6VniQW909aX3GT1bvLFLaLVGK9DIyPcfkG7HzpFjVQUGjPEgxeZQTepHTcOL/F9RM9TWF8m+N+qSdZUys9Scnra0bqSe4X3DMU75+wvodLCxx+psnzKJ5h8Npp0t2Qi8CzBIxepZM3uSV7lNL80vUTPU1hfKvjPqknWa2lnmSsP5jX14DUk6wvkGcY8f2LdfYwPJSho7hO7a/Ou3iWEMArDepNFoXj+ZJ5fun6WXqa0v7Ceq3qSQIt9yQDen3Rp55kXUE9w4juX6x/mneuDdIBlLdWmbsyzz4d4OpZBvFKA3uT0nFKgpcpXL+y7GlK+3PxWtWThLbwJK29vkaknmRd4TxD9/tXWZvlylfeJZ48zdDoKBOTWVauLEXiWVp7pSG9Sem47GUart/q/on+nqa0PxevVT3Jw4kjHKs+0cO7Yu/1NS71JCGkZxjh/dvY2ICNDfKFGL3fGmW6f5k5HD1Lj/y80lDepMVx4/wF4fqVgnuakscaxGtVTxJa70lWStZeH6gn6VrknmTA+2dfnEQMKg8cPUuP/LzSUN5kgOOe80teZWUjsKdp8lhtjtemniTQck8SS68P9SRNtdKTtLl/vuurnGFiKsPWao6NnSIkehkYGeX0bo656s//OHmWtl6pgzdpPG4xv3T9jJ6mNH4Ar9Ur9STbwpO08/pAPUnfWuxJivfPuL41thghe2GI7mQH8fLe9Xvr+kL9j/AI+XuRdl6pizdpPi7PL10/s6cpjW/vtXqlCtAJSz1JTWtuClycpNST1LSmp58kNU3TDOknSU3TNEP6JqlpmmZIPcknIvUkNS1s6km2oDAeoFvqSUpF4UlKx00eIgieouQ5SuM7epDS/myun2l/rudH4Vn6pZ5kk3PxADVDLfYkpeOShyh5ipLXKI0fhQdp2p90/aT9uZ7v6lmaUk+yplZ6klYeYINTT3K/4J6kdFzyEI2eouQ55oXxJe/R1oM07E+6fpIX6XS+xfUB1JM80Z7k0Xw8wEamnmR9gTxJ6bjkIRZOmz1FyXMs9JnHl7xHSw/S+vk9ev1CeJuBzpeuj+RZCqknCbTckzySlwfY2NSTrCuoJykdTwgeouhRCl6jNH4EHmSQ5/fY9esM6G0GPt/RsxRSTxLaxJOs5uP5NTT1JOsK7Ela319/D1HyKCWvURrfyYMM8vzicf0s9ud6vpNnqZ6k9QIjHKs+0SMM4QFGn3qSEM6T/MJiRvQmjR5iSfYojV6jhbfo4kEGeX69rp/N/lzPd/Is1ZO0qNWeZE02HqB6km5F7UlWtpdEb9LoIUqeYk2eXmPlgbW3GMaDrGzbPb9+1892f67n2+zvYAvqSZ48TxIwen77qSfpXys9Sem40UNE9hQlz1Ea38mDtNifdP2k/bme7+pZmlJPsk08STB7fvupJ+lTiz1JKbOHKHuKkucoje/mQQpZXD/j/lzPF9evnuQTlXqSmtbcFLg4SaknqWlNTz9JapqmGdJPkpqmaYb0TVLTNM2QepJPROpJalrY1JNsQZ7eYCLDyNQE5zK9JDvKFB/nWL0zX6ckhU89SSlXT1I8X1ifdL7oIQrjS+f3jd1gfDBNqruDeLnI4/z3uTu/wIPNit38wvVxXR+YPUkXL1JKPckm5+fx9U9dZbw3x/z1Gb5fTJAZnebF6Wm2vvSy8bdvtGot9iTF84X12cxv9BAt9m88v7jJ6t1lCttFKrFeBsbHuHwDdr50i1zF4nzh+riuT/IkbdcXJvUka2qlJ5lJdbC1vsRy9dc2H9xd4fzwCKd7YyzZ/MB9BKknuV9wT1I8X1if1fwmD9Fm/4bzHy4tcPiZK8+jeIbBa6dJd0POZn4we5uO65M8Sqv1hUw9yWqt9iTXN4pkMwP0deZ4uAM9g2foKuZ4N9ecN0j1JOsL40lK57vOH9ZDDHp+rLOH4aEMHcV1an91Xzo/yPUJtD5Lj9L1+vilniTQDp7kw9kXWLj6Ta596729/5Ut5rj91VlPNi761JOsK4QnKZ3vOr+Lh2h7fqx/mneuDdIBlLdWmbsyzz5dIJ0f6PoEXZ/obbpfH1PqSUJbeJLpka8x2lvg9ldnyBU7OH3uImOvTLP1lVd50Oh/wkE9yboCe5LS+a7z4+Yh2p5fWZvlylfeJZ48zdDoKBOTWVauLMnnF4Ndn8Dr2wLJk3S9PqbUkzycOMKx6hM9vuslRkd7Kcy/wFL1a/uN+QW5rvwqAAAJ7UlEQVQy77zC+eEUDxabYUuqJwnhPEnRU3Sc36sgHmKQ8zc2NmBjg3whRu+3RpnuX2bW4/tJtefHkvbXJ9T6CvYepbS/MKknCa33JGNDJOLg9ZV1LHH8+6fqSboVtScpnW9bkPODeIjhz4/j8fgdO7+St/dSQ60vgCcZbH92qScJtN6TXCJXGOfcyCTZ7busFxOkz47T37HF/Yf1nyLVk/SvlZ6kdL60PuP5lh6i7/iljPn8RD8TUxm2VnNs7BQh0cvAyCind3PM5SpW89t4qaHXh+BJOnqRUupJtokneWdmlvjkGKM33mQyXmb3cZ77r81y68jfbqsn6VMrPUmb803ru9ttPv+RhYdoHF84v7TGFiNkLwzRnewgXt67f29dX9j7J41jbh6j8/qQPMoI1mdIFaATlnqSmtbcFLg4SaknqWlNTz9JapqmGdJPkpqmaYb0TVLTNM2QepJPROpJalrY1JNsQZ7eXlc/E5NjDGVO0cEuj/Mr3Jlb8PQMg6eepJSrJwmQGbvBxeEMpxJ7/6Trwtwtaw/S2WvE7C3aPF/G9QvepDi/xf0xzS96mxbrC5t6kk3Oz9sbuTZNlvvMvXSdPCkGx19k+lqZ5y/dac1CT1ot9iS7sje5eq6TtfkrvFpIkp2cYvraNs9fqvm3rwXv0MVrlLxF6fkS1y94k6L3KNwfaX7x/lh4mGFTT7KmlnmSnWfp7y2z/toCDzYqwDaLC/cY/HqWi+k7TftUrp7kfsE9ycFsBtZmeGNlb78LC6v0v57lYnrx8P5J3qGD12j0Fi2eL2n9kjcpeo/C/ZHml+6PlYcZMvUkq7XUk6w+MZXaccplyvFuTvUmIN/4X/tRT7K+QJ5kbIB0CjaWHx3+t0KOQvk5etOdkN/7lCB5h6G9RslblJ6vQp/V+g+mO+pNWnqPvllev9pMXqefhxk29SSBlnuS20vkty4wlM3C2t6vUvWPZukFNruSeNMXUaaeZF1BPclEkmS8THG3QnriG9zoz/PGSyuUSpBK7ik1knfo5DWK3qLwfFmsHwzeZKfsPZqvt938pvtjXJ9j6klCW3iSd2dvk5oe5733LgC7PF5dYW2rm1QzfndcPcm6wnuSZcrFHYrFEqUjL07JO3T3Gs3eot3z5b9+MHuT0vx2meffz8/rNK8vfOpJHk4c4Vj12XiEpfwi17+4SKKzi9LONsQGuPqdLMViSMkicOpJQkhP8kqRYjlOR0eCjcUrfHkRiA1wNgEln/sneYeBvMYrsrdofL5K9uv39CbXg3uP9Zu1n1/yNm09zCCpJwmt9yRrl7Kzt8HOwWEybLLoAQuqJ+lW5J5kpUR+8zIjmdOwXP2Rl94MvfEt1j2+nwaydxjIa6yUrL1Fz+er8iDw+veqepMhvMe6LOcP7nX6e5hBUk8SaL0nCYm+s4x0b/Nws0SiZ4jR0TNU1mZYPPK/KepJ+tdKT3J1OcfIxBhTQ0UWC0myE4MkC9W/2Za8wwi8RqO3iPx8GdcveZMW80v3xzi/dH8s1ueSepJt4kmW6SYzMsq57g4oPia38hqXFtaO/Tn1JH1qpScJbC9f4dWuG1ycuMmb8b0fhp6d3f+SUPIO3T1Eo7eI/HwZ1y95kxbzS/fHOL94f+T1uaQK0AlLPUlNa24KXJyk1JPUtKannyQ1TdMM6SdJTdM0Q/omebTTaZ75p5/h6d6Pt3ol7ZleH+0JK9SbZOfQDd752oj5D6XG+MZ3v85ICmw9Q6txG9xTvzrAxy/9KrFf+kRL19Gu6fXRml+azi/+gJ+dXuZTXVH8il2wgv+cZCzD6GgvuYXZACfZeYY7K9f5/ug7XB1Y5lWv303UjD31m/+U+D/7OZ7+1DNQLvHR//aveP9f/EWrl9X2fey/+i0+8eVPA9uUv/gOlU14Zvb3+PjPPwN/tU5p7F9D988Q+8oAz/x8F0/xPj/Z+jE/+bP/gw/etLi+yc/w7Ld/lac9Dn349f+RD/40XXP8Q/7+x3/LRz/4t3xw8/BnfZ9+6Xlin+niqU8l+BglPvp3f07lzT/io8JHe38g8Q955tIwsc9+mqc6PuQnj3/ER+/8K8p/3KDf2PoPfodP/+6X4Y/P88NHv8inJy/zUwB8wEe7P+SDjXv8aPntxsztV88b/Mz5X+Hp//s1/uLb/0tkwz71C7/wC6TTaZ599lmrExKDIwzG11leb8yb2NLqLmfODTVkbJt+8i9+n9I/eZ33327WrwNG01Mv/XOe/eIp+LMHvP/f/wHvf+9HfOzzv86zv5mUTw7QSb0+dnXx9Gc/AYk0T/+j+s8PsVd/g/h/0cXHdrf56C/+FpJdPPPPft5y3A/4yeMf1/zf37D3t6Ufwu6HNX/uQ37y4w/gU5/imV/6ZRJ3/kl18jTP/NIpPlb5MR/92Z/z4dbTPP3zP8+zr/7awZlPX/oNPv5Ln+Zjxcd8+G9/DN2niP13v84zNb+oE2Uf7z/PT73/J/xw/S/r9vnh3/w/0NHDJzJT/Ox/+1pjJm9yz/zwhz/kp3/6p/nMZz7DD37wA95//33jCf2DGcjNHIcsE2nOTk0xeuYU8dIjVu+HQ9w2Vx+ydW6Yka6lY79t4tfHfvO3+MQXn+HDP32Gpz/7cT76w3X+/rO/wDPdf8eHX/sO5T/+Oz72j3+J+Bd+jqe6EzzVAX//V9t8+L1/TfkPfrQ3xu/8cz7xhU8djPnRd96ueyOI/f5LxE/9DR9+7zEf+8enebrjAz783nt88PYmkCL+L79A7FM/5IP/5l/y4RY888ZLfPwflahcfZPynwHJFM9c+mVin+3iqXj108If/xEfvPmX1Qn+4d7x//I/4qkO+Mnjx3z0zh8dfhKIpfn4e7/OM2xQ/nqJp3/7NE99CviLP+Xv3vwkH//8J/joG+/wwR/87d6f//5f8EHyd3n28z/Hx/i5J/v6XPw/rZ6jn/y4xFOf7eFjmymeosRPdhN734/q7uPpU8DuI/7ut/7w8Ln7nOU7UDFP+XdrfqOou49nf/+Xebr853zwxx9xAEPu/jn/72/9ISR/hvgbv0Hsp0/z7G9+n/f/1zzl3/tL/r7wQXWvKf7+f/4CsU+dIv4ZKP8AnvrZBJT/nL/73XsAPPXSl/ipzyd46qeBIy/F8L/WWi1xnuR//h/zwZ9N1//39/+E/+t/moLEr/Ef/vYsn/zkr/DpMz+z90b6dIZ/8PnLdP5naeLPwod//QN2/2SWv35UvS6JX+aTn5/ikz09xJ6pfhp99AY/+t//6Pj8T/8in/rtN0h2wwf/ZprNf/cr/Ke/de7wy+L/5DI/O30Z+IDS0i/yV4/K8HSGjqHLJE+niT8Dle0/YWflf+Bvf/TX8PSv0fV7s/yDZ35AceXf84lf/EVi/Hve/zez/Gj1j/j/AAT/iT3BzQ2OAAAAAElFTkSuQmCC"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![image.png](attachment:image.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Note that when the weights are randomly initialized, we expect each token in 0-50256 range to be equally likely.  Thus we expect loss to be around -ln(1/50257) = 10.82. Currently, the loss starts around this value, which is a good sanity check.\n",
    "- Also, we can make sure that the training is set up correctly by overfitting on a single batch.  Running the training loop for 500 iterations on the same batch yields: iteration 499, loss = 0.0008159472490660846"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Loader Lite"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- To build a simple data loader, advance by batches of size B*T, set the input *x* and output *y* as before, and loop around to the start of our tokens if run out of them: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DataLoaderLite:\n",
    "    def __init__(self, B, T):\n",
    "        self.B = B\n",
    "        self.T = T\n",
    "        # load the data from disk into memory\n",
    "        with open(\"input.txt\", \"r\") as f:\n",
    "            text = f.read()\n",
    "        enc = tiktoken.get_encoding('gpt2')\n",
    "        self.tokens = torch.tensor(enc.encode(text))\n",
    "        print(f\"total tokens: {len(self.tokens)}\")\n",
    "        print(f\"1 epoch = {len(self.tokens)//(B*T)} batches\")\n",
    "    \n",
    "        self.current_position = 0\n",
    "    \n",
    "    def next_batch(self):\n",
    "        B, T = self.B, self.T\n",
    "        buf = self.tokens[self.current_position:self.current_position+B*T+1]\n",
    "        x = (buf[:-1]).view(B,T)\n",
    "        y = (buf[1:]).view(B,T)\n",
    "        self.current_position += B*T\n",
    "        # if run out of tokens, loop around to zero\n",
    "        if self.current_position + B*T >= len(self.tokens):\n",
    "            self.current_position = 0\n",
    "        return x, y\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Using this data loader, loading and training can now be done concisely as below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_loader = DataLoaderLite(B=4,T=32)\n",
    "model = GPT(GPTConfig())\n",
    "model.to(device)\n",
    "# create an optimizer object\n",
    "optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4)\n",
    "for i in range(50):\n",
    "    x,y = train_loader.next_batch()\n",
    "    x,y = x.to(device), y.to(device)\n",
    "    optimizer.zero_grad()\n",
    "    logits, loss = model(x,y)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    print(f'iteration {i}, loss = {loss.item()}')\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Note that even with different batches of text loaded at every step, the loss still goes down quickly because many of the tokens will not occur in out data set and GPT2 can, for example, drive down the biases (mask) of all the logits that don't occur in our data to negative infinity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parameter sharing via weight tying"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Observe that transformer.wte.weight and lm_head.weight are of the same shape (50257*768).\n",
    "- In addition, we can below that these point to the same tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "(sd_hf['lm_head.weight']==sd_hf['transformer.wte.weight']).all() # tensor(True)\n",
    "sd_hf['lm_head.weight'].data_ptr()==sd_hf['transformer.wte.weight'].data_ptr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- This weight tying scheme comes from p.5 of *Attention is All You Need* paper, which in turn used the work of Press and Wolf (2017).  The idea is that if two tokens are very similar semantically, they should be close in the token embedding space.  For the same reason, we expect these tokens to have similar probabilities in the output of a transformer.  Press and Wolf (2017) argue that tying these weights leads to better performance. The weight scheme can be implemented in code as"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "self.transformer.wte.weight = self.lm_head.weight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Note that this weight sharing scheme also entails a significant memory savings of 38,597,376 parameters (768*50257) for the 124M model, which amounts to about 30% of the weights."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model initialization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- OpenAI initialized the weights with mean 0 and standard deviation with 0.02 and the bias with 0.  They initialized the token embeddings with mean 0 and standard deviation with 0.02 and position embeddings with 0.01. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Following Xavier initialization, the standard deviation should be 1/sqrt(in_features_for_layer), and with 768 to 1600 features used by GPT-2 models of different sizes, 0.02 is approximately correct.  Ideally, though, one would want the standard deviation to scale down more precisely with the model size."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- In code, this leads to"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GPT(nn.Module):\n",
    "    ...\n",
    "    def _init_weights(self, module):\n",
    "        if isinstance(module, nn.Linear):\n",
    "            torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)\n",
    "            if module.bias is not None:\n",
    "                torch.nn.init.zeros_(module.bias)\n",
    "        elif isinstance(module, nn.Embedding):\n",
    "            torch.nn.init.normal_(module.weight, mean=0.0, std = 0.02)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- In addition, observe that our *Block's* forward method contains a residual stream:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Block(nn.Module):\n",
    "    ...\n",
    "    def forward(self, x):\n",
    "        x = x + self.attn(self.ln_1(x))\n",
    "        x = x + self.mlp(self.ln_2(x))\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Not accounting for this in initialization would lead to exploding variance as seen from the following simple example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(10.0776)\n"
     ]
    }
   ],
   "source": [
    "x = torch.zeros(768)\n",
    "n = 100\n",
    "for _ in range(n):\n",
    "    x += torch.randn(768)\n",
    "print(x.std())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- However, scaling it by 1/sqrt(N) approximately yields the original standard deviation of 1 as seen below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.9894)\n"
     ]
    }
   ],
   "source": [
    "x = torch.zeros(768)\n",
    "n = 100\n",
    "for _ in range(n):\n",
    "    x += n**-.5*torch.randn(768)\n",
    "print(x.std())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- To implement it in our transformer, add flags in the attention and mlp modules since these have residual connections:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CausalSelfAttention(nn.Module):\n",
    "    def __init__(self, config) -> None:\n",
    "        ...\n",
    "        self.c_proj.NANOGPT_SCALE_INIT = 1\n",
    "        ...\n",
    "class MLP(nn.Module):\n",
    "    def __init__(self,config):\n",
    "        ...\n",
    "        self.c_proj.NANOGPT_SCALE_INIT = 1\n",
    "     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Then in the GPT module, check if a module has this attribute and adjust the standard deviation accordingly.  Note that *2* multiplies self.config.n_layers because the residual connection is used twice: once for attention and once for the mlp module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GPT(nn.Module):\n",
    "    ...\n",
    "\n",
    "    def _init_weights(self, module):\n",
    "        if isinstance(module, nn.Linear):\n",
    "            std = 0.02\n",
    "            if hasattr(module, \"NANOGPT_SCALE_INIT\"):\n",
    "                # number of residual layers is double self.config.n_layers\n",
    "                # one for attention, one for mlp\n",
    "                std *= (2*self.config.n_layers)**-0.5\n",
    "            torch.nn.init.normal_(module.weight, mean=0.0, std=std) \n",
    "            if module.bias is not None:\n",
    "                torch.nn.init.zeros_(module.bias)\n",
    "        elif isinstance(module, nn.Embedding):\n",
    "            torch.nn.init.normal_(module.weight, mean=0.0, std = 0.02)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Press, O., & Wolf, L. (2017). Using the output embedding to improve language models. arXiv preprint arXiv:1608.05859v3.\n",
    "\n",
    "Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need.  arXiv preprint arXiv:1706.03762 . \n",
    "\n",
    "Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., & Liu, T. (2020). On Layer Normalization in the Transformer Architecture. arXiv preprint arXiv:2002.04745."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mintonano",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}