Tabular Notebooks/02_Regression_and_Permutation_Importance.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 02 - Tabular Regression and Permutation Importance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Rossmann"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rossmann's Kaggle competition was a business sales prediction competition for $35,000 dollars to the winners.\n",
    "\n",
    "The premise is that we're given the sales data and information of their stores for the past few years. We need to make a model that can predict the number of sales that will be made in the future.\n",
    "\n",
    "We can do this through a tabular regression model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Jeremy walks through feature engineering for this problem, for today though we will download a clean engineered dataset straight from Kaggle. To download it:\n",
    "1. Go to: https://www.kaggle.com/init27/fastai-v3-rossman-data-clean\n",
    "2. Go to `output`\n",
    "3. Right click the download button\n",
    "4. Click \"copy link location\" for both train and test\n",
    "5. `!wget {url}`\n",
    "\n",
    "(Note: to walk through the feature engineering, see [this](https://github.com/fastai/fastai2/blob/master/nbs/course/rossman_data_clean.ipynb) notebook)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = 'ENTER_URL_HERE'\n",
    "test = 'ENTER_URL_HERE'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget {train} -q\n",
    "!wget {test} -q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now that we have our data, let's install the `fastai2` library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pip install fastai2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Time-Series Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai2.tabular.all import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = pd.read_pickle('train_clean')\n",
    "test_df = pd.read_pickle('test_clean')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>index</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Store</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DayOfWeek</th>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <td>2015-07-31 00:00:00</td>\n",
       "      <td>2015-07-31 00:00:00</td>\n",
       "      <td>2015-07-31 00:00:00</td>\n",
       "      <td>2015-07-31 00:00:00</td>\n",
       "      <td>2015-07-31 00:00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sales</th>\n",
       "      <td>5263</td>\n",
       "      <td>6064</td>\n",
       "      <td>8314</td>\n",
       "      <td>13995</td>\n",
       "      <td>4822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>StateHoliday_bw</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Promo_bw</th>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SchoolHoliday_fw</th>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>StateHoliday_fw</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Promo_fw</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>93 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    0  ...                    4\n",
       "index                               0  ...                    4\n",
       "Store                               1  ...                    5\n",
       "DayOfWeek                           5  ...                    5\n",
       "Date              2015-07-31 00:00:00  ...  2015-07-31 00:00:00\n",
       "Sales                            5263  ...                 4822\n",
       "...                               ...  ...                  ...\n",
       "StateHoliday_bw                     0  ...                    0\n",
       "Promo_bw                            5  ...                    5\n",
       "SchoolHoliday_fw                    7  ...                    1\n",
       "StateHoliday_fw                     0  ...                    0\n",
       "Promo_fw                            5  ...                    1\n",
       "\n",
       "[93 rows x 5 columns]"
      ]
     },
     "execution_count": null,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head().T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With our time-series based approach, the feature engineering made a bunch of date-related categorical columns that we can utilize in our **embeddings**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_vars = ['Store', 'DayOfWeek', 'Year', 'Month', 'Day', 'StateHoliday', 'CompetitionMonthsOpen',\n",
    "    'Promo2Weeks', 'StoreType', 'Assortment', 'PromoInterval', 'CompetitionOpenSinceYear', 'Promo2SinceYear',\n",
    "    'State', 'Week', 'Events', 'Promo_fw', 'Promo_bw', 'StateHoliday_fw', 'StateHoliday_bw',\n",
    "    'SchoolHoliday_fw', 'SchoolHoliday_bw', 'Promo', 'SchoolHoliday']\n",
    "\n",
    "cont_vars = ['CompetitionDistance', 'Max_TemperatureC', 'Mean_TemperatureC', 'Min_TemperatureC',\n",
    "   'Max_Humidity', 'Mean_Humidity', 'Min_Humidity', 'Max_Wind_SpeedKm_h', \n",
    "   'Mean_Wind_SpeedKm_h', 'CloudCover', 'trend', 'trend_DE',\n",
    "   'AfterStateHoliday', 'BeforeStateHoliday']\n",
    "   \n",
    "dep_var = 'Sales'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When doing Regression with these large numbers, we often use the **log** of these values for our `y`'s. Let's transform them real quick:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df[dep_var] = np.log(train_df[dep_var])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's declare our `procs`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "procs = [FillMissing, Normalize, Categorify]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And splits. Now we want to ensure that (since this is time series) our validation and test "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(844338, 41088)"
      ]
     },
     "execution_count": null,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_df), len(test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Timestamp('2015-08-01 00:00:00'), Timestamp('2015-09-17 00:00:00'))"
      ]
     },
     "execution_count": null,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df['Date'].min(), test_df['Date'].max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's find that particular index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "idx = train_df['Date'][(train_df['Date'] == train_df['Date'][len(test_df)])].index.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "41395"
      ]
     },
     "execution_count": null,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now our `splits` will be every index *after* the 41395'th item"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "splits = (L(range(idx, len(train_df))),L(range(idx)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((#802943) [41395,41396,41397,41398,41399,41400,41401,41402,41403,41404...],\n",
       " (#41395) [0,1,2,3,4,5,6,7,8,9...])"
      ]
     },
     "execution_count": null,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "splits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make our `TabularPandas`! Since we have a large `DataFrame`, we can enable `inplace` to `True` and `reduce_memory` to `True` to save on some storage space (**Note**: `reduce_memory` is by default `True`). To use `inplace`, we need to set `chained_assignment` to `None` in `pandas`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.options.mode.chained_assignment=None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "to = TabularPandas(train_df, procs, cat_vars, cont_vars, dep_var, y_block=RegressionBlock(),\n",
    "                   splits=splits, inplace=True, reduce_memory=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now let's build our `dataloaders`!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dls = to.dataloaders(bs=512)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Store</th>\n",
       "      <th>DayOfWeek</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>Day</th>\n",
       "      <th>StateHoliday</th>\n",
       "      <th>CompetitionMonthsOpen</th>\n",
       "      <th>Promo2Weeks</th>\n",
       "      <th>StoreType</th>\n",
       "      <th>Assortment</th>\n",
       "      <th>PromoInterval</th>\n",
       "      <th>CompetitionOpenSinceYear</th>\n",
       "      <th>Promo2SinceYear</th>\n",
       "      <th>State</th>\n",
       "      <th>Week</th>\n",
       "      <th>Events</th>\n",
       "      <th>Promo_fw</th>\n",
       "      <th>Promo_bw</th>\n",
       "      <th>StateHoliday_fw</th>\n",
       "      <th>StateHoliday_bw</th>\n",
       "      <th>SchoolHoliday_fw</th>\n",
       "      <th>SchoolHoliday_bw</th>\n",
       "      <th>Promo</th>\n",
       "      <th>SchoolHoliday</th>\n",
       "      <th>CompetitionDistance_na</th>\n",
       "      <th>CloudCover_na</th>\n",
       "      <th>CompetitionDistance</th>\n",
       "      <th>Max_TemperatureC</th>\n",
       "      <th>Mean_TemperatureC</th>\n",
       "      <th>Min_TemperatureC</th>\n",
       "      <th>Max_Humidity</th>\n",
       "      <th>Mean_Humidity</th>\n",
       "      <th>Min_Humidity</th>\n",
       "      <th>Max_Wind_SpeedKm_h</th>\n",
       "      <th>Mean_Wind_SpeedKm_h</th>\n",
       "      <th>CloudCover</th>\n",
       "      <th>trend</th>\n",
       "      <th>trend_DE</th>\n",
       "      <th>AfterStateHoliday</th>\n",
       "      <th>BeforeStateHoliday</th>\n",
       "      <th>Sales</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>407</td>\n",
       "      <td>3</td>\n",
       "      <td>2015</td>\n",
       "      <td>5</td>\n",
       "      <td>13</td>\n",
       "      <td>False</td>\n",
       "      <td>24</td>\n",
       "      <td>25</td>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "      <td>Feb,May,Aug,Nov</td>\n",
       "      <td>2003</td>\n",
       "      <td>2011</td>\n",
       "      <td>BY</td>\n",
       "      <td>20</td>\n",
       "      <td>Rain-Thunderstorm</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>5889.999991</td>\n",
       "      <td>21.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>15.000000</td>\n",
       "      <td>88.0</td>\n",
       "      <td>68.000000</td>\n",
       "      <td>56.000000</td>\n",
       "      <td>19.000000</td>\n",
       "      <td>10.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>77.0</td>\n",
       "      <td>76.0</td>\n",
       "      <td>12.000000</td>\n",
       "      <td>-0.999998</td>\n",
       "      <td>8.433594</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>377</td>\n",
       "      <td>6</td>\n",
       "      <td>2013</td>\n",
       "      <td>11</td>\n",
       "      <td>30</td>\n",
       "      <td>False</td>\n",
       "      <td>24</td>\n",
       "      <td>25</td>\n",
       "      <td>a</td>\n",
       "      <td>c</td>\n",
       "      <td>Feb,May,Aug,Nov</td>\n",
       "      <td>2010</td>\n",
       "      <td>2010</td>\n",
       "      <td>SH</td>\n",
       "      <td>48</td>\n",
       "      <td>Rain</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>100.000077</td>\n",
       "      <td>6.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>98.0</td>\n",
       "      <td>87.000000</td>\n",
       "      <td>74.000001</td>\n",
       "      <td>21.000000</td>\n",
       "      <td>13.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>77.0</td>\n",
       "      <td>76.0</td>\n",
       "      <td>58.000000</td>\n",
       "      <td>-25.000000</td>\n",
       "      <td>9.230339</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>812</td>\n",
       "      <td>3</td>\n",
       "      <td>2015</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>False</td>\n",
       "      <td>24</td>\n",
       "      <td>0</td>\n",
       "      <td>d</td>\n",
       "      <td>a</td>\n",
       "      <td>#na#</td>\n",
       "      <td>2012</td>\n",
       "      <td>1900</td>\n",
       "      <td>SN</td>\n",
       "      <td>6</td>\n",
       "      <td>Snow</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2589.999952</td>\n",
       "      <td>-2.0</td>\n",
       "      <td>-3.0</td>\n",
       "      <td>-4.000001</td>\n",
       "      <td>94.0</td>\n",
       "      <td>91.000000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>14.000000</td>\n",
       "      <td>10.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>62.0</td>\n",
       "      <td>66.0</td>\n",
       "      <td>34.000000</td>\n",
       "      <td>-58.000000</td>\n",
       "      <td>8.906936</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>464</td>\n",
       "      <td>4</td>\n",
       "      <td>2014</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>False</td>\n",
       "      <td>24</td>\n",
       "      <td>0</td>\n",
       "      <td>c</td>\n",
       "      <td>a</td>\n",
       "      <td>#na#</td>\n",
       "      <td>2009</td>\n",
       "      <td>1900</td>\n",
       "      <td>BE</td>\n",
       "      <td>14</td>\n",
       "      <td>#na#</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>519.999983</td>\n",
       "      <td>22.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>82.0</td>\n",
       "      <td>58.000001</td>\n",
       "      <td>26.000000</td>\n",
       "      <td>23.000000</td>\n",
       "      <td>8.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>56.0</td>\n",
       "      <td>57.0</td>\n",
       "      <td>91.999999</td>\n",
       "      <td>-15.000000</td>\n",
       "      <td>9.067970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>862</td>\n",
       "      <td>3</td>\n",
       "      <td>2014</td>\n",
       "      <td>8</td>\n",
       "      <td>27</td>\n",
       "      <td>False</td>\n",
       "      <td>24</td>\n",
       "      <td>25</td>\n",
       "      <td>a</td>\n",
       "      <td>c</td>\n",
       "      <td>Jan,Apr,Jul,Oct</td>\n",
       "      <td>2010</td>\n",
       "      <td>2011</td>\n",
       "      <td>SH</td>\n",
       "      <td>35</td>\n",
       "      <td>Fog</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2840.000018</td>\n",
       "      <td>22.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>100.0</td>\n",
       "      <td>61.000001</td>\n",
       "      <td>28.000000</td>\n",
       "      <td>14.000000</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>77.0</td>\n",
       "      <td>79.000000</td>\n",
       "      <td>-37.000000</td>\n",
       "      <td>9.033245</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>901</td>\n",
       "      <td>6</td>\n",
       "      <td>2015</td>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "      <td>False</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>a</td>\n",
       "      <td>c</td>\n",
       "      <td>#na#</td>\n",
       "      <td>2014</td>\n",
       "      <td>1900</td>\n",
       "      <td>SH</td>\n",
       "      <td>10</td>\n",
       "      <td>#na#</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>3170.000078</td>\n",
       "      <td>13.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>81.0</td>\n",
       "      <td>57.000000</td>\n",
       "      <td>32.000001</td>\n",
       "      <td>24.000000</td>\n",
       "      <td>14.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>57.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>65.000001</td>\n",
       "      <td>-27.000000</td>\n",
       "      <td>8.535230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>950</td>\n",
       "      <td>5</td>\n",
       "      <td>2014</td>\n",
       "      <td>9</td>\n",
       "      <td>5</td>\n",
       "      <td>False</td>\n",
       "      <td>24</td>\n",
       "      <td>0</td>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "      <td>#na#</td>\n",
       "      <td>1994</td>\n",
       "      <td>1900</td>\n",
       "      <td>BW</td>\n",
       "      <td>36</td>\n",
       "      <td>Fog</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>8459.999947</td>\n",
       "      <td>24.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>12.000000</td>\n",
       "      <td>100.0</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>48.000000</td>\n",
       "      <td>13.000001</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>62.0</td>\n",
       "      <td>78.000002</td>\n",
       "      <td>-28.000000</td>\n",
       "      <td>8.738255</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>148</td>\n",
       "      <td>1</td>\n",
       "      <td>2014</td>\n",
       "      <td>12</td>\n",
       "      <td>15</td>\n",
       "      <td>False</td>\n",
       "      <td>24</td>\n",
       "      <td>0</td>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "      <td>#na#</td>\n",
       "      <td>2008</td>\n",
       "      <td>1900</td>\n",
       "      <td>BE</td>\n",
       "      <td>51</td>\n",
       "      <td>Rain</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2090.000053</td>\n",
       "      <td>6.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>93.0</td>\n",
       "      <td>83.000000</td>\n",
       "      <td>63.000000</td>\n",
       "      <td>29.000000</td>\n",
       "      <td>14.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>78.0</td>\n",
       "      <td>79.0</td>\n",
       "      <td>73.000000</td>\n",
       "      <td>-10.000000</td>\n",
       "      <td>9.525297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>469</td>\n",
       "      <td>3</td>\n",
       "      <td>2014</td>\n",
       "      <td>8</td>\n",
       "      <td>27</td>\n",
       "      <td>False</td>\n",
       "      <td>24</td>\n",
       "      <td>0</td>\n",
       "      <td>c</td>\n",
       "      <td>c</td>\n",
       "      <td>#na#</td>\n",
       "      <td>2005</td>\n",
       "      <td>1900</td>\n",
       "      <td>BE</td>\n",
       "      <td>35</td>\n",
       "      <td>Fog</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>710.000229</td>\n",
       "      <td>21.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>94.0</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>22.000001</td>\n",
       "      <td>19.000000</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>77.0</td>\n",
       "      <td>79.000000</td>\n",
       "      <td>-37.000000</td>\n",
       "      <td>9.122929</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>917</td>\n",
       "      <td>2</td>\n",
       "      <td>2013</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>24</td>\n",
       "      <td>0</td>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "      <td>#na#</td>\n",
       "      <td>2010</td>\n",
       "      <td>1900</td>\n",
       "      <td>NW</td>\n",
       "      <td>40</td>\n",
       "      <td>#na#</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>7239.999993</td>\n",
       "      <td>17.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>81.0</td>\n",
       "      <td>64.000000</td>\n",
       "      <td>41.000000</td>\n",
       "      <td>23.000000</td>\n",
       "      <td>11.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>62.0</td>\n",
       "      <td>123.999998</td>\n",
       "      <td>-2.000002</td>\n",
       "      <td>8.837391</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "dls.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Model:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we're doing regression, we want to dictate what the maximum value to be (and minimum) so we will use a `y_range`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_log_y = np.max(train_df['Sales'])*1.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12.7616122408592"
      ]
     },
     "execution_count": null,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_log_y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we can make a `y_range`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 0.0000, 12.7616])"
      ]
     },
     "execution_count": null,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_range = torch.tensor([0, max_log_y]); y_range"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next comes our `Learner`. We'll walk through each step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = tabular_learner(dls, layers=[1000,500], ps=[0.001, 0.01],\n",
    "                       embed_p=0.04, y_range=y_range, metrics=exp_rmspe,\n",
    "                       loss_func=MSELossFlat())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we have a lot going on right there. `ps` is overall dropout (helps with overfitting), `embed_p` is dropout on the embedding weights, `exp_rmspe` is Root Mean Square Percentage Error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TabularModel (Input shape: ['512 x 26', '512 x 14'])\n",
       "================================================================\n",
       "Layer (type)         Output Shape         Param #    Trainable \n",
       "================================================================\n",
       "Embedding            512 x 81             90,396     True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 5              40         True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 3              12         True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 7              91         True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 11             352        True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 3              9          True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 10             260        True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 10             270        True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 4              20         True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 3              12         True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 3              12         True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 9              216        True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 5              45         True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 7              91         True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 15             795        True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 9              198        True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 5              35         True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 5              35         True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 3              12         True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 3              12         True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 5              45         True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 5              45         True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 3              9          True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 3              9          True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 3              9          True      \n",
       "________________________________________________________________\n",
       "Embedding            512 x 3              9          True      \n",
       "________________________________________________________________\n",
       "Dropout              512 x 223            0          False     \n",
       "________________________________________________________________\n",
       "BatchNorm1d          512 x 14             28         True      \n",
       "________________________________________________________________\n",
       "BatchNorm1d          512 x 237            474        True      \n",
       "________________________________________________________________\n",
       "Dropout              512 x 237            0          False     \n",
       "________________________________________________________________\n",
       "Linear               512 x 1000           237,000    True      \n",
       "________________________________________________________________\n",
       "ReLU                 512 x 1000           0          False     \n",
       "________________________________________________________________\n",
       "BatchNorm1d          512 x 1000           2,000      True      \n",
       "________________________________________________________________\n",
       "Dropout              512 x 1000           0          False     \n",
       "________________________________________________________________\n",
       "Linear               512 x 500            500,000    True      \n",
       "________________________________________________________________\n",
       "ReLU                 512 x 500            0          False     \n",
       "________________________________________________________________\n",
       "Linear               512 x 1              501        True      \n",
       "________________________________________________________________\n",
       "SigmoidRange         512 x 1              0          False     \n",
       "________________________________________________________________\n",
       "\n",
       "Total params: 833,042\n",
       "Total trainable params: 833,042\n",
       "Total non-trainable params: 0\n",
       "\n",
       "Optimizer used: <function Adam at 0x7f3ec01936a8>\n",
       "Loss function: FlattenedLoss of MSELoss()\n",
       "\n",
       "Callbacks:\n",
       "  - TrainEvalCallback\n",
       "  - Recorder\n",
       "  - ProgressCallback"
      ]
     },
     "execution_count": null,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "Now let's fit!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "(0.0009120108559727668, 0.0002290867705596611)"
      ]
     },
     "execution_count": null,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEKCAYAAAAYd05sAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nO3dd3hc1Z3/8fdXvUu2Jduy3Cu42whj\nMAYCppsSp9ADCYGEZNmwSTabPNlsNpvNL9nNQkJJCD2QUEMvocWYajCWu417VbEtuajLanN+f2hM\nhCLZkjV37szo83oePWhmNDofCemj63PvnGPOOUREJPbE+R1ARES8oYIXEYlRKngRkRilghcRiVEq\neBGRGKWCFxGJUQl+B2gvNzfXjRw50u8YIiJRY9myZfucc3mdPRZRBT9y5EiKior8jiEiEjXMbGdX\nj2mKRkQkRqngRURilApeRCRGqeBFRGKUCl5EJEap4EVEYpQKXkTER5+UVbNs5wG8WLpdBS8i4qPb\n3tzI1x8uorElEPLPrYIXEfHJtopaFm4o55rZI0hJjA/551fBi4j45KEPdpAYF8fVJ4/w5POr4EVE\nfFBZ38TTy0q4ePoQBmameDKGCl5ExAePfbyLhuZWrj91lGdjqOBFRMKsqSXAw4t3cOrYXI7Pz/Js\nHBW8iEiY/XXNbvZWN3p69A4qeBGRsHLO8cD72xmTl87p4ztdxj1kVPAiImG0oriSNaVVXDdnFHFx\n5ulYKngRkTD684c7yUhO4PMzCjwfSwUvIhIm+2sbeXn1br4ws4CMZO831FPBi4iEyVNFJTS1Brh6\ntjcvbOpIBS8iEgatAcejS3Yye3R/xg3KDMuYKngRkTB4Z1M5JQcbuGb2yLCNqYIXEQmDRz7cycDM\nZM6ZNChsY6rgRUQ8tmt/Pe9squCKWcNJjA9f7argRUQ89ujHO4kz44pZw8M6rmcFb2YTzGxlu7dq\nM7vFq/FERCJRY0srfykqYd7xAxmc7c2qkV3x7EJM59xGYDqAmcUDpcBzXo0nIhKJXl2zhwN1TWG7\nNLK9cE3RnAVsdc7tDNN4IiIR4c8f7WTkgDTmjMkN+9jhKvjLgcc7e8DMbjSzIjMrqqioCFMcERHv\nbdhTTdHOg1x10gjP153pjOcFb2ZJwMXAXzp73Dl3r3Ou0DlXmJfn7cpqIiLh9OhHu0hKiOOLJwz1\nZfxwHMGfDyx3zu0Nw1giIhGhrrGF51aUMn9qPv3Sk3zJEI6Cv4IupmdERGLV8ytLqW1s8eXk6mGe\nFryZpQNnA896OY6ISCRxzvHoR7uYmJ/FjGE5vuXwtOCdc3XOuQHOuSovxxERiSSrSqr4ZHc1V540\nHLPwn1w9TK9kFREJsceX7CItKZ5Lpg/xNYcKXkQkhKoPNfPiqjIunjaEzJREX7Oo4EVEQuiFFaU0\nNLdy5UnhXXemMyp4EZEQcc7x6JJdTBqSxZSCbL/jqOBFREJlRXElG/bU+H5y9TAVvIhIiPz95GqB\n31EAFbyISEhUH2rmpdVlXDJ9CBnJni3U2yMqeBGREHhhZRmHmgNh39TjSFTwIiIh8OTSXRyfHxkn\nVw9TwYuI9NLa0irWllZzxaxhEXFy9TAVvIhILz2xdBfJCXFcMi0yTq4epoIXEemF+qYWXlhRxoVT\n8slO8/eVqx2p4EVEeuGva/ZQ09jCZScO8zvKP1DBi4j0whMf72J0bjqzRvX3O8o/UMGLiByjLeU1\nFO08yGUnRtbJ1cNU8CIix+jJpcUkxBkLZvqz5+rRqOBFRI5BU0uAZ5aXMu/4QeRlJvsdp1MqeBGR\nY7Bw/V4O1DVx2azIO7l6mApeROQYPLG0mPzsFE4bl+d3lC6p4EVEeqissoF3N1fwpROGEh8XeSdX\nD1PBi4j00F+KSnAOvlQYudMzoIIXEemRQMDxVFExp47NZVj/NL/jHJEKXkSkBz7Yuo/Syga+HIGv\nXO1IBS8i0gNPFZWQk5bIORMH+R3lqFTwIiLdVH2omTfW7eHiaUNISYz3O85RqeBFRLrp1TW7aWwJ\nROwrVztSwYuIdNMzy0sZnZvOtKGRs2vTkXha8GaWY2ZPm9kGM1tvZid7OZ6IiFeKD9Tz8fYDLJhZ\nEJELi3XG662/bwdec8590cySgMi+pkhEpAvPrygF4NIZkbVr05F4VvBmlg2cBlwH4JxrApq8Gk9E\nxCvOOZ5dUcpJo/oztF/0HKd6OUUzCqgAHjKzFWZ2v5mlezieiIgnVhZXsn1fHV+IkpOrh3lZ8AnA\nTOBu59wMoA74YccPMrMbzazIzIoqKio8jCMicmyeXV5KckIc508Z7HeUHvGy4EuAEufckuDtp2kr\n/M9wzt3rnCt0zhXm5UXuqmwi0jc1tQR4aXUZ50waTGZKZG2qfTSeFbxzbg9QbGYTgnedBXzi1Xgi\nIl5YtLGcyvpmFkTRydXDvL6K5mbg0eAVNNuAr3o8nohISD2/opTcjCTmjsv1O0qPeVrwzrmVQKGX\nY4iIeKWqvpmF68u5avZwEuKj73Wh0ZdYRCRMXlmzm6bWAAtmRNfVM4ep4EVEuvDcihLG5KUzuSDL\n7yjHRAUvItKJ4gP1LN1xkAUzh0bN0gQdqeBFRDpxeGmCS6YP8TnJsVPBi4h04JzjuShcmqAjFbyI\nSAerSqrYtq+Oz0fhte/tqeBFRDp4YWUpSfFxnD8l3+8ovaKCFxFppzXgeGX1bs6YkEd2anQtTdCR\nCl5EpJ0l2/dTXtPIxVF8cvUwFbyISDsvrSojLSmes44b5HeUXlPBi4gENbUEeHXtHs6eOIjUpHi/\n4/SaCl5EJOj9LRVU1jdz0dTon54BFbyIyKdeXFlGdmoip42Pjb0pVPAiIkBDUytvfLKX8ycPJikh\nNqoxNr4KEZFeemtDOfVNrVw0LTamZ0AFLyICwIurSsnNSGb26AF+RwkZFbyI9Hk1h5pZtLGC+VPz\niY+LzpUjO6OCF5E+72/r99LUEmD+1OhemqAjFbyI9Hkvr9pNfnYKM4f38ztKSKngRaRPq6pv5t3N\nbdMzcTE0PQMqeBHp415ft4fmVsf8GHlxU3sqeBHp015aXcbw/mlMHZrtd5SQU8GLSJ+1v7aRxVv3\nc+HU/Kjdd/VIVPAi0me9tm4PrQEXM2vPdKSCF5E+6+VVuxmdl87x+Zl+R/GECl5E+qSKmkaWbN/P\n/KlDYnJ6BlTwItJHvfHJHgIOLpgy2O8onlHBi0if9NraPYzKTWfCoNicngFI8PKTm9kOoAZoBVqc\nc4Vejici0h1V9c18uHU/X587OmanZ6CbBW9m6UCDcy5gZuOB44BXnXPN3Xj655xz+3oTUkQklN5c\nv5eWgOP8ybE7PQPdn6J5F0gxswLgDeAa4I9ehRIR8dJra/cwJDslJl/c1F53C96cc/XAAuD3zrkv\nAZO68TwHvGFmy8zsxk4/sdmNZlZkZkUVFRXdjCMicmxqG1t4d3MF504eHNPTM9CDgjezk4GrgFeC\n93Vny/FTnXMzgfOBb5vZaR0/wDl3r3Ou0DlXmJcXG/sgikjkWrShnKaWAOdPjq2lgTvT3YK/BfgR\n8Jxzbp2ZjQYWHe1JzrnS4H/LgeeAWccaVEQkFF5bt4fcjGROGBFbSwN3plsnWZ1z7wDvAJhZHLDP\nOffPR3pO8MRsnHOuJvj+OcB/9TKviMgxO9TcyqIN5Vw6oyCmdm7qSreO4M3sMTPLChb1WuATM/vX\nozxtEPC+ma0CPgZecc691ru4IiLH7r3N+6hvao35q2cO6+518BOdc9VmdhXwKvBDYBnw666e4Jzb\nBkzrfUQRkdB4be0eslISYmpj7SPp7hx8opklApcCLwavf3fexRIRCa3m1gALN+xl3sRBJMb3jRfx\nd/ervAfYAaQD75rZCKDaq1AiIqH28fYDVNY3c+6kvjE9A90/yXoHcEe7u3aa2ee8iSQiEnqvr9tD\nSmIcp43rO5djd/cka7aZ3Xb4BUlmdittR/MiIhEvEHC8vm4PZ4wfSGpSd17CExu6O0XzIG2Lhn05\n+FYNPORVKBGRUFpVUsne6kbOnTzI7yhh1d2raMY4577Q7vbPzGylF4FERELttXV7SIgzzpzQtwq+\nu0fwDWZ26uEbZjYHaPAmkohI6DjneGPdXk4eM4DstES/44RVd4/gvwk8YmaHl147CFzrTSQRkdDZ\nXF7L9n11XH/qKL+jhF13r6JZBUwzs6zg7WozuwVY7WU4EZHeem3tHszgnIl9a3oGerhln3Ou2jl3\n+Pr373qQR0QkpF5ft4cZw3IYmJXid5Sw683LuWJ/pR4RiWolB+tZV1bdp17c1F5vCl5LFYhIRHtj\n3V4AzumjBX/EOXgzq6HzIjcg1ZNEIiIh8sYnexg/KINRuX3zdZlHLHjnXGa4goiIhNKBuiY+3n6A\nb39urN9RfNM3llQTkT5n4fq9BBycM7FvTs+ACl5EYtTr6/YyJDuFyQVZfkfxjQpeRGJOfVML722u\n4JxJgzHruxf8qeBFJOa8u6mCxpZAn3xxU3sqeBGJOW+s20t2aiKzRvX3O4qvursWjYhIxNq0t4bH\nluyi5lAL9U0tvLOpgvMmDyahj2zN1xUVvIhEtZdWlfGDp1fjcAxITyY1KZ4JgzO5ZvYIv6P5TgUv\nIlGppTXAr17dwP3vb6dwRD9+f9XMPrnezJGo4EUk6jS3BrjxkSIWbazg2pNH8OMLJ5KU0LenYzoT\nEwW/YU81Y/My+vx8m0hf4Jzj359by6KNFfz8kklcc/JIvyNFrKhvxKr6Zi675yPm3/k+H27d73cc\nEfHYXW9t4cmiYm4+c6zK/SiivuCzUhP4ny9MoeZQC1fc9xHffmw5pZXaTVAkFj23ooRb39zEghkF\nfPfs8X7HiXhRP0VjZpw3OZ/Txw/knne3cvfbW3lz3V4+P6OAG08fzZi8DL8jikgIbN9Xxw+eXs3J\nowfwqy9M7dOvUO0uzwvezOKBIqDUOTffq3FSk+K5Zd54vnjCUO55ZxtPFRXz1LJizhifx9B+aaQk\nxpGSGE9BTirH52cxflAmqUnxXsURkRC7c+Fm4uOM26+YrhOq3RSOI/jvAOuBsKz4M7RfGj+/dDLf\nmTeOhxfv4PmVpawsruRQc4BDLa244Or2ZjAkO5V+6YnkpCYxICOJ08blcfakQWSl9K2d10Ui3daK\nWp5fWcrX545mYKYuhewuTwvezIYCFwK/IMx7uOZmJPO9cybwvXMmfHpfIOAoPljP+t01rN9dTfGB\neiobmqmsb2LT1hpeWFlG0rNxnD4hj1kj+zN8QBrD+6cxID2JQ80BGppbaW4NMDAzmbzMZP0TUSRM\n7li4meSEeG48bbTfUaKK10fwvwV+AHS5cYiZ3QjcCDB8+HBPw8TFGSMGpDNiQDrnTf7sGtHOOVYU\nV/LK6t28umY3b36y94ifKzkhjoJ+qQxITyIlMZ7UxHgyUhIYnJVCfnYKA7NSSIgzWgOOgIP05Hhy\nM5LJzUgmKzWB1oCjucXREgiQEB9HckIcSfFxVB9qZteBenYdqOdAXRNpSQlkJMeTnpxASmI8ifFx\nJMYbOWlJDM5KIT5Of2Qktm0pr+HFVWXceNpocjOS/Y4TVTwreDObD5Q755aZ2RldfZxz7l7gXoDC\nwkLf9nk1M2YO78fM4f34yfyJVNU3s/NAHbsO1HOwvpnUYIknxBt7qw9RcrCB4gP1VDU0U9vYQkVN\nIzWHWthbfYiWQHi+jIQ4Y0hOKiMGpDFtaA7ThuUwfVgOeZn6JZDYcfvCLaQmxvON08b4HSXqeHkE\nPwe42MwuAFKALDP7s3Puag/HDJnstESmpuUwdWhOj54XCDj21TVSXt1IwDnizDCDusZW9tU2sq+2\n7Q9BQpyREB9HQpzREnA0trTS2BwgIznh06mh3IxkGppaqW1sobaxhaaWAE2trTS1BDhQ10zxwXpK\nDjawpbyWu9/ZSmvwD0tuRhKj8zIYk5fB0H6pZKUmkp2aSL+0RPKzUxmSk0JaUtRfQCV9wKa9Nby8\nuoybTh9D//Qkv+NEHc9+y51zPwJ+BBA8gv9+tJR7b8TFGQMzU8J+IqihqZW1ZVWsKq5kS3ktW8pr\neW3tbg7WN3f68TlpifRPTyIrJZGsYPkPzExmYGYKg7JTGNov9dPzDzrXIH555MMdJCfEccNczb0f\nCx3GxYjUpHhOHNmfE0d+dv3rQ82tVDc0U9XQzIG6JnZXHaK0soHdVQ1U1jdTfaiF6oZmtu+rpby6\nkcaWwGeen5YUz5CcVPKzUxiclcKw/mmMG5jBuEEZjBiQTqKWhxCPNLUEeHn1bs6ZOJh+Ono/JmEp\neOfc28Db4RhLPislMZ6UxPhurbLnnKM6eB6hOHiid9eBenZXHmJP9SE27a2gvKbx00tNkxLimDk8\nhzljcjllbC5TCrJ1fbKEzKKN5VTWN/P5mQV+R4laOoKXT5kZ2cH5+vGDOr/wqaGpla0VtWwur2Fd\naTWLt+7n1jc3ceubm0iKj2PC4EwmF2QzY1gOp4wdwNB+aWH+KiRWPL+ilNyMJOaOzfU7StRSwUuP\npCbFM7kgm8kF2Xx+Rtt9B+qaWLJtPytLKllbWsUrq8t4/ONdAIwYkMacsbmcP3kwJ48eoBU/pVuq\n6ptZuL6cq2YP189ML6jgpdf6pydx/pR8zp+SD7RN9Wwur+WDLfv4YMt+XlxZxmNLdpGbkcQFU/K5\naNoQThjejzhdwy9deGXNbppaA3x+hqZnekMFLyFnZowflMn4QZl8dc4oDjW38vbGCl5aVcaTS4t5\n5MOd5GenMH9qPpdML2ByQbbfkSXCPL+ilDF56UzRz0avqODFcymJ8Zw3eTDnTR5MbWMLC9fv5cWV\nZfxx8Q7ue287U4dmc+Ws4Vw0bQjpyfqR7OuKD9Tz8Y4DfP+c8bpEt5f02yRhlZGcwCXTC7hkegGV\n9U08v6KUxz7exQ+fXcMvXlnPV+eM5Pq5o8lO1YJvfdXzK0oBuGS6pmd6SwUvvslJS+K6OaO49pSR\nLNt5kAc/2M4db23hocU7uGHuaL526igydETfpzS3BniyqJhZo/ozrL+uwOot/faI78yMwpH9KRzZ\nn0/KqvnN3zZx25ubeHjxDm4+cyxXnjRC19f3Ec8tL6XkYAM/u3iS31Fign5rJKJMHJLFfV8p5IVv\nz2H8oEz+86VPmHfbO7y0qgznfFuLTsKguTXAXYu2MKUgmzOPG+h3nJiggpeING1YDo/dcBIPf20W\n6ckJ3Pz4Cq554GN27KvzO5p45LkVpew6UM8t88bp5GqIqOAlYpkZp4/P4+WbT+Xnl0xiVXEl5/z2\nXe5cuJnm1sDRP4FEjebWAHe9paP3UFPBS8SLjzOuOXkkf/ve6Zw9cRC3vrmJL969mJ37dTQfKw4f\nvX/nLB29h5IKXqLGoKwUfnflTO6+aiY79tdzwe3v8ezyEs3NR7nm1gC/W7SFyQVZnHW8jt5DSQUv\nUef8Kfm8+p25TCrI5rtPreJ7T62ioanV71hyjJ5YWszO/fXccpZe2BRqKniJSkNyUnn8htncMm8c\nz60sZcHdi9m1v97vWNJDNYea+e2bmzhpVH8dvXtABS9RKz7OuGXeeB687kTKKhuYf+d7LNpQ7ncs\n6YE/vLOV/XVN/PjC43X07gEVvES9z00YyEv/dCpD+6XxtYeX8siHO/yOJN1QVtnA/e9t59LpQ3q8\n97F0jwpeYsLwAWk8c9MpnHXcIP7jhXX88tX1BAI6+RrJ/u+NjTjg++dO8DtKzFLBS8xITYrnnmtO\n4OrZw7nnnW1858mVNLbo5GskWltaxXMrSvnanFHa9ctDWotGYkp8nPHzSyZTkJPG/7y2gaqGZu65\n+gRSk+L9jiZBTS0B/u2Z1fRPS+Jbnxvjd5yYpiN4iTlmxk1njOF/vzCV9zdXcO1DH1NzqNnvWBJ0\n11ubWVdWzf9bMIWsFC0L7SUVvMSsL584jNsvn8HynQe5+v4lVNY3+R2pz1tVXMnv3t7KgpkFnDtp\nsN9xYp4KXmLaRdOG8IerT2D9nhouv/cjDtSp5P1yqLmV7z61koGZyfz0Ii0HHA4qeIl58yYO4oFr\nC9m+r44r7/uI/bWNfkfqk259YyNbK+r49RenaceuMFHBS58wd1weD153Ijv213HFfR9RUaOSDyfn\nHE8vK2H+1HxOHZfrd5w+QwUvfcacsbk8eN2J7DpQr5IPs5KDDRysb+ak0QP8jtKnqOClTzllTC4P\nXTeL0oMNXHW/pmvCZW1pFQBTC7J9TtK3eFbwZpZiZh+b2SozW2dmP/NqLJGeOHnMAB64rpBdB+q5\n6v4lOvEaBqtLq0iMN47Lz/Q7Sp/i5RF8I3Cmc24aMB04z8xmezieSLedMiaX+79yItv31XH1/Us4\nqJL31JqSKiYMziQ5QS84CyfPCt61qQ3eTAy+aXEQiRinjsvlvq8UsqWilqsf0HXyXnHOsaa0iima\nngk7T+fgzSzezFYC5cCbzrklXo4n0lOnjc/jvq8Usrm8livv05G8F4oPNFDV0MyUAq0YGW6eFrxz\nrtU5Nx0YCswys8kdP8bMbjSzIjMrqqio8DKOSKdOD5b8lopartJ0TcitLq0EYOpQHcGHW1iuonHO\nVQKLgPM6eexe51yhc64wLy8vHHFE/kH7ktcllKG1prSKpPg4xg/SCdZw8/Iqmjwzywm+nwqcDWzw\najyR3jp9fB4PXFvIzv31fPmeDymtbPA7UkxYU1LFcfmZJCXoquxw8/I7ng8sMrPVwFLa5uBf9nA8\nkV6bOy6PP10/i301jXzp7sVs31fnd6SophOs/vLyKprVzrkZzrmpzrnJzrn/8moskVAqHNmfx2+c\nzaGWAF/6w2JWFlf6HSlq7dxfT82hFhW8T/RvJpFOTC7I5qlvnExqUjyX3fMhf12z2+9IUWl18BWs\nU3SC1RcqeJEujB2YwXPfmsOkIVl869Hl/P7tLTinl3L0xJqSSpISdILVLyp4kSPIzUjmsRtmc/G0\nIfzvaxv59mPLqarX7lDdtaa0ion5WSTGq2r8oO+6yFGkJMZz++XT+dH5x/HGur1ccMd7LN1xwO9Y\nES8QcKwtrdb8u49U8CLdYGZ84/QxPH3TKSTEG5fd8yG/enUD1drrtUsvr9lNbWOL5t99pIIX6YHp\nw3J4+eZTWTBzKH94Zyun/e8i7n13K4eaW/2OFlGeXlbCLU+s4IQR/bhwSr7fcfosi6STRoWFha6o\nqMjvGCLdsra0il+/vpF3NlUwMDOZBTOHsmBmQZ8/ofinD3fwkxfWMWfsAO77SiFpSQl+R4ppZrbM\nOVfY6WMqeJHe+XDrfu57bxvvbKqgNeCYXJDFmRMGMmvUAGaOyOlTBXf/e9v471fWM+/4Qdx15QxS\nErU8sNdU8CJhsK+2kRdXlvHCqjLWlFQScJAQ17bJxfGDszg+v+1tckEWmSmxt+n0Qx9s52cvfcIF\nUwZz++UzdOVMmKjgRcKs5lAzy3dVsmTbftaUVrF+dzX7attWqTSDMXkZTBuaw9xxuXxuwkCy06K7\n8P/00U5+8vxazp00iLuunKlyDyMVvEgEqKhpZF1ZFatLqlhVXMnK4kr21zWREGecNLo/F08bwqUz\nCqJu16Onlhbzg2dWM+/4gfz+qhO0qFiYqeBFIlAg4FhVUskbn+zl9XV72FZRx8DMZL4+dxRXnjSC\njOTIn7vfub+Os297l5NG9+f+awuj7o9TLFDBi0Q45xwfbNnP3e9s4YMt+8lOTeT7507gylnDiY8z\nv+N16esPL+XDrft56/tnMCgrxe84fdKRCl7/lhKJAGbGqeNyefTrs3n+23OYmJ/FT55fy4K7F7M2\nuGBXpFm0sZy/rS/n5rPGqdwjlApeJMJMH5bDYzecxO2XT6f0YAMX3/U+P3tpHXWNLX5H+1RTS4Cf\nv/QJo3PT+dqcUX7HkS6o4EUikJlxyfQCFn7vdK46aQR/XLyDc37zLm9vLPc7GgAPfrCdbfvq+I+L\nJuqkagTT/xmRCJadmsjPL53M099sW5v+uoeWcvPjKyjzcTvBLeW13LlwM/OOH8QZEwb6lkOOTgUv\nEgVOGNGfV/75VG6ZN4431u3hzFvf5rY3N1HfFN5pm7WlVXz5ng9JTUrgpxdNDOvY0nMqeJEokZwQ\nzy3zxrPwe6dz9sTB3LFwM5/7v7d5/ONdtLQGPB9/6Y4DXHHvR6QmxvOXb57MsP5pno8pvaOCF4ky\nQ/ulcecVM3jmppMpyEnlR8+u4ezfvMtLq8oIBLy57Plvn+zlmgeWkJeZzF++eTKjctM9GUdCSwUv\nEqVOGNGfZ246hfu+UkhSfBw3P76Ci3/3Pou37gvZGM2tAX751/V8/ZEixg7M4KlvnsyQnNSQfX7x\nll7oJBIDWgOOF1aWcusbmyitbODM4wbyw/OP69XSxWWVDfzTY8tZvquSq2cP598vnKjVISOQXskq\n0kccam7lj4t38LtFW6hrbOGyE4fxL/PGM7CHL0TaW32Ii+58n/qmVn65YAoXTRviUWLpLRW8SB9z\nsK6JO97azJ8/2klifBw3zB3N104dRXbq0VetbGxp5fJ7P2LjnhqeuekUjs/PCkNiOVZaqkCkj+mX\nnsRPL5rEm/9yOmdMyOP2hZuZ86u3+OVf17O3+lCXz3PO8R/Pr2PFrkpu+/I0lXuUi/zl6kTkmI3M\nTef3V53A2tIq7nl3G/e9t42HPtjBeZMH8/mZBcwdm0tCu7Xb/7xkF08WFXPzmWM5b7L2Uo12mqIR\n6UN27q/jgfe388LKMqoamsnNSGbm8BwO1jdRXtNI8YF6zpgwkPu/UkhcBK9iKX+nOXgR+YzGllbe\n3ljBc8tL2VpRS25GMnmZyQzvn8aNp48mKwa3FIxVRyp4z6ZozGwY8AgwCHDAvc65270aT0S6Lzkh\nnnMnDebcSYP9jiIe8nIOvgX4nnNuuZllAsvM7E3n3CcejikiIkGeXUXjnNvtnFsefL8GWA8UeDWe\niIh8VlgukzSzkcAMYEknj91oZkVmVlRRURGOOCIifYLnBW9mGcAzwC3OueqOjzvn7nXOFTrnCvPy\n8ryOIyLSZ3ha8GaWSFu5P+qce9bLsURE5LM8K3gzM+ABYL1z7javxhERkc55eQQ/B7gGONPMVgbf\nLvBwPBERacezyySdc+8DendjzTsAAAb9SURBVCmciIhPIuqVrGZWAewEsoGqdg8dvt3+/o735QI9\n2emg4xjdebyrXN15P5Jz9jbj0XJ29Vg05DxSxmjMGW0/m9GSs+N9iT3MeLScR/odynHOdX6FinMu\n4t5oe9XrP9xuf3/H+4Ci3ozRnce7ytWd9yM5Z28zHi1nV49FQ84jZYzGnNH2sxktOTve59fvUMe3\nSF0u+KUubr90lPt6M0Z3Hu8qV3fej+Scvc14tOd29Vg05DxSxo63oyFntP1stn8/knN29XhPhOJ3\n6DMiaoqmN8ysyHWx4E4kiYac0ZARlDPUlDN0IiVjpB7BH4t7/Q7QTdGQMxoygnKGmnKGTkRkjJkj\neBER+axYOoIXEZF2VPAiIjFKBS8iEqP6RMGb2Vwz+4OZ3W9mi/3O0xkzizOzX5jZnWZ2rd95umJm\nZ5jZe8Hv5xl+5zkSM0sPLkU93+8sXTGz44Pfy6fN7Ca/83TFzC41s/vM7EkzO8fvPJ0xs9Fm9oCZ\nPe13lo6CP4sPB7+HV4Vr3IgveDN70MzKzWxth/vPM7ONZrbFzH54pM/hnHvPOfdN4GXg4UjMCFwC\nDAWagZJQZwxhTgfUAikRnhPg34CnvMgYzBOKn831wZ/NL9O2flOk5nzeOXcD8E3gsgjNuM05d32o\ns3Wlh5kXAE8Hv4cXhytjj15p5ccbcBowE1jb7r54YCswGkgCVgETgSm0lXj7t4HtnvcUkBmJGYEf\nAt8IPvfpSP1eAnHB5w2ibRnoSM15NnA5cB0wP1JzBp9zMfAqcGUk5ww+71ZgZoRn9OT3p5eZfwRM\nD37MY+HI55zzdE/WkHDOvRvcEaq9WcAW59w2ADN7ArjEOfdLoNN/jpvZcKDKtW0fGHEZzawEaAre\nbA11xlDlbOcgkBypOYPTR+m0/XI1mNlfnXOBSMsZ/DwvAi+a2SvAY6HMGKqcweW/fwW86oJbcUZa\nxnDrSWba/rU7FFhJGGdOIr7gu1AAFLe7XQKcdJTnXA885Fmif9TTjM8Cd5rZXOBdL4N10KOcZrYA\nOBfIAe7yNtpn9Cinc+7HAGZ2HbAv1OV+BD39fp5B2z/fk4G/eprss3r683kzMA/INrOxzrk/eBku\nqKffywHAL4AZZvaj4B+CcOsq8x3AXWZ2Ib1bwqJHorXge8w591O/MxyJc66etj9CEc217cwVNbtz\nOef+6HeGI3HOvQ287XOMo3LO3UFbSUUs59x+2s4RRBznXB3w1XCPG/EnWbtQCgxrd3to8L5IEg0Z\nQTlDTTlDJxoydhRRmaO14JcC48xslJkl0XYy7UWfM3UUDRlBOUNNOUMnGjJ2FFmZw3U2txdnqh8H\ndvP3ywevD95/AbCJtjPWP1ZG5VTO6M0ZDRmjMbMWGxMRiVHROkUjIiJHoYIXEYlRKngRkRilghcR\niVEqeBGRGKWCFxGJUSp4iWhmVhvm8UKyX4C1rZtfZWYrzWyDmf1fN55zqZlNDMX4IqCClz7GzI64\n/pJz7pQQDveec246MAOYb2ZHW+/9UtpWvxQJCRW8RB0zG2Nmr5nZMmvbXeq44P0XmdkSM1thZn8z\ns0HB+//TzP5kZh8AfwreftDM3jazbWb2z+0+d23wv2cEH386eAT+aHDJXMzsguB9y8zsDjN7+Uh5\nnXMNtC0TWxB8/g1mttTMVpnZM2aWZman0LYu/K+DR/1juvo6RbpLBS/R6F7gZufcCcD3gd8H738f\nmO2cmwE8Afyg3XMmAvOcc1cEbx9H27LHs4CfmlliJ+PMAG4JPnc0MMfMUoB7gPOD4+cdLayZ9QPG\n8fdloJ91zp3onJsGrKftJe6LaVuz5F+dc9Odc1uP8HWKdEufWS5YYoOZZQCnAH8JHlDD3zceGQo8\naWb5tO2ms73dU18MHkkf9opzrhFoNLNy2nao6rgF4cfOuZLguCuBkbRtV7jNOXf4cz8O3NhF3Llm\ntoq2cv+tc25P8P7JZvbftK2pnwG83sOvU6RbVPASbeKAyuDcdkd3Arc5514MbqTxn+0eq+vwsY3t\n3m+l89+F7nzMkbznnJtvZqOAj8zsKefcSuCPwKXOuVXBDUnO6OS5R/o6RbpFUzQSVZxz1cB2M/sS\ntG0lZ2bTgg9n8/e1t6/1KMJGYHS7rdqOugF18Gj/V7RtAg6QCewOTgtd1e5Da4KPHe3rFOkWFbxE\nujQzK2n39l3aSvH64PTHOtr2vIS2I/a/mNkyYJ8XYYLTPN8CXguOUwNUdeOpfwBOC/5h+AmwBPgA\n2NDuY54A/jV4kngMXX+dIt2i5YJFesjMMpxztcGran4HbHbO/cbvXCId6QhepOduCJ50XUfbtNA9\nPucR6ZSO4EVEYpSO4EVEYpQKXkQkRqngRURilApeRCRGqeBFRGKUCl5EJEb9fy5L7JMdnOIBAAAA\nAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>_exp_rmspe</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.026195</td>\n",
       "      <td>0.026274</td>\n",
       "      <td>0.182667</td>\n",
       "      <td>00:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.015253</td>\n",
       "      <td>0.015979</td>\n",
       "      <td>0.133926</td>\n",
       "      <td>00:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.011368</td>\n",
       "      <td>0.012961</td>\n",
       "      <td>0.114459</td>\n",
       "      <td>00:47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.008773</td>\n",
       "      <td>0.011173</td>\n",
       "      <td>0.109166</td>\n",
       "      <td>00:47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.007508</td>\n",
       "      <td>0.010604</td>\n",
       "      <td>0.107100</td>\n",
       "      <td>00:47</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(5, 3e-3, wd=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For comparison, an `exp_rmspe` of 0.108 was 10th place"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference on a test set:\n",
    "Now that we have a model, we want to make our final competitions that would be submitted to a competition, as such, we can only make the predictions and submit them as a CSV.\n",
    "\n",
    "We'll practice bringing in an exported model just to show that you do not need the original data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.export('myModel')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "del learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = load_learner('myModel')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we generate our `test_dl` from our `test_df`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dl = learn.dls.test_dl(test_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to tell the `Learner` to return the predictions for the test set we added"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "raw_test_preds = learn.get_preds(dl=dl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a peek"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[ 8.3808],\n",
       "         [ 8.8781],\n",
       "         [ 9.1845],\n",
       "         ...,\n",
       "         [ 8.7501],\n",
       "         [10.0406],\n",
       "         [ 8.9215]]), None)"
      ]
     },
     "execution_count": null,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_test_preds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll notice `[0]` contains our predictions, and `[1]` contains any labels (if we had any). This is nice because if we accidently run `learn.validate()` on a non-labeled test set, we get the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "(#2) [None,None]"
      ]
     },
     "execution_count": null,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.validate(dl=dl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Still runs, just there are no labels so it's `None`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now back to our predictions! We need to undo our `log` transform first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 4362.4321],\n",
       "        [ 7173.3774],\n",
       "        [ 9745.0059],\n",
       "        ...,\n",
       "        [ 6311.3501],\n",
       "        [22939.8164],\n",
       "        [ 7490.9766]])"
      ]
     },
     "execution_count": null,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.exp(raw_test_preds[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_preds = np.exp(raw_test_preds[0]).numpy().T[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(If you want to learn about what `T` does, compare below):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 8.380785],\n",
       "       [ 8.878132],\n",
       "       [ 9.18451 ],\n",
       "       ...,\n",
       "       [ 8.750105],\n",
       "       [10.040629],\n",
       "       [ 8.921454]], dtype=float32)"
      ]
     },
     "execution_count": null,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_test_preds[0].numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 4362.432 ,  7173.3774,  9745.006 , ...,  6311.35  , 22939.816 ,\n",
       "        7490.9766], dtype=float32)"
      ]
     },
     "execution_count": null,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_preds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can submit to Kaggle!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df['Sales'] = test_preds\n",
    "test_df[['Id', \"Sales\"]] = test_df[['Id', 'Sales']].astype('int')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally we make our submission. **NOTE:** remove the index when generating your submission always!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df[['Id', 'Sales']].to_csv('submission.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Permutation Importance\n",
    "\n",
    "Permutation importance is a technique in which we shuffle each column in a dataframe and analyze how changing a particular column affected our `y` values. The more that it was affected, the more \"important\" we can (generally) call a variable in our neural network. Let's build a quick algorithm to do this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PermutationImportance():\n",
    "  \"Calculate and plot the permutation importance\"\n",
    "  def __init__(self, learn:Learner, df=None, bs=None):\n",
    "    \"Initialize with a test dataframe, a learner, and a metric\"\n",
    "    self.learn = learn\n",
    "    self.df = df\n",
    "    bs = bs if bs is not None else learn.dls.bs\n",
    "    if self.df is not None:\n",
    "      self.dl = learn.dls.test_dl(self.df, bs=bs)\n",
    "    else:\n",
    "      self.dl = learn.dls[1]\n",
    "    self.x_names = learn.dls.x_names.filter(lambda x: '_na' not in x)\n",
    "    self.na = learn.dls.x_names.filter(lambda x: '_na' in x)\n",
    "    self.y = dls.y_names\n",
    "    self.results = self.calc_feat_importance()\n",
    "    self.plot_importance(self.ord_dic_to_df(self.results))\n",
    "\n",
    "  def measure_col(self, name:str):\n",
    "    \"Measures change after column shuffle\"\n",
    "    col = [name]\n",
    "    if f'{name}_na' in self.na: col.append(name)\n",
    "    orig = self.dl.items[col].values\n",
    "    perm = np.random.permutation(len(orig))\n",
    "    self.dl.items[col] = self.dl.items[col].values[perm]\n",
    "    metric = learn.validate(dl=self.dl)[1]\n",
    "    self.dl.items[col] = orig\n",
    "    return metric\n",
    "\n",
    "  def calc_feat_importance(self):\n",
    "    \"Calculates permutation importance by shuffling a column on a percentage scale\"\n",
    "    print('Getting base error')\n",
    "    base_error = self.learn.validate(dl=self.dl)[1]\n",
    "    self.importance = {}\n",
    "    pbar = progress_bar(self.x_names)\n",
    "    print('Calculating Permutation Importance')\n",
    "    for col in pbar:\n",
    "      self.importance[col] = self.measure_col(col)\n",
    "    for key, value in self.importance.items():\n",
    "      self.importance[key] = (base_error-value)/base_error #this can be adjusted\n",
    "    return OrderedDict(sorted(self.importance.items(), key=lambda kv: kv[1], reverse=True))\n",
    "\n",
    "  def ord_dic_to_df(self, dict:OrderedDict):\n",
    "    return pd.DataFrame([[k, v] for k, v in dict.items()], columns=['feature', 'importance'])\n",
    "\n",
    "  def plot_importance(self, df:pd.DataFrame, limit=20, asc=False, **kwargs):\n",
    "    \"Plot importance with an optional limit to how many variables shown\"\n",
    "    df_copy = df.copy()\n",
    "    df_copy['feature'] = df_copy['feature'].str.slice(0,25)\n",
    "    df_copy = df_copy.sort_values(by='importance', ascending=asc)[:limit].sort_values(by='importance', ascending=not(asc))\n",
    "    ax = df_copy.plot.barh(x='feature', y='importance', sort_columns=True, **kwargs)\n",
    "    for p in ax.patches:\n",
    "      ax.annotate(f'{p.get_width():.4f}', ((p.get_width() * 1.005), p.get_y()  * 1.005))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we can simply call `PermutationImportance` to run it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Getting base error\n"
     ]
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calculating Permutation Importance\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='38' class='' max='38', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [38/38 00:07<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfkAAAD5CAYAAAAkwUxQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nOzdd5zV1Z3/8ddbUDESsKGiqCMzggUQ\nZCjWxRYFyagRAUNUlKy9YfSHu7gazSokGo2KJXaJhjGYVbEwdqKiiJSRZgEBV9Q1QGwojDB8fn+c\nc4fvDPfeKUzn83w87oNvOfd8z3fczbnfct5HZoZzzjnnmp8tGroBzjnnnKsb3sk755xzzZR38s45\n51wz5Z28c84510x5J++cc841U97JO+ecc81Uy4ZugGu6dtppJ8vJyWnoZjjnXIP55ptv+PTTTwHY\naaed2HXXXcvtX79+PUuXLuWHH36gRYsWdOzYkXnz5q0AegHvAx/GotPM7DwASUOA0UAL4FkzGxW3\nDwduAj6L3xlnZvdna5938q7GcnJymDFjRkM3wznnGkRpaSmdOnViwYIFdOjQgV69ejF+/Hj233//\nsjJ33XUXc+bM4Z577qGwsJAnn3ySefPmfRJ3f2xm3ZN1StqR0JH3NLPlkh6RdLSZvRKLPG5mF1W1\njX673jnnnKuB6dOnk5eXR8eOHdlqq60YOnQoTz/9dLkyTz/9NGeeeSYAgwYN4pVXXklXVVJHYKGZ\nLY/rLwOn1LSNfiUfSTLgMTP7VVxvCXwBvGNmA2vpGMOB/OSvMElTgCvMbJMviSU9D/zSzL6usP23\nwCozu1nS9cDrZvaypMuAe83sh009dkPIueq5hm6Cc24ztHTsCQB89tln7LHHHmXbO3TowDvvvFOu\nbLJMy5Ytadu2LStXrkz1vXtLmg18C1xtZm8Ai4DOknKAZcBJwFaJKk+RdATwETDSzD7N1la/kt/g\ne6CLpG3i+rFseO7RJJjZgIodfJoy15jZy3H1MuAndd8y55xzFXwB7GlmPYDLgb9KamNmXwHnA48D\nbwBLgdL4nWeAHDPrBrwEPFLZQbyTL+954IS4fBowIbVDUm9Jb0uaLektSZ3j9pGSHozLXSXNk1Sj\njlPSqsTyIEkPx+WHJd0taZqkxZL6SXpQ0vupMrHcUkk7xeXRkj6S9CbQOVHm4Vj3JcBuwGuSXpN0\ntqQ/Jcr9u6Rba3Iezjm3Odh9993LXroDWLZsGbvvvnvGMuvWreObb74BWGdmJWa2EsDMZgIfA53i\n+jNm1sfMDia8mPdR3L7SzEpi1fcDPStro3fy5RUCQyW1AroByfsuHwCHx19d1wA3xu23AXmSTgYe\nAs6t5Pb3EEnFqQ+QX8W2bQ8cDIwEJgG3AgcAXSVVfHGjJzAU6A4MILzFWY6Z3Q58DhxpZkcCfwN+\nLmnLWOQs4MGK35N0jqQZkmYsX7684m7nnNts9OrVi4ULF7JkyRJ+/PFHCgsLKSgoKFemoKCARx4J\nF9xPPPEERx11FACS2klqEZc7AvsAi+P6zvHf7YELCB06ktonqya8nZ+VP5NPMLM58TnIaYSr+qS2\nwCOS9gEM2DJ+Z3181j4H+LOZTa3kMI+neSZfFc+YmUmaC3xpZnPj9+cDOUBxouzhwJOpHxuSJlVW\nuZmtkvQqMFDS+8CWqWNUKHcvcC9Afn6+T2HonNtstWzZknHjxnHcccdRWlrK2WefzQEHHMA111xD\nfn4+BQUFjBgxgtNPP528vDx22GEHCgsLmThxIsARwPWS1gLrgfPM7F+x6tskHRiXrzezj+LyJZIK\ngHXAv4DhlbaxNk+4mZgE3Az0A3ZMbP8d8JqZnRx/CExJ7NsHWEW4/b0pkp1mqwr7Urdo1ieWU+u1\n9d/xfuA/CXctHqqlOp1zrtkaMGAAAwYMKLft+uuvL1tu1apVqlMvx8z+Dvw9XZ1mdlqG7f8B/Ed1\n2ued/MYeBL42s7mS+iW2t2XDi3jDUxsltQVuJ/wqGydpkJk9UcNjfylpP8IzmJOB72pYz+vAw5LG\nEP4b/xz4c5py3wE/BVYAmNk7kvYADiI8rmjUUm+4OuecS8+fyVdgZsvi8+qK/gCMicMdkj+ObgXu\njLdTRgBjU89TauAq4FngLcKblzViZrMIb2a+B0wG3s1Q9F6gSNJriW1/A6bGNzydc841ZWbmH/+U\nfQg/Mo6uStmePXuaqx2TJ0+2Tp06WW5uro0ZM2aj/WvWrLHBgwdbbm6u9e7d25YsWWJmZitWrLB+\n/frZtttuaxdeeGG578yYMcO6dOliubm5dvHFF9v69evNzOzqq6+2rl272oEHHmjHHnusffbZZ3V+\nfs65DYAZVk//m+5X8g4ASdtJ+ghYbRviE109KC0t5cILL2Ty5MksWLCACRMmsGDBgnJlHnjgAbbf\nfnsWLVrEyJEjGTVqFBCe9/3ud7/j5ptv3qje888/n/vuu4+FCxeycOFCioqKALjyyiuZM2cOxcXF\nDBw4sNzzQ+dc8+KdfB2QdFZymFz83FmDekor1HFVLbezn6RDAMzsazPrZGan1uYxXOVqGo1pZmy7\n7bYcdthhtGpV/j3NL774gm+//Za+ffsiiTPOOIOnnnoKgDZt2pSV+/7775FUx2fonGso/uJdHTCz\nh6idt9NXW4XJC2pZP8KogLfq8Bh1pjnE2i4de8KmRGOy0047pa33s88+o0OHDuXq/OyzDQGOo0eP\nZvz48bRt25bXXnstXRXOuWbAr+SbGEnHS5qYWO8n6dm4/LOYyjdL0kRJreP2pZKui9vnSto3DgM8\nDxgZ7xIcLunUmNj3nqTXG+L8XP244YYb+PTTTxk2bBjjxo1r6OY45+qId/KN2zYVbtcPIcxI1EfS\ntrHMEKAwxtleDRxjZgcBMwh5yCkr4va7CRPiLAXuAW41s+4WJka4BjjOzA4kpCm5elDTaMwdd9yR\nTHbffXeWLVuWtU6AYcOG8fe/px2q65xrBryTb9xWxw449XnczNYBRYQI2paErP2ngb7A/sDUGJd7\nJrBXoq7/if/OJCTkpTOVML7+34EW6Qp4rG3tq2k0ZrZn6e3bt6dNmzZMmzYNM2P8+PGceOKJACxc\nuLCs3NNPP82+++5bB2flnGsM/Jl801QIXESINZxhZt8p/C/+S5YhKYkNKXmlZPjvbmbnSepD+OEw\nU1JPixMoJMp4rG0tq2k0ZkpOTg7ffvstP/74I0899RQvvvgi+++/P3fddRfDhw9n9erV9O/fn/79\n+wNw1VVX8eGHH7LFFluw1157cc899zTUqTvn6pjCkD3XGElaZWat02xvQZix6F1gopn9TVI7wlX6\nUWa2KN7O393MPpK0lDCP/QpJ+cDNZtZP0m+ANmZ2baw318w+jsvvAv9uZsUVj5+Sn59vM2bMqOWz\nds655k3STDOr6uRkm8Rv1zduFZ/JjwUws1JCaE3/+C9mtpwQtztB0hzgbaCy+7DPACenXrwDboov\n5s0jvHH/Xp2clXPOuXrhV/KuxvxK3jnnqs+v5J1rQEVFRXTu3Jm8vDzGjh270f6SkhKGDBlCXl4e\nffr0YenSpWX7xowZQ15eHp07d+aFF14o973S0lJ69OjBwIEDy7YNGzaMzp0706VLF84++2zWrl1b\nZ+flnNv8eCfvXMKmRMwuWLCAwsJC5s+fT1FRERdccAGlpaVl37vtttvYb7/9ytU1bNgwPvjgA+bO\nncvq1au5//776/4knXObDe/knUvYlIjZp59+mqFDh7L11luz9957k5eXx/Tp04EwTv25557j17/+\ndbm6BgwYgCQk0bt373Jj251zblM1qiF0kgx4zMx+FddbEqZcfcfMBmb98qYf+0lgb6A10A5YEndd\nYGaNKvY1ptUdYmZ/rcU6WwN/BI4BvibMNT/KzN7J+sUGVNuxtpsaMfvZZ5/Rt2/fct9NRcledtll\n/OEPf+C7775Le+y1a9fyl7/8hdtuu61Wz8k5t3lrbFfy3wNdJG0T148FPstSvtaY2ckxJ/7XwBuJ\nAJoG6eDjD5xMcoBf1qDOtAE30f2Ecff7mFlP4CwgfTC6q5Znn32WnXfemZ49e2Ysc8EFF3DEEUdw\n+OGH12PLnHPNXWPr5AGeJ4SxAJwGTEjtkLStpAclTZc0W9KJcXuOpDdiNvus1MxqMdd9iqQnJH0g\n6TFVY8otSe0k/V3Su/FzaNz+W0mPxGN+IukXkv4Qh58VSdoyllua2D5dUl4V6v2LpKnAXzKdFzAW\nODwOfRspabikcYl2PyupX1xeJemPkt4DDpb0q9iWYkl/ltRCUi7QB7jazNYDmNkSM2v6M8BU06ZE\nzGb67tSpU5k0aRI5OTkMHTqUV199lV/96ldl5a677jqWL1/OLbfcUsdn55zb3DTGTr4QGCqpFdAN\nSN4rHQ28ama9gSMJ47q3Bf4JHBuz2YcAtye+0wO4jBD52hE4tBptuY2Q7d4LOIVwtZuSCxxFyHh/\nFHjNzLoCq9nwIwXgm7h9HPCnKtS7PyF//rQs53UVG+423FrJOWxLeNxxILAy1nNovGtRCgwDDgCK\n4/j7rJp7rO2mRMwWFBRQWFhISUkJS5YsYeHChfTu3ZsxY8awbNkyli5dSmFhIUcddRSPPvooAPff\nfz8vvPACEyZMYIstGuP/OzrnmrJG9UwewMzmxGfOpxGu6pN+BhRIuiKutwL2BD4HxklKdVydEt+Z\nbmbLAGKmew7wZhWbcwywf+Liv018dg0w2czWSppLyHkvitvnUj4bfkLi31SHnK3eSWa2Oi5vmeW8\nqqoUSM1AcjTQE3g3Hnsbwg+JWVWtrLnH2m5KxOwBBxzA4MGD2X///WnZsiV33nknLVpke0IC5513\nHnvttRcHH3wwAL/4xS+45ppr6vw8nXObh0bXyUeTgJsJ850np9oScIqZfZgsLOm3wJfAgYS7E2sS\nu0sSyxlz2zPYAuhrZsn6UhODlACY2XpJa21DqtD6CsewNMvZ6v0+sWkkmc8raR3l78q0SiyvSVyh\nC3jEzP6jwnFzgQMltajK1XxjsXTsCZUXqoEBAwYwYMCActuuv/76suVWrVoxceLEil8Dwjzto0eP\nzlh3v3796NevX9n6unXrNq2xzjmXRWO9P/ggcJ2Zza2w/QXg4tRzdUk94va2wBfxefLpZJhBrQZe\nBC5OrcQr6uoakvj37WrWm+m8vgN+mii3FOguaQtJewC9M9T3CjBI0s7xuDtI2ivm1c8Arkv8bXMk\n1U0v6pxzrl40yk7ezJaZ2e1pdv2OcAt7jqT5cR3gLuDM+HLZvpS/Gt4UlwD5kuZIWgCcV4M6to9Z\n8pcSrsyrU2+m85oDlEp6T9JIwhSxS4AFhOf2aW+/m9kCwpzzL8Y2vQS0j7t/DewCLIrZ9Q8TbuU7\n55xrqszMP3X0IVxh79TQ7airT8+ePa25mTx5snXq1Mlyc3NtzJgxG+1fs2aNDR482HJzc6137962\nZMmSsn033nij5ebmWqdOnayoqKjc99atW2fdu3e3E044oWzbHXfcYbm5uQbY8uXL6+ycnHONC2GK\n8Hr53+lGeSXvXEOo70jbQw89lJdffpm99tqr7k/OObdZ2iw7eUlPqvwUrsWSjqvt45hZjpmtqO16\nXd2o70jbHj16kJOTUy/n5pzbPNXZ2/Wq44ja+ILYckJC21eS2hOG0h1uZm/GMssJz7J/D9xi4Zk0\nZnZyFepfZWatM+zbgjDm/SjCG/NrgMFmtiRd+dogaSmQb2Yrkm2TNCC25Vgz+2QT6s94vo1Vbcba\nNmSkrXPO1ZW6vJKv04ja+FxjGnBw3HQIMDv+i6TOwEozW2lmv0518LVkCLAb0M1C0M3JhLz3eiXp\naMKLdv03pYN3dacqkbbOOVdX6vp2fbaI2t6S3laIp30rdsrEmNYH43JXSfMk/SRD/W8RO/X4762U\n7/SnxnqmSMqPy6sk3RDfTJ8maZe4fe/YnrmS/ruS82rPhqFtWBgN8FWi/lslzZf0iqR2cXuuQuTt\nzBhVu2/cninidkdJL8Z67ieMcS8j6QjgPmCghSFwSHpY0t3xvBYrxPo+KOl9SQ9Xck6k+7ukKdNs\nE+8aItLWOefqUl138tkiaj8g3FrvAVwD3Bi33wbkSToZeAg418x+yFD/VDZ08r2BJ4HU/dZDCD8C\nKtoWmGYh5vV14N8Tx707Xpl/Ucl5/Q34eXyW/8fEeP1U/TPM7ADgH8C1cfu9wMUWJn+5gjA8LnXc\ndBG31wJvxnqeJCT7pWwNPAWcZGYfVGjb9oQfOiMJoUK3EmJru1Yyzj/T36UcM7vXzPLNLL9du3ZZ\nqmt66jvS1jnn6lqddvJmNocQ8ZouorYtMDGOyU51RMSr4+HAX4B/mNnULId4F+ihkF+/pZmtAhYr\nTARTdiVfwY/As3F5JhsiaA9lw52Gv1RyXsuAzsB/EBLuXom3zonrj8flR4HDYmTtIfF8i4E/s2F8\n+jGE6NpiQqecirg9In4fCxPFfJVowlrCD5gRaZr3THyUMRf40szmxr/pfMrH7VaU6e+y2UhG2u63\n334MHjy4LNJ20qRJAIwYMYKVK1eSl5fHLbfcwtixY4HykbbHH398lSJtb7/9djp06MCyZcvo1q3b\nRi/mOefcJqursXnAqvjvNYSJUboSYmqfjdsfBi6JyznA0sR3jwP+D5hQheNMJ6TH3RXXLyeEzSwH\nFLdNIby0VtauuDwIeDgurwRaxuU2yXJVaMMVwB1xuTRRT0fCewJtCLf30313BdAqzfZioGNi/V/E\nMffAKuAnhAS9/0yUeRgYlPibzku3L9t/r4p/l2yf5jhO3jnn6hrNbJx8pojatmx4EW94aqOktoSX\nyY4AdpQ0qJL63yLMMpeKjH2bkC43Lf4xq2oqMDQuD8tWUNJBknaLy1sQHkWkXnzbgtBJQpjz/U0z\n+xZYIunU+B1JOjCWyRRx+3r8PpL6E27Dl7HwCOMEYJikdFf0zjnnNnN13slb5ojaPwBjJM2m/FC+\nW4E7zewjwu3osams9QymEq6YU538LKAD6Z/HZ3MpcKHCrHK7V1J2Z+CZ+KhhDmGCmNR87t8DveO+\no4DUzCbDgBExonY+cGLcnini9jrgiBjf+wvgfys2wsz+BRwPXC2poOJ+55xzmzdV72LXVaYpjjev\nqfz8fJsxY0ZDN6NWFBUVcemll1JaWsqvf/1rrrrqqnL7S0pKOOOMM5g5cyY77rgjjz/+eFmQzZgx\nY3jggQdo0aIFt99+O8cddxxr1qzhiCOOoKSkhHXr1jFo0CCuu+46AA4//PCyMfP//Oc/6d27N089\n9VS9nq9zruFImmlm+fVxrMY61axz9SYVZ/vSSy/RoUMHevXqRUFBAfvvv39ZmWScbWFhIaNGjeLx\nxx8vF2f7+eefc8wxx/DRRx+x9dZb8+qrr9K6dWvWrl3LYYcdRv/+/enbty9vvPFGWb2nnHIKJ554\nYrpmOefcJmsSsbaSzkoTQ3tnPRy3a5rjvpPtO439Kl7SO2nOqWtDt6sh1UWcrSRatw7/p7B27VrW\nrl2LVC7qgG+//ZZXX32Vk046qX5O1Dm32WkSV/Jm9hBhzPwmUTWjduPLgjWZQz7T8acAV5jZjLie\nQxht0KUW6s4HzjCzS9LsW0qMxAVKzax7PPYhZvbXTT12Q6mNWNu6jLMtLS2lZ8+eLFq0iAsvvJA+\nffqUq/Opp57i6KOPpk2bNpt8Hs45l06TuJKvRXUatduQzGxGug4+TblUeFAO8e19VzdatGhBcXEx\ny5YtY/r06cybN6/c/gkTJnDaaac1UOucc5uDza2Th+xRu9vGGNjpMW73xLg9J0bRzoqfVD5+vxiZ\n+4SkDyQ9por3ZKtI0nBJ4xLrz0rqF5dXSbopRty+rBAJPCVG1xYk2vJsXM4YiStpVVwcCxweb9eP\nlPR6MhFP0puJYX7Jdja7WNu6iLNN2m677TjyyCMpKioq27ZixQqmT5/OCSecgHPO1ZXNsZPPFrU7\nGnjVzHoDRwI3xTS9fxJmeTuIMDlNckhgD8I4/f0JQ/kOreT4j6WehbNxCmAm28Z2HQB8B/w34S7E\nyWwYopeULRI35SrgDTPrbma3Ag8Q8wokdSIE9LxX8UvWDGNt6yLOdvny5Xz9dZizaPXq1bz00kvs\nu+++ZfU98cQTDBw4kFatWtXfiTrnNjtN4pl8bTKzOfF5dLqo3Z8BBZKuiOutCB3k54To2e6ERLtO\nie9MtxBzS+y4c4A3szRhWMVn8lVo9o9A6jJwLlBiZmvjmP6cNOWPIIytx8yek/RVmjIVTQT+S9KV\nwNmEhLzNQjLOtrS0lLPPPrsszjY/P5+CggJGjBjB6aefTl5eHjvssAOFhYVA+Tjbli1blsXZfvHF\nF5x55pmUlpayfv16Bg8ezMCBG177KCws3GiYnnPO1bbNrpOPJgE3E2J2d0xsF3CKmX2YLCzpt8CX\nwIGEux9rErtLEsul1Pxvuo7yd1aSl3hrE+l961PHNLP18eXBTWZmP0h6iRDSMxho9HOjLh1be7e6\nBwwYwIABA8ptu/76DTdJWrVqxcSJE9N+d/To0YwePbrctm7dujF79uyMx5syZUrNG+ucc1W0Od6u\nh8xRuy8AF6eeq2vD7HJt2TC17OlA9plHamYp0F3SFpL2IMyqV1NZI3Gj74CfVth2P+FRxLsWp851\nzjnXdG2WnXyWqN3fAVsCc2Kc7O/i9ruAM2Mk7b6Et/Rr21RgCbCA0NHO2oS6Ko3EJcTxlirMHz8S\nwMxmAt9SC8MVnXPONQL1NROOfxr/B9gN+AjYoirlG8MsdJMnT7ZOnTpZbm6ujRkzZqP9a9asscGD\nB1tubq717t3blixZUrbvxhtvtNzcXOvUqZMVFRVVWufixYutd+/elpuba4MHD7aSkpI6PTfnXPNE\nM5uFzjUBks4gjDQYbeGxRKOXiqOdPHkyCxYsYMKECSxYsKBcmWQc7ciRIxk1ahRAuTjaoqIiLrjg\nAkpLS7PWOWrUKEaOHMmiRYvYfvvteeCBB+r9nJ1zrjq8k68Dkp5MEx17XEO3KxszG29me5hZ+rfL\nGqG6iKPNVKeZ8eqrrzJoUJhF+Mwzz/RJZZxzjV6z7eQljY5hMHNiJ9tH0mWSflKF71a13JQYJ5ta\nz5E0z8xOtjD+PPl5IZZJhtYUSEo7jioRWrPJksdsTtLF0aYiZdOVqRhHm+67mbavXLmS7bbbjpYt\nW2Y8lnPONTbNspOXdDAwEDjIzLoBxwCfEkJrKu28q1Fuk5jZJDMbW9fHcc45t3lqlp080B5YYWap\n8eQrgEGEF8tek/QagKS7Y0TrfEnXxW2XpCn3M0lvx0jbiZIqnWlOUitJD0maGyNyj0xTpizKVtLe\n8RhzJf13okxrSa/EY89NRO1eL+myRLkbJF2apUltJD0n6UNJ98SheqdKuiV+/1JJi+NyR0lTM5xX\no4m1rYs42kzbd9xxR77++mvWrVuX8VjOOdfo1NcbfvX5AVoDxYQ3xe8C/i1uXwrslCi3Q/y3BTAF\n6FaxHLATYdz5tnF9FHBNXJ4CfBiPVUwY/jYv7vsN8GBc3pcwjK0VIYDn2bh9ODAuLk8izCIHcCGw\nKi63BNok2rKIENqTA8yK27cAPgZ2zPD36EcI8OkYz/Ulwo+eXQlj4gGeAN4FdgfOBMZU9ndu6Lfr\n165da3vvvbctXrzYSkpKrFu3bjZv3rxyZcaNG2fnnnuumZlNmDDBTj31VDMzmzdvnnXr1s3WrFlj\nixcvtr333tvWrVuXtc5BgwbZhAkTzMzs3HPPtTvvvLMez9Y511xQj2/XN8vEOzNbJakncDghg/7x\nDM++B0s6h9CRtifkz8+pUKZv3D41ZuRsBbyd2J8ppvYw4I7Yng8kfUL5ONyKDgVOict/AX4flwXc\nKOkIQtrd7sAuZrZU0soY2LMLMNvMVmapf7qZpa7UJwCHmdkT8U7BT4E9gL8SInEPB/4nS12NQl3E\n0QJp6wT4/e9/z9ChQ7n66qvp0aMHI0aMaLBzd865qmiWnTyAmZUSrrSnxIz3M5P7Je0NXAH0MrOv\nJD1M+SjZsqLAS2ZWH3OCWpptw4B2QE8LefVL2dDO+wl3A3YlpPhVp+7U+lvAWYQ7Em8QcusPJtyJ\naPRqO442U50AHTt2ZPr06ZvYYuecqz/N8pm8pM6S9kls6g58Qvko1zaE5LpvJO0C9E+UT5abBhwq\nKS/WvW2cpa0ybxA66NSsbnsSOtJMpgJD4/KwxPa2wD9jB38ksFdi35PA8UAvQiRvNr3jc/8tCDPp\npSbReYPwY+d1YDbhzkeJmX1TSX3OOecaueZ6Jd8auEPSdoSJXxYB5xBmniuS9LmZHSlpNvAB4c37\n5Itm91YoNxyYIGnruP9qwvP+bO4C7o53EdYBw82sRJmnm78U+KukUUBysPdjwDOxnhmxvQCY2Y/x\n5cCv452LbN4FxgF5wGuEHwgQOvk9gNfNrFTSp8ljOOeca8Lq6+G/f2r/Q7gTUwzs0xDHb+gX78w8\n1tY51/TgsbauMpL2J9yheMXMFjZ0exqCx9o651x23sk3UWa2wMw6mlnZC3KSuqaJ032nIdtZlzzW\n1jnnsvNOvo5IKo2d7LwYoFMfCXpzbeM43T6xPeUieJsDj7V1zrnsvJOvO6tjJ9sF+BE4L7lTUnN9\n6dE551wj4Z18/XgDyIsTxbwhaRKwIFP0bYy7fUrSS5KWSrpI0uWxzDRJO8Ry3eP6HIWZ77avpB2n\nJ+4u9I51zJW0nYKVClPOImm8pGMrVuCxth5r65xrOryTr2Pxir0/MDduOgi41Mw6EeJrzcy6Eob3\nPSIpFXTTBfgFYQz8DcAPZtaDkLZ3RiwzHhhlYRKeucC1lTTnJ2bWHbiADeE5UwlpewcAiwlpdxAC\ncd6qWIGZ3Wtm+WaW365duyr+FepGr169WLhwIUuWLOHHH3+ksLCQgoKCcmUKCgp45JFHAHjiiSc4\n6qijkERBQQGFhYWUlJSwZMkSFi5cSO/evTPWKYkjjzySJ554AoBHHnmEE088sd7P2TnnqsM7+bqz\njaRiwtj2/wVSr2JPN7Mlcfkw4FEI0beEwJ5U0M5rZvadmS0HvgGeidvnAjmS2gLbmdk/4vZHCJG0\n2UyIx3qdMGHNdoS7DEfEz5gwOGgAACAASURBVN1AV0m7A1+Z2fc1O/X6kYy13W+//Rg8eHBZrO2k\nSZMAGDFiBCtXriQvL49bbrmFsWPDpH/JWNvjjz++LNY2U50QYm1vueUW8vLyWLlypcfaOucaPYUh\ne662SVplZq0rbOsHXGFmA+P6k8AdZvZqXH+DcHV/EJBvZhfF7Uvj+ooYzJMPjAbmmtmesUwuMNHM\nDsrQninAdWaWmlnvf4GuhOS/xwk/MEYDtwEvA3sm39xPJz8/32bMmFGNv4pzzjlJM82sXl6E9iv5\nhlXd6NsyFmJnv5KUur1+OvCPLF+BEGeLpMOAb8zsGzP7lDC73T4WJrB5kw0xt84555owf8O7YVU3\n+raiM4F74vC8xYSJZrJZE6N8tyRMRJPyDmEKWgg/PMawIdveOedcE+VX8nWk4q36uG1K6lZ9XF9j\nZmeZWVcz65G6lW5mD6du1cf1HDNbUXGfmRWbWV8z62ZmJ5nZV1na08/MLovH6WJm0xP7TjezX8bl\nt8xsC8s+bW2jUVRUROfOncnLyyt73p5UUlLCkCFDyMvLo0+fPixdurRs35gxY8jLy6Nz58688MIL\nlda5ZMkS+vTpQ15eHkOGDOHHH3+s03NzzrlN5Z28a7I81tY557LzTr6ZkXRnmmjbym7jN0kea+uc\nc9l5J19PYuDMBXVUdz9JzwKY2YUVo20Bk7Q8hukslPSCpEMS339Y0pLEj4KNxsc3Rh5r65xz2Xkn\nX3+2I4TQlFOP8baPx+fx+wBjgf+RtF9i/5WJHwaHZKjDOedcE+KdfP0ZC+TGK+V3K8TbtpB0U9w+\nR9K5UHaFPkXSE5I+kPSY4qv3ko6P22YRkvGqLL7gdy9wTnVPwmNtPdbWOdd0eCdff64CPo63z6+k\nfLztCMK49V6EGNt/l7R3/F4P4DJgf6AjcGiMvr0P+DnQE9i1Bu2ZBeybWL8pcbv+sUxf8lhbj7V1\nzjUdPk6+4STjbX8GdJM0KK63BfYhzF433cyWAcSY3BxgFbDEzBbG7Y9S/avyioPxrzSzJ6p9Fg0o\nGUFbWlrK2WefXRZrm5+fT0FBASNGjOD0008nLy+PHXbYgcLCQqB8rG3Lli3LYm2BtHVCiLUdOnQo\nV199NT169PBYW+dco+extvVEUg7wrJl1SRNv+3fgXjN7ocJ3KpYbR8jCLwZuN7Mj4vYC4JzkGPwK\n9QwnEZMbt10PtDazyyU9HNtWrU7eY22dc676PNa2efoO+GmGfS8A50vaEkLEraRts9T1AWGSmty4\nflp1GiLp3whX/vdV53vOOeeaFr9dX0/MbKWkqZLmAauBLxO77yfchp8VX6xbDpyUpa41ks4BnpP0\nAyGKNtMPiJQhMbP+J8AS4BQzez+x/yZJVyfWe5uZR7o551wT5lfy9cjMfhkjZXtViLddb2b/GeNt\nu5jZkXHymIoxuBeZ2cNxucjM9jWzg8zs0ky36mPZh82sXRwe18nMjjOzqYn9w81s7wrj65tEB++x\nts45l5l38k2ApNL41vs8SRPjhDSbPY+1dc657LyTbxpWx6vrLoQ37s9L7kwF6kg6K02k7Z0N0eD6\n4LG2zjmXnXfyTc8bQF4MykkG6rQCjiBMGWvAyDgm/11JT0l6SdJSSRdJujxG3E6TtAOApO5xfY6k\nJyVt32BnWEUea+ucc9l5J9+ExCv2/sDcuCkZqHMhYGbWlfC2/SOx4wfoQkjF6wXcAPxgZj2At4Ez\nYpnxwCgz6xbrv7YeTsk551wd8k6+adgmBuHMAP4XSD0MTgbqHAY8CmBmHwCfAJ3ivtfM7DszWw58\nAzwTt88lDMVrC2xnZv+I2x8h3BXYiMfaeqytc67pqLSTV/ArSdfE9T0l9a77prmE1Ym33i9OvPn+\nfRW/X5JYXp9YX081h1F6rK3H2jrnmo6qXMnfBRzMhsCV74Bm+zJXE/YGMAxCmA6wJ/BhVb5oZt8A\nX0k6PG46HfhHlq80CslY2/3224/BgweXxdpOmjQJgBEjRrBy5Ury8vK45ZZbyobEJWNtjz/++LJY\n20x1Qoi1veWWW8jLy2PlypUea+uca/QqjbWVNMvMDpI0Oz7HRdJ7ZnZgvbTQIWmVmbWusK0f5SNv\nWwF3A/nAOuByM3utYqStpKVxfUVyn6TuwD2EsJzFwFlm9lW2dnmsrXPOVV99xtpWpZN/BzgEeDd2\n9u2AF1Mdvtt8eSfvnHPV19iy628HngR2lnQD8CZwY522yjnnnHObLGsnL2kLQs75/wPGAF8AJ5nZ\nxHpom3OV8lhb55zLwsyyfoDZlZXxz+b56dmzpzWkdevWWceOHe3jjz+2kpIS69atm82fP79cmTvv\nvNPOPfdcMzObMGGCDR482MzM5s+fb926dbM1a9bY4sWLrWPHjrZu3bqsdZ566qk2YcIEMzM799xz\n7a677qrHs3XONRfADKun/52uyu36VySdEmdHa5IkjZY0P6a5FUvqI+myqmTAV6PcFEn5ifWcOONc\ntu/0k/RsXC6QdFWGcqsqO351SLop/j1uqs1665vH2jrnXHZV6eTPBSYCJZK+lfSdpG/ruF21RtLB\nwEDgIAtpbscAnwKXEd4kr0xVy20SM5tkZhvfb64b5wDdzOzKejpenfBYW+ecy67STt7MfmpmW5jZ\nVmbWJq63qY/G1ZL2wAozKwEwsxXAIGA34DVJrwFIujsmuc2XdF3cdkmacj+T9LakWXFGuNbpDpok\nqZWkhyTNjZnxR6YpM1zSuLi8dzzGXEn/nSjTWtIr8dhzJZ0Yt18v6bJEuRskXZqhLZOA1sBMSUMk\nzYzbD5RkkvaM6x9X5Q6Gc865xqsqiXdHpPvUR+NqyYvAHpI+knSXpH8zs9uBz4EjzSzV4Y62MKSh\nG/BvkrpVLCdpJ+Bq4BgzO4gQM3t54liPxccBxcDzie3ZcuXTuQ24O5b/IrF9DXByPPaRwB/jY5QH\niRn08WXJocSI24rMrIANCXqPA60ktQEOj+dzuKS9gH+a2Q8Vv++xth5r65xrOqpyu/7KxOe/CLnn\nv63DNtUqM1sF9CTcol4OPB5DYCoaLGkWMBs4ANg/TZm+cfvU2JGfCeyV2D8sdp7dgQGJ7dly5dM5\nFJgQl/+S2C7gRklzgJeB3YFdzGwpsFJSD+BnhJclV2apP+mteLwjCEMjjyB0+G+kK2weawt4rK1z\nrmmoNLfczH6eXJe0B/CnOmtRHTCzUmAKMEXSXELnXEbS3sAVQC8z+0rSw0C6K20BL5nZaWn21bZ0\nKUXDgHZATzNbG9PrUu28HxgO7Eq4sq+q1wmd+l7A08CoeOznatTqepSMoC0tLeXss88ui7XNz8+n\noKCAESNGcPrpp5OXl8cOO+xAYWEhUD7WtmXLlmWxtkDaOiHE2g4dOpSrr76aHj16eKytc67xq+7r\n+ISObkF9vf6/qR+gM7BPYv2/gXGEGdj2jtsOBN4j3NnYBfgSGB73Jcu1I8wClxfXtwU6xeUphIjY\n1HFygHlx+XLggbjciXAlvzXQD3g2bh8OjIvLk4BfxeXzgVVx+VLgjrh8JKEzzonrWxGy6hcDLSr5\nm6yq0M7/BR6N68/H9e0r+9s29BA655xriqjHIXSVXslLuoMNV5VbAN2BWZV9rxFpDdwhaTtCpvsi\nwq3704AiSZ9beN4+G/iA8Ob91MT3761QbjgwQdLWcf/VwEeVtOEu4O54F2Ed4QdESZZRiZcCf5U0\ninB1nfIY8EysZ0ZsLwBm9mN8OfBrC3cuqsTMlsbn+q/HTW8CHayS3HrnnHNNQGW/Agi3tlOfYcCh\n9fULxD9V/xB+gBWTuGtR15/GcCU/efJk69Spk+Xm5tqYMWM22r9mzRobPHiw5ebmWu/evW3JkiVl\n+2688UbLzc21Tp06WVFRUaV1Ll682Hr37m25ubk2ePBgKykpqdNzc841TzSyMJztzOyR+HnMzKZm\nGp7lGoak/Ql3KF4xs4UN3Z76UlpayoUXXsjkyZNZsGABEyZMYMGCBeXKPPDAA2y//fYsWrSIkSNH\nMmrUKAAWLFhAYWEh8+fPp6ioiAsuuIDS0tKsdY4aNYqRI0eyaNEitt9+ex544IF6P2fnnKuOqnTy\nZ6bZNryW29GoNfbEPDNbQAjtWZ7Y1zU1nE9Safz3ncra0ZR44p1zzmWX8Zm8pNOAXwJ7xwCVlJ8C\n/6rrhjUWFRLzSuJY+a2AxwnD4jYaS17BZVUst0nMbBLhhb3U+lzC+xOp+ei71+XxG0K6dLp33nkn\nY5mKiXd9+/Yt991Ugl26Oj3xzjnXFGV78e4tQhDLTsAfE9u/A+bUZaMamY0S8yok4a2w8ELe3UAv\nYBvgCTO7NkO5nwHXEd6u/xg4y8JY/oxicM7dQD7hxb3Lzey1CmWGE97uvygOCfwr4aXDpxNlUuvb\nA1sCV5vZ05KuB/5lZn+K5W4ghOHcVuO/mnPOuQaXsZM3s08IQ70Orr/mNEovAtdI+ogQQPO4md0u\n6XJCEt6KWG60mf1LUgvCpD7dKparkJj3fXx7/nLg+ljHY5JWx+WtgPVxuSwxT9K+wIuSsoXppBLz\nxku6MLE9lZj3bWzLtHiX5kHgf4A/JRLzetfw71VvqpN416FDhyol3gGVJt61bNnSE++cc01CVWJt\n+0p6V9IqST/G57tNZoKaTWWemFdOY4q19cQ755zLrtJx8oTgmKGEmejyCRnp2TqYZsc8MW9DpWb3\nErIDyM/PT3eMeuOJd845V4nKxtgRx/MBcxLbZtfXGL+G/uCJeY16nLxzzjU1NKbEO+AHSVsBxZL+\nQHgZrypD75oLT8xzzjnXJCn8qMhSIEw7+iXhSm8k0Ba4y8wW1X3zXH2JL9zNAk61Kgbq5Ofn24wZ\nM+q2Yc4518xImmlhavM6V+kVuYW37AW0N7PrzOxy7+Cbl6acmFdUVETnzp3Jy8tj7NixG+0vKSlh\nyJAh5OXl0adPH5YuXVq2b8yYMeTl5dG5c2deeOGFSutcsmQJffr0IS8vjyFDhvDjjz/W6bk559ym\nqsrb9T8nZKIXxfXuFcJxXBNnZgvMrKOZ/aah21IdHmvrnHPZVeXZ+m8JY6a/BjCzYmDvOmyTq4Sk\nWyVdllh/QdL9ifU/xvH51anzYUmDarOddc1jbZ1zLruqdPJrzeybCtsadOiUYypwCJQ9S9+JMDY/\n5RBCYmGzli7WtmLUbLZY23TfzbTdY22dc01RVTr5+ZJ+CbSQtE+cX77ZdyCN3FtsSCI8AJgHfCdp\n+/jW/n6ASfqHpJnxSr89gKRcSUVx+xsxQa8cSb+LV/Yt6uuEnHPO1b6MnbykVFLax4SOpISQovYt\nYdIV10DM7HNgnaQ9CVftbwPvEDr+fOB94FZgkJn1JITb3BC/fi9wcdx+BWF4XhlJNxHG85+Vbihd\nY0q8q06sLVClWNtM25OxtpmO5ZxzjU22K/meknYDhhAmqDmOEHn6R6DSqVNdnXuL0MGnOvm3E+uf\nAV2Al2J87tVAhzhBzSHAxLj9z4QJeFL+C2hrZudZhrGVZnavmeWbWX67du3q6NSqxmNtnXMuu2xh\nOPcArwAdCcEpKSI8k+9Yh+1ylUs9l+9KuF3/KfAbwp2WKcDuZlZuciFJbQhhN5mmnX2X8ONuBzNr\n9NMJe6ytc85lV5UwnLvN7Px6ao+rIkndCTPHLTazY+K2mYRJZ7oRrvRPN7O3JW1JiM+dL+kt4FYz\nm6gQmdfNzN6LefvPEn7EXQ78zMy+y9YGD8Nxzrnqa2xhON7BN05zCW/VT6uw7Rsz+ycwCPi9pPcI\nOQeHxDLDgBFx+3yg3D1nM5sI3AdMkrRN3Z6Cc865ulTplbxzmfiVvHPOVV+jupJ3rjHzWFvnnMvM\nO3nXZHmsrXPOZeedfDMkqVRSsaT5kt6T9JuYjNeseKytc85l1+z+h98BsNrMupvZAcCxQH/g2gZu\nU63zWFvnnMvOO/lmLr5pfw5wkYKcGGc7K35SGfjjJZ2U+p6kxyR52otzzjVh3slvBsxsMdAC2Bn4\nJ3CsmR1ESDO8PRZ7ABgOIKktYcjdcxXr8lhbj7V1zjUd3slvfrYE7pM0F5gI7A9gZv8A9pHUDjgN\n+LuZrav4ZY+19Vhb51zTkS3W1jUTkjoCpYSr+GuBL4EDCT/y1iSKjgd+BQwFzqrnZlabx9o651x2\nHobTDElaZWat43I74DHgbTO7VtKtwDIz+6Oks4AHzUyx7C7AdOD/zKxPZcfxMBznnKu++gzD8Sv5\n5mmbOMvclsA64C/ALXHfXcDfJZ0BFAHfp75kZl9Keh/wsWHOOdcMeCffDJlZiyz7FhImsEkZlVqQ\n9BNgH2BC3bXOOedcffEX7xwAko4B3gfuMLNvGro9VeWxts45l5l38g4AM3vZzPYysz81dFuqymNt\nnXMuO+/kmwBJJunRxHpLScslPVvD+raTdEFivV9N62pIHmvrnHPZeSffNHwPdEnM734ssCmZqtsB\nF1RaqpHzWFvnnMvOO/mm43nghLh8GomX4yTtIOkpSXMkTZPULW7/raQHJU2RtFjSJfErY4HcOInN\nTXFba0lPSPogRtqqvk7MOedc3fBOvukoBIZKakV4O/6dxL7rgNlm1g34T0KoTcq+wHFAb+BaSVsC\nVwEfx0lsrozlegCXERLwOgKHpmuEx9p6rK1zrunwTr6JMLM5QA7hKv75CrsPI4yFx8xeBXaU1Cbu\ne87MSsxsBSHxbpcMh5huZsvMbD1QHI+Vrh0ea4vH2jrnmgYfJ9+0TAJuBvoBO1bxOyWJ5VIy/zev\narlGw2NtnXMuO4+1bQJSMbWSOgC/MLPbJfUDrjCzgZJuB5ab2e/i9lvNrIek3wKrzOzmWM88YCDw\nHTDLzPaK28vqiuvjgBlm9nC2dnmsrXPOVZ/H2rq0zGwZG6aGTfot8KCkOcAPwJmV1LNS0tTY6U8m\nzZSyzjnnmj6/knc15lfyzjlXffV5Je8v3rkmzWNtnXMuM+/kXZPlsbbOOZedd/LNiII3JfVPbDtV\nUlFDtquueKytc85l5518M2LhBYvzgFsktZLUGrgRuLAm9Ulq1C9meqytc85l5518M2Nm84BnCPPE\nXwM8CoyWNF3SbEknAkjKkfSGpFnxc0jc3i9unwQsyHQc55xzjV+jvlJzNXYdMAv4EXgWeNXMzpa0\nHTBd0suE9LtjzWyNpH0IWfiptz0PArqY2ZKKFUs6BzgHYM8996z7M8miOrG2HTp0qFKsLVBprG3L\nli091tY51yT4lXwzZGbfA48Tom6PBa6SVAxMAVoBewJbAvdJmgtMJGTWp0xP18HHuj3WFo+1dc41\nDX4l33ytjx8Bp5jZh8mdMQ3vS+BAwo+9NYnd39dTGzeJx9o651x2HobTTKUibYEdgDbAxWZmknqY\n2WxJtwLLzOyPks4CHgy7y0fcZuNhOM45V30ehuNq0+8It+bnSJof1wHuAs6U9B5hOtomcfXunHOu\n6vxK3tWYX8k751z1+ZW8c1XksbbOOZeZd/KuyfJYW+ecy847+RqSVCqpWNJ8Se9J+o2kGv89JR0W\nA2s+iJ9zEvvaSXonhtlcK+lPiX1/juPeU+sXx/nlq3v830q6oqbtbwgea+ucc9l5J19zq82su5kd\nQBiL3h+4tiYVSdoV+CtwnpntCxwGnCvphFjkaGCumfUgzP1+SOLrBwJtJbWI64cAb9WkHU2Nx9o6\n51x23snXAjP7JyEF7qI4SUymyNjxkk5KfU/SYzFm9kLgYTObFetbAfw/QohNd+APwIkx0OZDoJOk\nbSS1BVYDxUDXWO0hwFRJuZKKJM2Mbdk3HrOdpL9Lejd+Dq14PpL+XdJkSdvUyR/MOedcvfBOvpaY\n2WKgBbAzGyJjDwKGAKnb5w8AwwFiB30I4cr8AGBmhSpnAAeYWTEhg/7xeOfgO2A20AvoC7wDTAMO\nkbQ7YcTEp8C9hLHxPYErCEPmAG4DbjWzXsApwP3Jg0q6CBgInGRmqyuep6RzJM2QNGP58uXV/0PV\nourE2gJVirXNtD0Za5vpWM4519h4J1830kbGmtk/gH0ktQNOA/5uZutqUP9bhB8IhwBvx09q/a04\n+9whwMR49f9noH387jHAuLh9EtAmlgc4g/DYYZCZlaQ7sMfaeqytc67p8FjbWiKpI1BKuIq/lsyR\nseOBXwFDgbPitgVATyD51lhPYH6Gw00lTCnbCrgTWE74IbGc8ANgC+BrM+ue5rtbAH3NLNkmJAHM\nBboDHYC02fWNicfaOudcdh6GU0OSVplZ67jcDngMeNvMrs0UGRvL7gJMB/7PzPrEbe0Jt90LzKxY\n0o5AEXC9mT0jaTiQb2YXxfLbE57NfxZfxkPSC8A+wKlmNlPSW4Tb8hMVevBuZvaepL8Cs83spvi9\n7vGYvyXE4E4D7gaOM7PPs/0NPAzHOeeqz8NwmoZtUkPogJeBFwlTvEKWyFgz+xJ4H3gose0LwtX9\nfZI+IFyNP2hmz6Q7sJl9RbhqT17pv014H+C9uD4MGBHbMB9I3Vu+BMiXNEfSAsIdgWTdbxKe4T8n\naadq/D2cc841Mn4lX88k/YRwW/wgM/umoduzKfxK3jnnqs+v5JspSccQruLvaOodfGNhZlxyySXk\n5eXRrVs3Zs2albbczJkz6dq1K3l5eVxyySWkftz+61//4thjj2Wfffbh2GOP5auvvspa7yeffMJB\nBx1E9+7dOeCAA7jnnnvq50Sdc64GvJOvR2b2spntZWZ/qry0q4rJkyezcOFCFi5cyL333sv555+f\nttz555/PfffdV1a2qKgIgLFjx3L00UezcOFCjj766LKs+kz1tm/fnrfffpvi4mLeeecdxo4dy+ef\nZ311wTnnGox38hVIMkmPJtZbSlou6dm4XiDpqhrUmyNpXoVttRYlK+n6eKeg4vZ+6dou6SRJ+9fG\nsetbzlXPkXPVc0CIrT3jjDOQRN++ffn666/54osvypX/4osv+Pbbb+nbty+SOOOMM8oiaZOxt8mo\n2kz1brXVVmy99dZAmPxm/fr19XXazjlXbd7Jb+x7oEsi7e1YoCy/1MwmmdnG0501MDO7xsxerqRM\nsu0nEcfvN2VVjbbt0KFD2jJffvkl7duHCIFdd92VL7/8stJ6P/30U7p168Yee+zBqFGj2G233erm\n5JxzbhN5J5/e80AqN/40YEJqh6ThksbF5Ycl3S7pLUmLJQ2q6QElTZGUH5d3krQ0cbynJL0kaamk\niyRdHiermSZph0RbBsXl4xUmuZkF/KJi22PMbgFwUxwhkBvLpsrtk1zfXEhK5QVktcceezBnzhwW\nLVrEI488UvbDwDnnGhvv5NMrBIZKagV0I4xhz6Q9YUKZgUBlV/i5sVMtjolz51VSPqULobPuBdwA\n/BDHx79NSKkrE9t8H/BzQqDOrhUrM7O3CGl3V8ao3I+Bb2JOPoSQnocqfi/W3+Cxtt/NepbPH7qY\n7t270759+ypF2y5btixtmV122aXs9v4XX3zBzjvvXPadyurdbbfd6NKlC2+88UbtnqBzztUS7+TT\nMLM5QA7hKv75Soo/ZWbrzWwBsEslZT+OnWr3mEZX1VezXzOz78xsOfANkBo/Pze2M2lfYImZLbTw\nCvmjVM39wFkKs9kNIcyKt5HGEGv704MGsttZd1BcXMxJJ53E+PHjMTOmTZtG27Zty26/p7Rv3542\nbdowbdo0zIzx48eXRdImY2+TUbUFBQVp6122bBmrV4dI/6+++oo333yTzp071+PZO+dc1XmsbWaT\ngJuBfsCOWcolM94rv9eb2To2/OhqleUY6xPr66m9/4Z/J8TxvgrMNLOVtVRvrVs69oSy5QEDBvD8\n88+Tl5fHT37yEx56aMMNiO7du1NcXAzAXXfdxfDhw1m9ejX9+/enf//+AFx11VUMHjyYBx54gL32\n2ou//e1vWet9//33+c1vfoMkzIwrrriCrl274pxzjZF38pk9SMh/nyupXz0cbynh9vp0oMbP9oEP\ngBxJufE2/GkZyn0H/DS1YmZrYjTu3UCTCWWXxJ133pl2X6qDB8jPz2fevHkbldlxxx155ZVXqlzv\nsccey5w5czahxc45V3/8dn0GZrbMzG6vvGStuRk4X9JsoMZxsnHimXMIsbSzCBPmpFMIXBlf4MuN\n2x4j3B14sabHd84513h4rK0rE8fstzWz/6pKeY+1dc656vNYW1fvJD1JeFP/toZuS3XUd6wtQIsW\nLejevTvdu3ffaP5655xrTLyTT5A0WtL8OENbsaQ+GcqVjZVPs69rcphc/GQcghfHvu8Ul1dV9TiJ\nMmWpeVVJvcvEzE42s25mtiJbucamvmNtAbbZZhuKi4spLi5m0qRJdX+SzjlXQ/7iXSTpYMJY94PM\nrCR2vFtVtx4zmwt0r7RgHTCzaxriuPUpFWcL4S37TPGzyWF0yVhboCzWtn///jz99NNMmTIFCLG2\n/fr14/e//32V6nXOucbOr+Q3aA+sMLMSADNbYWafS+oVE+3ekzRdUuqN9N0kFUlaKOkPqUoknSZp\nrqR5kn5f2faqitn3r8a7DK9I2jNNmaqk3vWW9HZ84e4tSZ3j9tcTYThIelPSgdVtZ31riFjbNWvW\nkJ+fT9++fcuy7p1zrjHyK/kNXgSukfQR8DLwOCFR7nFgiJm9K6kNsDqW7w70IIxZ/1DSHUAp8HvC\nULivgBclnUQYFrfRdjOr2ENsE5PwUnYgjNcHuAN4xMwekXQ2cDshf34jidS7o4BF8RxSPgAON7N1\n8db+jcApwAPAcOAySZ2AVmb2XhX+bs1GVWNtP/nkE3bffXcWL17MUUcdRdeuXcnNza30e845V9/8\nSj4ys1WETvgcYDmhYzwX+MLM3o1lvjWzdfErr5jZN3HI2gJgL0Ls7BQzWx7LPQYckWV7RasrJOIl\nb78fzIYUur8QonQzyZZ61xaYqDAj3q3AAXH7RGCgpC2Bs4GH01XssbaU/duxY0f69evH7Nmz6+As\nnXNu03knn2BmpWY28AF+HwAAEwZJREFUxcyuBS4icZs7jWQKXSlN567I7wgxuV0I+fatAMzsB+Al\n4ET+f3vnHl1Vdefxzy8JJEEeLSnjBKlCIo9SExCS2IHyUDCgtqKjoyhtgSkzIpLWtmrTVle1nTWt\nLTNIQaVdimkYmLLAZXTAilpJG8FCwiMgikBoRqMuWyITwlOS/OaPvXO5uSQk5HEf8fdZ66yce84+\n53z3vTf3d/Y+e39/cBvuRuQcPu22tkeOHOH0affRHz58mM2bNzNyZMwn8zMMo5sSK4Gpy/HPphtU\n9YDfNBp4G5guItm+u74PZ7vrm2Mb8Cs/aO8Izm1u6Xm2XwhbgJm4Vvws4HxZUc7netePs6lz54Qc\n9xTOF79EVY9coL6wEGxpC5Gxtb3rrruIi4ujoaGB/Px8C/KGYUQtFuTP0htYKiKfwfnIH8R13T/j\ntyfjAvw5U9QaUdUPRSQf2ITzsd+gqs8DtLT9AsgDnhGR+3GPE+aeR8cpEWl0vTuBuyFoHDD4C+C3\nIvIgsCHkuO0icpQWMtBFI+G2tR03bhx79uzpgGLDMIzwYY53RgARGQgUAyNUtaG18uZ4ZxixxZkz\nZ6iqquLUqVORlvKpICkpiUGDBtGjR48m28PpeGcteQMAEfkGLlf9d9sS4A3DiD2qqqro06cPgwcP\nbtNMEqP9qCrV1dVUVVUxZMiQiOmwgXcGAKpaqKqfV9W1kdZyIZitrWG0nVOnTpGSkmIBPgyICCkp\nKRHvNbEgH8M0Z8MrIveKSK82HNumctGO2doaxoVhAT58RMN7bUE+Rgmx4c3EDQh8D7gXaEvwbmu5\nqGJw/obAArRoPxtMsK2tiARsbRuPnz17NuBsbYO3t3ZewzCMaMeeyccu59jwisi3gIHAJhE5rKpX\ni8iTODOeZGCdqv64hXK5wCNAIlABzPUGQVFNS/azwXPlO9PWNjU1NWBrm5CQQH5+Pjfd1KzxoGFE\nPcG5IDqD0CmuzTFu3Di2bNnSqdc9H5WVlWzZsoU777wzbNeMJqwlH7u8DHxeRPaLyBMiMklVfwV8\nAFytqlf7cj/yozgzgUkikhlazs/ffxCYqqpjgDLgu81dNBoc77qKC7G1LSsrY/Xq1dx7771UVFSE\nQZ1hdA/CGeDr6uqorKxk9erVrRfupliQj1Gas+EVkTnNFL3NJ6nZibOwbc655Ut++2bvnT8bZ9Pb\n3HUj7nhntraGEbv07t0bgOLiYiZNmsSMGTNIS0sjPz+fVatWkZOTQ0ZGRuDmec6cOcyfP5+srCyG\nDRvG+vUua/apU6eYO3cuGRkZXHnllWzatAmAgoICbrzxRq655hqmTJlCfn4+JSUljB49msWLF1NZ\nWcmECRMYM2YMY8aMCdx0FBcXM3nyZG699VZGjBjBrFmzAgN0S0tLGTduHKNGjSInJ4fa2lrq6+u5\n//77yc7OJjMzk1//+tfhfivbhHXXxzCqWo+b114sIntwwTmAiAwB7gOyVfWIiBTgbWxDEOAVVb2j\nmX1RR58xX6HPmK+w6+c3sGHDBpYtW8bMmTPZunVrq7a2V111FYWFheTl5QFnbW3z8/PPsbVt7rxH\njhyhV69eJCYmBmxtH3jggbC/B4bRHSgvL+ftt9+mf//+pKWlMW/ePLZt28aSJUtYunQpjz32GOC6\n3Ldt20ZFRQVXX301Bw8e5PHHH0dE2LNnD/v27SM3N5f9+/cDsGPHDnbv3k3//v0pLi5m0aJFgZuD\nEydO8Morr5CUlMSBAwe44447aPT72LlzJ3v37mXgwIGMHz+ezZs3k5OTw+23386aNWvIzs7m6NGj\nJCcn8/TTT9OvXz9KS0s5ffo048ePJzc3N6LT5ZrDgnyM0oIN7/8Cg3HudoeBvsBxoEZELgauw90U\nANQGlfsz8LiIXK6qB0XkIuASVd0fpuq0GbO1NYzuQ3Z2duCmPD09ndzcXAAyMjICLXOA2267jbi4\nOIYOHUpaWhr79u3j9ddfD9ysjxgxgssuuywQ5K+99lr69+/f7DXPnDnDwoUL2bVrF/Hx8YFjAHJy\ncgLjd0aPHk1lZWXgBj87OxuAvn37AvDyyy+ze/du1q1bB0BNTQ0HDhywIG90Gi3Z8N4BvCQiH/jn\n7TtxXvbvAZuDjv9NSLk5wH+LSKLf/yAQdUE+FLO1NYzYJTExMbAeFxcXeB0XF0ddXV1gX+hYmdbG\nzlx00UUt7lu8eDEXX3wx5eXlNDQ0kJR0tnMzWE98fHwTDaGoKkuXLmXatGnn1RJp7Jl8jKKq21V1\nnKqOVNVMVf1HVT2sqktVdXjjwDtVnaOqw1R1ii9T4LeHlntNVbP9uTJV1SaAG4YRFaxdu5aGhgYq\nKio4dOgQw4cPZ8KECaxa5ZJl7t+/n3fffZfhw4efc2yfPn2ora0NvK6pqSE1NZW4uDhWrlxJfX39\nea89fPhwPvzwQ0pLSwGora2lrq6OadOm8eSTT3LmzJmAhuPHj3dWlTsNa8kbhmF8SmnLlLdo4NJL\nLyUnJ4ejR4+yfPlykpKSWLBgAXfffTcZGRkkJCRQUFDQpCXeSGZmJvHx8YwaNYo5c+awYMECbrnl\nFgoLC5k+ffp5W/0APXv2ZM2aNeTl5XHy5EmSk5N59dVXmTdvHpWVlYwZMwZVZcCAAQGfjahCVW2x\npV3L2LFjtTNoaGjQvLw8TU9P14yMDN2+fXuz5crKyvSKK67Q9PR0zcvL04aGBlVVra6u1qlTp+rl\nl1+uU6dO1Y8//lhVVYuKijQjI0NHjRqlY8eO1ZKSkibnq6mp0UsuuUTvueeeTqmHYUQ7b731VqQl\nXDCzZ8/WtWvXRlpGu2nuPQfKNEy/01HfXS8i9d6ytVxEdojIuDYc8y0ReVtEVnXC9S8WkfX++m+J\nyIt++2ARadVd4QLLvRmy7WERua+V4wpE5Fa//pSInDMKTETmiMiy1jREiq6ypp0yZQrl5eXs2rWL\nFStWMG/evCbne+ihh5g4cWLXVs4wDCOCxEJ3/UlVHQ0gItOAnwGTWjlmAc7YpaqVcvjzJqhqSyMs\nfoKbXrbEl8302wcDdwKtuSy0tVyHUdV5rZeKDoKdtqpfWkbSpRkM+cGLALx/8H0GLVxJQu+zo2Pr\njn3MRxUfMrOoGope5HiPDIqKirjuuut4/vnnKS4uBpw17eTJk3n00UcD83EBjh8/3mSwzvbt2/no\no4+YPn06li7XMKKXgoKCSEuIaaK+JR9CX+BI4wsRuV9ESn2Clkf8tuVAGvB7EfmOiPQXkSJf5s+N\nQdq3kleKyGZgpYgMEJFn/flKRWS8v0wqELhZUNXdfvXnwATfy/Ad3xIv8b0NwT0OoeXiReSXQbrv\nakvFRWS0179bRJ4Tkc82U6ZYRLL8+lzvhrcNGB9U5qsislVEdorIq76nIk5EDojIAF8mTkQONr7u\nauqPVRPf93OB1wl9UqivrW5apraahD4pgdfxfVJataYFeO655xgxYgQ33HADK1asAKChoYHvfe97\nLFq0qMvqZBjRiustNsJBNLzXsRDkk32A3Ac8BfwUwHutDwVycHPEx4rIRFWdz1nL1sU4P/ad6pK4\n/BAoDDr3SFyL/w5gCbBYVbOBW/y1AB4HnhaRTeKyvg302/OBElUd7a/zV+BadbawtwO/aqHcN4Ea\nf51s4F+8aQ1Auq/rLu88Nz9IayHwfV+PPcCPW3rDRCTV13s88GWauty9DnxJVa8Efgc8oC5//H8B\ns3yZqUC5qp7jWxvttrah1rQ333wz+/bto6ioiIceeghwc+avv/76Jn72hvFpICkpierq6qgIPt0d\nVZdPPniKXiSIte76fwAKReQKINcvjZ6ivXFB/08hx38ZF7RR1ddEJEVE+vp9L6jqSb8+FRgZFCD6\nikhvVd0oImnAdJyZzE5//VB6AMtEZDRQDwxroT65QGbjc3Sgn9e9H6horKuv78P+bz/gM6r6R7/r\nt8D58r5fBRQ3BmkRWROkZxDOAjcV6An8xW9fATwPPAb8M/AMzaCqv8HNsScrK6vdvxS1O9ZTW74R\ngMS/H0r90cOBfXW11cQHtdrBtdzrglr39bXV51jTpqamNrGmDWbixIkcOnSIw4cP88Ybb1BSUsIT\nTzzBsWPH+OSTT+jdu3fgWb5hdFcGDRpEVVUV0XiD3h1JSkqKeGMiFoJ8AFV9Q1wylQE4K9afqWpH\nDIODJzXG4Vq4p5q57se4Z+qrRWQ9MBGoDin2HeAjYJQ/1znn8QiQp6obm2wUGdwO/e1hKfCfqvqC\niEwGHgZQ1fdE5CMRuQbXOzKr5VN0nEZrWoATFaXUbl9Pry9M5JMP3iEusVeT5/EACb37E5eYzOn3\n99Fz4HCOvfkaM+b/BGjZmvbgwYOkp6cjIuzYsYPTp0+TkpISmFsL7nlfWVmZBXjjU0GPHj2izpHN\n6FpiKsiLyAggHhdgNwI/FZFVqnpMRC4BzqjqX0MOK8EFrJ/6oHZYVY/KuY5JLwN5wC/9tUar6i4f\n9P6sqidEpA+QDrwLNOBsYRvpB1SpaoOIzPY64ax9bCMbgbtF5DVVPSMiw4D3z1dvVa0RkSMiMkFV\nS4CvA388zyFbgSUikgIcBf4JKA/S2Xi92SHHPYXrtl+pzhe/ywien6t6PQsX/o2X1n2bvr168T8v\nrCYrKwtoak1bduvFzpp2y0m+eX3r1rTPPvsshYWF9OjRg+TkZNasWdOmLHOGYRjdBYn2ZzMiUo97\nBg2uFfxDVd3g930baBxRfgz4mqpWiEglkKUux3p/XFd0GnAC+FdV3e27wo+p6iJ/rs/hnr9/AXfz\n8ydVnS8i9wNzcdaxccAzqvofItIDF7BTgAJgPfAsoMBLwD2q2ruZckuAfwO+6uvzN+Am4LPAelUN\nPAoI1ugfAywHegGHcPneG5POrFfVdSJSDNynqmUiMhf4AfB/wC7gE1VdKCIzgMW4AYyv4ZLXTPbX\n64G7gcpR1X2tfTZZWVlqI9MNwzAuDBHZri4FeNdfK9qDvBE+/Mj8xao6oS3lLcgbhmFcOBbkjbAj\nIvnA3cAsVX29jcf8DZf5LlJ8DpdFL1Yx/ZEn1utg+iNPe+pwmaqGZYqyBXkjZhGRsnDdDXcFpj/y\nxHodTH/kifY6xMI8ecMwDMMw2oEFecMwDMPopliQN2KZ30RaQAcx/ZEn1utg+iNPVNfBnskbhmEY\nRjfFWvKGYRiG0U2xIG8YhmEY3RQL8kbUISLTReQdn+42v5n9iSKyxu/fGuz7LyI/8NvfEZFp4dQd\norFddfAJlDaJyDERWRZu3UH62qv/WhHZLiJ7/N9rwq3d62iv/pygTJDlInJzuLUHaWz3/4Hff6n/\nHt0XLs0h12/vZzBYRE4GfQ7Lw63d6+jI71CmiLwhInv9/0LkUtGpqi22RM2C8/yvwNkQ98R57o8M\nKbMAWO7XZwJr/PpIXz4RGOLPEx9jdbgIlzlxPrAsBj+DK4GBfv0K4P0Y098LSPDrqbgU0gmxVIeg\n/etw2SrviyX9wGDgzXBr7kT9CcBuYJR/nRKJ36HGxVryRrSRAxxU1UOq+gku5/2MkDIzcOl2wf2Q\nTRGXeWYG8DtVPa2qfwEO+vOFm3bXQVWPq3McbCmLYTjoiP6dqvqB374XSBaRxLCoPktH9J9Q1Tq/\nPQmXiyISdOT/ABG5CZdGem+Y9IbSIf1RQEf05wK7VbUcQFWrtYsTfp0PC/JGtHEJ8F7Q6yq/rdky\n/ge5Bne33JZjw0FH6hANdJb+W4Adqnq6i3S2RIf0i8hVIrIXlxhrflDQDyftroOI9Aa+DzwSBp0t\n0dHv0BAR2SkifxSRNuXS6GQ6on8YoCKyUUR2iMgDYdDbIjGVatYwjNhARL4IPIpr1cQUqroV+KKI\nfAH4rYj8XlUj2bNyoTyMSzR1LHoaxhfEh8ClqlotImOBIhH5oqoejbSwNpKAe+SWjct8+gdxCWn+\nEAkx1pI3oo33gc8HvR7ktzVbRkQSgH64FLltOTYcdKQO0UCH9IvIIOA54BuqWtHlas+lU95/VX0b\nl8L6CsJPR+pwFfALcSm37wV+KCILu1pwS9o8bdbvH7dVA6jqdtyz8WFdrrgFbZ4Lef+rcKnKD6vq\nCeBFYEyXK24BC/JGtFEKDBWRISLSEzeg5YWQMi8As/36rcBr6ka4vADM9KNehwBDgW1h0h1MR+oQ\nDbRbv4h8BtgA5Kvq5rApbkpH9A/xP9iIyGXACKAyPLKb0O46qOoEVR2sqoOBx4B/V9Vwz9ToyGcw\nQETiAUQkDfd/fChMuhvpyP/wRiBDRHr579Ik4K0w6T6XSI34s8WWlhbgemA/7g7+R37bT4Ab/XoS\nbtTwQVwQTws69kf+uHeA62K0DpXAx7hWZBUho3qjWT/wIHAc2BW0/F0M6f86brDaLmAHcFMsfoeC\nzvEwERhd38HP4JaQz+CrsaTf7/uar8ObwC8i9R1SVbO1NQzDMIzuinXXG4ZhGEY3xYK8YRiGYXRT\nLMgbhmEYRjfFgrxhGIZhdFMsyBuGYRhGN8WCvGEYhmF0UyzIG4ZhGEY35f8Bm4DTk4NJpbcAAAAA\nSUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "res = PermutationImportance(learn, train_df.iloc[:1000], bs=64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'AfterStateHoliday': -0.006495955086819835,\n",
       " 'Assortment': -0.15894991142484757,\n",
       " 'BeforeStateHoliday': -0.0034450730077863977,\n",
       " 'CloudCover': -0.021922465471671267,\n",
       " 'CompetitionDistance': -0.07073883242654737,\n",
       " 'CompetitionMonthsOpen': -0.048707575175682846,\n",
       " 'CompetitionOpenSinceYear': -0.006830320781063511,\n",
       " 'Day': 0.0,\n",
       " 'DayOfWeek': 0.0,\n",
       " 'Events': 0.010310428429607058,\n",
       " 'Max_Humidity': 0.05946607822128924,\n",
       " 'Max_TemperatureC': -0.029806386487200524,\n",
       " 'Max_Wind_SpeedKm_h': 0.004062584161248675,\n",
       " 'Mean_Humidity': 0.0036539491262186446,\n",
       " 'Mean_TemperatureC': 0.0041852054038987,\n",
       " 'Mean_Wind_SpeedKm_h': -0.017164310518782103,\n",
       " 'Min_Humidity': -0.0003381559916355307,\n",
       " 'Min_TemperatureC': -0.012325842237373623,\n",
       " 'Month': 0.0,\n",
       " 'Promo': 0.0,\n",
       " 'Promo2SinceYear': -0.026129080933810448,\n",
       " 'Promo2Weeks': -0.3059371320786493,\n",
       " 'PromoInterval': -0.11177023585996265,\n",
       " 'Promo_bw': 0.0,\n",
       " 'Promo_fw': -0.0034969078856317715,\n",
       " 'SchoolHoliday': -0.00045760157971400057,\n",
       " 'SchoolHoliday_bw': -0.061669982493948075,\n",
       " 'SchoolHoliday_fw': -0.06475180159493665,\n",
       " 'State': -0.000484645863807239,\n",
       " 'StateHoliday': 0.0,\n",
       " 'StateHoliday_bw': 0.0,\n",
       " 'StateHoliday_fw': 0.0,\n",
       " 'Store': -19.48458644833461,\n",
       " 'StoreType': -0.012292753965547427,\n",
       " 'Week': 0.0,\n",
       " 'Year': 0.0,\n",
       " 'trend': -0.01690329220109433,\n",
       " 'trend_DE': 0.0}"
      ]
     },
     "execution_count": null,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res.importance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}