ch14/ch14_part2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Python Machine Learning 3rd Edition* by [Sebastian Raschka](https://sebastianraschka.com) & [Vahid Mirjalili](http://vahidmirjalili.com), Packt Publishing Ltd. 2019\n",
    "\n",
    "Code Repository: https://github.com/rasbt/python-machine-learning-book-3rd-edition\n",
    "\n",
    "Code License: [MIT License](https://github.com/rasbt/python-machine-learning-book-3rd-edition/blob/master/LICENSE.txt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 14: Going Deeper -- the Mechanics of TensorFlow (Part 2/3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the optional watermark extension is a small IPython notebook plugin that I developed to make the code reproducible. You can just skip the following line(s)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sebastian Raschka & Vahid Mirjalili \n",
      "last updated: 2019-10-29 \n",
      "\n",
      "numpy 1.17.2\n",
      "scipy 1.3.1\n",
      "matplotlib 3.1.0\n",
      "tensorflow 2.0.0\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -a \"Sebastian Raschka & Vahid Mirjalili\" -u -d -p numpy,scipy,matplotlib,tensorflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TensorFlow Estimators\n",
    "\n",
    "##### Steps for using pre-made estimators\n",
    "\n",
    " * **Step 1:** Define the input function for importing the data   \n",
    " * **Step 2:**  Define the feature columns to bridge between the estimator and the data   \n",
    " * **Step 3:** Instantiate an estimator or convert a Keras model to an estimator   \n",
    " * **Step 4:** Use the estimator: train() evaluate() predict()   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(1)\n",
    "np.random.seed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with feature columns\n",
    "\n",
    "\n",
    " * See definition: https://developers.google.com/machine-learning/glossary/#feature_columns\n",
    " * Documentation: https://www.tensorflow.org/api_docs/python/tf/feature_column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MPG</th>\n",
       "      <th>Cylinders</th>\n",
       "      <th>Displacement</th>\n",
       "      <th>Horsepower</th>\n",
       "      <th>Weight</th>\n",
       "      <th>Acceleration</th>\n",
       "      <th>ModelYear</th>\n",
       "      <th>Origin</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>393</th>\n",
       "      <td>27.0</td>\n",
       "      <td>4</td>\n",
       "      <td>140.0</td>\n",
       "      <td>86.0</td>\n",
       "      <td>2790.0</td>\n",
       "      <td>15.6</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>394</th>\n",
       "      <td>44.0</td>\n",
       "      <td>4</td>\n",
       "      <td>97.0</td>\n",
       "      <td>52.0</td>\n",
       "      <td>2130.0</td>\n",
       "      <td>24.6</td>\n",
       "      <td>82</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>395</th>\n",
       "      <td>32.0</td>\n",
       "      <td>4</td>\n",
       "      <td>135.0</td>\n",
       "      <td>84.0</td>\n",
       "      <td>2295.0</td>\n",
       "      <td>11.6</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>396</th>\n",
       "      <td>28.0</td>\n",
       "      <td>4</td>\n",
       "      <td>120.0</td>\n",
       "      <td>79.0</td>\n",
       "      <td>2625.0</td>\n",
       "      <td>18.6</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>397</th>\n",
       "      <td>31.0</td>\n",
       "      <td>4</td>\n",
       "      <td>119.0</td>\n",
       "      <td>82.0</td>\n",
       "      <td>2720.0</td>\n",
       "      <td>19.4</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      MPG  Cylinders  Displacement  Horsepower  Weight  Acceleration  \\\n",
       "393  27.0          4         140.0        86.0  2790.0          15.6   \n",
       "394  44.0          4          97.0        52.0  2130.0          24.6   \n",
       "395  32.0          4         135.0        84.0  2295.0          11.6   \n",
       "396  28.0          4         120.0        79.0  2625.0          18.6   \n",
       "397  31.0          4         119.0        82.0  2720.0          19.4   \n",
       "\n",
       "     ModelYear  Origin  \n",
       "393         82       1  \n",
       "394         82       2  \n",
       "395         82       1  \n",
       "396         82       1  \n",
       "397         82       1  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset_path = tf.keras.utils.get_file(\"auto-mpg.data\", \n",
    "                                       (\"http://archive.ics.uci.edu/ml/machine-learning-databases\"\n",
    "                                        \"/auto-mpg/auto-mpg.data\"))\n",
    "\n",
    "column_names = ['MPG', 'Cylinders', 'Displacement', 'Horsepower',\n",
    "                'Weight', 'Acceleration', 'ModelYear', 'Origin']\n",
    "\n",
    "df = pd.read_csv(dataset_path, names=column_names,\n",
    "                 na_values = \"?\", comment='\\t',\n",
    "                 sep=\" \", skipinitialspace=True)\n",
    "\n",
    "df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MPG             0\n",
      "Cylinders       0\n",
      "Displacement    0\n",
      "Horsepower      6\n",
      "Weight          0\n",
      "Acceleration    0\n",
      "ModelYear       0\n",
      "Origin          0\n",
      "dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MPG</th>\n",
       "      <th>Cylinders</th>\n",
       "      <th>Displacement</th>\n",
       "      <th>Horsepower</th>\n",
       "      <th>Weight</th>\n",
       "      <th>Acceleration</th>\n",
       "      <th>ModelYear</th>\n",
       "      <th>Origin</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>387</th>\n",
       "      <td>27.0</td>\n",
       "      <td>4</td>\n",
       "      <td>140.0</td>\n",
       "      <td>86.0</td>\n",
       "      <td>2790.0</td>\n",
       "      <td>15.6</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>388</th>\n",
       "      <td>44.0</td>\n",
       "      <td>4</td>\n",
       "      <td>97.0</td>\n",
       "      <td>52.0</td>\n",
       "      <td>2130.0</td>\n",
       "      <td>24.6</td>\n",
       "      <td>82</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>389</th>\n",
       "      <td>32.0</td>\n",
       "      <td>4</td>\n",
       "      <td>135.0</td>\n",
       "      <td>84.0</td>\n",
       "      <td>2295.0</td>\n",
       "      <td>11.6</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>390</th>\n",
       "      <td>28.0</td>\n",
       "      <td>4</td>\n",
       "      <td>120.0</td>\n",
       "      <td>79.0</td>\n",
       "      <td>2625.0</td>\n",
       "      <td>18.6</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>391</th>\n",
       "      <td>31.0</td>\n",
       "      <td>4</td>\n",
       "      <td>119.0</td>\n",
       "      <td>82.0</td>\n",
       "      <td>2720.0</td>\n",
       "      <td>19.4</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      MPG  Cylinders  Displacement  Horsepower  Weight  Acceleration  \\\n",
       "387  27.0          4         140.0        86.0  2790.0          15.6   \n",
       "388  44.0          4          97.0        52.0  2130.0          24.6   \n",
       "389  32.0          4         135.0        84.0  2295.0          11.6   \n",
       "390  28.0          4         120.0        79.0  2625.0          18.6   \n",
       "391  31.0          4         119.0        82.0  2720.0          19.4   \n",
       "\n",
       "     ModelYear  Origin  \n",
       "387         82       1  \n",
       "388         82       2  \n",
       "389         82       1  \n",
       "390         82       1  \n",
       "391         82       1  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(df.isna().sum())\n",
    "\n",
    "df = df.dropna()\n",
    "df = df.reset_index(drop=True)\n",
    "df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>MPG</th>\n",
       "      <td>313.0</td>\n",
       "      <td>23.404153</td>\n",
       "      <td>7.666909</td>\n",
       "      <td>9.0</td>\n",
       "      <td>17.5</td>\n",
       "      <td>23.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>46.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Cylinders</th>\n",
       "      <td>313.0</td>\n",
       "      <td>5.402556</td>\n",
       "      <td>1.701506</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Displacement</th>\n",
       "      <td>313.0</td>\n",
       "      <td>189.512780</td>\n",
       "      <td>102.675646</td>\n",
       "      <td>68.0</td>\n",
       "      <td>104.0</td>\n",
       "      <td>140.0</td>\n",
       "      <td>260.0</td>\n",
       "      <td>455.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Horsepower</th>\n",
       "      <td>313.0</td>\n",
       "      <td>102.929712</td>\n",
       "      <td>37.919046</td>\n",
       "      <td>46.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>92.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>230.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Weight</th>\n",
       "      <td>313.0</td>\n",
       "      <td>2961.198083</td>\n",
       "      <td>848.602146</td>\n",
       "      <td>1613.0</td>\n",
       "      <td>2219.0</td>\n",
       "      <td>2755.0</td>\n",
       "      <td>3574.0</td>\n",
       "      <td>5140.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Acceleration</th>\n",
       "      <td>313.0</td>\n",
       "      <td>15.704473</td>\n",
       "      <td>2.725399</td>\n",
       "      <td>8.5</td>\n",
       "      <td>14.0</td>\n",
       "      <td>15.5</td>\n",
       "      <td>17.3</td>\n",
       "      <td>24.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ModelYear</th>\n",
       "      <td>313.0</td>\n",
       "      <td>75.929712</td>\n",
       "      <td>3.675305</td>\n",
       "      <td>70.0</td>\n",
       "      <td>73.0</td>\n",
       "      <td>76.0</td>\n",
       "      <td>79.0</td>\n",
       "      <td>82.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Origin</th>\n",
       "      <td>313.0</td>\n",
       "      <td>1.591054</td>\n",
       "      <td>0.807923</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              count         mean         std     min     25%     50%     75%  \\\n",
       "MPG           313.0    23.404153    7.666909     9.0    17.5    23.0    29.0   \n",
       "Cylinders     313.0     5.402556    1.701506     3.0     4.0     4.0     8.0   \n",
       "Displacement  313.0   189.512780  102.675646    68.0   104.0   140.0   260.0   \n",
       "Horsepower    313.0   102.929712   37.919046    46.0    75.0    92.0   120.0   \n",
       "Weight        313.0  2961.198083  848.602146  1613.0  2219.0  2755.0  3574.0   \n",
       "Acceleration  313.0    15.704473    2.725399     8.5    14.0    15.5    17.3   \n",
       "ModelYear     313.0    75.929712    3.675305    70.0    73.0    76.0    79.0   \n",
       "Origin        313.0     1.591054    0.807923     1.0     1.0     1.0     2.0   \n",
       "\n",
       "                 max  \n",
       "MPG             46.6  \n",
       "Cylinders        8.0  \n",
       "Displacement   455.0  \n",
       "Horsepower     230.0  \n",
       "Weight        5140.0  \n",
       "Acceleration    24.8  \n",
       "ModelYear       82.0  \n",
       "Origin           3.0  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sklearn\n",
    "import sklearn.model_selection\n",
    "\n",
    "\n",
    "df_train, df_test = sklearn.model_selection.train_test_split(df, train_size=0.8)\n",
    "train_stats = df_train.describe().transpose()\n",
    "train_stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MPG</th>\n",
       "      <th>Cylinders</th>\n",
       "      <th>Displacement</th>\n",
       "      <th>Horsepower</th>\n",
       "      <th>Weight</th>\n",
       "      <th>Acceleration</th>\n",
       "      <th>ModelYear</th>\n",
       "      <th>Origin</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>203</th>\n",
       "      <td>28.0</td>\n",
       "      <td>-0.824303</td>\n",
       "      <td>-0.901020</td>\n",
       "      <td>-0.736562</td>\n",
       "      <td>-0.950031</td>\n",
       "      <td>0.255202</td>\n",
       "      <td>76</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>255</th>\n",
       "      <td>19.4</td>\n",
       "      <td>0.351127</td>\n",
       "      <td>0.413800</td>\n",
       "      <td>-0.340982</td>\n",
       "      <td>0.293190</td>\n",
       "      <td>0.548737</td>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>13.0</td>\n",
       "      <td>1.526556</td>\n",
       "      <td>1.144256</td>\n",
       "      <td>0.713897</td>\n",
       "      <td>1.339617</td>\n",
       "      <td>-0.625403</td>\n",
       "      <td>72</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>235</th>\n",
       "      <td>30.5</td>\n",
       "      <td>-0.824303</td>\n",
       "      <td>-0.891280</td>\n",
       "      <td>-1.053025</td>\n",
       "      <td>-1.072585</td>\n",
       "      <td>0.475353</td>\n",
       "      <td>77</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>14.0</td>\n",
       "      <td>1.526556</td>\n",
       "      <td>1.563051</td>\n",
       "      <td>1.636916</td>\n",
       "      <td>1.470420</td>\n",
       "      <td>-1.359240</td>\n",
       "      <td>71</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      MPG  Cylinders  Displacement  Horsepower    Weight  Acceleration  \\\n",
       "203  28.0  -0.824303     -0.901020   -0.736562 -0.950031      0.255202   \n",
       "255  19.4   0.351127      0.413800   -0.340982  0.293190      0.548737   \n",
       "72   13.0   1.526556      1.144256    0.713897  1.339617     -0.625403   \n",
       "235  30.5  -0.824303     -0.891280   -1.053025 -1.072585      0.475353   \n",
       "37   14.0   1.526556      1.563051    1.636916  1.470420     -1.359240   \n",
       "\n",
       "     ModelYear  Origin  \n",
       "203         76       3  \n",
       "255         78       1  \n",
       "72          72       1  \n",
       "235         77       1  \n",
       "37          71       1  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "numeric_column_names = ['Cylinders', 'Displacement', 'Horsepower', 'Weight', 'Acceleration']\n",
    "\n",
    "df_train_norm, df_test_norm = df_train.copy(), df_test.copy()\n",
    "\n",
    "for col_name in numeric_column_names:\n",
    "    mean = train_stats.loc[col_name, 'mean']\n",
    "    std  = train_stats.loc[col_name, 'std']\n",
    "    df_train_norm.loc[:, col_name] = (df_train_norm.loc[:, col_name] - mean)/std\n",
    "    df_test_norm.loc[:, col_name] = (df_test_norm.loc[:, col_name] - mean)/std\n",
    "    \n",
    "df_train_norm.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Numeric Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[NumericColumn(key='Cylinders', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None),\n",
       " NumericColumn(key='Displacement', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None),\n",
       " NumericColumn(key='Horsepower', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None),\n",
       " NumericColumn(key='Weight', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None),\n",
       " NumericColumn(key='Acceleration', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None)]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "numeric_features = []\n",
    "\n",
    "for col_name in numeric_column_names:\n",
    "    numeric_features.append(tf.feature_column.numeric_column(key=col_name))\n",
    "    \n",
    "numeric_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[BucketizedColumn(source_column=NumericColumn(key='ModelYear', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), boundaries=(73, 76, 79))]\n"
     ]
    }
   ],
   "source": [
    "feature_year = tf.feature_column.numeric_column(key=\"ModelYear\")\n",
    "\n",
    "bucketized_features = []\n",
    "\n",
    "bucketized_features.append(tf.feature_column.bucketized_column(\n",
    "    source_column=feature_year,\n",
    "    boundaries=[73, 76, 79]))\n",
    "\n",
    "print(bucketized_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='Origin', vocabulary_list=(1, 2, 3), dtype=tf.int64, default_value=-1, num_oov_buckets=0))]\n"
     ]
    }
   ],
   "source": [
    "feature_origin = tf.feature_column.categorical_column_with_vocabulary_list(\n",
    "    key='Origin',\n",
    "    vocabulary_list=[1, 2, 3])\n",
    "\n",
    "categorical_indicator_features = []\n",
    "categorical_indicator_features.append(tf.feature_column.indicator_column(feature_origin))\n",
    "\n",
    "print(categorical_indicator_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Machine learning with pre-made Estimators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keys: dict_keys(['Cylinders', 'Displacement', 'Horsepower', 'Weight', 'Acceleration', 'ModelYear', 'Origin'])\n",
      "Batch Model Years: tf.Tensor([82 78 76 72 78 73 70 78], shape=(8,), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "def train_input_fn(df_train, batch_size=8):\n",
    "    df = df_train.copy()\n",
    "    train_x, train_y = df, df.pop('MPG')\n",
    "    dataset = tf.data.Dataset.from_tensor_slices((dict(train_x), train_y))\n",
    "\n",
    "    # shuffle, repeat, and batch the examples\n",
    "    return dataset.shuffle(1000).repeat().batch(batch_size)\n",
    "\n",
    "## inspection\n",
    "ds = train_input_fn(df_train_norm)\n",
    "batch = next(iter(ds))\n",
    "print('Keys:', batch[0].keys())\n",
    "print('Batch Model Years:', batch[0]['ModelYear'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[NumericColumn(key='Cylinders', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), NumericColumn(key='Displacement', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), NumericColumn(key='Horsepower', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), NumericColumn(key='Weight', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), NumericColumn(key='Acceleration', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), BucketizedColumn(source_column=NumericColumn(key='ModelYear', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), boundaries=(73, 76, 79)), IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='Origin', vocabulary_list=(1, 2, 3), dtype=tf.int64, default_value=-1, num_oov_buckets=0))]\n"
     ]
    }
   ],
   "source": [
    "all_feature_columns = (numeric_features + \n",
    "                       bucketized_features + \n",
    "                       categorical_indicator_features)\n",
    "\n",
    "print(all_feature_columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "INFO:tensorflow:Using config: {'_model_dir': 'models/autompg-dnnregressor/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true\n",
      "graph_options {\n",
      "  rewrite_options {\n",
      "    meta_optimizer_iterations: ONE\n",
      "  }\n",
      "}\n",
      ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fcff2243358>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
     ]
    }
   ],
   "source": [
    "regressor = tf.estimator.DNNRegressor(\n",
    "    feature_columns=all_feature_columns,\n",
    "    hidden_units=[32, 10],\n",
    "    model_dir='models/autompg-dnnregressor/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Steps: 40000\n",
      "WARNING:tensorflow:From /Users/sebastian/miniconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n",
      "WARNING:tensorflow:From /Users/sebastian/miniconda3/lib/python3.7/site-packages/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "WARNING:tensorflow:Layer dnn is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2.  The layer has dtype float32 because it's dtype defaults to floatx.\n",
      "\n",
      "If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n",
      "\n",
      "To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n",
      "\n",
      "WARNING:tensorflow:From /Users/sebastian/miniconda3/lib/python3.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:4276: IndicatorColumn._variable_shape (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n",
      "WARNING:tensorflow:From /Users/sebastian/miniconda3/lib/python3.7/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:4331: VocabularyListCategoricalColumn._num_buckets (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n",
      "WARNING:tensorflow:From /Users/sebastian/miniconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/head/regression_head.py:156: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.cast` instead.\n",
      "WARNING:tensorflow:From /Users/sebastian/miniconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/optimizer_v2/adagrad.py:108: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into models/autompg-dnnregressor/model.ckpt.\n",
      "INFO:tensorflow:loss = 965.9457, step = 0\n",
      "INFO:tensorflow:global_step/sec: 846.167\n",
      "INFO:tensorflow:loss = 435.55515, step = 100 (0.119 sec)\n",
      "INFO:tensorflow:global_step/sec: 1771.39\n",
      "INFO:tensorflow:loss = 486.4853, step = 200 (0.056 sec)\n",
      "INFO:tensorflow:global_step/sec: 1727.5\n",
      "INFO:tensorflow:loss = 355.30582, step = 300 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1743.04\n",
      "INFO:tensorflow:loss = 568.7341, step = 400 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1780.05\n",
      "INFO:tensorflow:loss = 512.99524, step = 500 (0.056 sec)\n",
      "INFO:tensorflow:global_step/sec: 1770.88\n",
      "INFO:tensorflow:loss = 565.2082, step = 600 (0.056 sec)\n",
      "INFO:tensorflow:global_step/sec: 1777.4\n",
      "INFO:tensorflow:loss = 443.9086, step = 700 (0.056 sec)\n",
      "INFO:tensorflow:global_step/sec: 1732.8\n",
      "INFO:tensorflow:loss = 655.3407, step = 800 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1738.68\n",
      "INFO:tensorflow:loss = 484.80676, step = 900 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1758.05\n",
      "INFO:tensorflow:loss = 888.8401, step = 1000 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1728.08\n",
      "INFO:tensorflow:loss = 355.9267, step = 1100 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1717.15\n",
      "INFO:tensorflow:loss = 552.66125, step = 1200 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1755.4\n",
      "INFO:tensorflow:loss = 689.29297, step = 1300 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1702.65\n",
      "INFO:tensorflow:loss = 527.3436, step = 1400 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1607.97\n",
      "INFO:tensorflow:loss = 312.59235, step = 1500 (0.062 sec)\n",
      "INFO:tensorflow:global_step/sec: 1719.49\n",
      "INFO:tensorflow:loss = 492.2902, step = 1600 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1277.77\n",
      "INFO:tensorflow:loss = 515.0305, step = 1700 (0.078 sec)\n",
      "INFO:tensorflow:global_step/sec: 1651.75\n",
      "INFO:tensorflow:loss = 381.9837, step = 1800 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1711.09\n",
      "INFO:tensorflow:loss = 457.89188, step = 1900 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1606.99\n",
      "INFO:tensorflow:loss = 437.80118, step = 2000 (0.062 sec)\n",
      "INFO:tensorflow:global_step/sec: 1669.39\n",
      "INFO:tensorflow:loss = 718.0835, step = 2100 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1642.74\n",
      "INFO:tensorflow:loss = 614.21, step = 2200 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1737.2\n",
      "INFO:tensorflow:loss = 722.5958, step = 2300 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1728.73\n",
      "INFO:tensorflow:loss = 521.09766, step = 2400 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1584.2\n",
      "INFO:tensorflow:loss = 423.88907, step = 2500 (0.063 sec)\n",
      "INFO:tensorflow:global_step/sec: 1444.69\n",
      "INFO:tensorflow:loss = 340.96692, step = 2600 (0.069 sec)\n",
      "INFO:tensorflow:global_step/sec: 1719.84\n",
      "INFO:tensorflow:loss = 379.07098, step = 2700 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1615.35\n",
      "INFO:tensorflow:loss = 367.67584, step = 2800 (0.062 sec)\n",
      "INFO:tensorflow:global_step/sec: 1727.86\n",
      "INFO:tensorflow:loss = 569.1295, step = 2900 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1733.52\n",
      "INFO:tensorflow:loss = 591.14777, step = 3000 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1741.62\n",
      "INFO:tensorflow:loss = 382.24747, step = 3100 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1742.8\n",
      "INFO:tensorflow:loss = 559.0973, step = 3200 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1717.27\n",
      "INFO:tensorflow:loss = 334.82214, step = 3300 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1742.4\n",
      "INFO:tensorflow:loss = 273.33005, step = 3400 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1732.32\n",
      "INFO:tensorflow:loss = 511.62704, step = 3500 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1727.93\n",
      "INFO:tensorflow:loss = 449.68387, step = 3600 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1717.08\n",
      "INFO:tensorflow:loss = 631.39966, step = 3700 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1736.51\n",
      "INFO:tensorflow:loss = 376.1551, step = 3800 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1714.3\n",
      "INFO:tensorflow:loss = 515.2615, step = 3900 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1645.84\n",
      "INFO:tensorflow:loss = 614.921, step = 4000 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1646.91\n",
      "INFO:tensorflow:loss = 517.6301, step = 4100 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1738.55\n",
      "INFO:tensorflow:loss = 427.694, step = 4200 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1682.94\n",
      "INFO:tensorflow:loss = 290.24667, step = 4300 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1592.46\n",
      "INFO:tensorflow:loss = 288.75388, step = 4400 (0.063 sec)\n",
      "INFO:tensorflow:global_step/sec: 1708.5\n",
      "INFO:tensorflow:loss = 292.9913, step = 4500 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1730.88\n",
      "INFO:tensorflow:loss = 408.25854, step = 4600 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1704.75\n",
      "INFO:tensorflow:loss = 327.53836, step = 4700 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1681.91\n",
      "INFO:tensorflow:loss = 281.63214, step = 4800 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1571.71\n",
      "INFO:tensorflow:loss = 386.23837, step = 4900 (0.064 sec)\n",
      "INFO:tensorflow:global_step/sec: 1722.17\n",
      "INFO:tensorflow:loss = 339.5951, step = 5000 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1660.68\n",
      "INFO:tensorflow:loss = 379.98798, step = 5100 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1733.61\n",
      "INFO:tensorflow:loss = 324.04468, step = 5200 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1717.72\n",
      "INFO:tensorflow:loss = 367.2947, step = 5300 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1716.5\n",
      "INFO:tensorflow:loss = 312.40906, step = 5400 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1665.75\n",
      "INFO:tensorflow:loss = 373.8691, step = 5500 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1709.8\n",
      "INFO:tensorflow:loss = 267.36197, step = 5600 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1730.55\n",
      "INFO:tensorflow:loss = 340.65076, step = 5700 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1685.01\n",
      "INFO:tensorflow:loss = 282.98227, step = 5800 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1690.76\n",
      "INFO:tensorflow:loss = 292.66016, step = 5900 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1703.41\n",
      "INFO:tensorflow:loss = 309.35815, step = 6000 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1702.15\n",
      "INFO:tensorflow:loss = 535.7477, step = 6100 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1709\n",
      "INFO:tensorflow:loss = 337.63654, step = 6200 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1705.79\n",
      "INFO:tensorflow:loss = 351.9597, step = 6300 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1696.15\n",
      "INFO:tensorflow:loss = 370.8979, step = 6400 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1624.48\n",
      "INFO:tensorflow:loss = 407.55585, step = 6500 (0.062 sec)\n",
      "INFO:tensorflow:global_step/sec: 1545.62\n",
      "INFO:tensorflow:loss = 348.00806, step = 6600 (0.065 sec)\n",
      "INFO:tensorflow:global_step/sec: 1677.35\n",
      "INFO:tensorflow:loss = 316.6603, step = 6700 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1669.89\n",
      "INFO:tensorflow:loss = 455.61356, step = 6800 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1675.52\n",
      "INFO:tensorflow:loss = 372.7221, step = 6900 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1603.67\n",
      "INFO:tensorflow:loss = 200.061, step = 7000 (0.062 sec)\n",
      "INFO:tensorflow:global_step/sec: 1711.07\n",
      "INFO:tensorflow:loss = 231.79959, step = 7100 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1698.2\n",
      "INFO:tensorflow:loss = 478.46185, step = 7200 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1718.33\n",
      "INFO:tensorflow:loss = 250.39194, step = 7300 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1662.21\n",
      "INFO:tensorflow:loss = 493.37506, step = 7400 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1700.16\n",
      "INFO:tensorflow:loss = 247.21848, step = 7500 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1706.17\n",
      "INFO:tensorflow:loss = 243.20918, step = 7600 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1659.72\n",
      "INFO:tensorflow:loss = 214.57013, step = 7700 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1483.77\n",
      "INFO:tensorflow:loss = 240.91962, step = 7800 (0.067 sec)\n",
      "INFO:tensorflow:global_step/sec: 1697.13\n",
      "INFO:tensorflow:loss = 368.7198, step = 7900 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1709.55\n",
      "INFO:tensorflow:loss = 379.6798, step = 8000 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1717.15\n",
      "INFO:tensorflow:loss = 172.78369, step = 8100 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1699.47\n",
      "INFO:tensorflow:loss = 162.06506, step = 8200 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1727.74\n",
      "INFO:tensorflow:loss = 193.4512, step = 8300 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1711.83\n",
      "INFO:tensorflow:loss = 299.81866, step = 8400 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1714.94\n",
      "INFO:tensorflow:loss = 142.07172, step = 8500 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1726.85\n",
      "INFO:tensorflow:loss = 225.7315, step = 8600 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1670.12\n",
      "INFO:tensorflow:loss = 227.37274, step = 8700 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1702.65\n",
      "INFO:tensorflow:loss = 176.44263, step = 8800 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1641.58\n",
      "INFO:tensorflow:loss = 144.32623, step = 8900 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1725.09\n",
      "INFO:tensorflow:loss = 194.21887, step = 9000 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1707.42\n",
      "INFO:tensorflow:loss = 231.46078, step = 9100 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1703.03\n",
      "INFO:tensorflow:loss = 194.99727, step = 9200 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1721.38\n",
      "INFO:tensorflow:loss = 96.3828, step = 9300 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1711.16\n",
      "INFO:tensorflow:loss = 107.715965, step = 9400 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1715.15\n",
      "INFO:tensorflow:loss = 132.06448, step = 9500 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1659.72\n",
      "INFO:tensorflow:loss = 225.05287, step = 9600 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1688.11\n",
      "INFO:tensorflow:loss = 172.19652, step = 9700 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1689.11\n",
      "INFO:tensorflow:loss = 182.46667, step = 9800 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1686.85\n",
      "INFO:tensorflow:loss = 167.6417, step = 9900 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1662.67\n",
      "INFO:tensorflow:loss = 239.03984, step = 10000 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1649.25\n",
      "INFO:tensorflow:loss = 106.3367, step = 10100 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1696\n",
      "INFO:tensorflow:loss = 117.52605, step = 10200 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1704.05\n",
      "INFO:tensorflow:loss = 144.02449, step = 10300 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1701.49\n",
      "INFO:tensorflow:loss = 168.28322, step = 10400 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1727.44\n",
      "INFO:tensorflow:loss = 217.90193, step = 10500 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1464.84\n",
      "INFO:tensorflow:loss = 185.32053, step = 10600 (0.068 sec)\n",
      "INFO:tensorflow:global_step/sec: 1690.93\n",
      "INFO:tensorflow:loss = 96.76355, step = 10700 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1709.37\n",
      "INFO:tensorflow:loss = 146.97032, step = 10800 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1633.27\n",
      "INFO:tensorflow:loss = 121.82247, step = 10900 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1727.83\n",
      "INFO:tensorflow:loss = 93.18637, step = 11000 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1703.23\n",
      "INFO:tensorflow:loss = 216.08694, step = 11100 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1687.45\n",
      "INFO:tensorflow:loss = 93.29232, step = 11200 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1701.43\n",
      "INFO:tensorflow:loss = 84.61253, step = 11300 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1697.13\n",
      "INFO:tensorflow:loss = 94.3682, step = 11400 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1690.13\n",
      "INFO:tensorflow:loss = 100.659424, step = 11500 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1718.51\n",
      "INFO:tensorflow:loss = 73.45858, step = 11600 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1699.73\n",
      "INFO:tensorflow:loss = 66.8405, step = 11700 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1737.02\n",
      "INFO:tensorflow:loss = 119.53209, step = 11800 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1696.87\n",
      "INFO:tensorflow:loss = 141.3996, step = 11900 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1734.19\n",
      "INFO:tensorflow:loss = 95.78476, step = 12000 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1734.24\n",
      "INFO:tensorflow:loss = 102.20946, step = 12100 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1696.04\n",
      "INFO:tensorflow:loss = 85.47716, step = 12200 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1676.96\n",
      "INFO:tensorflow:loss = 111.60966, step = 12300 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1737.28\n",
      "INFO:tensorflow:loss = 82.85508, step = 12400 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1669.71\n",
      "INFO:tensorflow:loss = 61.08092, step = 12500 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1686.87\n",
      "INFO:tensorflow:loss = 76.87332, step = 12600 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1762.47\n",
      "INFO:tensorflow:loss = 114.90727, step = 12700 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1631.48\n",
      "INFO:tensorflow:loss = 135.31578, step = 12800 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1644.14\n",
      "INFO:tensorflow:loss = 82.34329, step = 12900 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1704.85\n",
      "INFO:tensorflow:loss = 51.209923, step = 13000 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1666.09\n",
      "INFO:tensorflow:loss = 111.13637, step = 13100 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1736.62\n",
      "INFO:tensorflow:loss = 131.1045, step = 13200 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1720.05\n",
      "INFO:tensorflow:loss = 46.694324, step = 13300 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1683.13\n",
      "INFO:tensorflow:loss = 75.09001, step = 13400 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1674.87\n",
      "INFO:tensorflow:loss = 66.24064, step = 13500 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1740.31\n",
      "INFO:tensorflow:loss = 31.624622, step = 13600 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1720.97\n",
      "INFO:tensorflow:loss = 60.937103, step = 13700 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1696.95\n",
      "INFO:tensorflow:loss = 100.58494, step = 13800 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1670.31\n",
      "INFO:tensorflow:loss = 64.88651, step = 13900 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1713.47\n",
      "INFO:tensorflow:loss = 75.23317, step = 14000 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1715.86\n",
      "INFO:tensorflow:loss = 46.643867, step = 14100 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1716.53\n",
      "INFO:tensorflow:loss = 98.56179, step = 14200 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1701.61\n",
      "INFO:tensorflow:loss = 138.51083, step = 14300 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1684.35\n",
      "INFO:tensorflow:loss = 101.08188, step = 14400 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1756.39\n",
      "INFO:tensorflow:loss = 92.856705, step = 14500 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1626.23\n",
      "INFO:tensorflow:loss = 101.42168, step = 14600 (0.062 sec)\n",
      "INFO:tensorflow:global_step/sec: 1726.36\n",
      "INFO:tensorflow:loss = 44.813534, step = 14700 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1702.07\n",
      "INFO:tensorflow:loss = 35.623688, step = 14800 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1662.45\n",
      "INFO:tensorflow:loss = 27.96667, step = 14900 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1451.25\n",
      "INFO:tensorflow:loss = 78.90332, step = 15000 (0.069 sec)\n",
      "INFO:tensorflow:global_step/sec: 1477.08\n",
      "INFO:tensorflow:loss = 54.82135, step = 15100 (0.068 sec)\n",
      "INFO:tensorflow:global_step/sec: 1591.55\n",
      "INFO:tensorflow:loss = 71.83845, step = 15200 (0.063 sec)\n",
      "INFO:tensorflow:global_step/sec: 1716.8\n",
      "INFO:tensorflow:loss = 78.07594, step = 15300 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1670.23\n",
      "INFO:tensorflow:loss = 54.60871, step = 15400 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1713\n",
      "INFO:tensorflow:loss = 62.458935, step = 15500 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1714.23\n",
      "INFO:tensorflow:loss = 43.720818, step = 15600 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1651.39\n",
      "INFO:tensorflow:loss = 80.34233, step = 15700 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1728.7\n",
      "INFO:tensorflow:loss = 40.205956, step = 15800 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1723.25\n",
      "INFO:tensorflow:loss = 53.602367, step = 15900 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1704.95\n",
      "INFO:tensorflow:loss = 36.53252, step = 16000 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1673.19\n",
      "INFO:tensorflow:loss = 32.232708, step = 16100 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1740.37\n",
      "INFO:tensorflow:loss = 37.462452, step = 16200 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1733.31\n",
      "INFO:tensorflow:loss = 21.674019, step = 16300 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1690.02\n",
      "INFO:tensorflow:loss = 16.769125, step = 16400 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1705.67\n",
      "INFO:tensorflow:loss = 57.544434, step = 16500 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1469.83\n",
      "INFO:tensorflow:loss = 40.77007, step = 16600 (0.068 sec)\n",
      "INFO:tensorflow:global_step/sec: 1522.95\n",
      "INFO:tensorflow:loss = 35.78828, step = 16700 (0.066 sec)\n",
      "INFO:tensorflow:global_step/sec: 1599.66\n",
      "INFO:tensorflow:loss = 54.513, step = 16800 (0.063 sec)\n",
      "INFO:tensorflow:global_step/sec: 1650.05\n",
      "INFO:tensorflow:loss = 18.221458, step = 16900 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1697.65\n",
      "INFO:tensorflow:loss = 43.658375, step = 17000 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1724.32\n",
      "INFO:tensorflow:loss = 32.004093, step = 17100 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1675.5\n",
      "INFO:tensorflow:loss = 38.456337, step = 17200 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1698.22\n",
      "INFO:tensorflow:loss = 31.469858, step = 17300 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1666.72\n",
      "INFO:tensorflow:loss = 76.53949, step = 17400 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1691.67\n",
      "INFO:tensorflow:loss = 24.958817, step = 17500 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1696.38\n",
      "INFO:tensorflow:loss = 33.607086, step = 17600 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1708.82\n",
      "INFO:tensorflow:loss = 44.80117, step = 17700 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1699.98\n",
      "INFO:tensorflow:loss = 30.39735, step = 17800 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1464.78\n",
      "INFO:tensorflow:loss = 19.683651, step = 17900 (0.068 sec)\n",
      "INFO:tensorflow:global_step/sec: 1545.91\n",
      "INFO:tensorflow:loss = 25.447193, step = 18000 (0.065 sec)\n",
      "INFO:tensorflow:global_step/sec: 1498.04\n",
      "INFO:tensorflow:loss = 28.694195, step = 18100 (0.067 sec)\n",
      "INFO:tensorflow:global_step/sec: 1657.11\n",
      "INFO:tensorflow:loss = 73.98665, step = 18200 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1691.27\n",
      "INFO:tensorflow:loss = 36.695335, step = 18300 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1710.63\n",
      "INFO:tensorflow:loss = 16.46688, step = 18400 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1733.41\n",
      "INFO:tensorflow:loss = 43.327297, step = 18500 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1702.53\n",
      "INFO:tensorflow:loss = 14.0220375, step = 18600 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1654.81\n",
      "INFO:tensorflow:loss = 13.240676, step = 18700 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1730.34\n",
      "INFO:tensorflow:loss = 27.294521, step = 18800 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1654.6\n",
      "INFO:tensorflow:loss = 19.232815, step = 18900 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1715.7\n",
      "INFO:tensorflow:loss = 22.65117, step = 19000 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1721.14\n",
      "INFO:tensorflow:loss = 31.615631, step = 19100 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1713.91\n",
      "INFO:tensorflow:loss = 32.04344, step = 19200 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1668.22\n",
      "INFO:tensorflow:loss = 25.548805, step = 19300 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1644.14\n",
      "INFO:tensorflow:loss = 25.413357, step = 19400 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1680.28\n",
      "INFO:tensorflow:loss = 25.79448, step = 19500 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1710.72\n",
      "INFO:tensorflow:loss = 22.91647, step = 19600 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1739.7\n",
      "INFO:tensorflow:loss = 23.934977, step = 19700 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1719.37\n",
      "INFO:tensorflow:loss = 14.1135025, step = 19800 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1717.14\n",
      "INFO:tensorflow:loss = 19.30763, step = 19900 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1651.56\n",
      "INFO:tensorflow:loss = 18.60486, step = 20000 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1692.62\n",
      "INFO:tensorflow:loss = 53.408043, step = 20100 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1669.98\n",
      "INFO:tensorflow:loss = 34.334156, step = 20200 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1727.89\n",
      "INFO:tensorflow:loss = 33.181496, step = 20300 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1695.49\n",
      "INFO:tensorflow:loss = 40.705566, step = 20400 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1744.75\n",
      "INFO:tensorflow:loss = 25.2642, step = 20500 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1707.01\n",
      "INFO:tensorflow:loss = 25.285252, step = 20600 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1741.37\n",
      "INFO:tensorflow:loss = 60.51396, step = 20700 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1728.35\n",
      "INFO:tensorflow:loss = 34.346, step = 20800 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1677.11\n",
      "INFO:tensorflow:loss = 30.538233, step = 20900 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1738.71\n",
      "INFO:tensorflow:loss = 17.264736, step = 21000 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1735.09\n",
      "INFO:tensorflow:loss = 35.94033, step = 21100 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1562.33\n",
      "INFO:tensorflow:loss = 29.519213, step = 21200 (0.064 sec)\n",
      "INFO:tensorflow:global_step/sec: 1359.14\n",
      "INFO:tensorflow:loss = 21.723145, step = 21300 (0.074 sec)\n",
      "INFO:tensorflow:global_step/sec: 1647.25\n",
      "INFO:tensorflow:loss = 22.438433, step = 21400 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1627.66\n",
      "INFO:tensorflow:loss = 26.390743, step = 21500 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1723.81\n",
      "INFO:tensorflow:loss = 78.44823, step = 21600 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1655.44\n",
      "INFO:tensorflow:loss = 27.885046, step = 21700 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1658.43\n",
      "INFO:tensorflow:loss = 27.214973, step = 21800 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1692.59\n",
      "INFO:tensorflow:loss = 23.467308, step = 21900 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1717.95\n",
      "INFO:tensorflow:loss = 25.617788, step = 22000 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1629.03\n",
      "INFO:tensorflow:loss = 19.72428, step = 22100 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1700.42\n",
      "INFO:tensorflow:loss = 55.037613, step = 22200 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1710.42\n",
      "INFO:tensorflow:loss = 67.73604, step = 22300 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1712.74\n",
      "INFO:tensorflow:loss = 26.427261, step = 22400 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1707.42\n",
      "INFO:tensorflow:loss = 59.923656, step = 22500 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1711.86\n",
      "INFO:tensorflow:loss = 26.327465, step = 22600 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1684.24\n",
      "INFO:tensorflow:loss = 36.743195, step = 22700 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1750.89\n",
      "INFO:tensorflow:loss = 30.160175, step = 22800 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1721.71\n",
      "INFO:tensorflow:loss = 36.566135, step = 22900 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1664.44\n",
      "INFO:tensorflow:loss = 9.562983, step = 23000 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1727.86\n",
      "INFO:tensorflow:loss = 17.603941, step = 23100 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1732.63\n",
      "INFO:tensorflow:loss = 10.009632, step = 23200 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1721.5\n",
      "INFO:tensorflow:loss = 21.46733, step = 23300 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1682.54\n",
      "INFO:tensorflow:loss = 20.99877, step = 23400 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1640.6\n",
      "INFO:tensorflow:loss = 65.844574, step = 23500 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1580.68\n",
      "INFO:tensorflow:loss = 20.14148, step = 23600 (0.063 sec)\n",
      "INFO:tensorflow:global_step/sec: 1713.8\n",
      "INFO:tensorflow:loss = 13.220474, step = 23700 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1721.14\n",
      "INFO:tensorflow:loss = 34.298286, step = 23800 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1728.9\n",
      "INFO:tensorflow:loss = 13.647365, step = 23900 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1586.72\n",
      "INFO:tensorflow:loss = 9.190278, step = 24000 (0.063 sec)\n",
      "INFO:tensorflow:global_step/sec: 1715.45\n",
      "INFO:tensorflow:loss = 24.68869, step = 24100 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1689.53\n",
      "INFO:tensorflow:loss = 23.603409, step = 24200 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1707.6\n",
      "INFO:tensorflow:loss = 45.551582, step = 24300 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1704.16\n",
      "INFO:tensorflow:loss = 12.447203, step = 24400 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1684.8\n",
      "INFO:tensorflow:loss = 14.388153, step = 24500 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1710.05\n",
      "INFO:tensorflow:loss = 25.630888, step = 24600 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1692.02\n",
      "INFO:tensorflow:loss = 10.933075, step = 24700 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1715.97\n",
      "INFO:tensorflow:loss = 7.9647584, step = 24800 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1755.53\n",
      "INFO:tensorflow:loss = 23.671944, step = 24900 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1735.66\n",
      "INFO:tensorflow:loss = 36.395546, step = 25000 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1716.35\n",
      "INFO:tensorflow:loss = 90.33572, step = 25100 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1742.04\n",
      "INFO:tensorflow:loss = 19.85579, step = 25200 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1703.23\n",
      "INFO:tensorflow:loss = 32.168686, step = 25300 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1680.7\n",
      "INFO:tensorflow:loss = 49.993244, step = 25400 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1662.46\n",
      "INFO:tensorflow:loss = 10.653378, step = 25500 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1633.64\n",
      "INFO:tensorflow:loss = 21.233597, step = 25600 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1647.83\n",
      "INFO:tensorflow:loss = 29.218864, step = 25700 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1696.66\n",
      "INFO:tensorflow:loss = 21.293818, step = 25800 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1705.62\n",
      "INFO:tensorflow:loss = 15.644426, step = 25900 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1684.24\n",
      "INFO:tensorflow:loss = 11.822701, step = 26000 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1713.62\n",
      "INFO:tensorflow:loss = 26.101885, step = 26100 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1713.15\n",
      "INFO:tensorflow:loss = 16.975256, step = 26200 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1613.71\n",
      "INFO:tensorflow:loss = 16.675241, step = 26300 (0.062 sec)\n",
      "INFO:tensorflow:global_step/sec: 1720.7\n",
      "INFO:tensorflow:loss = 5.992489, step = 26400 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1727.89\n",
      "INFO:tensorflow:loss = 22.450819, step = 26500 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1739.77\n",
      "INFO:tensorflow:loss = 24.40264, step = 26600 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1723.77\n",
      "INFO:tensorflow:loss = 56.10045, step = 26700 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1703.44\n",
      "INFO:tensorflow:loss = 18.507614, step = 26800 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1706.19\n",
      "INFO:tensorflow:loss = 22.865948, step = 26900 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1715.35\n",
      "INFO:tensorflow:loss = 11.989531, step = 27000 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1637.52\n",
      "INFO:tensorflow:loss = 18.527515, step = 27100 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1690.01\n",
      "INFO:tensorflow:loss = 23.829687, step = 27200 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1632.07\n",
      "INFO:tensorflow:loss = 22.312756, step = 27300 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1666.47\n",
      "INFO:tensorflow:loss = 58.509857, step = 27400 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1676.59\n",
      "INFO:tensorflow:loss = 41.749958, step = 27500 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1678.11\n",
      "INFO:tensorflow:loss = 29.38898, step = 27600 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1632.25\n",
      "INFO:tensorflow:loss = 23.380291, step = 27700 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1658.49\n",
      "INFO:tensorflow:loss = 29.288826, step = 27800 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1704.25\n",
      "INFO:tensorflow:loss = 29.598656, step = 27900 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1677.25\n",
      "INFO:tensorflow:loss = 12.27314, step = 28000 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1587.56\n",
      "INFO:tensorflow:loss = 14.931431, step = 28100 (0.063 sec)\n",
      "INFO:tensorflow:global_step/sec: 1711.88\n",
      "INFO:tensorflow:loss = 24.772148, step = 28200 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1701.32\n",
      "INFO:tensorflow:loss = 18.881832, step = 28300 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1701.45\n",
      "INFO:tensorflow:loss = 4.597565, step = 28400 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1693.8\n",
      "INFO:tensorflow:loss = 17.753885, step = 28500 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1723.19\n",
      "INFO:tensorflow:loss = 13.964875, step = 28600 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1740.89\n",
      "INFO:tensorflow:loss = 23.599941, step = 28700 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1732.3\n",
      "INFO:tensorflow:loss = 30.815712, step = 28800 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1696.35\n",
      "INFO:tensorflow:loss = 7.126786, step = 28900 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1677.29\n",
      "INFO:tensorflow:loss = 10.343699, step = 29000 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1708.52\n",
      "INFO:tensorflow:loss = 11.170483, step = 29100 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1701.46\n",
      "INFO:tensorflow:loss = 12.889736, step = 29200 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1718.04\n",
      "INFO:tensorflow:loss = 30.62483, step = 29300 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1683.76\n",
      "INFO:tensorflow:loss = 18.144823, step = 29400 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1656.43\n",
      "INFO:tensorflow:loss = 18.740158, step = 29500 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1685.54\n",
      "INFO:tensorflow:loss = 20.274807, step = 29600 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1642.84\n",
      "INFO:tensorflow:loss = 25.71187, step = 29700 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1366.31\n",
      "INFO:tensorflow:loss = 9.564793, step = 29800 (0.073 sec)\n",
      "INFO:tensorflow:global_step/sec: 1554.7\n",
      "INFO:tensorflow:loss = 25.677887, step = 29900 (0.064 sec)\n",
      "INFO:tensorflow:global_step/sec: 1586.85\n",
      "INFO:tensorflow:loss = 25.293968, step = 30000 (0.063 sec)\n",
      "INFO:tensorflow:global_step/sec: 1619.17\n",
      "INFO:tensorflow:loss = 21.639647, step = 30100 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1637.01\n",
      "INFO:tensorflow:loss = 13.2285795, step = 30200 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1687.05\n",
      "INFO:tensorflow:loss = 7.3189907, step = 30300 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1688.45\n",
      "INFO:tensorflow:loss = 8.997917, step = 30400 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1704.72\n",
      "INFO:tensorflow:loss = 15.5277605, step = 30500 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1675.34\n",
      "INFO:tensorflow:loss = 18.557573, step = 30600 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1696.42\n",
      "INFO:tensorflow:loss = 15.780535, step = 30700 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1737.82\n",
      "INFO:tensorflow:loss = 61.81815, step = 30800 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1762.33\n",
      "INFO:tensorflow:loss = 10.845165, step = 30900 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1763.79\n",
      "INFO:tensorflow:loss = 15.872068, step = 31000 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1716.86\n",
      "INFO:tensorflow:loss = 12.260029, step = 31100 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1707.21\n",
      "INFO:tensorflow:loss = 17.910793, step = 31200 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1634.61\n",
      "INFO:tensorflow:loss = 13.206525, step = 31300 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1543.85\n",
      "INFO:tensorflow:loss = 14.904988, step = 31400 (0.065 sec)\n",
      "INFO:tensorflow:global_step/sec: 1762.55\n",
      "INFO:tensorflow:loss = 12.759709, step = 31500 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1706.4\n",
      "INFO:tensorflow:loss = 10.563314, step = 31600 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1686.74\n",
      "INFO:tensorflow:loss = 16.083445, step = 31700 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1651.61\n",
      "INFO:tensorflow:loss = 9.154925, step = 31800 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1688.37\n",
      "INFO:tensorflow:loss = 25.338978, step = 31900 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1687.39\n",
      "INFO:tensorflow:loss = 27.875525, step = 32000 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1725.57\n",
      "INFO:tensorflow:loss = 12.497484, step = 32100 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1698.83\n",
      "INFO:tensorflow:loss = 41.029713, step = 32200 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1714.35\n",
      "INFO:tensorflow:loss = 12.936128, step = 32300 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1678.64\n",
      "INFO:tensorflow:loss = 8.013531, step = 32400 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1735.03\n",
      "INFO:tensorflow:loss = 15.885679, step = 32500 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1710.45\n",
      "INFO:tensorflow:loss = 9.087262, step = 32600 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1711.1\n",
      "INFO:tensorflow:loss = 10.564496, step = 32700 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1641.66\n",
      "INFO:tensorflow:loss = 16.833069, step = 32800 (0.061 sec)\n",
      "INFO:tensorflow:global_step/sec: 1604.49\n",
      "INFO:tensorflow:loss = 3.2030706, step = 32900 (0.062 sec)\n",
      "INFO:tensorflow:global_step/sec: 1658.54\n",
      "INFO:tensorflow:loss = 21.9673, step = 33000 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1675.66\n",
      "INFO:tensorflow:loss = 50.941628, step = 33100 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1723.22\n",
      "INFO:tensorflow:loss = 8.902422, step = 33200 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1737.23\n",
      "INFO:tensorflow:loss = 9.243065, step = 33300 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1696.09\n",
      "INFO:tensorflow:loss = 19.546772, step = 33400 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1393.26\n",
      "INFO:tensorflow:loss = 6.3463593, step = 33500 (0.071 sec)\n",
      "INFO:tensorflow:global_step/sec: 1685.35\n",
      "INFO:tensorflow:loss = 23.06793, step = 33600 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1691.7\n",
      "INFO:tensorflow:loss = 12.391812, step = 33700 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1693.29\n",
      "INFO:tensorflow:loss = 25.455618, step = 33800 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1701.54\n",
      "INFO:tensorflow:loss = 27.495245, step = 33900 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1706.66\n",
      "INFO:tensorflow:loss = 25.125893, step = 34000 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1687.51\n",
      "INFO:tensorflow:loss = 10.645081, step = 34100 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1729.45\n",
      "INFO:tensorflow:loss = 8.829086, step = 34200 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1720.19\n",
      "INFO:tensorflow:loss = 12.165992, step = 34300 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1738.05\n",
      "INFO:tensorflow:loss = 18.061028, step = 34400 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1743.65\n",
      "INFO:tensorflow:loss = 16.256449, step = 34500 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1734.15\n",
      "INFO:tensorflow:loss = 14.68507, step = 34600 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1746.61\n",
      "INFO:tensorflow:loss = 8.875821, step = 34700 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1756.42\n",
      "INFO:tensorflow:loss = 12.63289, step = 34800 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1740.7\n",
      "INFO:tensorflow:loss = 5.436164, step = 34900 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1660.6\n",
      "INFO:tensorflow:loss = 6.5834293, step = 35000 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1710.05\n",
      "INFO:tensorflow:loss = 7.114298, step = 35100 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1721.38\n",
      "INFO:tensorflow:loss = 14.999966, step = 35200 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1760.75\n",
      "INFO:tensorflow:loss = 22.257206, step = 35300 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1759.26\n",
      "INFO:tensorflow:loss = 4.1086226, step = 35400 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1730.79\n",
      "INFO:tensorflow:loss = 26.967918, step = 35500 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1761.07\n",
      "INFO:tensorflow:loss = 8.1224785, step = 35600 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1748.06\n",
      "INFO:tensorflow:loss = 11.74909, step = 35700 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1764.07\n",
      "INFO:tensorflow:loss = 7.3952994, step = 35800 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1715.12\n",
      "INFO:tensorflow:loss = 22.704182, step = 35900 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1688.31\n",
      "INFO:tensorflow:loss = 6.311548, step = 36000 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1736.47\n",
      "INFO:tensorflow:loss = 8.6354885, step = 36100 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1771.86\n",
      "INFO:tensorflow:loss = 16.167746, step = 36200 (0.056 sec)\n",
      "INFO:tensorflow:global_step/sec: 1695.21\n",
      "INFO:tensorflow:loss = 5.6035967, step = 36300 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1735.38\n",
      "INFO:tensorflow:loss = 13.955162, step = 36400 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1722.83\n",
      "INFO:tensorflow:loss = 8.607983, step = 36500 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1723.36\n",
      "INFO:tensorflow:loss = 16.449093, step = 36600 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1753.63\n",
      "INFO:tensorflow:loss = 9.816775, step = 36700 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1707.06\n",
      "INFO:tensorflow:loss = 17.795435, step = 36800 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1724.44\n",
      "INFO:tensorflow:loss = 13.82176, step = 36900 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1744.74\n",
      "INFO:tensorflow:loss = 18.744055, step = 37000 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1706.37\n",
      "INFO:tensorflow:loss = 9.39481, step = 37100 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1740.58\n",
      "INFO:tensorflow:loss = 14.207376, step = 37200 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1736.54\n",
      "INFO:tensorflow:loss = 19.826979, step = 37300 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1765.94\n",
      "INFO:tensorflow:loss = 42.82961, step = 37400 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1731.09\n",
      "INFO:tensorflow:loss = 11.905295, step = 37500 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1732.44\n",
      "INFO:tensorflow:loss = 13.5831585, step = 37600 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1731.78\n",
      "INFO:tensorflow:loss = 50.488914, step = 37700 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1721.09\n",
      "INFO:tensorflow:loss = 12.250552, step = 37800 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1695.92\n",
      "INFO:tensorflow:loss = 5.4371953, step = 37900 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1742.15\n",
      "INFO:tensorflow:loss = 21.536736, step = 38000 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1658.04\n",
      "INFO:tensorflow:loss = 11.87853, step = 38100 (0.060 sec)\n",
      "INFO:tensorflow:global_step/sec: 1736.44\n",
      "INFO:tensorflow:loss = 6.905135, step = 38200 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1722.8\n",
      "INFO:tensorflow:loss = 4.582919, step = 38300 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1745.57\n",
      "INFO:tensorflow:loss = 13.756659, step = 38400 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1736.45\n",
      "INFO:tensorflow:loss = 7.1715574, step = 38500 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1713.53\n",
      "INFO:tensorflow:loss = 3.4536357, step = 38600 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1729.14\n",
      "INFO:tensorflow:loss = 8.142284, step = 38700 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1721.14\n",
      "INFO:tensorflow:loss = 13.480332, step = 38800 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1728.94\n",
      "INFO:tensorflow:loss = 3.3697133, step = 38900 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1748.59\n",
      "INFO:tensorflow:loss = 17.561552, step = 39000 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1733.43\n",
      "INFO:tensorflow:loss = 10.607781, step = 39100 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1752.14\n",
      "INFO:tensorflow:loss = 16.86771, step = 39200 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1696.53\n",
      "INFO:tensorflow:loss = 12.833585, step = 39300 (0.059 sec)\n",
      "INFO:tensorflow:global_step/sec: 1731.6\n",
      "INFO:tensorflow:loss = 4.351342, step = 39400 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1757.13\n",
      "INFO:tensorflow:loss = 7.428226, step = 39500 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1774.78\n",
      "INFO:tensorflow:loss = 17.150429, step = 39600 (0.056 sec)\n",
      "INFO:tensorflow:global_step/sec: 1737.77\n",
      "INFO:tensorflow:loss = 19.651794, step = 39700 (0.058 sec)\n",
      "INFO:tensorflow:global_step/sec: 1745.93\n",
      "INFO:tensorflow:loss = 18.86742, step = 39800 (0.057 sec)\n",
      "INFO:tensorflow:global_step/sec: 1731.27\n",
      "INFO:tensorflow:loss = 10.241472, step = 39900 (0.058 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 40000 into models/autompg-dnnregressor/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 7.7539473.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow_estimator.python.estimator.canned.dnn.DNNRegressorV2 at 0x7fcfd08c6ef0>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "EPOCHS = 1000\n",
    "BATCH_SIZE = 8\n",
    "total_steps = EPOCHS * int(np.ceil(len(df_train) / BATCH_SIZE))\n",
    "print('Training Steps:', total_steps)\n",
    "\n",
    "regressor.train(\n",
    "    input_fn=lambda:train_input_fn(df_train_norm, batch_size=BATCH_SIZE),\n",
    "    steps=total_steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "INFO:tensorflow:Using config: {'_model_dir': 'models/autompg-dnnregressor/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true\n",
      "graph_options {\n",
      "  rewrite_options {\n",
      "    meta_optimizer_iterations: ONE\n",
      "  }\n",
      "}\n",
      ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fcfb16a71d0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
     ]
    }
   ],
   "source": [
    "reloaded_regressor = tf.estimator.DNNRegressor(\n",
    "    feature_columns=all_feature_columns,\n",
    "    hidden_units=[32, 10],\n",
    "    warm_start_from='models/autompg-dnnregressor/',\n",
    "    model_dir='models/autompg-dnnregressor/')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n",
      "WARNING:tensorflow:Layer dnn is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2.  The layer has dtype float32 because it's dtype defaults to floatx.\n",
      "\n",
      "If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n",
      "\n",
      "To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n",
      "\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-10-29T21:16:01Z\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from models/autompg-dnnregressor/model.ckpt-40000\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-10-29-21:16:01\n",
      "INFO:tensorflow:Saving dict for global step 40000: average_loss = 21.015877, global_step = 40000, label/mean = 23.611393, loss = 20.888556, prediction/mean = 21.884321\n",
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 40000: models/autompg-dnnregressor/model.ckpt-40000\n",
      "average_loss    21.01587677001953\n",
      "label/mean      23.611392974853516\n",
      "loss            20.8885555267334\n",
      "prediction/mean 21.884321212768555\n",
      "global_step     40000\n",
      "Average-Loss 21.0159\n"
     ]
    }
   ],
   "source": [
    "def eval_input_fn(df_test, batch_size=8):\n",
    "    df = df_test.copy()\n",
    "    test_x, test_y = df, df.pop('MPG')\n",
    "    dataset = tf.data.Dataset.from_tensor_slices((dict(test_x), test_y))\n",
    "\n",
    "    return dataset.batch(batch_size)\n",
    "\n",
    "eval_results = reloaded_regressor.evaluate(\n",
    "    input_fn=lambda:eval_input_fn(df_test_norm, batch_size=8))\n",
    "\n",
    "for key in eval_results:\n",
    "    print('{:15s} {}'.format(key, eval_results[key]))\n",
    "    \n",
    "print('Average-Loss {:.4f}'.format(eval_results['average_loss']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n",
      "WARNING:tensorflow:Layer dnn is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2.  The layer has dtype float32 because it's dtype defaults to floatx.\n",
      "\n",
      "If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n",
      "\n",
      "To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n",
      "\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from models/autompg-dnnregressor/model.ckpt-40000\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "{'predictions': array([24.132158], dtype=float32)}\n"
     ]
    }
   ],
   "source": [
    "pred_res = regressor.predict(input_fn=lambda: eval_input_fn(df_test_norm, batch_size=8))\n",
    "\n",
    "print(next(iter(pred_res)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Boosted Tree Regressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "WARNING:tensorflow:Using temporary folder as model directory: /var/folders/jg/tpqyh1fd5js5wsr1d138k3n40000gn/T/tmpoibe_re7\n",
      "INFO:tensorflow:Using config: {'_model_dir': '/var/folders/jg/tpqyh1fd5js5wsr1d138k3n40000gn/T/tmpoibe_re7', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true\n",
      "graph_options {\n",
      "  rewrite_options {\n",
      "    meta_optimizer_iterations: ONE\n",
      "  }\n",
      "}\n",
      ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fcfa072e080>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "WARNING:tensorflow:From /Users/sebastian/miniconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/boosted_trees.py:214: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.cast` instead.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "WARNING:tensorflow:Issue encountered when serializing resources.\n",
      "Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.\n",
      "'_Resource' object has no attribute 'name'\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "WARNING:tensorflow:Issue encountered when serializing resources.\n",
      "Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.\n",
      "'_Resource' object has no attribute 'name'\n",
      "INFO:tensorflow:Saving checkpoints for 0 into /var/folders/jg/tpqyh1fd5js5wsr1d138k3n40000gn/T/tmpoibe_re7/model.ckpt.\n",
      "WARNING:tensorflow:Issue encountered when serializing resources.\n",
      "Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.\n",
      "'_Resource' object has no attribute 'name'\n",
      "INFO:tensorflow:loss = 537.5112, step = 0\n",
      "WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.\n",
      "WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.\n",
      "WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.\n",
      "WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.\n",
      "WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.\n",
      "INFO:tensorflow:loss = 310.72693, step = 80 (0.282 sec)\n",
      "INFO:tensorflow:global_step/sec: 247.848\n",
      "INFO:tensorflow:loss = 111.53148, step = 180 (0.205 sec)\n",
      "INFO:tensorflow:global_step/sec: 942.979\n",
      "INFO:tensorflow:loss = 63.24356, step = 280 (0.111 sec)\n",
      "INFO:tensorflow:global_step/sec: 924.163\n",
      "INFO:tensorflow:loss = 10.18588, step = 380 (0.112 sec)\n",
      "INFO:tensorflow:global_step/sec: 893.822\n",
      "INFO:tensorflow:loss = 3.1144676, step = 480 (0.113 sec)\n",
      "INFO:tensorflow:global_step/sec: 858.769\n",
      "INFO:tensorflow:loss = 2.7045398, step = 580 (0.099 sec)\n",
      "INFO:tensorflow:global_step/sec: 930.655\n",
      "INFO:tensorflow:loss = 2.359455, step = 680 (0.107 sec)\n",
      "INFO:tensorflow:global_step/sec: 977.919\n",
      "INFO:tensorflow:loss = 1.143165, step = 780 (0.110 sec)\n",
      "INFO:tensorflow:global_step/sec: 945.332\n",
      "INFO:tensorflow:loss = 2.290662, step = 880 (0.113 sec)\n",
      "INFO:tensorflow:global_step/sec: 891.957\n",
      "INFO:tensorflow:loss = 0.6972606, step = 980 (0.115 sec)\n",
      "INFO:tensorflow:global_step/sec: 876.355\n",
      "INFO:tensorflow:loss = 4.0084243, step = 1080 (0.112 sec)\n",
      "INFO:tensorflow:global_step/sec: 873.761\n",
      "INFO:tensorflow:loss = 2.0189037, step = 1180 (0.101 sec)\n",
      "INFO:tensorflow:global_step/sec: 910.43\n",
      "INFO:tensorflow:loss = 3.7696376, step = 1280 (0.106 sec)\n",
      "INFO:tensorflow:global_step/sec: 992.566\n",
      "INFO:tensorflow:loss = 1.4588101, step = 1380 (0.111 sec)\n",
      "INFO:tensorflow:global_step/sec: 941.808\n",
      "INFO:tensorflow:loss = 3.6786046, step = 1480 (0.114 sec)\n",
      "INFO:tensorflow:global_step/sec: 885.455\n",
      "INFO:tensorflow:loss = 2.9153657, step = 1580 (0.112 sec)\n",
      "INFO:tensorflow:global_step/sec: 889.436\n",
      "INFO:tensorflow:loss = 2.9118788, step = 1680 (0.115 sec)\n",
      "INFO:tensorflow:global_step/sec: 838.757\n",
      "INFO:tensorflow:loss = 1.1475714, step = 1780 (0.104 sec)\n",
      "INFO:tensorflow:global_step/sec: 884.087\n",
      "INFO:tensorflow:loss = 0.25348487, step = 1880 (0.111 sec)\n",
      "INFO:tensorflow:global_step/sec: 970.214\n",
      "INFO:tensorflow:loss = 1.1675123, step = 1980 (0.115 sec)\n",
      "INFO:tensorflow:global_step/sec: 888.786\n",
      "INFO:tensorflow:loss = 1.0550041, step = 2080 (0.118 sec)\n",
      "INFO:tensorflow:global_step/sec: 862.642\n",
      "INFO:tensorflow:loss = 1.143069, step = 2180 (0.112 sec)\n",
      "INFO:tensorflow:global_step/sec: 895.551\n",
      "INFO:tensorflow:loss = 0.35377735, step = 2280 (0.110 sec)\n",
      "INFO:tensorflow:global_step/sec: 876.333\n",
      "INFO:tensorflow:loss = 2.2801447, step = 2380 (0.106 sec)\n",
      "INFO:tensorflow:global_step/sec: 855.031\n",
      "INFO:tensorflow:loss = 1.3852522, step = 2480 (0.117 sec)\n",
      "INFO:tensorflow:global_step/sec: 909.015\n",
      "INFO:tensorflow:loss = 2.0988648, step = 2580 (0.115 sec)\n",
      "INFO:tensorflow:global_step/sec: 913.209\n",
      "INFO:tensorflow:loss = 0.98262966, step = 2680 (0.116 sec)\n",
      "INFO:tensorflow:global_step/sec: 873.904\n",
      "INFO:tensorflow:loss = 0.73869824, step = 2780 (0.117 sec)\n",
      "INFO:tensorflow:global_step/sec: 858.85\n",
      "INFO:tensorflow:loss = 0.72617865, step = 2880 (0.112 sec)\n",
      "INFO:tensorflow:global_step/sec: 860.347\n",
      "INFO:tensorflow:loss = 1.8807092, step = 2980 (0.108 sec)\n",
      "INFO:tensorflow:global_step/sec: 839.546\n",
      "INFO:tensorflow:loss = 1.21948, step = 3080 (0.114 sec)\n",
      "INFO:tensorflow:global_step/sec: 928.488\n",
      "INFO:tensorflow:loss = 0.18839535, step = 3180 (0.117 sec)\n",
      "INFO:tensorflow:global_step/sec: 896.096\n",
      "INFO:tensorflow:loss = 1.800826, step = 3280 (0.119 sec)\n",
      "INFO:tensorflow:global_step/sec: 850.234\n",
      "INFO:tensorflow:loss = 1.0338854, step = 3380 (0.115 sec)\n",
      "INFO:tensorflow:global_step/sec: 871.194\n",
      "INFO:tensorflow:loss = 0.7889396, step = 3480 (0.116 sec)\n",
      "INFO:tensorflow:global_step/sec: 837.956\n",
      "INFO:tensorflow:loss = 0.66842735, step = 3580 (0.106 sec)\n",
      "INFO:tensorflow:global_step/sec: 870.011\n",
      "INFO:tensorflow:loss = 0.9957332, step = 3680 (0.110 sec)\n",
      "INFO:tensorflow:global_step/sec: 919.245\n",
      "INFO:tensorflow:loss = 0.12767088, step = 3780 (0.121 sec)\n",
      "INFO:tensorflow:global_step/sec: 890.838\n",
      "INFO:tensorflow:loss = 0.3221328, step = 3880 (0.120 sec)\n",
      "INFO:tensorflow:global_step/sec: 845.685\n",
      "INFO:tensorflow:loss = 0.37988144, step = 3980 (0.121 sec)\n",
      "INFO:tensorflow:global_step/sec: 825.226\n",
      "INFO:tensorflow:loss = 0.35624224, step = 4080 (0.117 sec)\n",
      "INFO:tensorflow:global_step/sec: 796.394\n",
      "INFO:tensorflow:loss = 0.20606172, step = 4180 (0.115 sec)\n",
      "INFO:tensorflow:global_step/sec: 841.412\n",
      "INFO:tensorflow:loss = 0.40514332, step = 4280 (0.114 sec)\n",
      "INFO:tensorflow:global_step/sec: 917.725\n",
      "INFO:tensorflow:loss = 0.48485115, step = 4380 (0.125 sec)\n",
      "INFO:tensorflow:global_step/sec: 839.243\n",
      "INFO:tensorflow:loss = 0.16458231, step = 4480 (0.122 sec)\n",
      "INFO:tensorflow:global_step/sec: 827.267\n",
      "INFO:tensorflow:loss = 0.67558205, step = 4580 (0.120 sec)\n",
      "INFO:tensorflow:global_step/sec: 841.034\n",
      "INFO:tensorflow:loss = 0.24442446, step = 4680 (0.120 sec)\n",
      "INFO:tensorflow:global_step/sec: 811.912\n",
      "INFO:tensorflow:loss = 0.7619127, step = 4780 (0.108 sec)\n",
      "INFO:tensorflow:global_step/sec: 861.061\n",
      "INFO:tensorflow:loss = 0.36964735, step = 4880 (0.115 sec)\n",
      "INFO:tensorflow:global_step/sec: 902.504\n",
      "INFO:tensorflow:loss = 0.43948948, step = 4980 (0.122 sec)\n",
      "INFO:tensorflow:global_step/sec: 857.096\n",
      "INFO:tensorflow:loss = 0.31855115, step = 5080 (0.128 sec)\n",
      "INFO:tensorflow:global_step/sec: 758.996\n",
      "INFO:tensorflow:loss = 0.22875834, step = 5180 (0.144 sec)\n",
      "INFO:tensorflow:global_step/sec: 686.092\n",
      "INFO:tensorflow:loss = 0.21667434, step = 5280 (0.125 sec)\n",
      "INFO:tensorflow:global_step/sec: 823.601\n",
      "INFO:tensorflow:loss = 0.24828078, step = 5380 (0.110 sec)\n",
      "INFO:tensorflow:global_step/sec: 829.965\n",
      "INFO:tensorflow:loss = 0.05486358, step = 5480 (0.116 sec)\n",
      "INFO:tensorflow:global_step/sec: 909.332\n",
      "INFO:tensorflow:loss = 0.333002, step = 5580 (0.122 sec)\n",
      "INFO:tensorflow:global_step/sec: 858.788\n",
      "INFO:tensorflow:loss = 0.22008666, step = 5680 (0.122 sec)\n",
      "INFO:tensorflow:global_step/sec: 819.751\n",
      "INFO:tensorflow:loss = 0.114844896, step = 5780 (0.127 sec)\n",
      "INFO:tensorflow:global_step/sec: 763.897\n",
      "INFO:tensorflow:loss = 0.26149094, step = 5880 (0.126 sec)\n",
      "INFO:tensorflow:global_step/sec: 805.211\n",
      "INFO:tensorflow:loss = 0.44472122, step = 5980 (0.111 sec)\n",
      "INFO:tensorflow:global_step/sec: 823.431\n",
      "INFO:tensorflow:loss = 0.17810586, step = 6080 (0.118 sec)\n",
      "INFO:tensorflow:global_step/sec: 894.279\n",
      "INFO:tensorflow:loss = 0.13973756, step = 6180 (0.121 sec)\n",
      "INFO:tensorflow:global_step/sec: 853.948\n",
      "INFO:tensorflow:loss = 0.16352013, step = 6280 (0.128 sec)\n",
      "INFO:tensorflow:global_step/sec: 795.608\n",
      "INFO:tensorflow:loss = 0.16917916, step = 6380 (0.124 sec)\n",
      "INFO:tensorflow:global_step/sec: 802.061\n",
      "INFO:tensorflow:loss = 0.2607274, step = 6480 (0.121 sec)\n",
      "INFO:tensorflow:global_step/sec: 809.258\n",
      "INFO:tensorflow:loss = 0.10278683, step = 6580 (0.113 sec)\n",
      "INFO:tensorflow:global_step/sec: 807.056\n",
      "INFO:tensorflow:loss = 0.07342141, step = 6680 (0.120 sec)\n",
      "INFO:tensorflow:global_step/sec: 882.448\n",
      "INFO:tensorflow:loss = 0.17932981, step = 6780 (0.124 sec)\n",
      "INFO:tensorflow:global_step/sec: 834.578\n",
      "INFO:tensorflow:loss = 0.10977276, step = 6880 (0.125 sec)\n",
      "INFO:tensorflow:global_step/sec: 809.913\n",
      "INFO:tensorflow:loss = 0.13595888, step = 6980 (0.126 sec)\n",
      "INFO:tensorflow:global_step/sec: 796.058\n",
      "INFO:tensorflow:loss = 0.20784737, step = 7080 (0.122 sec)\n",
      "INFO:tensorflow:global_step/sec: 791.823\n",
      "INFO:tensorflow:loss = 0.12233504, step = 7180 (0.114 sec)\n",
      "INFO:tensorflow:global_step/sec: 807.011\n",
      "INFO:tensorflow:loss = 0.06943629, step = 7280 (0.124 sec)\n",
      "INFO:tensorflow:global_step/sec: 862.805\n",
      "INFO:tensorflow:loss = 0.23862602, step = 7380 (0.127 sec)\n",
      "INFO:tensorflow:global_step/sec: 807.689\n",
      "INFO:tensorflow:loss = 0.087960616, step = 7480 (0.124 sec)\n",
      "INFO:tensorflow:global_step/sec: 813.226\n",
      "INFO:tensorflow:loss = 0.024548963, step = 7580 (0.129 sec)\n",
      "INFO:tensorflow:global_step/sec: 776.778\n",
      "INFO:tensorflow:loss = 0.14796612, step = 7680 (0.126 sec)\n",
      "INFO:tensorflow:global_step/sec: 772.105\n",
      "INFO:tensorflow:loss = 0.24338263, step = 7780 (0.120 sec)\n",
      "INFO:tensorflow:global_step/sec: 780.013\n",
      "INFO:tensorflow:loss = 0.0901476, step = 7880 (0.125 sec)\n",
      "INFO:tensorflow:global_step/sec: 855.16\n",
      "INFO:tensorflow:loss = 0.12800561, step = 7980 (0.125 sec)\n",
      "INFO:tensorflow:global_step/sec: 821.849\n",
      "INFO:tensorflow:loss = 0.025041103, step = 8080 (0.128 sec)\n",
      "INFO:tensorflow:global_step/sec: 778.817\n",
      "INFO:tensorflow:loss = 0.10714712, step = 8180 (0.131 sec)\n",
      "INFO:tensorflow:global_step/sec: 768.037\n",
      "INFO:tensorflow:loss = 0.16210106, step = 8280 (0.126 sec)\n",
      "INFO:tensorflow:global_step/sec: 777.315\n",
      "INFO:tensorflow:loss = 0.10621644, step = 8380 (0.120 sec)\n",
      "INFO:tensorflow:global_step/sec: 775.446\n",
      "INFO:tensorflow:loss = 0.03812648, step = 8480 (0.123 sec)\n",
      "INFO:tensorflow:global_step/sec: 843.989\n",
      "INFO:tensorflow:loss = 0.10982646, step = 8580 (0.132 sec)\n",
      "INFO:tensorflow:global_step/sec: 795.678\n",
      "INFO:tensorflow:loss = 0.019496983, step = 8680 (0.130 sec)\n",
      "INFO:tensorflow:global_step/sec: 771.808\n",
      "INFO:tensorflow:loss = 0.10744777, step = 8780 (0.134 sec)\n",
      "INFO:tensorflow:global_step/sec: 752.738\n",
      "INFO:tensorflow:loss = 0.06604447, step = 8880 (0.127 sec)\n",
      "INFO:tensorflow:global_step/sec: 762.445\n",
      "INFO:tensorflow:loss = 0.11459672, step = 8980 (0.121 sec)\n",
      "INFO:tensorflow:global_step/sec: 773.156\n",
      "INFO:tensorflow:loss = 0.059055243, step = 9080 (0.126 sec)\n",
      "INFO:tensorflow:global_step/sec: 844.337\n",
      "INFO:tensorflow:loss = 0.01626624, step = 9180 (0.128 sec)\n",
      "INFO:tensorflow:global_step/sec: 801.065\n",
      "INFO:tensorflow:loss = 0.07157644, step = 9280 (0.131 sec)\n",
      "INFO:tensorflow:global_step/sec: 770.523\n",
      "INFO:tensorflow:loss = 0.06485769, step = 9380 (0.135 sec)\n",
      "INFO:tensorflow:global_step/sec: 739.546\n",
      "INFO:tensorflow:loss = 0.0943073, step = 9480 (0.128 sec)\n",
      "INFO:tensorflow:global_step/sec: 757.65\n",
      "INFO:tensorflow:loss = 0.031422477, step = 9580 (0.125 sec)\n",
      "INFO:tensorflow:global_step/sec: 746.298\n",
      "INFO:tensorflow:loss = 0.025782982, step = 9680 (0.130 sec)\n",
      "INFO:tensorflow:global_step/sec: 801.256\n",
      "INFO:tensorflow:loss = 0.06699596, step = 9780 (0.140 sec)\n",
      "INFO:tensorflow:global_step/sec: 746.029\n",
      "INFO:tensorflow:loss = 0.031459384, step = 9880 (0.138 sec)\n",
      "INFO:tensorflow:global_step/sec: 729.198\n",
      "INFO:tensorflow:loss = 0.020690419, step = 9980 (0.141 sec)\n",
      "INFO:tensorflow:global_step/sec: 716.286\n",
      "INFO:tensorflow:loss = 0.07409065, step = 10080 (0.132 sec)\n",
      "INFO:tensorflow:global_step/sec: 730.157\n",
      "INFO:tensorflow:loss = 0.027148174, step = 10180 (0.127 sec)\n",
      "INFO:tensorflow:global_step/sec: 728.693\n",
      "INFO:tensorflow:loss = 0.034020357, step = 10280 (0.132 sec)\n",
      "INFO:tensorflow:global_step/sec: 797.499\n",
      "INFO:tensorflow:loss = 0.026023386, step = 10380 (0.135 sec)\n",
      "INFO:tensorflow:global_step/sec: 763.866\n",
      "INFO:tensorflow:loss = 0.03268505, step = 10480 (0.136 sec)\n",
      "INFO:tensorflow:global_step/sec: 750.564\n",
      "INFO:tensorflow:loss = 0.02815911, step = 10580 (0.139 sec)\n",
      "INFO:tensorflow:global_step/sec: 714.108\n",
      "INFO:tensorflow:loss = 0.01720243, step = 10680 (0.145 sec)\n",
      "INFO:tensorflow:global_step/sec: 674.318\n",
      "INFO:tensorflow:loss = 0.017777737, step = 10780 (0.129 sec)\n",
      "INFO:tensorflow:global_step/sec: 735.532\n",
      "INFO:tensorflow:loss = 0.023739574, step = 10880 (0.135 sec)\n",
      "INFO:tensorflow:global_step/sec: 763.365\n",
      "INFO:tensorflow:loss = 0.021578457, step = 10980 (0.138 sec)\n",
      "INFO:tensorflow:global_step/sec: 755.457\n",
      "INFO:tensorflow:loss = 0.027984615, step = 11080 (0.138 sec)\n",
      "INFO:tensorflow:global_step/sec: 735.83\n",
      "INFO:tensorflow:loss = 0.02471932, step = 11180 (0.138 sec)\n",
      "INFO:tensorflow:global_step/sec: 717.489\n",
      "INFO:tensorflow:loss = 0.025597647, step = 11280 (0.141 sec)\n",
      "INFO:tensorflow:global_step/sec: 702.119\n",
      "INFO:tensorflow:loss = 0.04579971, step = 11380 (0.129 sec)\n",
      "INFO:tensorflow:global_step/sec: 725.774\n",
      "INFO:tensorflow:loss = 0.05098244, step = 11480 (0.134 sec)\n",
      "INFO:tensorflow:global_step/sec: 768.621\n",
      "INFO:tensorflow:loss = 0.02209612, step = 11580 (0.138 sec)\n",
      "INFO:tensorflow:global_step/sec: 755.076\n",
      "INFO:tensorflow:loss = 0.008423474, step = 11680 (0.141 sec)\n",
      "INFO:tensorflow:global_step/sec: 714.369\n",
      "INFO:tensorflow:loss = 0.026257418, step = 11780 (0.141 sec)\n",
      "INFO:tensorflow:global_step/sec: 692.756\n",
      "INFO:tensorflow:loss = 0.0071309824, step = 11880 (0.141 sec)\n",
      "INFO:tensorflow:global_step/sec: 706.453\n",
      "INFO:tensorflow:loss = 0.019367374, step = 11980 (0.135 sec)\n",
      "INFO:tensorflow:global_step/sec: 695.557\n",
      "INFO:tensorflow:loss = 0.012905819, step = 12080 (0.137 sec)\n",
      "INFO:tensorflow:global_step/sec: 753.936\n",
      "INFO:tensorflow:loss = 0.026504947, step = 12180 (0.144 sec)\n",
      "INFO:tensorflow:global_step/sec: 721.454\n",
      "INFO:tensorflow:loss = 0.021863198, step = 12280 (0.149 sec)\n",
      "INFO:tensorflow:global_step/sec: 670.731\n",
      "INFO:tensorflow:loss = 0.027671717, step = 12380 (0.145 sec)\n",
      "INFO:tensorflow:global_step/sec: 679.223\n",
      "INFO:tensorflow:loss = 0.009333629, step = 12480 (0.143 sec)\n",
      "INFO:tensorflow:global_step/sec: 705.009\n",
      "INFO:tensorflow:loss = 0.012834126, step = 12580 (0.132 sec)\n",
      "INFO:tensorflow:global_step/sec: 715.15\n",
      "INFO:tensorflow:loss = 0.010697606, step = 12680 (0.136 sec)\n",
      "INFO:tensorflow:global_step/sec: 757.495\n",
      "INFO:tensorflow:loss = 0.031983756, step = 12780 (0.143 sec)\n",
      "INFO:tensorflow:global_step/sec: 727.438\n",
      "INFO:tensorflow:loss = 0.006336843, step = 12880 (0.142 sec)\n",
      "INFO:tensorflow:global_step/sec: 697.238\n",
      "INFO:tensorflow:loss = 0.013784962, step = 12980 (0.145 sec)\n",
      "INFO:tensorflow:global_step/sec: 694.762\n",
      "INFO:tensorflow:loss = 0.010662235, step = 13080 (0.148 sec)\n",
      "INFO:tensorflow:global_step/sec: 666.81\n",
      "INFO:tensorflow:loss = 0.007105016, step = 13180 (0.135 sec)\n",
      "INFO:tensorflow:global_step/sec: 706.709\n",
      "INFO:tensorflow:loss = 0.0048833443, step = 13280 (0.139 sec)\n",
      "INFO:tensorflow:global_step/sec: 735.212\n",
      "INFO:tensorflow:loss = 0.01628526, step = 13380 (0.143 sec)\n",
      "INFO:tensorflow:global_step/sec: 723.15\n",
      "INFO:tensorflow:loss = 0.01734767, step = 13480 (0.147 sec)\n",
      "INFO:tensorflow:global_step/sec: 678.113\n",
      "INFO:tensorflow:loss = 0.024584802, step = 13580 (0.146 sec)\n",
      "INFO:tensorflow:global_step/sec: 688.057\n",
      "INFO:tensorflow:loss = 0.0061204783, step = 13680 (0.145 sec)\n",
      "INFO:tensorflow:global_step/sec: 686.342\n",
      "INFO:tensorflow:loss = 0.0216946, step = 13780 (0.135 sec)\n",
      "INFO:tensorflow:global_step/sec: 692.381\n",
      "INFO:tensorflow:loss = 0.0059639085, step = 13880 (0.144 sec)\n",
      "INFO:tensorflow:global_step/sec: 711.588\n",
      "INFO:tensorflow:loss = 0.0065877875, step = 13980 (0.150 sec)\n",
      "INFO:tensorflow:global_step/sec: 697.593\n",
      "INFO:tensorflow:loss = 0.0064577665, step = 14080 (0.148 sec)\n",
      "INFO:tensorflow:global_step/sec: 665.318\n",
      "INFO:tensorflow:loss = 0.010996201, step = 14180 (0.153 sec)\n",
      "INFO:tensorflow:global_step/sec: 669.362\n",
      "INFO:tensorflow:loss = 0.0031180899, step = 14280 (0.148 sec)\n",
      "INFO:tensorflow:global_step/sec: 658.887\n",
      "INFO:tensorflow:loss = 0.012393519, step = 14380 (0.135 sec)\n",
      "INFO:tensorflow:global_step/sec: 691.644\n",
      "INFO:tensorflow:loss = 0.0042724945, step = 14480 (0.144 sec)\n",
      "INFO:tensorflow:global_step/sec: 715.18\n",
      "INFO:tensorflow:loss = 0.013512075, step = 14580 (0.147 sec)\n",
      "INFO:tensorflow:global_step/sec: 708.351\n",
      "INFO:tensorflow:loss = 0.0103187775, step = 14680 (0.152 sec)\n",
      "INFO:tensorflow:global_step/sec: 661.936\n",
      "INFO:tensorflow:loss = 0.012806257, step = 14780 (0.155 sec)\n",
      "INFO:tensorflow:global_step/sec: 640.533\n",
      "INFO:tensorflow:loss = 0.0032162615, step = 14880 (0.151 sec)\n",
      "INFO:tensorflow:global_step/sec: 656.897\n",
      "INFO:tensorflow:loss = 0.0018870407, step = 14980 (0.142 sec)\n",
      "INFO:tensorflow:global_step/sec: 667.762\n",
      "INFO:tensorflow:loss = 0.008410793, step = 15080 (0.148 sec)\n",
      "INFO:tensorflow:global_step/sec: 693.288\n",
      "INFO:tensorflow:loss = 0.006150189, step = 15180 (0.149 sec)\n",
      "INFO:tensorflow:global_step/sec: 691.416\n",
      "INFO:tensorflow:loss = 0.014513381, step = 15280 (0.155 sec)\n",
      "INFO:tensorflow:global_step/sec: 646.725\n",
      "INFO:tensorflow:loss = 0.013656415, step = 15380 (0.155 sec)\n",
      "INFO:tensorflow:global_step/sec: 644.084\n",
      "INFO:tensorflow:loss = 0.0037059453, step = 15480 (0.152 sec)\n",
      "INFO:tensorflow:global_step/sec: 645.603\n",
      "INFO:tensorflow:loss = 0.0027904203, step = 15580 (0.144 sec)\n",
      "INFO:tensorflow:global_step/sec: 663.459\n",
      "INFO:tensorflow:loss = 0.0081713535, step = 15680 (0.149 sec)\n",
      "INFO:tensorflow:global_step/sec: 685.904\n",
      "INFO:tensorflow:loss = 0.004991539, step = 15780 (0.157 sec)\n",
      "INFO:tensorflow:global_step/sec: 658.814\n",
      "INFO:tensorflow:loss = 0.010882616, step = 15880 (0.156 sec)\n",
      "INFO:tensorflow:global_step/sec: 642.588\n",
      "INFO:tensorflow:loss = 0.008557976, step = 15980 (0.159 sec)\n",
      "INFO:tensorflow:global_step/sec: 627.829\n",
      "INFO:tensorflow:loss = 0.006327864, step = 16080 (0.151 sec)\n",
      "INFO:tensorflow:global_step/sec: 662.91\n",
      "INFO:tensorflow:loss = 0.006493992, step = 16180 (0.144 sec)\n",
      "INFO:tensorflow:global_step/sec: 657.454\n",
      "INFO:tensorflow:loss = 0.0042195152, step = 16280 (0.148 sec)\n",
      "INFO:tensorflow:global_step/sec: 693.064\n",
      "INFO:tensorflow:loss = 0.0031367687, step = 16380 (0.158 sec)\n",
      "INFO:tensorflow:global_step/sec: 644.753\n",
      "INFO:tensorflow:loss = 0.0021976228, step = 16480 (0.164 sec)\n",
      "INFO:tensorflow:global_step/sec: 610.623\n",
      "INFO:tensorflow:loss = 0.0020480475, step = 16580 (0.164 sec)\n",
      "INFO:tensorflow:global_step/sec: 599.489\n",
      "INFO:tensorflow:loss = 0.0005994295, step = 16680 (0.160 sec)\n",
      "INFO:tensorflow:global_step/sec: 622.801\n",
      "INFO:tensorflow:loss = 0.00608566, step = 16780 (0.147 sec)\n",
      "INFO:tensorflow:global_step/sec: 663.971\n",
      "INFO:tensorflow:loss = 0.004730368, step = 16880 (0.149 sec)\n",
      "INFO:tensorflow:global_step/sec: 688.9\n",
      "INFO:tensorflow:loss = 0.0044580232, step = 16980 (0.155 sec)\n",
      "INFO:tensorflow:global_step/sec: 653.826\n",
      "INFO:tensorflow:loss = 0.006871547, step = 17080 (0.157 sec)\n",
      "INFO:tensorflow:global_step/sec: 643.319\n",
      "INFO:tensorflow:loss = 0.0019816638, step = 17180 (0.156 sec)\n",
      "INFO:tensorflow:global_step/sec: 645.26\n",
      "INFO:tensorflow:loss = 0.0026867283, step = 17280 (0.156 sec)\n",
      "INFO:tensorflow:global_step/sec: 634.18\n",
      "INFO:tensorflow:loss = 0.009364197, step = 17380 (0.148 sec)\n",
      "INFO:tensorflow:global_step/sec: 606.27\n",
      "INFO:tensorflow:loss = 0.0071661435, step = 17480 (0.167 sec)\n",
      "INFO:tensorflow:global_step/sec: 644.903\n",
      "INFO:tensorflow:loss = 0.003446314, step = 17580 (0.160 sec)\n",
      "INFO:tensorflow:global_step/sec: 645.653\n",
      "INFO:tensorflow:loss = 0.0034257886, step = 17680 (0.163 sec)\n",
      "INFO:tensorflow:global_step/sec: 605.628\n",
      "INFO:tensorflow:loss = 0.0034906813, step = 17780 (0.167 sec)\n",
      "INFO:tensorflow:global_step/sec: 612.37\n",
      "INFO:tensorflow:loss = 0.0010763546, step = 17880 (0.158 sec)\n",
      "INFO:tensorflow:global_step/sec: 622.902\n",
      "INFO:tensorflow:loss = 0.0012766183, step = 17980 (0.152 sec)\n",
      "INFO:tensorflow:global_step/sec: 614.182\n",
      "INFO:tensorflow:loss = 0.004087025, step = 18080 (0.162 sec)\n",
      "INFO:tensorflow:global_step/sec: 638.816\n",
      "INFO:tensorflow:loss = 0.003000428, step = 18180 (0.165 sec)\n",
      "INFO:tensorflow:global_step/sec: 613.88\n",
      "INFO:tensorflow:loss = 0.002123837, step = 18280 (0.170 sec)\n",
      "INFO:tensorflow:global_step/sec: 598.612\n",
      "INFO:tensorflow:loss = 0.0017298514, step = 18380 (0.166 sec)\n",
      "INFO:tensorflow:global_step/sec: 605.833\n",
      "INFO:tensorflow:loss = 0.0024391166, step = 18480 (0.172 sec)\n",
      "INFO:tensorflow:global_step/sec: 558.893\n",
      "INFO:tensorflow:loss = 0.0012706239, step = 18580 (0.160 sec)\n",
      "INFO:tensorflow:global_step/sec: 611.793\n",
      "INFO:tensorflow:loss = 0.0022460993, step = 18680 (0.157 sec)\n",
      "INFO:tensorflow:global_step/sec: 658.52\n",
      "INFO:tensorflow:loss = 0.00330067, step = 18780 (0.166 sec)\n",
      "INFO:tensorflow:global_step/sec: 608.365\n",
      "INFO:tensorflow:loss = 0.0014178703, step = 18880 (0.161 sec)\n",
      "INFO:tensorflow:global_step/sec: 631.44\n",
      "INFO:tensorflow:loss = 0.00064154173, step = 18980 (0.160 sec)\n",
      "INFO:tensorflow:global_step/sec: 624.344\n",
      "INFO:tensorflow:loss = 0.0014936434, step = 19080 (0.158 sec)\n",
      "INFO:tensorflow:global_step/sec: 629.434\n",
      "INFO:tensorflow:loss = 0.0012685368, step = 19180 (0.153 sec)\n",
      "INFO:tensorflow:global_step/sec: 614.5\n",
      "INFO:tensorflow:loss = 0.0009501026, step = 19280 (0.163 sec)\n",
      "INFO:tensorflow:global_step/sec: 640.467\n",
      "INFO:tensorflow:loss = 0.0032116016, step = 19380 (0.161 sec)\n",
      "INFO:tensorflow:global_step/sec: 627.011\n",
      "INFO:tensorflow:loss = 0.00020568268, step = 19480 (0.168 sec)\n",
      "INFO:tensorflow:global_step/sec: 597.5\n",
      "INFO:tensorflow:loss = 0.0015133687, step = 19580 (0.169 sec)\n",
      "INFO:tensorflow:global_step/sec: 595.167\n",
      "INFO:tensorflow:loss = 0.00062260585, step = 19680 (0.164 sec)\n",
      "INFO:tensorflow:global_step/sec: 609.871\n",
      "INFO:tensorflow:loss = 0.0004801547, step = 19780 (0.151 sec)\n",
      "INFO:tensorflow:global_step/sec: 627.667\n",
      "INFO:tensorflow:loss = 0.000921194, step = 19880 (0.163 sec)\n",
      "INFO:tensorflow:global_step/sec: 626.187\n",
      "INFO:tensorflow:loss = 0.0043853344, step = 19980 (0.167 sec)\n",
      "INFO:tensorflow:global_step/sec: 608.324\n",
      "INFO:tensorflow:loss = 0.0013903134, step = 20080 (0.162 sec)\n",
      "INFO:tensorflow:global_step/sec: 626.429\n",
      "INFO:tensorflow:loss = 0.001209721, step = 20180 (0.161 sec)\n",
      "INFO:tensorflow:global_step/sec: 622.166\n",
      "INFO:tensorflow:loss = 0.0032171812, step = 20280 (0.160 sec)\n",
      "INFO:tensorflow:global_step/sec: 620.875\n",
      "INFO:tensorflow:loss = 0.0012091644, step = 20380 (0.153 sec)\n",
      "INFO:tensorflow:global_step/sec: 617.897\n",
      "INFO:tensorflow:loss = 0.000616746, step = 20480 (0.159 sec)\n",
      "INFO:tensorflow:global_step/sec: 659.07\n",
      "INFO:tensorflow:loss = 0.00043906303, step = 20580 (0.163 sec)\n",
      "INFO:tensorflow:global_step/sec: 618.697\n",
      "INFO:tensorflow:loss = 0.00034749185, step = 20680 (0.162 sec)\n",
      "INFO:tensorflow:global_step/sec: 623.5\n",
      "INFO:tensorflow:loss = 0.0006929866, step = 20780 (0.162 sec)\n",
      "INFO:tensorflow:global_step/sec: 621.933\n",
      "INFO:tensorflow:loss = 0.0012473831, step = 20880 (0.162 sec)\n",
      "INFO:tensorflow:global_step/sec: 612.531\n",
      "INFO:tensorflow:loss = 0.0014690836, step = 20980 (0.158 sec)\n",
      "INFO:tensorflow:global_step/sec: 587.212\n",
      "INFO:tensorflow:loss = 0.00048632748, step = 21080 (0.164 sec)\n",
      "INFO:tensorflow:global_step/sec: 641.986\n",
      "INFO:tensorflow:loss = 0.0018161723, step = 21180 (0.171 sec)\n",
      "INFO:tensorflow:global_step/sec: 584.532\n",
      "INFO:tensorflow:loss = 0.00064360513, step = 21280 (0.170 sec)\n",
      "INFO:tensorflow:global_step/sec: 600.218\n",
      "INFO:tensorflow:loss = 0.0005411065, step = 21380 (0.173 sec)\n",
      "INFO:tensorflow:global_step/sec: 576.153\n",
      "INFO:tensorflow:loss = 0.00068329973, step = 21480 (0.166 sec)\n",
      "INFO:tensorflow:global_step/sec: 603.905\n",
      "INFO:tensorflow:loss = 0.00052374724, step = 21580 (0.167 sec)\n",
      "INFO:tensorflow:global_step/sec: 564.66\n",
      "INFO:tensorflow:loss = 0.0027153534, step = 21680 (0.166 sec)\n",
      "INFO:tensorflow:global_step/sec: 617.917\n",
      "INFO:tensorflow:loss = 0.001638851, step = 21780 (0.168 sec)\n",
      "INFO:tensorflow:global_step/sec: 613.895\n",
      "INFO:tensorflow:loss = 0.0010489854, step = 21880 (0.175 sec)\n",
      "INFO:tensorflow:global_step/sec: 577.005\n",
      "INFO:tensorflow:loss = 0.0013357425, step = 21980 (0.170 sec)\n",
      "INFO:tensorflow:global_step/sec: 585.483\n",
      "INFO:tensorflow:loss = 0.0011078725, step = 22080 (0.192 sec)\n",
      "INFO:tensorflow:global_step/sec: 499.468\n",
      "INFO:tensorflow:loss = 0.00070813263, step = 22180 (0.211 sec)\n",
      "INFO:tensorflow:global_step/sec: 451.412\n",
      "INFO:tensorflow:loss = 0.0010020329, step = 22280 (0.213 sec)\n",
      "INFO:tensorflow:global_step/sec: 481.974\n",
      "INFO:tensorflow:loss = 0.0007271028, step = 22380 (0.215 sec)\n",
      "INFO:tensorflow:global_step/sec: 484.349\n",
      "INFO:tensorflow:loss = 0.000567897, step = 22480 (0.205 sec)\n",
      "INFO:tensorflow:global_step/sec: 483.276\n",
      "INFO:tensorflow:loss = 0.0010279386, step = 22580 (0.210 sec)\n",
      "INFO:tensorflow:global_step/sec: 478.625\n",
      "INFO:tensorflow:loss = 0.0009803168, step = 22680 (0.217 sec)\n",
      "INFO:tensorflow:global_step/sec: 453.93\n",
      "INFO:tensorflow:loss = 0.0004008984, step = 22780 (0.199 sec)\n",
      "INFO:tensorflow:global_step/sec: 479.393\n",
      "INFO:tensorflow:loss = 0.0002498337, step = 22880 (0.209 sec)\n",
      "INFO:tensorflow:global_step/sec: 498.562\n",
      "INFO:tensorflow:loss = 0.0005590151, step = 22980 (0.207 sec)\n",
      "INFO:tensorflow:global_step/sec: 488.429\n",
      "INFO:tensorflow:loss = 0.00052176055, step = 23080 (0.205 sec)\n",
      "INFO:tensorflow:global_step/sec: 497.493\n",
      "INFO:tensorflow:loss = 0.0005056064, step = 23180 (0.198 sec)\n",
      "INFO:tensorflow:global_step/sec: 494.963\n",
      "INFO:tensorflow:loss = 0.0007074217, step = 23280 (0.215 sec)\n",
      "INFO:tensorflow:global_step/sec: 476.034\n",
      "INFO:tensorflow:loss = 0.00033772603, step = 23380 (0.194 sec)\n",
      "INFO:tensorflow:global_step/sec: 478.735\n",
      "INFO:tensorflow:loss = 0.00042475882, step = 23480 (0.197 sec)\n",
      "INFO:tensorflow:global_step/sec: 522.584\n",
      "INFO:tensorflow:loss = 0.0010735475, step = 23580 (0.191 sec)\n",
      "INFO:tensorflow:global_step/sec: 539.621\n",
      "INFO:tensorflow:loss = 0.00019561013, step = 23680 (0.198 sec)\n",
      "INFO:tensorflow:global_step/sec: 521.083\n",
      "INFO:tensorflow:loss = 0.00067777047, step = 23780 (0.181 sec)\n",
      "INFO:tensorflow:global_step/sec: 546.421\n",
      "INFO:tensorflow:loss = 0.0002681838, step = 23880 (0.192 sec)\n",
      "INFO:tensorflow:global_step/sec: 473.689\n",
      "INFO:tensorflow:loss = 0.00019832201, step = 23980 (0.221 sec)\n",
      "INFO:tensorflow:global_step/sec: 461.295\n",
      "INFO:tensorflow:Saving checkpoints for 24000 into /var/folders/jg/tpqyh1fd5js5wsr1d138k3n40000gn/T/tmpoibe_re7/model.ckpt.\n",
      "WARNING:tensorflow:Issue encountered when serializing resources.\n",
      "Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.\n",
      "'_Resource' object has no attribute 'name'\n",
      "INFO:tensorflow:Loss for final step: 0.00047454803.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-10-29T21:16:37Z\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from /var/folders/jg/tpqyh1fd5js5wsr1d138k3n40000gn/T/tmpoibe_re7/model.ckpt-24000\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-10-29-21:16:38\n",
      "INFO:tensorflow:Saving dict for global step 24000: average_loss = 11.731688, global_step = 24000, label/mean = 23.611393, loss = 11.6518, prediction/mean = 22.625761\n",
      "WARNING:tensorflow:Issue encountered when serializing resources.\n",
      "Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.\n",
      "'_Resource' object has no attribute 'name'\n",
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 24000: /var/folders/jg/tpqyh1fd5js5wsr1d138k3n40000gn/T/tmpoibe_re7/model.ckpt-24000\n",
      "{'average_loss': 11.731688, 'label/mean': 23.611393, 'loss': 11.6518, 'prediction/mean': 22.625761, 'global_step': 24000}\n",
      "Average-Loss 11.7317\n"
     ]
    }
   ],
   "source": [
    "boosted_tree = tf.estimator.BoostedTreesRegressor(\n",
    "    feature_columns=all_feature_columns,\n",
    "    n_batches_per_layer=20,\n",
    "    n_trees=200)\n",
    "\n",
    "boosted_tree.train(\n",
    "    input_fn=lambda:train_input_fn(df_train_norm, batch_size=BATCH_SIZE))\n",
    "\n",
    "eval_results = boosted_tree.evaluate(\n",
    "    input_fn=lambda:eval_input_fn(df_test_norm, batch_size=8))\n",
    "\n",
    "print(eval_results)\n",
    "\n",
    "print('Average-Loss {:.4f}'.format(eval_results['average_loss']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "Readers may ignore the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[NbConvertApp] Converting notebook ch14_part2.ipynb to script\n",
      "[NbConvertApp] Writing 6276 bytes to ch14_part2.py\n"
     ]
    }
   ],
   "source": [
    "! python ../.convert_notebook_to_script.py --input ch14_part2.ipynb --output ch14_part2.py"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}