notebooks/top_k.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 탑-k 방법\n",
    "\n",
    "탑-k 방법은 모델의 결과를 조사하는 유용한 방법입니다. 단순하게 **가장 성공적인 샘플과 가장 성공적이지 않은 샘플**을 살펴 보고 그 안의 패턴을 찾는 것입니다. 이런 패턴을 사용해 새로운 특성을 고안하거나 기존 특성을 반복할 수 있습니다.\n",
    "\n",
    "먼저 데이터를 로드합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from pathlib import Path\n",
    "import joblib\n",
    "\n",
    "import sys\n",
    "sys.path.append(\"..\")\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "from ml_editor.data_processing import (\n",
    "    format_raw_df, get_split_by_author, \n",
    "    add_text_features_to_df, \n",
    "    get_vectorized_series, \n",
    "    get_feature_vector_and_label\n",
    ")\n",
    "from ml_editor.model_evaluation import get_top_k\n",
    "\n",
    "data_path = Path('../data/writers.csv')\n",
    "df = pd.read_csv(data_path)\n",
    "df = format_raw_df(df.copy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "그다음 특성을 추가하고 데이터셋을 분할합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = add_text_features_to_df(df.loc[df[\"is_question\"]].copy())\n",
    "train_df, test_df = get_split_by_author(df, test_size=0.2, random_state=40)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "훈련된 모델을 로드하고 특성을 벡터화합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model_path = Path(\"../models/model_1.pkl\")\n",
    "clf = joblib.load(model_path) \n",
    "vectorizer_path = Path(\"../models/vectorizer_1.pkl\")\n",
    "vectorizer = joblib.load(vectorizer_path) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('../models/model_1.pkl')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df[\"vectors\"] = get_vectorized_series(train_df[\"full_text\"].copy(), vectorizer)\n",
    "test_df[\"vectors\"] = get_vectorized_series(test_df[\"full_text\"].copy(), vectorizer)\n",
    "\n",
    "features = [\n",
    "                \"action_verb_full\",\n",
    "                \"question_mark_full\",\n",
    "                \"text_len\",\n",
    "                \"language_question\",\n",
    "            ]\n",
    "X_train, y_train = get_feature_vector_and_label(train_df, features)\n",
    "X_test, y_test = get_feature_vector_and_label(test_df, features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 탑-k 방법을 사용해 다음을 조사합니다:\n",
    "\n",
    "- 각 클래스에서 (높은 점수와 낮은 점수를 내는) k 개의 최상의 샘플\n",
    "- 각 클래스에서 k 개의 최악의 샘플\n",
    "- 모델 예측 확률이 0.5에 가까운 가장 불확실한 k 개 샘플\n",
    "\n",
    "이런 특정 샘플을 출력하는 것이 모델 반복에 어떻게 도움이 되는지 알려면 이 책의 5장을 참고하세요."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_analysis_df = test_df.copy()\n",
    "y_predicted_proba = clf.predict_proba(X_test)\n",
    "test_analysis_df[\"predicted_proba\"] = y_predicted_proba[:, 1]\n",
    "test_analysis_df[\"true_label\"] = y_test\n",
    "\n",
    "to_display = [\n",
    "    \"predicted_proba\",\n",
    "    \"true_label\",\n",
    "    \"Title\",\n",
    "    \"body_text\",\n",
    "    \"text_len\",\n",
    "    \"action_verb_full\",\n",
    "    \"question_mark_full\",\n",
    "    \"language_question\",\n",
    "]\n",
    "threshold = 0.5\n",
    "\n",
    "\n",
    "top_pos, top_neg, worst_pos, worst_neg, unsure = get_top_k(test_analysis_df, \"predicted_proba\", \"true_label\", k=2)\n",
    "pd.options.display.max_colwidth = 500"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "가장 올바르게 정답을 맞춘 양성 예측"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>predicted_proba</th>\n",
       "      <th>true_label</th>\n",
       "      <th>Title</th>\n",
       "      <th>body_text</th>\n",
       "      <th>text_len</th>\n",
       "      <th>action_verb_full</th>\n",
       "      <th>question_mark_full</th>\n",
       "      <th>language_question</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6793</th>\n",
       "      <td>0.74</td>\n",
       "      <td>True</td>\n",
       "      <td>Non-cheap ways to make villains evil?</td>\n",
       "      <td>Do you have any tried and true techniques to make villains of your stories truly hated by the audience?\\nI mean, frequently it's \"eh, sure, that's bad, he's got to be stopped\" but the audience would rather observe the villain more, learn, maybe try to get them to change their ways. Or worst of all, pity the villain in the end for failing to execute their just revenge, or not getting along with their plan for what would -really- be a better future, even if through baptism of fire.\\nNow what t...</td>\n",
       "      <td>1448</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4107</th>\n",
       "      <td>0.72</td>\n",
       "      <td>True</td>\n",
       "      <td>Single character POV vs. two POVs - how to decide?</td>\n",
       "      <td>I'm starting to look at my next novel, and I'm trying to decide whether I should tell it from one POV or two.  I've used both techniques in the past, so I'm aware of the basic advantages/disadvantages, but I'm still having trouble deciding which is best for the story I want to tell.  \\nI realize that it's impossible to answer that question without knowing the details of my story, but I'm hunting for some sort of framework for my thoughts, so: in general, when is it advisable to stick with a ...</td>\n",
       "      <td>1406</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      predicted_proba  true_label  \\\n",
       "Id                                  \n",
       "6793             0.74        True   \n",
       "4107             0.72        True   \n",
       "\n",
       "                                                   Title  \\\n",
       "Id                                                         \n",
       "6793               Non-cheap ways to make villains evil?   \n",
       "4107  Single character POV vs. two POVs - how to decide?   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                body_text  \\\n",
       "Id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          \n",
       "6793  Do you have any tried and true techniques to make villains of your stories truly hated by the audience?\\nI mean, frequently it's \"eh, sure, that's bad, he's got to be stopped\" but the audience would rather observe the villain more, learn, maybe try to get them to change their ways. Or worst of all, pity the villain in the end for failing to execute their just revenge, or not getting along with their plan for what would -really- be a better future, even if through baptism of fire.\\nNow what t...   \n",
       "4107  I'm starting to look at my next novel, and I'm trying to decide whether I should tell it from one POV or two.  I've used both techniques in the past, so I'm aware of the basic advantages/disadvantages, but I'm still having trouble deciding which is best for the story I want to tell.  \\nI realize that it's impossible to answer that question without knowing the details of my story, but I'm hunting for some sort of framework for my thoughts, so: in general, when is it advisable to stick with a ...   \n",
       "\n",
       "      text_len  action_verb_full  question_mark_full  language_question  \n",
       "Id                                                                       \n",
       "6793      1448             False                True              False  \n",
       "4107      1406              True                True              False  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 가장 올바르게 예측한 양성 샘플\n",
    "top_pos[to_display]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "가장 올바르게 정답을 맞춘 음성 예측"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>predicted_proba</th>\n",
       "      <th>true_label</th>\n",
       "      <th>Title</th>\n",
       "      <th>body_text</th>\n",
       "      <th>text_len</th>\n",
       "      <th>action_verb_full</th>\n",
       "      <th>question_mark_full</th>\n",
       "      <th>language_question</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7488</th>\n",
       "      <td>0.09</td>\n",
       "      <td>False</td>\n",
       "      <td>Releases needed for picture books?</td>\n",
       "      <td>Do you need location releases for national parks and model releases for Pets to use in picture books?\\n</td>\n",
       "      <td>137</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34919</th>\n",
       "      <td>0.13</td>\n",
       "      <td>False</td>\n",
       "      <td>Is Rob Zombie's Red Red Kroovy Trochee style?</td>\n",
       "      <td>https://genius.com/Rob-zombie-never-gonna-stop-\\nYeah, I'm on Durango number 95\\nTake me to the home, kick boots and ultra live\\nSee heaven flash a horror show\\nKnock it nice and smooth step back and watch it blow, yeah\\nNever gonna stop me, never gonna stop\\nNever gonna stop me, never gonna stop\\nNever gonna stop me, never gonna stop\\nNever gonna stop me, never gonna stop\\nGive it to me, give it to me\\nYeah, the devil ride it down the shore\\nHe paint the monster red so the blood don't stain...</td>\n",
       "      <td>783</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       predicted_proba  true_label  \\\n",
       "Id                                   \n",
       "7488              0.09       False   \n",
       "34919             0.13       False   \n",
       "\n",
       "                                               Title  \\\n",
       "Id                                                     \n",
       "7488              Releases needed for picture books?   \n",
       "34919  Is Rob Zombie's Red Red Kroovy Trochee style?   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 body_text  \\\n",
       "Id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           \n",
       "7488                                                                                                                                                                                                                                                                                                                                                                                                               Do you need location releases for national parks and model releases for Pets to use in picture books?\\n   \n",
       "34919  https://genius.com/Rob-zombie-never-gonna-stop-\\nYeah, I'm on Durango number 95\\nTake me to the home, kick boots and ultra live\\nSee heaven flash a horror show\\nKnock it nice and smooth step back and watch it blow, yeah\\nNever gonna stop me, never gonna stop\\nNever gonna stop me, never gonna stop\\nNever gonna stop me, never gonna stop\\nNever gonna stop me, never gonna stop\\nGive it to me, give it to me\\nYeah, the devil ride it down the shore\\nHe paint the monster red so the blood don't stain...   \n",
       "\n",
       "       text_len  action_verb_full  question_mark_full  language_question  \n",
       "Id                                                                        \n",
       "7488        137             False                True              False  \n",
       "34919       783             False                True              False  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 가장 올바르게 예측한 음성 샘플\n",
    "top_neg[to_display]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "올바르게 예측한 음성 예측의 대부분은 **길이가 짧습니다**. 이 결과는 가중 중요한 특성 중의 하나가 질문의 길이라는 특성 중요도 분석의 결과를 뒷받침합니다.\n",
    "\n",
    "가장 확실하게 틀린 양성 예측을 살펴 보죠."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>predicted_proba</th>\n",
       "      <th>true_label</th>\n",
       "      <th>Title</th>\n",
       "      <th>body_text</th>\n",
       "      <th>text_len</th>\n",
       "      <th>action_verb_full</th>\n",
       "      <th>question_mark_full</th>\n",
       "      <th>language_question</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14157</th>\n",
       "      <td>0.14</td>\n",
       "      <td>True</td>\n",
       "      <td>Do you bold punctuation directly after bold text?</td>\n",
       "      <td>Do you bold punctuation directly after bold, linked or italic text? \\n</td>\n",
       "      <td>119</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17543</th>\n",
       "      <td>0.15</td>\n",
       "      <td>True</td>\n",
       "      <td>Using Pronoun 'It' repetitvely for emphasis?</td>\n",
       "      <td>I'd like to know if using \"It\" repetitively (for emphasis) in this context is okay grammatically.\\n\\nTV has become the modern day baby sitter.  It is raising our children.  It is dictating the cultural narrative and shaping future society.  It is raising the bored inattentive child.  It is raising the consumer child. It is raising the aggressive child. It is raising the obese child. It is raising the misinformed and complacent child. It is raising the disenchanted child.  And what’s more, it...</td>\n",
       "      <td>591</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       predicted_proba  true_label  \\\n",
       "Id                                   \n",
       "14157             0.14        True   \n",
       "17543             0.15        True   \n",
       "\n",
       "                                                   Title  \\\n",
       "Id                                                         \n",
       "14157  Do you bold punctuation directly after bold text?   \n",
       "17543       Using Pronoun 'It' repetitvely for emphasis?   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 body_text  \\\n",
       "Id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           \n",
       "14157                                                                                                                                                                                                                                                                                                                                                                                                                                               Do you bold punctuation directly after bold, linked or italic text? \\n   \n",
       "17543  I'd like to know if using \"It\" repetitively (for emphasis) in this context is okay grammatically.\\n\\nTV has become the modern day baby sitter.  It is raising our children.  It is dictating the cultural narrative and shaping future society.  It is raising the bored inattentive child.  It is raising the consumer child. It is raising the aggressive child. It is raising the obese child. It is raising the misinformed and complacent child. It is raising the disenchanted child.  And what’s more, it...   \n",
       "\n",
       "       text_len  action_verb_full  question_mark_full  language_question  \n",
       "Id                                                                        \n",
       "14157       119             False                True              False  \n",
       "17543       591             False                True              False  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 가장 틀리게 예측한 양성 샘플\n",
    "worst_pos[to_display]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "반대로 모델이 틀린 질문에서 높은 점수를 가진 짧은 질문이 잘 나타나있습니다.\n",
    "\n",
    "그다음 가장 확실하게 틀린 음성 예측을 살펴 보겠습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>predicted_proba</th>\n",
       "      <th>true_label</th>\n",
       "      <th>Title</th>\n",
       "      <th>body_text</th>\n",
       "      <th>text_len</th>\n",
       "      <th>action_verb_full</th>\n",
       "      <th>question_mark_full</th>\n",
       "      <th>language_question</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7878</th>\n",
       "      <td>0.86</td>\n",
       "      <td>False</td>\n",
       "      <td>When quoting a person's informal speech, how much liberty do you have to make changes to what they say?</td>\n",
       "      <td>Even during a formal interview for a news article, people speak informally. They say \"uhm\", they cut off sentences half-way through, they interject phrases like \"you know?\", and they make innocent grammatical mistakes.\\nAs somebody who wants to fairly and accurately report the discussion that takes place in an interview, what guidelines should I use in making changes to what a person says?\\nWhile the simplest solution is to write exactly what they say and [sic] any errors they make, that can...</td>\n",
       "      <td>694</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24995</th>\n",
       "      <td>0.72</td>\n",
       "      <td>False</td>\n",
       "      <td>Self-translating into English</td>\n",
       "      <td>I am finishing writing my first book (in Slovak, SF) and will be looking for publishers soon. I was considering self-publishing but I don't think I can do more then them in this field. Well, except for the translation.\\nWe have all heard the 3% problem where only this many books are translated to english. So I think about translating my book to english by my own money (i.e. paying someone to translate it - I'm not the one doing this).\\nFew reasons: 1) publishers in my country don't try hard ...</td>\n",
       "      <td>917</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       predicted_proba  true_label  \\\n",
       "Id                                   \n",
       "7878              0.86       False   \n",
       "24995             0.72       False   \n",
       "\n",
       "                                                                                                         Title  \\\n",
       "Id                                                                                                               \n",
       "7878   When quoting a person's informal speech, how much liberty do you have to make changes to what they say?   \n",
       "24995                                                                            Self-translating into English   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 body_text  \\\n",
       "Id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           \n",
       "7878   Even during a formal interview for a news article, people speak informally. They say \"uhm\", they cut off sentences half-way through, they interject phrases like \"you know?\", and they make innocent grammatical mistakes.\\nAs somebody who wants to fairly and accurately report the discussion that takes place in an interview, what guidelines should I use in making changes to what a person says?\\nWhile the simplest solution is to write exactly what they say and [sic] any errors they make, that can...   \n",
       "24995  I am finishing writing my first book (in Slovak, SF) and will be looking for publishers soon. I was considering self-publishing but I don't think I can do more then them in this field. Well, except for the translation.\\nWe have all heard the 3% problem where only this many books are translated to english. So I think about translating my book to english by my own money (i.e. paying someone to translate it - I'm not the one doing this).\\nFew reasons: 1) publishers in my country don't try hard ...   \n",
       "\n",
       "       text_len  action_verb_full  question_mark_full  language_question  \n",
       "Id                                                                        \n",
       "7878        694              True                True              False  \n",
       "24995       917              True                True              False  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 가장 틀리게 예측한 음성 샘플\n",
    "worst_neg[to_display]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "마지막으로 모델의 확률이 모든 클래스에 동일한 가장 불확실한 질문입니다(두 개의 클래스라면 확률이 `0.5`에 가까운 샘플)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>predicted_proba</th>\n",
       "      <th>true_label</th>\n",
       "      <th>Title</th>\n",
       "      <th>body_text</th>\n",
       "      <th>text_len</th>\n",
       "      <th>action_verb_full</th>\n",
       "      <th>question_mark_full</th>\n",
       "      <th>language_question</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>21955</th>\n",
       "      <td>0.5</td>\n",
       "      <td>True</td>\n",
       "      <td>How to make a dark story not-so-dark (Shining the light in darkness)</td>\n",
       "      <td>I'm writing a war story, and it's dark. However, I find that every scene turns out to be depressing because of it. Readers will be overwhelmed. Are there ways I can induce hope/shine the light in the darkness in my novel?\\n</td>\n",
       "      <td>291</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23599</th>\n",
       "      <td>0.5</td>\n",
       "      <td>False</td>\n",
       "      <td>Quotes around long backstory narrated by a character</td>\n",
       "      <td>I am working on a novel in which the characters talk to the protagonist and explain lot of backstory. It can run into tens of pages - essentially the entire story is told by the character to the protagonist and the reader is a third person learning about it in parallel.\\nMy question is: what are the rules for quotes in these kind of conversations.\\nFor instance (H: hero, C: a local character)\\nPage 1: H: So, how did you end up here?\\nPage 1 - 15: C: tells a 15-page long story ......\\nPage 16...</td>\n",
       "      <td>831</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       predicted_proba  true_label  \\\n",
       "Id                                   \n",
       "21955              0.5        True   \n",
       "23599              0.5       False   \n",
       "\n",
       "                                                                      Title  \\\n",
       "Id                                                                            \n",
       "21955  How to make a dark story not-so-dark (Shining the light in darkness)   \n",
       "23599                  Quotes around long backstory narrated by a character   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 body_text  \\\n",
       "Id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           \n",
       "21955                                                                                                                                                                                                                                                                                      I'm writing a war story, and it's dark. However, I find that every scene turns out to be depressing because of it. Readers will be overwhelmed. Are there ways I can induce hope/shine the light in the darkness in my novel?\\n   \n",
       "23599  I am working on a novel in which the characters talk to the protagonist and explain lot of backstory. It can run into tens of pages - essentially the entire story is told by the character to the protagonist and the reader is a third person learning about it in parallel.\\nMy question is: what are the rules for quotes in these kind of conversations.\\nFor instance (H: hero, C: a local character)\\nPage 1: H: So, how did you end up here?\\nPage 1 - 15: C: tells a 15-page long story ......\\nPage 16...   \n",
       "\n",
       "       text_len  action_verb_full  question_mark_full  language_question  \n",
       "Id                                                                        \n",
       "21955       291              True                True              False  \n",
       "23599       831              True                True              False  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unsure[to_display]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "새로운 후보 특성을 찾기 위해 탑-k 방법과 특성 중요도, 벡터화 방법을 함께 사용하는 것을 추천합니다."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}