{"cells":[{"metadata":{},"cell_type":"markdown","source":"**Notebook Objective**: \n\nSimple approach to build a sincere/insincere classifier for Quora questions.\n\n- **Data preprocssing**\n \n Data Exploration: few statistics about the questions text.\n \n Building the vocabulary and then tokenizing the questions text.\n \n\n\n- **Modelisation**\n \n Straight-forward Many-to-One approach with a LSTM layer. LSTM are particular RNN that prevent vanishing gradient and helps the model to better 'remember' when dealing with long sequences. For a better understanding of what is going on under the hood see: [Understanding LSTM](http://colah.github.io/posts/2015-08-Understanding-LSTMs/).\n \n We fed the model with embedded words. In this notebook we will use our own embeddings (trained on the top of the model) rather than the pre-trained ones given as auxiliary inputs. Despite an additional training step for our model, the idea here is that our embedding will be more dedicated to the specific task we are doing than a pre-trained embedding coming from Wikipedia pages.\n \n \n- **Predictions and submission**"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport spacy\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import f1_score\n\nimport tensorflow as tf\nfrom keras.models import Sequential\nfrom keras.preprocessing import sequence\nfrom keras.layers import Dense, Embedding, Dropout, LSTM\nfrom keras import Model\nfrom keras.optimizers import Adam \nimport keras.backend as K\nfrom keras.callbacks import Callback\n","execution_count":1,"outputs":[{"output_type":"stream","text":"Using TensorFlow backend.\n","name":"stderr"}]},{"metadata":{},"cell_type":"markdown","source":"# Data Exploration and Processing"},{"metadata":{},"cell_type":"markdown","source":"In this section we will explore and make some preprocessing on the data to feed our model."},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"../input/train.csv\")\ntest = pd.read_csv(\"../input/test.csv\")\nprint(\"Train shape : \", train.shape)\nprint(\"Test shape : \", test.shape)\ntrain.head()","execution_count":2,"outputs":[{"output_type":"stream","text":"Train shape : (1306122, 3)\nTest shape : (375806, 2)\n","name":"stdout"},{"output_type":"execute_result","execution_count":2,"data":{"text/plain":" qid ... target\n0 00002165364db923c7e6 ... 0\n1 000032939017120e6e44 ... 0\n2 0000412ca6e4628ce2cf ... 0\n3 000042bf85aa498cd78e ... 0\n4 0000455dfa3e01eae3af ... 0\n\n[5 rows x 3 columns]","text/html":"
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
qidquestion_texttarget
000002165364db923c7e6How did Quebec nationalists see their province...0
1000032939017120e6e44Do you have an adopted dog, how would you enco...0
20000412ca6e4628ce2cfWhy does velocity affect time? Does velocity a...0
3000042bf85aa498cd78eHow did Otto von Guericke used the Magdeburg h...0
40000455dfa3e01eae3afCan I convert montra helicon D to a mountain b...0
\n
"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Look at the classes distribution.\ntrain.describe()","execution_count":3,"outputs":[{"output_type":"execute_result","execution_count":3,"data":{"text/plain":" target\ncount 1.306122e+06\nmean 6.187018e-02\nstd 2.409197e-01\nmin 0.000000e+00\n25% 0.000000e+00\n50% 0.000000e+00\n75% 0.000000e+00\nmax 1.000000e+00","text/html":"
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
target
count1.306122e+06
mean6.187018e-02
std2.409197e-01
min0.000000e+00
25%0.000000e+00
50%0.000000e+00
75%0.000000e+00
max1.000000e+00
\n
"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"We can see that we have unbalanced classes: 6.2% insincere vs 93.8% sincere questions. Hence, the performance metrics can't be the accuracy, we will look at the F1-score (as mentioned in the challenge description)."},{"metadata":{},"cell_type":"markdown","source":"## Some statistics"},{"metadata":{"trusted":true},"cell_type":"code","source":"train['lenght_sentence'] = train['question_text'].apply(lambda x: len(x.split()))\nprint('Min questions lenght:', np.min(train['lenght_sentence'] ))\nprint('Max questions lenght:', np.max(train['lenght_sentence'] ))\nprint('Mean questions lenght:', np.mean(train['lenght_sentence'] ))\nprint('Standard deviation questions lenght:', np.std(train['lenght_sentence'] ))\n\n\n# Plot the distribution of the lenght of the questions\nplt.hist(train['lenght_sentence'], 100);","execution_count":4,"outputs":[{"output_type":"stream","text":"Min questions lenght: 1\nMax questions lenght: 134\nMean questions lenght: 12.803609463740754\nStandard deviation questions lenght: 7.052434330971179\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"
","image/png":"iVBORw0KGgoAAAANSUhEUgAAAZEAAAD8CAYAAAC2PJlnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAE89JREFUeJzt3X+s3fV93/HnazjQ/GhiE1yP2s7MUq8VRS0hFvGUasrCCoZUNZFYZloVL0Nxp4CWTJFak0gjS1qJaG2yorXuaPAwUYpDCRlWS+q6DlK0PyCYhPAzlFtCii2DXUwgXbSkJO/9cT4Oh5tzf32u7Xsufj6ko/M97+/n+/2+z1e+9+Xvj3NuqgpJknr8k4VuQJK0eBkikqRuhogkqZshIknqZohIkroZIpKkboaIJKmbISJJ6jZjiCRZneSuJI8keTjJB1r9o0kOJLm/PS4ZWuaaJBNJHkty0VB9Q6tNJNk6VD8ryT2t/rkkp7b6ae31RJu/5li+eUnS/GSmT6wnORM4s6q+muQngfuAS4H3AP9QVb83afzZwC3A+cBPA38N/Is2+2+AXwb2A/cCl1fVI0luBW6vqp1J/hj4elVtS/J+4Beq6j8m2QS8u6r+3XT9nnHGGbVmzZo57AJJ0n333ff3VbV8rsstmWlAVR0EDrbp7yR5FFg5zSIbgZ1V9T3gm0kmGAQKwERVPQGQZCewsa3vncCvtTE7gI8C29q6PtrqtwH/I0lqmuRbs2YN+/btm+ltSZKGJPlWz3JzuibSTie9Bbinla5O8kCS7UmWtdpK4Kmhxfa32lT1NwLfrqoXJ9Vftq42//k2fnJfW5LsS7Lv8OHDc3lLkqR5mHWIJHkd8Hngg1X1AoMjhTcD5zI4Uvn949LhLFTVDVW1rqrWLV8+56MxSVKnWYVIklcxCJDPVtXtAFX1TFX9oKp+CPwJL52yOgCsHlp8VatNVX8WWJpkyaT6y9bV5r+hjZckjYHZ3J0V4Ebg0ar65FD9zKFh7wYeatO7gE3tzqqzgLXAVxhcSF/b7sQ6FdgE7GrXN+4CLmvLbwbuGFrX5jZ9GfCl6a6HSJJOrBkvrANvB34DeDDJ/a32YeDyJOcCBTwJ/CZAVT3c7rZ6BHgRuKqqfgCQ5GpgN3AKsL2qHm7r+21gZ5LfAb7GILRoz59pF+ePMAgeSdKYmPEW38Vm3bp15d1ZkjQ3Se6rqnVzXc5PrEuSuhkikqRuhogkqdtsLqxrjtZs/YsfTT953bsWsBNJOr48EpEkdTNEJEndDBFJUjdDRJLUzRCRJHUzRCRJ3QwRSVI3Q0SS1M0QkSR1M0QkSd0MEUlSN0NEktTNEJEkdTNEJEndDBFJUjdDRJLUzRCRJHUzRCRJ3QwRSVI3Q0SS1M0QkSR1M0QkSd0MEUlSN0NEktTNEJEkdTNEJEndDBFJUjdDRJLUzRCRJHUzRCRJ3WYMkSSrk9yV5JEkDyf5QKufnmRPksfb87JWT5Lrk0wkeSDJeUPr2tzGP55k81D9rUkebMtcnyTTbUOSNB5mcyTyIvChqjobWA9cleRsYCuwt6rWAnvba4CLgbXtsQXYBoNAAK4F3gacD1w7FArbgPcNLbeh1afahiRpDMwYIlV1sKq+2qa/AzwKrAQ2AjvasB3ApW16I3BzDdwNLE1yJnARsKeqjlTVc8AeYEOb9/qquruqCrh50rpGbUOSNAbmdE0kyRrgLcA9wIqqOthmPQ2saNMrgaeGFtvfatPV94+oM802JEljYNYhkuR1wOeBD1bVC8Pz2hFEHePeXma6bSTZkmRfkn2HDx8+nm1IkobMKkSSvIpBgHy2qm5v5WfaqSja86FWPwCsHlp8VatNV181oj7dNl6mqm6oqnVVtW758uWzeUuSpGNgNndnBbgReLSqPjk0axdw9A6rzcAdQ/Ur2l1a64Hn2ymp3cCFSZa1C+oXArvbvBeSrG/bumLSukZtQ5I0BpbMYszbgd8AHkxyf6t9GLgOuDXJlcC3gPe0eXcClwATwHeB9wJU1ZEkHwfubeM+VlVH2vT7gZuAVwNfbA+m2YYkaQzMGCJV9X+ATDH7ghHjC7hqinVtB7aPqO8DzhlRf3bUNiRJ48FPrEuSuhkikqRuhogkqZshIknqZohIkroZIpKkboaIJKmbISJJ6maISJK6GSKSpG6GiCSpmyEiSepmiEiSuhkikqRuhogkqZshIknqZohIkroZIpKkboaIJKmbISJJ6maISJK6GSKSpG6GiCSpmyEiSepmiEiSuhkikqRuhogkqZshIknqZohIkroZIpKkboaIJKmbISJJ6maISJK6GSKSpG4zhkiS7UkOJXloqPbRJAeS3N8elwzNuybJRJLHklw0VN/QahNJtg7Vz0pyT6t/LsmprX5aez3R5q85Vm9aknRszOZI5CZgw4j6p6rq3Pa4EyDJ2cAm4OfbMn+U5JQkpwB/CFwMnA1c3sYCfKKt62eA54ArW/1K4LlW/1QbJ0kaIzOGSFV9GTgyy/VtBHZW1feq6pvABHB+e0xU1RNV9X1gJ7AxSYB3Are15XcAlw6ta0ebvg24oI2XJI2J+VwTuTrJA+1017JWWwk8NTRmf6tNVX8j8O2qenFS/WXravOfb+MlSWOiN0S2AW8GzgUOAr9/zDrqkGRLkn1J9h0+fHghW5Gkk0pXiFTVM1X1g6r6IfAnDE5XARwAVg8NXdVqU9WfBZYmWTKp/rJ1tflvaONH9XNDVa2rqnXLly/veUuSpA5dIZLkzKGX7waO3rm1C9jU7qw6C1gLfAW4F1jb7sQ6lcHF911VVcBdwGVt+c3AHUPr2tymLwO+1MZLksbEkpkGJLkFeAdwRpL9wLXAO5KcCxTwJPCbAFX1cJJbgUeAF4GrquoHbT1XA7uBU4DtVfVw28RvAzuT/A7wNeDGVr8R+EySCQYX9jfN+91Kko6pGUOkqi4fUb5xRO3o+N8FfndE/U7gzhH1J3jpdNhw/f8B/3am/iRJC8dPrEuSuhkikqRuhogkqZshIknqZohIkroZIpKkboaIJKmbISJJ6maISJK6zfiJdR0/a7b+xY+mn7zuXQvYiST18UhEktTNEJEkdTNEJEndDBFJUjdDRJLUzRCRJHUzRCRJ3QwRSVI3P2x4jAx/cFCSThYeiUiSuhkikqRuns6aI7/vSpJe4pGIJKmbISJJ6maISJK6GSKSpG6GiCSpmyEiSepmiEiSuhkikqRuhogkqZshIknqZohIkrr53Vnz4Ne/SzrZzXgkkmR7kkNJHhqqnZ5kT5LH2/OyVk+S65NMJHkgyXlDy2xu4x9Psnmo/tYkD7Zlrk+S6bYhSRofszmddROwYVJtK7C3qtYCe9trgIuBte2xBdgGg0AArgXeBpwPXDsUCtuA9w0tt2GGbUiSxsSMIVJVXwaOTCpvBHa06R3ApUP1m2vgbmBpkjOBi4A9VXWkqp4D9gAb2rzXV9XdVVXAzZPWNWobkqQx0XtNZEVVHWzTTwMr2vRK4Kmhcftbbbr6/hH16baxqEy+buLfIJH0SjLvu7PaEUQdg166t5FkS5J9SfYdPnz4eLYiSRrSGyLPtFNRtOdDrX4AWD00blWrTVdfNaI+3TZ+TFXdUFXrqmrd8uXLO9+SJGmuekNkF3D0DqvNwB1D9SvaXVrrgefbKandwIVJlrUL6hcCu9u8F5Ksb3dlXTFpXaO2IUkaEzNeE0lyC/AO4Iwk+xncZXUdcGuSK4FvAe9pw+8ELgEmgO8C7wWoqiNJPg7c28Z9rKqOXqx/P4M7wF4NfLE9mGYbkqQxMWOIVNXlU8y6YMTYAq6aYj3bge0j6vuAc0bUnx21DUnS+PBrTyRJ3QwRSVI3Q0SS1M0QkSR1M0QkSd0MEUlSN/+eyAnm3yCR9ErikYgkqZshIknqZohIkroZIpKkboaIJKmbISJJ6maISJK6GSKSpG6GiCSpmyEiSepmiEiSuhkikqRuhogkqZshIknqZohIkroZIpKkbv5RqjE3/EesnrzuXQvYiST9OI9EJEndDBFJUjdDRJLUzRCRJHUzRCRJ3QwRSVI3Q0SS1M0QkSR1M0QkSd0MEUlSt3mFSJInkzyY5P4k+1rt9CR7kjzenpe1epJcn2QiyQNJzhtaz+Y2/vEkm4fqb23rn2jLZj79SpKOrWNxJPKvq+rcqlrXXm8F9lbVWmBvew1wMbC2PbYA22AQOsC1wNuA84FrjwZPG/O+oeU2HIN+JUnHyPE4nbUR2NGmdwCXDtVvroG7gaVJzgQuAvZU1ZGqeg7YA2xo815fVXdXVQE3D61LkjQG5vstvgX8VZIC/mdV3QCsqKqDbf7TwIo2vRJ4amjZ/a02XX3/iPorkt/WK2kxmm+I/FJVHUjyU8CeJN8YnllV1QLmuEqyhcEpMt70pjcd781Jkpp5nc6qqgPt+RDwBQbXNJ5pp6Joz4fa8APA6qHFV7XadPVVI+qj+rihqtZV1brly5fP5y1JkuagO0SSvDbJTx6dBi4EHgJ2AUfvsNoM3NGmdwFXtLu01gPPt9Neu4ELkyxrF9QvBHa3eS8kWd/uyrpiaF2SpDEwn9NZK4AvtLtulwB/WlV/meRe4NYkVwLfAt7Txt8JXAJMAN8F3gtQVUeSfBy4t437WFUdadPvB24CXg18sT0kSWOiO0Sq6gngF0fUnwUuGFEv4Kop1rUd2D6ivg84p7dHSdLx5SfWJUnd5nt3lo6D4dt9JWmceSQiSepmiEiSuhkikqRuhogkqZsX1l8B/N4tSQvFIxFJUjdDRJLUzRCRJHUzRCRJ3QwRSVI3Q0SS1M0QkSR1M0QkSd38sOEi4ocKJY0bj0QkSd0MEUlSN09nLVJz/cNVngqTdDwYIrPgXxqUpNE8nSVJ6maISJK6GSKSpG6GiCSpmxfWX2G8C0vSieSRiCSpm0cir2DemizpeDNE9COeCpM0V57OkiR1M0QkSd08nXUS8rSVpGPFEDnJefFd0nwYIlPwl+toHsVIGuY1EUlSt7E/EkmyAfgD4BTg01V13QK3dFKYz5GYRyvSyWOsQyTJKcAfAr8M7AfuTbKrqh5Z2M4Exy5owLCRFquxDhHgfGCiqp4ASLIT2AgYIovEfK8teVQjjbdxD5GVwFNDr/cDb1ugXnQczSZsZjPGoJFOrHEPkVlJsgXY0l7+Q5LH5riKM4C/P7ZdnRCLse/j2nM+cbzWvCj3Ndj3ibQYe4aX+v5nPQuPe4gcAFYPvV7Vai9TVTcAN/RuJMm+qlrXu/xCWYx9L8aewb5PtMXY92LsGebf97jf4nsvsDbJWUlOBTYBuxa4J0lSM9ZHIlX1YpKrgd0MbvHdXlUPL3BbkqRmrEMEoKruBO48zpvpPhW2wBZj34uxZ7DvE20x9r0Ye4Z59p2qOlaNSJJOMuN+TUSSNMZO6hBJsiHJY0kmkmxd6H6mkmR1kruSPJLk4SQfaPXTk+xJ8nh7XrbQvY6S5JQkX0vy5+31WUnuafv9c+2mibGSZGmS25J8I8mjSf7luO/vJP+5/ft4KMktSX5iHPd1ku1JDiV5aKg2ct9m4PrW/wNJzhuzvv9b+zfyQJIvJFk6NO+a1vdjSS5amK5H9z0070NJKskZ7fWc9/dJGyJDX6lyMXA2cHmSsxe2qym9CHyoqs4G1gNXtV63Anurai2wt70eRx8AHh16/QngU1X1M8BzwJUL0tX0/gD4y6r6OeAXGfQ/tvs7yUrgPwHrquocBjeibGI89/VNwIZJtan27cXA2vbYAmw7QT2OchM/3vce4Jyq+gXgb4BrANrP5ybg59syf9R+5yyEm/jxvkmyGrgQ+Luh8pz390kbIgx9pUpVfR84+pUqY6eqDlbVV9v0dxj8QlvJoN8dbdgO4NKF6XBqSVYB7wI+3V4HeCdwWxsydn0neQPwr4AbAarq+1X1bcZ/fy8BXp1kCfAa4CBjuK+r6svAkUnlqfbtRuDmGrgbWJrkzBPT6cuN6ruq/qqqXmwv72bwWTYY9L2zqr5XVd8EJhj8zjnhptjfAJ8CfgsYvjA+5/19MofIqK9UWblAvcxakjXAW4B7gBVVdbDNehpYsUBtTee/M/iH+sP2+o3At4d+8MZxv58FHAb+VzsN9+kkr2WM93dVHQB+j8H/Kg8CzwP3Mf77+qip9u1i+jn9D8AX2/RY951kI3Cgqr4+adac+z6ZQ2TRSfI64PPAB6vqheF5NbjNbqxutUvyK8ChqrpvoXuZoyXAecC2qnoL8H+ZdOpq3PZ3u4awkUEA/jTwWkacwlgMxm3fzkaSjzA47fzZhe5lJkleA3wY+C/HYn0nc4jM6itVxkWSVzEIkM9W1e2t/MzRQ832fGih+pvC24FfTfIkg9OF72RwrWFpO+UC47nf9wP7q+qe9vo2BqEyzvv73wDfrKrDVfWPwO0M9v+47+ujptq3Y/9zmuTfA78C/Hq99JmJce77zQz+s/H19rO5Cvhqkn9KR98nc4gsmq9UadcRbgQerapPDs3aBWxu05uBO050b9OpqmuqalVVrWGwf79UVb8O3AVc1oaNY99PA08l+dlWuoDBnx8Y5/39d8D6JK9p/16O9jzW+3rIVPt2F3BFu2toPfD80GmvBZfBH837LeBXq+q7Q7N2AZuSnJbkLAYXqr+yED1OVlUPVtVPVdWa9rO5Hziv/buf+/6uqpP2AVzC4I6KvwU+stD9TNPnLzE4vH8AuL89LmFwfWEv8Djw18DpC93rNO/hHcCft+l/zuAHagL4M+C0he5vRL/nAvvaPv/fwLJx39/AfwW+ATwEfAY4bRz3NXALg+s2/9h+gV051b4FwuAuyr8FHmRw99k49T3B4BrC0Z/LPx4a/5HW92PAxePU96T5TwJn9O5vP7EuSep2Mp/OkiTNkyEiSepmiEiSuhkikqRuhogkqZshIknqZohIkroZIpKkbv8fhV8RhnUkAeMAAAAASUVORK5CYII=\n"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"test['lenght_sentence'] = test['question_text'].apply(lambda x: len(x.split()))\nprint('Min questions lenght:', np.min(test['lenght_sentence'] ))\nprint('Max questions lenght:', np.max(test['lenght_sentence'] ))\nprint('Mean questions lenght:', np.mean(test['lenght_sentence'] ))\nprint('Standard deviation questions lenght:', np.std(test['lenght_sentence'] ))\n\n# Plot the distribution of the lenght of the questions\nplt.hist(test['lenght_sentence'], 100);","execution_count":5,"outputs":[{"output_type":"stream","text":"Min questions lenght: 1\nMax questions lenght: 87\nMean questions lenght: 12.81084123191221\nStandard deviation questions lenght: 7.04485108423754\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"
","image/png":"iVBORw0KGgoAAAANSUhEUgAAAYcAAAD8CAYAAACcjGjIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAFRFJREFUeJzt3X+sHeWd3/H3Zw0kbNKsTbhFXtup6WJt5CDFEBe8yqpKoQFDVjUr0Sy0DRZi461i1KRK25j8w+YHEpG6oYuUoHqDF1Nl4yCSFVbirNciSNv8geESWMAQxC0/1rYM3I35kTQq1Oy3f5zHyannXt+f9rm+9/2SRmfmO8/Mec7R2J87M885J1WFJEn9fm3QHZAkzT2GgySpw3CQJHUYDpKkDsNBktRhOEiSOgwHSVKH4SBJ6jAcJEkdpw26A9N19tln18qVKwfdDUk6pTzyyCN/X1VDE7U7ZcNh5cqVDA8PD7obknRKSfLiZNp5WUmS1GE4SJI6DAdJUofhIEnqMBwkSR2GgySpw3CQJHUYDpKkDsNBktRxyn5C+lSxcsv3fzn/wq0fG2BPJGnyPHOQJHVMGA5J3pnkoSR/m2Rfki+0+l1Jnk/yWJvWtHqS3J5kJMnjSS7s29fGJM+2aWNf/UNJnmjb3J4kJ+LFSpImZzKXld4ELqmqnyc5HfhRkh+0df+5qu49pv0VwKo2XQzcAVyc5CzgZmAtUMAjSXZW1autzSeBvcAuYD3wA05R/ZeSJOlUNOGZQ/X8vC2e3qY6ziYbgLvbdg8Ci5MsBS4H9lTV4RYIe4D1bd17qurBqirgbuCqGbwmSdIMTeqeQ5JFSR4DXqH3H/zetuqWdunotiTvaLVlwP6+zQ+02vHqB8aoS5IGZFLhUFVvV9UaYDlwUZLzgZuA9wP/DDgL+NwJ62WTZFOS4STDo6OjJ/rpJGnBmtJopap6DXgAWF9Vh9qlozeBPwcuas0OAiv6NlveaserLx+jPtbzb62qtVW1dmhowh8ykiRN02RGKw0lWdzmzwQ+Cvyk3SugjSy6CniybbITuK6NWloHvF5Vh4DdwGVJliRZAlwG7G7r3kiyru3rOuC+2X2ZkqSpmMxopaXA9iSL6IXJPVX1vSQ/TDIEBHgM+Pet/S7gSmAE+AVwPUBVHU7yJeDh1u6LVXW4zX8KuAs4k94opVN2pJIkzQcThkNVPQ5cMEb9knHaF7B5nHXbgG1j1IeB8yfqiyTp5PAT0pKkDsNBktRhOEiSOgwHSVKHX9l9Evn13ZJOFZ45SJI6DAdJUofhIEnqMBwkSR2GgySpw3CQJHUYDpKkDsNBktRhOEiSOgwHSVKH4SBJ6jAcJEkdhoMkqcNwkCR1GA6SpI4JwyHJO5M8lORvk+xL8oVWPzfJ3iQjSb6d5IxWf0dbHmnrV/bt66ZWfybJ5X319a02kmTL7L9MSdJUTObM4U3gkqr6ILAGWJ9kHfAV4LaqOg94Fbihtb8BeLXVb2vtSLIauAb4ALAe+HqSRUkWAV8DrgBWA9e2tpKkAZkwHKrn523x9DYVcAlwb6tvB65q8xvaMm39pUnS6juq6s2qeh4YAS5q00hVPVdVbwE7WltJ0oBM6mdC21/3jwDn0fsr/38Br1XVkdbkALCszS8D9gNU1ZEkrwPvbfUH+3bbv83+Y+oXT/mVDIA/+ylpvprUDemqeruq1gDL6f2l//4T2qtxJNmUZDjJ8Ojo6CC6IEkLwpRGK1XVa8ADwO8Ai5McPfNYDhxs8weBFQBt/W8AP+2vH7PNePWxnn9rVa2tqrVDQ0NT6bokaQomM1ppKMniNn8m8FHgaXohcXVrthG4r83vbMu09T+sqmr1a9popnOBVcBDwMPAqjb66Qx6N613zsaLkyRNz2TuOSwFtrf7Dr8G3FNV30vyFLAjyZeBR4E7W/s7gf+RZAQ4TO8/e6pqX5J7gKeAI8DmqnobIMmNwG5gEbCtqvbN2iuUJE3ZhOFQVY8DF4xRf47e/Ydj6/8H+Nfj7OsW4JYx6ruAXZPoryTpJJjUaCXNPkc6SZrL/PoMSVKH4SBJ6jAcJEkdhoMkqcNwkCR1GA6SpA7DQZLUYThIkjoMB0lSh+EgSeowHCRJHYaDJKnDcJAkdRgOkqQOw0GS1GE4SJI6DAdJUofhIEnqMBwkSR0ThkOSFUkeSPJUkn1JPt3qf5zkYJLH2nRl3zY3JRlJ8kySy/vq61ttJMmWvvq5Sfa2+reTnDHbL1SSNHmTOXM4Any2qlYD64DNSVa3dbdV1Zo27QJo664BPgCsB76eZFGSRcDXgCuA1cC1ffv5StvXecCrwA2z9PokSdMwYThU1aGq+nGb/xnwNLDsOJtsAHZU1ZtV9TwwAlzUppGqeq6q3gJ2ABuSBLgEuLdtvx24arovSJI0c1O655BkJXABsLeVbkzyeJJtSZa02jJgf99mB1ptvPp7gdeq6sgx9bGef1OS4STDo6OjU+m6JGkKJh0OSd4NfAf4TFW9AdwB/BawBjgE/MkJ6WGfqtpaVWurau3Q0NCJfjpJWrBOm0yjJKfTC4ZvVtV3Aarq5b71fwZ8ry0eBFb0bb681Rin/lNgcZLT2tlDf/sFYeWW7/9y/oVbPzbAnkhSz2RGKwW4E3i6qr7aV1/a1+z3gSfb/E7gmiTvSHIusAp4CHgYWNVGJp1B76b1zqoq4AHg6rb9RuC+mb0sSdJMTObM4cPAJ4AnkjzWap+nN9poDVDAC8AfAVTVviT3AE/RG+m0uareBkhyI7AbWARsq6p9bX+fA3Yk+TLwKL0wkiQNyIThUFU/AjLGql3H2eYW4JYx6rvG2q6qnqM3mkmSNAf4CWlJUofhIEnqMBwkSR2GgySpw3CQJHUYDpKkDsNBktRhOEiSOgwHSVKH4SBJ6jAcJEkdhoMkqcNwkCR1GA6SpI5J/RKcfqX/V9skab7yzEGS1GE4SJI6DAdJUof3HOaYY+9pvHDrxwbUE0kL2YRnDklWJHkgyVNJ9iX5dKuflWRPkmfb45JWT5Lbk4wkeTzJhX372tjaP5tkY1/9Q0meaNvcnmSs36yWJJ0kk7msdAT4bFWtBtYBm5OsBrYA91fVKuD+tgxwBbCqTZuAO6AXJsDNwMXARcDNRwOltflk33brZ/7SJEnTNWE4VNWhqvpxm/8Z8DSwDNgAbG/NtgNXtfkNwN3V8yCwOMlS4HJgT1UdrqpXgT3A+rbuPVX1YFUVcHffviRJAzClG9JJVgIXAHuBc6rqUFv1EnBOm18G7O/b7ECrHa9+YIy6JGlAJh0OSd4NfAf4TFW90b+u/cVfs9y3sfqwKclwkuHR0dET/XSStGBNKhySnE4vGL5ZVd9t5ZfbJSHa4yutfhBY0bf58lY7Xn35GPWOqtpaVWurau3Q0NBkui5JmobJjFYKcCfwdFV9tW/VTuDoiKONwH199evaqKV1wOvt8tNu4LIkS9qN6MuA3W3dG0nWtee6rm9fkqQBmMznHD4MfAJ4IsljrfZ54FbgniQ3AC8CH2/rdgFXAiPAL4DrAarqcJIvAQ+3dl+sqsNt/lPAXcCZwA/aJEkakAnDoap+BIz3uYNLx2hfwOZx9rUN2DZGfRg4f6K+SJJODr8+Q5LUYThIkjoMB0lSh+EgSeowHCRJHYaDJKnDcJAkdRgOkqQOw0GS1GE4SJI6/A3pU0j/70v729KSTiTPHCRJHYaDJKnDcJAkdRgOkqQOw0GS1GE4SJI6DAdJUofhIEnqMBwkSR0ThkOSbUleSfJkX+2PkxxM8libruxbd1OSkSTPJLm8r76+1UaSbOmrn5tkb6t/O8kZs/kCJUlTN5kzh7uA9WPUb6uqNW3aBZBkNXAN8IG2zdeTLEqyCPgacAWwGri2tQX4StvXecCrwA0zeUGSpJmbMByq6m+Aw5Pc3wZgR1W9WVXPAyPARW0aqarnquotYAewIUmAS4B72/bbgaum+BokSbNsJvccbkzyeLvstKTVlgH7+9ocaLXx6u8FXquqI8fUJUkDNN1wuAP4LWANcAj4k1nr0XEk2ZRkOMnw6OjoyXhKSVqQphUOVfVyVb1dVf8A/Bm9y0YAB4EVfU2Xt9p49Z8Ci5Ocdkx9vOfdWlVrq2rt0NDQdLouSZqEaf2eQ5KlVXWoLf4+cHQk007gL5J8FfhNYBXwEBBgVZJz6f3nfw3wb6qqkjwAXE3vPsRG4L7pvpiFxN92kHQiTRgOSb4FfAQ4O8kB4GbgI0nWAAW8APwRQFXtS3IP8BRwBNhcVW+3/dwI7AYWAduqal97is8BO5J8GXgUuHPWXp0kaVomDIequnaM8rj/gVfVLcAtY9R3AbvGqD/Hry5LSZLmAD8hLUnqMBwkSR2GgySpw3CQJHUYDpKkDsNBktRhOEiSOgwHSVKH4SBJ6pjWdystNP3fYyRJC4HhMA/4JXySZpuXlSRJHYaDJKnDcJAkdRgOkqQOw0GS1GE4SJI6DAdJUofhIEnqMBwkSR0ThkOSbUleSfJkX+2sJHuSPNsel7R6ktyeZCTJ40ku7NtmY2v/bJKNffUPJXmibXN7ksz2i5QkTc1kzhzuAtYfU9sC3F9Vq4D72zLAFcCqNm0C7oBemAA3AxcDFwE3Hw2U1uaTfdsd+1yagpVbvv/LSZKma8JwqKq/AQ4fU94AbG/z24Gr+up3V8+DwOIkS4HLgT1VdbiqXgX2AOvbuvdU1YNVVcDdffuSJA3IdO85nFNVh9r8S8A5bX4ZsL+v3YFWO179wBj1MSXZlGQ4yfDo6Og0uy5JmsiMb0i3v/hrFvoymefaWlVrq2rt0NDQyXhKSVqQphsOL7dLQrTHV1r9ILCir93yVjteffkYdUnSAE03HHYCR0ccbQTu66tf10YtrQNeb5efdgOXJVnSbkRfBuxu695Isq6NUrqub1+SpAGZ8Md+knwL+AhwdpID9EYd3Qrck+QG4EXg4635LuBKYAT4BXA9QFUdTvIl4OHW7otVdfQm96fojYg6E/hBmyRJAzRhOFTVteOsunSMtgVsHmc/24BtY9SHgfMn6ock6eTxZ0LnMX8+VNJ0+fUZkqQOw0GS1GE4SJI6DAdJUofhIEnqMBwkSR2GgySpw885LBB+5kHSVHjmIEnqMBwkSR2GgySpw3CQJHUYDpKkDkcrLXCOYpI0Fs8cJEkdnjmMo/8vaklaaDxzkCR1GA6SpI4ZhUOSF5I8keSxJMOtdlaSPUmebY9LWj1Jbk8ykuTxJBf27Wdja/9sko0ze0marpVbvv/LSdLCNhtnDv+iqtZU1dq2vAW4v6pWAfe3ZYArgFVt2gTcAb0wAW4GLgYuAm4+GiiSpME4EZeVNgDb2/x24Kq++t3V8yCwOMlS4HJgT1UdrqpXgT3A+hPQL0nSJM10tFIBf52kgP9eVVuBc6rqUFv/EnBOm18G7O/b9kCrjVfXAPn5B2lhm2k4/G5VHUzyj4E9SX7Sv7KqqgXHrEiyid4lKd73vvfN1m4lSceY0WWlqjrYHl8B/pLePYOX2+Ui2uMrrflBYEXf5stbbbz6WM+3tarWVtXaoaGhmXRdknQc0w6HJO9K8o+OzgOXAU8CO4GjI442Ave1+Z3AdW3U0jrg9Xb5aTdwWZIl7Ub0Za2mOcJRTNLCM5PLSucAf5nk6H7+oqr+KsnDwD1JbgBeBD7e2u8CrgRGgF8A1wNU1eEkXwIebu2+WFWHZ9AvSdIMTTscquo54INj1H8KXDpGvYDN4+xrG7Btun2RJM0uPyEtSerwi/c0JQ5xlRYGzxwkSR2GgySpw8tKmjYvMUnzl2cOkqQOzxw06zyjkE59njlIkjoMB0lSh5eVdEJ5iUk6NXnmIEnq8MxBJ41nEdKpwzMHSVKHZw4aCM8ipLnNcOjjj9kMhkEhzT1eVpIkdXjmoDnFswhpbjAcdEo49pKfwSGdWIaDTkmeYUgnluGgU95kBhIYINLUzJlwSLIe+FNgEfCNqrp1wF3SPDLemYbBIo1tToRDkkXA14CPAgeAh5PsrKqnBtszyQDRwjQnwgG4CBipqucAkuwANgCGg04544WJAaJTyVwJh2XA/r7lA8DFA+qLdELM5EOWBotOtlTVoPtAkquB9VX1h235E8DFVXXjMe02AZva4m8Dz0zyKc4G/n6Wujvf+N6Mz/dmfL4345vr780/qaqhiRrNlTOHg8CKvuXlrfb/qaqtwNap7jzJcFWtnX735i/fm/H53ozP92Z88+W9mStfn/EwsCrJuUnOAK4Bdg64T5K0YM2JM4eqOpLkRmA3vaGs26pq34C7JUkL1pwIB4Cq2gXsOkG7n/KlqAXE92Z8vjfj870Z37x4b+bEDWlJ0twyV+45SJLmkHkfDknWJ3kmyUiSLYPuzyAlWZHkgSRPJdmX5NOtflaSPUmebY9LBt3XQUiyKMmjSb7Xls9NsrcdO99ugyUWnCSLk9yb5CdJnk7yOx4zPUn+Y/u39GSSbyV553w5buZ1OPR9LccVwGrg2iSrB9urgToCfLaqVgPrgM3t/dgC3F9Vq4D72/JC9Gng6b7lrwC3VdV5wKvADQPp1eD9KfBXVfV+4IP03qMFf8wkWQb8B2BtVZ1PbzDNNcyT42ZehwN9X8tRVW8BR7+WY0GqqkNV9eM2/zN6/8iX0XtPtrdm24GrBtPDwUmyHPgY8I22HOAS4N7WZKG+L78B/HPgToCqequqXsNj5qjTgDOTnAb8OnCIeXLczPdwGOtrOZYNqC9zSpKVwAXAXuCcqjrUVr0EnDOgbg3SfwP+C/APbfm9wGtVdaQtL9Rj51xgFPjzdsntG0nehccMVXUQ+K/A39ELhdeBR5gnx818DweNIcm7ge8An6mqN/rXVW/42oIawpbk94BXquqRQfdlDjoNuBC4o6ouAP43x1xCWojHDEC7z7KBXoD+JvAuYP1AOzWL5ns4TOprORaSJKfTC4ZvVtV3W/nlJEvb+qXAK4Pq34B8GPhXSV6gd+nxEnrX2Re3ywWwcI+dA8CBqtrblu+lFxYL/ZgB+JfA81U1WlX/F/guvWNpXhw38z0c/FqOPu06+p3A01X11b5VO4GNbX4jcN/J7tsgVdVNVbW8qlbSO0Z+WFX/FngAuLo1W3DvC0BVvQTsT/LbrXQpva/SX9DHTPN3wLokv97+bR19b+bFcTPvPwSX5Ep615OPfi3HLQPu0sAk+V3gfwJP8Ktr65+nd9/hHuB9wIvAx6vq8EA6OWBJPgL8p6r6vST/lN6ZxFnAo8C/q6o3B9m/QUiyht6N+jOA54Dr6f1hueCPmSRfAP6A3kjAR4E/pHeP4ZQ/buZ9OEiSpm6+X1aSJE2D4SBJ6jAcJEkdhoMkqcNwkCR1GA6SpA7DQZLUYThIkjr+HxMBqgsqUdyQAAAAAElFTkSuQmCC\n"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"All the sentences are not of the same size, and we need them to be all at the same format to feed them to our model. \n\nTo cope with this issue we will truncate too long sentences and use a 0 padding for the short sentences. Thanks to the statistics we have extracted we will use a max_sentence_length = 20 (mean+stddev)."},{"metadata":{},"cell_type":"markdown","source":"## Building the vocabulary"},{"metadata":{},"cell_type":"markdown","source":"We are going to build a vocabulary of all the unique words in the Train and Test sets. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# First let's lower all the words in our train and test sets\ntrain['question_text_truncated'] = train['question_text'].apply(lambda x: \" \".join([word.lower() for word in x.split()[:20]]))\ntest['question_text_truncated'] = test['question_text'].apply(lambda x: \" \".join([word.lower() for word in x.split()[:20]]))\n\ntrain.head()","execution_count":6,"outputs":[{"output_type":"execute_result","execution_count":6,"data":{"text/plain":" qid ... question_text_truncated\n0 00002165364db923c7e6 ... how did quebec nationalists see their province...\n1 000032939017120e6e44 ... do you have an adopted dog, how would you enco...\n2 0000412ca6e4628ce2cf ... why does velocity affect time? does velocity a...\n3 000042bf85aa498cd78e ... how did otto von guericke used the magdeburg h...\n4 0000455dfa3e01eae3af ... can i convert montra helicon d to a mountain b...\n\n[5 rows x 5 columns]","text/html":"
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
qidquestion_texttargetlenght_sentencequestion_text_truncated
000002165364db923c7e6How did Quebec nationalists see their province...013how did quebec nationalists see their province...
1000032939017120e6e44Do you have an adopted dog, how would you enco...016do you have an adopted dog, how would you enco...
20000412ca6e4628ce2cfWhy does velocity affect time? Does velocity a...010why does velocity affect time? does velocity a...
3000042bf85aa498cd78eHow did Otto von Guericke used the Magdeburg h...09how did otto von guericke used the magdeburg h...
40000455dfa3e01eae3afCan I convert montra helicon D to a mountain b...015can i convert montra helicon d to a mountain b...
\n
"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Add all the questions\nlist_questions = list(train['question_text_truncated']) + list(test['question_text_truncated'])\n\n# Split the questions into words then join them all together and finally we remove duplicates\nunique_words = set((\" \".join(list_questions)).split())","execution_count":7,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Give an index to each word staring from 2.\nindex_from = 2\n\n# Making the vocabulary\nvocabulary = {k: (v + index_from) for v, k in enumerate(unique_words)}\n\nvocabulary[\"\"] = 0\nvocabulary[\"\"] = 1","execution_count":8,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"vocabulary","execution_count":9,"outputs":[{"output_type":"execute_result","execution_count":9,"data":{"text/plain":"{'orthotolidine?': 2,\n 'wellness/lifestyle': 3,\n 'special\"': 4,\n '\"hatiyar\"': 5,\n 'syndactyly.': 6,\n 'rope/reedfish': 7,\n 'mno2?': 8,\n 'redbubble': 9,\n '\"tsurupika': 10,\n 'phd-electrical': 11,\n 'pacifism': 12,\n 'cessation': 13,\n 'diagrams,': 14,\n 'khalifa?': 15,\n '2nights': 16,\n 'schizophrenia,': 17,\n 'r03': 18,\n 'placement.?': 19,\n 'bertoli': 20,\n 'demodulate': 21,\n 'expirstion': 22,\n 'accommodate?': 23,\n 'i18n': 24,\n 'scrach': 25,\n 'hospital/treatment': 26,\n 'relativization': 27,\n 'william,': 28,\n 'crisper': 29,\n 'overbite.': 30,\n 'navigation?': 31,\n 'qishan,': 32,\n 'ias…i': 33,\n 'phenergan?': 34,\n '51.': 35,\n 'drinking/smoking': 36,\n \"assis'\": 37,\n 'ozforex': 38,\n 'punjabees?': 39,\n 'danaans': 40,\n 'love.': 41,\n 'spectroscope)': 42,\n 'c.s.': 43,\n 'is/means': 44,\n 'over-taxation?': 45,\n '(nothing,': 46,\n '\"300\"?': 47,\n 'computers,': 48,\n 'communty': 49,\n \"'dangal'\": 50,\n '64-bits': 51,\n 'wholeheartedly': 52,\n 'decomposed?': 53,\n 'scet': 54,\n '“this': 55,\n '2^x-2^(-x)': 56,\n 'cleartrip?': 57,\n 'cancellation?': 58,\n '\"alone': 59,\n \"'decision'\": 60,\n '\"deathly': 61,\n 'deployed?': 62,\n 'pakprep.com': 63,\n 'e^-x]?': 64,\n '(n5110)': 65,\n '\"prophet\"': 66,\n '28,': 67,\n \"davis's\": 68,\n 'poles,': 69,\n 'hawkins?': 70,\n \"exists'\": 71,\n 'b.p.d': 72,\n 'multistate': 73,\n 'weeks\"?': 74,\n 'how…?': 75,\n '(jtrs': 76,\n 'unfocused.': 77,\n 'rukku?': 78,\n 'misconceptions/stigma': 79,\n 'rusenshirikan?': 80,\n 'washrooms?': 81,\n 'thigpen': 82,\n 'fruit,': 83,\n 'boardgame': 84,\n 'tum': 85,\n 'genitalia,': 86,\n 'range/distribution': 87,\n 'alifa': 88,\n \"''how\": 89,\n 'die-off': 90,\n 'stansberry': 91,\n 'sext?': 92,\n 'post-work?': 93,\n 'rosalind': 94,\n 'resilience': 95,\n \"sanyasi's\": 96,\n '$17?': 97,\n 'noncitizen': 98,\n 'song\"?': 99,\n '96.76': 100,\n '(compliment': 101,\n \"x'former,\": 102,\n 'welham': 103,\n 'consumption,': 104,\n 'paschimamoolaprati,': 105,\n 'backed,': 106,\n '1029,': 107,\n '753?': 108,\n \"'tora\": 109,\n 'patriotic': 110,\n 'gan)?': 111,\n 'dorsi': 112,\n 'sadhanas': 113,\n '4–5': 114,\n 'semotics?': 115,\n 'zwinglianism': 116,\n 'dank,': 117,\n 'incus': 118,\n 'place)': 119,\n 't，': 120,\n 'been\"': 121,\n 'parx': 122,\n 'octogon?': 123,\n 'it…': 124,\n 'neft/imps': 125,\n 'strated?': 126,\n 'iq?(indian': 127,\n 'planets/galaxies/star': 128,\n 'harrat': 129,\n 'hydrolic': 130,\n 'aloud/lecturing': 131,\n '(3,2,1)': 132,\n 'office)': 133,\n '15au620tx': 134,\n 'water.how': 135,\n 'fischer?': 136,\n 'stamcells,': 137,\n 'riel': 138,\n 'mirisch': 139,\n '850/m': 140,\n 'svgs': 141,\n 'cymbalta?': 142,\n 'hibiclens': 143,\n \"blanche'?\": 144,\n 'consumables?': 145,\n \"audible's\": 146,\n 'maax': 147,\n '3ax': 148,\n 'system,\"': 149,\n '(nlp)': 150,\n 'makeup/faces': 151,\n 'gls': 152,\n 'factory\"': 153,\n 'bandaranaike': 154,\n 'thumma': 155,\n 'report:': 156,\n \"u.s.'s\": 157,\n 'bi-product': 158,\n 'frying?': 159,\n 'moxycheese': 160,\n 'terrain': 161,\n 'call()': 162,\n 'ergonomic': 163,\n 'igims': 164,\n 'are.': 165,\n 'widom?': 166,\n 'provenance': 167,\n \"ptb's\": 168,\n 'bfr,': 169,\n 'gymnasium': 170,\n '[math]\\\\sin(\\\\alpha-\\\\beta)=\\\\frac{5}{13}[/math],': 171,\n 'harry': 172,\n 'gatorade?': 173,\n \"'86\": 174,\n 'peanuts,': 175,\n '38m/s': 176,\n 'sylaabus': 177,\n 'placket?': 178,\n 'hard-boiled': 179,\n 'timofey': 180,\n 'franhise?': 181,\n 'heals': 182,\n '“wow,': 183,\n 'frameworks,': 184,\n 'pong': 185,\n 'illusionist': 186,\n '.enc': 187,\n 'preference.': 188,\n 'e.gold': 189,\n 'nosey?': 190,\n 'bivins': 191,\n 'vogue.': 192,\n 'f(1)': 193,\n 'germeny?': 194,\n 'betting:': 195,\n 'quanta': 196,\n '(rainfall)': 197,\n \"hickam's\": 198,\n 'mgcl2.8h2o': 199,\n '16-gauge': 200,\n 'counter-balance': 201,\n 'liberalised': 202,\n 'mis:': 203,\n 'againest': 204,\n 'holsell': 205,\n 'bookcase?': 206,\n 'ican?': 207,\n \"nervous'?\": 208,\n 'is/are': 209,\n '4520s?': 210,\n 'sanjukta?': 211,\n 'pune/navi': 212,\n 'boycott?': 213,\n 'c-suite': 214,\n '/(x-5)': 215,\n 'babypips': 216,\n 'abarbanel?': 217,\n '\\\\alpha': 218,\n '7.0)?': 219,\n 'yoshino': 220,\n 'assignments/projects': 221,\n 'background-image': 222,\n '2015,havent': 223,\n '(become)': 224,\n 'co.': 225,\n 'salmons': 226,\n 'coupled': 227,\n 'moulton': 228,\n 'kanigiri?': 229,\n 'inductionmotor': 230,\n 'magnetisation': 231,\n 'freahers': 232,\n 'lean-7': 233,\n '(them)': 234,\n 'casino,': 235,\n 'clumpy': 236,\n 'rape?\"': 237,\n \"carmichael's\": 238,\n 'vng': 239,\n 'merch,': 240,\n 'accts,': 241,\n '3yrs,': 242,\n 'it…?\"': 243,\n 'stateprovider': 244,\n 'autonomic': 245,\n 'boa,': 246,\n '2mg/tid': 247,\n 's.h.i.e.l.d?': 248,\n '\"correlated': 249,\n \"gödel's\": 250,\n 'audiable.com': 251,\n 'bangle?': 252,\n 'african-descended': 253,\n 'ornament\"?': 254,\n \"emperor's\": 255,\n 'sexuality.': 256,\n 'chappan': 257,\n 'cdn': 258,\n 'jio-netflix': 259,\n 'explained)': 260,\n 'ttsm': 261,\n 'c9': 262,\n 'homesick,': 263,\n 'redheads': 264,\n 'low-level': 265,\n \"tone's\": 266,\n 'protrude,': 267,\n '52789': 268,\n 'opomoney.bid': 269,\n 'panic!\"': 270,\n '^-1=b/a?': 271,\n 'teams)': 272,\n 'nuturing': 273,\n 'arledge': 274,\n 'levels,': 275,\n 'vaatu': 276,\n '2.0.': 277,\n 'pro\"': 278,\n 'pleasures': 279,\n 'shell_exec': 280,\n 'unprogram': 281,\n 'non-ac': 282,\n \"banana'\": 283,\n '18khz': 284,\n '3+?': 285,\n 'physisian': 286,\n 'abstinence-only': 287,\n 'unforeseeable': 288,\n 'exception': 289,\n \"craven's\": 290,\n 'secreatly?': 291,\n 'www.thebodhiman.com?': 292,\n '(-1)': 293,\n 'mendatory?': 294,\n 'mayans,': 295,\n '‘speaks’?': 296,\n \"pharmaceuticals'\": 297,\n 'beamdog?': 298,\n 'netting': 299,\n 'indemand': 300,\n 'workaway.info,': 301,\n 'junipera': 302,\n 'path/program': 303,\n 'dystopia': 304,\n '“highly': 305,\n 'fring': 306,\n 'calment,': 307,\n 'whiny': 308,\n 'creepiest?': 309,\n 'familiarize': 310,\n 'jadwiga': 311,\n 'corsaut': 312,\n 'chami': 313,\n 'g.p.a': 314,\n 'doctrine?)': 315,\n 'angular?': 316,\n 'fraudsters': 317,\n \"takeshi's\": 318,\n 'morai': 319,\n 'kuhn?': 320,\n 'primer?': 321,\n '(during)': 322,\n 'shuttering': 323,\n 'certificate.how': 324,\n '137975': 325,\n '948': 326,\n 'checkbook?': 327,\n 'earlobe': 328,\n '(kuwait).': 329,\n \"jmc's\": 330,\n 'family).': 331,\n 'kovils': 332,\n \"'best\": 333,\n 'purandar-mahabaleshwar?': 334,\n 'hysteroscopy?': 335,\n 'purushottam': 336,\n 'jethmalani': 337,\n 'daraz.pk': 338,\n 'beano': 339,\n 'quiz,': 340,\n 'gratification.': 341,\n 'parent.': 342,\n \"baswedan's\": 343,\n '“da”': 344,\n 'province\"': 345,\n 'eisen?': 346,\n 'fosdick': 347,\n 'ethel': 348,\n 'creator.': 349,\n 'erythroplakia': 350,\n 'bonno?': 351,\n '2l': 352,\n 'tips/options': 353,\n 'fluctuates': 354,\n 'del': 355,\n '$65,000': 356,\n 'favorite': 357,\n 'praising': 358,\n 'locomotion': 359,\n 'shield': 360,\n 'dreamland': 361,\n 'frame…': 362,\n 'saif,': 363,\n 'educe': 364,\n 'repeatable?': 365,\n 'zhinaren(支那人)?': 366,\n \"nilsson's\": 367,\n 'traininhs': 368,\n 'ghoster?': 369,\n '\"ghar': 370,\n 'splitting?': 371,\n 'barriers,': 372,\n '1•5': 373,\n 'fauxhawk': 374,\n 'rich-but': 375,\n 'forex?': 376,\n 'intricacies': 377,\n 'reatment?': 378,\n 'croc': 379,\n 'inpainting': 380,\n 'premechanism?': 381,\n 'pedophiles:': 382,\n 'apler': 383,\n '-cgi': 384,\n 'education-focused': 385,\n '\"progress\"': 386,\n 'desam': 387,\n 'trekkies': 388,\n 'pruitt': 389,\n '\"ghayal': 390,\n 'grief\"?': 391,\n '(stakeholders)': 392,\n \"'stole'\": 393,\n 'gloat?': 394,\n 'ethnicity/ethnicities': 395,\n 'norris?': 396,\n 'tsllest': 397,\n 'fearfull/judging': 398,\n 'snoring': 399,\n 'conbritute': 400,\n '1.100*10': 401,\n 'conman,': 402,\n 'pokemon’s': 403,\n 'deleted],': 404,\n 'backbend': 405,\n '\"taliban': 406,\n '128.107.255.254': 407,\n 'birdies.': 408,\n 'christoph': 409,\n 'high-carbon': 410,\n 'frienships?': 411,\n 'colposcopy': 412,\n 'byzantines)?': 413,\n 'aconcagua': 414,\n 'congestion': 415,\n 'muralitharan': 416,\n '(cateogary:': 417,\n \"fender's\": 418,\n 'gimbal': 419,\n 'inventus': 420,\n 'bicyclists': 421,\n 'moana': 422,\n 'refridgerate': 423,\n 'cybermen': 424,\n '\"netanyahu': 425,\n \"'islamist'\": 426,\n 'bundestag,': 427,\n 'sanganer': 428,\n 'google-ing': 429,\n 'psat?': 430,\n 'pinacole': 431,\n 'assualted': 432,\n '/p\"': 433,\n 'me,so': 434,\n 'failed.if': 435,\n \"salazar's\": 436,\n 'recognize/tell': 437,\n \"hogwarts'\": 438,\n '(tools': 439,\n 'talks,': 440,\n 'indication?': 441,\n '(fall)?': 442,\n '/4^n,': 443,\n 'declaired': 444,\n 'starfighter)': 445,\n 'slayer?': 446,\n 'econometrics.?': 447,\n 'masturbators': 448,\n 'achievership': 449,\n 'carpel?': 450,\n 'interior,': 451,\n '(netflix)?': 452,\n 'artist\"?': 453,\n 'way/approach': 454,\n 'whent': 455,\n 'despotism?': 456,\n 'spritituality?': 457,\n 'sel,': 458,\n 'eyes.she': 459,\n 'eryka': 460,\n 'btwin': 461,\n 'other\\u200b': 462,\n 'patronuses': 463,\n 'covacrete?': 464,\n 'youtuber)': 465,\n 'sniper': 466,\n 'workout/practice': 467,\n '1992\"?': 468,\n 'pictones': 469,\n '$@?': 470,\n 'karib': 471,\n '\"elevator': 472,\n 'unsampled': 473,\n 'badshah?': 474,\n '\\\\displaystyle\\\\int': 475,\n 'eeriest': 476,\n 'comprise': 477,\n 'airtv,': 478,\n 'granulation': 479,\n 'electropneumatics?': 480,\n 'demotic': 481,\n 'mindset?': 482,\n 'bastards?': 483,\n 'boisar,': 484,\n 'disciple\"?': 485,\n 'parkes': 486,\n 'much…': 487,\n 'gambesons': 488,\n 'tetm?': 489,\n '(lda)': 490,\n 'naseem': 491,\n 'stalking?': 492,\n \"'testimony\": 493,\n 'fatalities?': 494,\n 'andhrabank': 495,\n 'finsta': 496,\n 'superior': 497,\n 'diminuendos': 498,\n 'bitch,': 499,\n 'andreas,': 500,\n \"'dick\": 501,\n 'smyth?': 502,\n 'pvt/ltd': 503,\n 'hawala': 504,\n 'sktcho.com?': 505,\n 'jnu-ceeb': 506,\n 'slit?': 507,\n \"xd's\": 508,\n 'e-series': 509,\n 'desperate': 510,\n 'nonroutine': 511,\n \"papa's\": 512,\n \"'teen\": 513,\n 'yz^2+xz^2+y^2z+x^2y+x^2z+xy^2+2xyz?': 514,\n 'transplant': 515,\n 'h160': 516,\n 'anti-democracy': 517,\n 'pump\"?': 518,\n '#100daysofcode': 519,\n 'studio/distributor': 520,\n 'mrityunjaya': 521,\n '3√5÷4√2?': 522,\n 'peril?': 523,\n 'bachelord': 524,\n 'aaptards': 525,\n 'remuneration?': 526,\n 'lebensraum?': 527,\n 'constitutuons': 528,\n '\"imitation\"': 529,\n 'spamming/whooping': 530,\n 'norte': 531,\n 'perpendicularly?': 532,\n '\"mechanical\"?': 533,\n 'repellor?': 534,\n 'xenophobes?': 535,\n 'synthesis?': 536,\n 'e^-0?': 537,\n 'kundliya\"in': 538,\n '$31,561,200': 539,\n 'else!': 540,\n 'bsin': 541,\n 'yantai,': 542,\n 'nates': 543,\n '\"uk\"': 544,\n 'lingayats?': 545,\n 'expexted': 546,\n 'tattoos/illustrations': 547,\n 'alissa': 548,\n 'harlequinn?': 549,\n 'necessary)?': 550,\n 'structure\"?': 551,\n 'a^2-b^2=90?': 552,\n 'itm': 553,\n 'scolds': 554,\n 'purches': 555,\n 'doordash?': 556,\n 'moderates': 557,\n 'disgree': 558,\n 'nantucket': 559,\n 'bumi': 560,\n 'vais': 561,\n 'amrita': 562,\n 'buck': 563,\n 'biological/athletic': 564,\n 'domesticates': 565,\n 'arginine,': 566,\n 'permenganate': 567,\n 'stroud': 568,\n 'lebron': 569,\n \"'qalam'\": 570,\n '\"besi': 571,\n '“willpower': 572,\n 'mikael': 573,\n 'poseidon,': 574,\n 'skikda,': 575,\n 'disabled?': 576,\n 'kropotkin?': 577,\n 'samosa,': 578,\n '535?': 579,\n 'taehynung': 580,\n \"asic's\": 581,\n 'game’s': 582,\n 'c7h8obr3?': 583,\n 'powerlifter)': 584,\n 'pre-molded': 585,\n 'murabba?': 586,\n \"'return'\": 587,\n 'karkota': 588,\n 'defferant': 589,\n 'road)?': 590,\n \"'intimidate\": 591,\n 'quartering': 592,\n 'pentoarbital': 593,\n 'bedridden': 594,\n 'defilement?': 595,\n 'homesman\"': 596,\n 'esurance?': 597,\n '(chromecast)': 598,\n 'sharbhi': 599,\n 'retires': 600,\n 'counter/delay': 601,\n '\"rip': 602,\n 'bosses/managers?': 603,\n '(baptism,': 604,\n 'vertex-disjoint': 605,\n 'diwani?': 606,\n 'salamanders?': 607,\n '\"facetards\"?': 608,\n 'anjani': 609,\n 'euromillions': 610,\n 'rbcoe': 611,\n 's*x?': 612,\n 'galilei': 613,\n '$120?': 614,\n 'bran?': 615,\n 'aerobe?': 616,\n \"'tolerant'\": 617,\n 'sfw': 618,\n 'rollout?': 619,\n \"manetti's\": 620,\n 'raf?': 621,\n 'ever.is': 622,\n '‘legally': 623,\n '6.64': 624,\n 'movie.': 625,\n 'physic?': 626,\n 'illinios': 627,\n 'khalistan': 628,\n 'iduce': 629,\n 'copen': 630,\n 'zebpay,': 631,\n 'rhinosinusitis?': 632,\n 'feared.now': 633,\n \"income'\": 634,\n 'multi-view': 635,\n 'kosh': 636,\n 'rollup?': 637,\n \"'mom'\": 638,\n 'surface-enhanced': 639,\n 'bottomers': 640,\n '-2.00.': 641,\n 'indo-greek': 642,\n '(watercolor,': 643,\n 'namikaze)': 644,\n '\\\\frac{adjacent}{hypotenuse}[/math],': 645,\n 'hendrix': 646,\n ':ryt]': 647,\n 'balconies': 648,\n 'to-do-list?': 649,\n '*top': 650,\n '$300/month': 651,\n \"genetic's\": 652,\n '-maybe': 653,\n 'poa?': 654,\n 'regality?': 655,\n 'kesey': 656,\n '\"unfollow\"': 657,\n 'developing/third': 658,\n '[owner/operator]': 659,\n 'watsap': 660,\n 'payback': 661,\n 'nonsecular': 662,\n 'seminoles': 663,\n 'breakage/formation': 664,\n 'tycoonia': 665,\n 'radioactivity?': 666,\n '\"added': 667,\n 'snort?': 668,\n 'drief': 669,\n '360,000': 670,\n 'sub-ethnicity': 671,\n 'pumps?': 672,\n 'residues': 673,\n 'aravrit': 674,\n 'specializing?': 675,\n 'landings.': 676,\n 'thereof?': 677,\n 'meccano': 678,\n 'npat?': 679,\n '11-year-old?': 680,\n 'instable?': 681,\n 'mention.': 682,\n 'works.?': 683,\n 'pureit': 684,\n '1.0g': 685,\n 'paulianity?': 686,\n 'opposition\\u200b': 687,\n 'succulent?': 688,\n 'loaned': 689,\n 'jaxp,': 690,\n 'axl': 691,\n 'na2co3,': 692,\n 'pepsi)?': 693,\n 't-90ms': 694,\n 'iims,': 695,\n '\"penis\"?': 696,\n '45000?': 697,\n 'blueprism': 698,\n '=90,': 699,\n '\"wing\"': 700,\n '(shirt,': 701,\n 'powar': 702,\n 'suspension': 703,\n 'vakde?': 704,\n 'buckminster': 705,\n 'dulcificum': 706,\n 'nahda?': 707,\n 'lonavala?': 708,\n '\"icarly\",': 709,\n 'refreshable': 710,\n 'ligature': 711,\n \"non-indian's\": 712,\n '“grantville”': 713,\n 'syphillis': 714,\n 'amaranth?': 715,\n 'villiers?': 716,\n 'hand-made': 717,\n 'requries': 718,\n 'woop': 719,\n 'w3schools.com': 720,\n 'diva': 721,\n 'jed': 722,\n 'weither': 723,\n 'darndest': 724,\n 'introspection': 725,\n 'self-respect': 726,\n '(pvcs)': 727,\n 'interrogated/beaten?': 728,\n 'ambassadors,': 729,\n 'weirdest/funniest/craziest': 730,\n 'self-righteous?': 731,\n 'other.can': 732,\n 'mutton': 733,\n 'amout': 734,\n 'billets?': 735,\n \"'me\": 736,\n '2000-01': 737,\n '(nyc)?': 738,\n 'coap': 739,\n '127v': 740,\n 'microbiologist': 741,\n 'schools/colleges': 742,\n 'manganin': 743,\n 'non-caucasian': 744,\n 'vigna?': 745,\n 'greenwood,': 746,\n 'www.spainbearing.com?': 747,\n '10000-20000?': 748,\n 'flebicite': 749,\n 'rice-sufficiency,': 750,\n 'impersonator': 751,\n '$860': 752,\n 'gunther': 753,\n 'trinomial?': 754,\n 'labo': 755,\n 'isin\\\\displaystyle\\\\sum[/math]?': 756,\n 'chefs': 757,\n 'recycling/refurbishing/reusing': 758,\n 'microcephaly': 759,\n 'womam': 760,\n 'globe,': 761,\n 'tansen?': 762,\n 'lasky?': 763,\n \"kartagener's\": 764,\n 'pre-workout,': 765,\n 'mk7': 766,\n 'hava.io': 767,\n 'philosophers,': 768,\n 'manhole?': 769,\n 'l-hopital': 770,\n 'red-eye': 771,\n 'tallinn,': 772,\n '-2x^2+4x+1?': 773,\n 'kagamine': 774,\n 'trejo': 775,\n '\"abigail\"': 776,\n 'permiability?': 777,\n 'ceus?': 778,\n 'lacto?': 779,\n '9.7/10': 780,\n 'sensitve': 781,\n 'relocate,': 782,\n 'overrated': 783,\n '3.kritva?': 784,\n 'braille?': 785,\n 'terrible?': 786,\n 'e^xtan': 787,\n 'all]': 788,\n 'chinese-russian': 789,\n \"'fulfilled\": 790,\n 'blinders\"?': 791,\n 'txt?': 792,\n 'laungauge': 793,\n 'provides?': 794,\n 'ststistical': 795,\n \"'vertalers'\": 796,\n 'assam,': 797,\n '\"ee-woss,\"': 798,\n \"padhuka's\": 799,\n 'lector': 800,\n 'reviews.': 801,\n 'aes,': 802,\n 'ejacualated?': 803,\n '115.50': 804,\n 'e-lottery?': 805,\n 'stroganoff?': 806,\n 'diagramming': 807,\n \"dell's\": 808,\n 'santner?': 809,\n 'teaghan': 810,\n 'csf': 811,\n 'enide': 812,\n '\"consumed': 813,\n 'complicate': 814,\n 'eldercare?': 815,\n '(detailed)': 816,\n \"bro's.\": 817,\n 'lightening': 818,\n '\"guessing': 819,\n 'millenials,': 820,\n 'enland': 821,\n 'feed)': 822,\n 'cite,': 823,\n 'worsening': 824,\n '0.050m': 825,\n 'permeable?': 826,\n 'syrua': 827,\n 'droppes': 828,\n 'ssbb5?': 829,\n 'phenyl)': 830,\n 'mccabe?': 831,\n 'cook\"': 832,\n '\"european\"items?': 833,\n 'officers/government': 834,\n '400': 835,\n 'ocl-': 836,\n 'dating/close': 837,\n 'aroused.': 838,\n 'tp=2,': 839,\n 'arousal-drugs?': 840,\n 'cenema?': 841,\n 'dissatisfies': 842,\n 'firms,': 843,\n 'satim': 844,\n 'tannirbhavi': 845,\n 'joule-thomson': 846,\n 'logics': 847,\n 'delivers,': 848,\n 'remake,': 849,\n 'cara\"': 850,\n 'astrophysicists?': 851,\n 'army/iranian': 852,\n '(3%)': 853,\n 'batra': 854,\n 'v.a.': 855,\n 'deductive/investigative': 856,\n 'logarithms': 857,\n 'mukherjee?': 858,\n '\"soulfulness\"?': 859,\n '\"jip,': 860,\n 'fear/anxiety': 861,\n 'siya': 862,\n 'markipler': 863,\n 'shortly\"?': 864,\n '5.07': 865,\n 'wirree': 866,\n 'ahoy?': 867,\n \"yoga's\": 868,\n 'funtioning': 869,\n 'surrendered?': 870,\n 'lezierae': 871,\n 'letters)': 872,\n 'mantri?': 873,\n 'eastbourne': 874,\n 'sin(e^t)': 875,\n '<': 876,\n 'vehicles?': 877,\n 'post-modern,': 878,\n 'confucian': 879,\n 'rafel': 880,\n 'i^?': 881,\n '(4n+27),': 882,\n \"'customized'\": 883,\n 'defying': 884,\n 'research-based': 885,\n 'graphene-based': 886,\n '5-3×2+3?': 887,\n 'runners.': 888,\n 'karui,': 889,\n 'son”': 890,\n 'kansas': 891,\n 'majora?': 892,\n 'moranis?': 893,\n 'benefits\"?': 894,\n 'austronesia': 895,\n 'zero-suit': 896,\n 'tupe': 897,\n '0.05)': 898,\n 'resources/tutorial/articles': 899,\n 'blogs,': 900,\n 'cultivate': 901,\n 'ishta': 902,\n \"'associate\": 903,\n \"mozilla's\": 904,\n '(lithuania)': 905,\n 'priciple?': 906,\n 'unclear?': 907,\n 'elevate?': 908,\n 'paraphrasing,': 909,\n 'protoplast?': 910,\n 'kalkutta': 911,\n 'kind\"?': 912,\n '288.': 913,\n '4040': 914,\n 'ewm?': 915,\n 'literete': 916,\n 'sacrificing': 917,\n 'whasapp': 918,\n 'book/theory/preaching': 919,\n 'manikonda': 920,\n '784.14': 921,\n '150mh/s': 922,\n \"morty'\": 923,\n 'icmr-jrf?': 924,\n 'iit/nit..what': 925,\n '569': 926,\n 'provider(s)': 927,\n 'cecile': 928,\n 'unserved': 929,\n 'datastage?': 930,\n 'prohibition': 931,\n 'noise/way': 932,\n 'marroww': 933,\n '“has': 934,\n 'hydra\"': 935,\n 'pakixtan?': 936,\n 'decussate': 937,\n 'victms': 938,\n 'finna': 939,\n 'cython?': 940,\n 'thornwood': 941,\n 'grant,': 942,\n 'selfishlandlady?': 943,\n '5662': 944,\n 'hoisting?': 945,\n 'podiatry?': 946,\n 'reprinted?': 947,\n 'cadd9,': 948,\n 'non-belligerent),': 949,\n '(bnf)': 950,\n 'decsion': 951,\n 'freeze.': 952,\n 'technoloyy?': 953,\n '(navy': 954,\n 'skinheads': 955,\n 'colonization.': 956,\n 'hotstar?': 957,\n 'violation?': 958,\n \"pacific'\": 959,\n 'almeria?': 960,\n 'at/listen': 961,\n 'factory\"?': 962,\n 'money-saving': 963,\n 'cultural)?': 964,\n 'ahmediyyas': 965,\n 'along?': 966,\n '\"endeavour\"': 967,\n '[/math]p(x)[/math]': 968,\n \"brozik's\": 969,\n '100’s?': 970,\n 'pit/pi?': 971,\n 'semen-filled': 972,\n 'avow': 973,\n 'kweisi': 974,\n 'amplifier': 975,\n 'necasssy': 976,\n 'non-selfish': 977,\n 'stenotype': 978,\n '[6]': 979,\n 'sittinh': 980,\n 'bronchoconstriction?': 981,\n 'holobionts?': 982,\n '32inch': 983,\n 'digipad': 984,\n 'gacy,': 985,\n 'acceleration…': 986,\n 'kardar?': 987,\n 'hi-power': 988,\n 'book/series': 989,\n 'smashburgers?': 990,\n '2000mah': 991,\n 'meghan?': 992,\n 'nhf2?': 993,\n '£45k.': 994,\n 'wring': 995,\n 'cryptocuttrncy?': 996,\n 'g350e,': 997,\n '49000?': 998,\n 'volunteers)?': 999,\n 'monocyte': 1000,\n 'tribes:': 1001,\n ...}"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Vocabulary length:', len(vocabulary))","execution_count":10,"outputs":[{"output_type":"stream","text":"Vocabulary length: 497852\n","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"## Tokenization of the questions"},{"metadata":{},"cell_type":"markdown","source":"Now that we have our vocabulary, we will tokenize our sentences and pad with 0 the smaller ones to reach a 134 sequence size."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Tokenization of all words in a sentence using our vocabulary\ndef sentence_tokenization(sentence, vocabulary):\n tokenized_sentence = []\n for word in sentence.split():\n tokenized_sentence.append(vocabulary[word])\n return tokenized_sentence\n \n\ntrain[\"question_tokenized\"] = train[\"question_text_truncated\"].apply(lambda x: sentence_tokenization(x, vocabulary))\ntest[\"question_tokenized\"] = test[\"question_text_truncated\"].apply(lambda x: sentence_tokenization(x, vocabulary))\ntrain.head()","execution_count":11,"outputs":[{"output_type":"execute_result","execution_count":11,"data":{"text/plain":" qid ... question_tokenized\n0 00002165364db923c7e6 ... [407716, 413381, 43056, 256481, 405959, 273237...\n1 000032939017120e6e44 ... [354193, 316362, 423148, 140019, 177645, 60715...\n2 0000412ca6e4628ce2cf ... [119044, 358720, 490330, 268179, 15699, 358720...\n3 000042bf85aa498cd78e ... [407716, 413381, 434631, 381917, 247766, 85088...\n4 0000455dfa3e01eae3af ... [446096, 113851, 452152, 268495, 45502, 60160,...\n\n[5 rows x 6 columns]","text/html":"
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
qidquestion_texttargetlenght_sentencequestion_text_truncatedquestion_tokenized
000002165364db923c7e6How did Quebec nationalists see their province...013how did quebec nationalists see their province...[407716, 413381, 43056, 256481, 405959, 273237...
1000032939017120e6e44Do you have an adopted dog, how would you enco...016do you have an adopted dog, how would you enco...[354193, 316362, 423148, 140019, 177645, 60715...
20000412ca6e4628ce2cfWhy does velocity affect time? Does velocity a...010why does velocity affect time? does velocity a...[119044, 358720, 490330, 268179, 15699, 358720...
3000042bf85aa498cd78eHow did Otto von Guericke used the Magdeburg h...09how did otto von guericke used the magdeburg h...[407716, 413381, 434631, 381917, 247766, 85088...
40000455dfa3e01eae3afCan I convert montra helicon D to a mountain b...015can i convert montra helicon d to a mountain b...[446096, 113851, 452152, 268495, 45502, 60160,...
\n
"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"input_max_length = 20\n# 0 padding of the tokenized questions\nX = sequence.pad_sequences(train['question_tokenized'], maxlen = input_max_length, padding = \"post\", truncating= \"post\", value = 0)\nX_test = sequence.pad_sequences(test['question_tokenized'], maxlen = input_max_length, padding = \"post\", truncating= \"post\", value = 0)\n\ny = train['target']","execution_count":12,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# We prepare our data for the training and validation steps which we will make to avoid overfitting\n# Train/Validate split is less time consuming than several folds cross-validation\nX_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Modelisation: Many-to-One model with LSTM layer and Embedding on top"},{"metadata":{"trusted":true},"cell_type":"code","source":"embedding_vector_length = 150\ntotal_words = len(vocabulary) \ninputs_max_length = 20\n\nmodel = Sequential()\nmodel.add(Embedding(total_words, embedding_vector_length, input_length = inputs_max_length))\nmodel.add(LSTM(units = 256))\nmodel.add(Dropout(0.3))\nmodel.add(Dense(1, activation='sigmoid'))\n\nprint(model.summary())","execution_count":13,"outputs":[{"output_type":"stream","text":"WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\nInstructions for updating:\nColocations handled automatically by placer.\nWARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\nInstructions for updating:\nPlease use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n_________________________________________________________________\nLayer (type) Output Shape Param # \n=================================================================\nembedding_1 (Embedding) (None, 20, 150) 74677800 \n_________________________________________________________________\nlstm_1 (LSTM) (None, 256) 416768 \n_________________________________________________________________\ndropout_1 (Dropout) (None, 256) 0 \n_________________________________________________________________\ndense_1 (Dense) (None, 1) 257 \n=================================================================\nTotal params: 75,094,825\nTrainable params: 75,094,825\nNon-trainable params: 0\n_________________________________________________________________\nNone\n","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"We can see above that the Embedding task gathers about 99% of our total number of parameters!"},{"metadata":{"trusted":true},"cell_type":"code","source":"# We must build a custom F1 metrics to plug it into our training steps with Keras\ndef f1(y_true, y_pred):\n '''\n metric from here \n https://stackoverflow.com/questions/43547402/how-to-calculate-f1-macro-in-keras\n '''\n def recall(y_true, y_pred):\n \"\"\"Recall metric.\n\n Only computes a batch-wise average of recall.\n\n Computes the recall, a metric for multi-label classification of\n how many relevant items are selected.\n \"\"\"\n true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n recall = true_positives / (possible_positives + K.epsilon())\n return recall\n\n def precision(y_true, y_pred):\n \"\"\"Precision metric.\n\n Only computes a batch-wise average of precision.\n\n Computes the precision, a metric for multi-label classification of\n how many selected items are relevant.\n \"\"\"\n true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n precision = true_positives / (predicted_positives + K.epsilon())\n return precision\n \n precision = precision(y_true, y_pred)\n recall = recall(y_true, y_pred)\n return 2*((precision*recall)/(precision+recall+K.epsilon()))","execution_count":14,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Compile and fit the model on our Train/Validate datasets\n\n#model.compile(loss='binary_crossentropy', optimizer=Adam(lr=0.001), metrics=[f1])\n#model.fit(X_train, y_train, epochs=5, batch_size=64, validation_data=(X_val, y_val))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Training on the whole dataset\nmodel.compile(loss='binary_crossentropy', optimizer=Adam(lr=0.001), metrics=[f1])\nmodel.fit(X, y, epochs=3, batch_size=64)","execution_count":17,"outputs":[{"output_type":"stream","text":"WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\nInstructions for updating:\nUse tf.cast instead.\nEpoch 1/3\n1306122/1306122 [==============================] - 1077s 824us/step - loss: 0.1230 - f1: 0.4914\nEpoch 2/3\n1306122/1306122 [==============================] - 1060s 812us/step - loss: 0.0936 - f1: 0.6337\nEpoch 3/3\n1306122/1306122 [==============================] - 1054s 807us/step - loss: 0.0749 - f1: 0.7118\n","name":"stdout"},{"output_type":"execute_result","execution_count":17,"data":{"text/plain":""},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"# Predictions and submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_test = np.where(model.predict(X_test, batch_size=1024) < 0.5, 0, 1)","execution_count":18,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\npredictions = pd.DataFrame({\"qid\":test[\"qid\"].values})\npredictions['prediction'] = pred_test\npredictions.head()","execution_count":21,"outputs":[{"output_type":"execute_result","execution_count":21,"data":{"text/plain":" qid prediction\n0 0000163e3ea7c7a74cd7 1\n1 00002bd4fb5d505b9161 0\n2 00007756b4a147d2b0b3 0\n3 000086e4b7e1c7146103 0\n4 0000c4c3fbe8785a3090 0","text/html":"
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
qidprediction
00000163e3ea7c7a74cd71
100002bd4fb5d505b91610
200007756b4a147d2b0b30
3000086e4b7e1c71461030
40000c4c3fbe8785a30900
\n
"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions.to_csv('submission.csv', index=False, sep=',')","execution_count":22,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}