{"cells":[{"metadata":{"_uuid":"01f361ddc47e0b386595316fe3d7f4dabbd260db"},"cell_type":"markdown","source":"# Introduction\n\nThis notebook is focus on [Quora Insincere Questions Classification](https://www.kaggle.com/c/quora-insincere-questions-classification) . This problem is basically **text classification** where we have to classify quora topics to be either *sincere* or *insincere*. To have a glance of the example quora topics see this [kernel](https://www.kaggle.com/konohayui/topic-modeling-on-quora-insincere-questions) and this [kernel](https://www.kaggle.com/thebrownviking20/analyzing-quora-for-the-insinceres). Note that this problem is 'kernel-only' submission with limited time of 2 hours.\n \n**update Nov, 16, 2018** -- Add F1 directly as an evaluation metric. Credit this [kernel](https://www.kaggle.com/applecer/use-f1-to-select-model-lstm-based).\n**update Nov, 20, 2018** -- Add more potential ideas to fight with overfitting\n\n![Picture from flickr.com with ](https://c2.staticflickr.com/4/3781/9200265759_897d96f81c_b.jpg)\n\nThis work is mainly based on [SRK's excellent kernel \"A look at different embeding\"](https://www.kaggle.com/sudalairajkumar/a-look-at-different-embeddings). In SRK's work, we can see that the 3 pretrained embedding matrices + GRU give good performance  (and their ensemble improves the final performance). Therefore, it is interesting to study them further, and see how far can the model go. This analysis can also be apply to any RNN types e.g. LSTM or a stack of GRU/LSTM and so on.\n\nHere, we would like to explore the limit of this neural architecture (we choose GloVe as the representative embedding since all of the embedding methods have similar performances) by answering the following questions :\n\n## 1) Is this architecture's capacity overfit or underfit the problem? \n- Can we improve the performance by running more and more epochs?\n- Should we increase the capacity of the network, i.e. increases more latent dimension or add more layers? (underfitting case)\n- In case of overfitting (as will be seen soon), what should we do?\n\n## 2) At its best, what kind of predictions do the network trying to make?\nSince the 'insincere' class has only a small number of data, our network has to be careful when it will predict 'class 1'.\n- Does it try to make only a sure prediction? (try a small number of class-1 prediction, but each one of them is precisely correct)\n- OR, Does it make class-1 prediction a lot (in order to cover most class 1 data), but hopefully not make too much mistakes to predict class 0 as class 1. \n\n## 3) At its best, what kind of errors do the network make?\n- what are *insincere topics* where the network strongly believe to be *sincere* ?\n- what are *sincere topics* where the network strongly believe to be *insincere* ?\n- what are *insincere topics* where the network are most uncertain how to classify ?\n- what are *sincere topics* where the network are most uncertain how to classify?\n- (Not error, but good to see) what are *insincere topics* where the network strongly believe *correctly* ?\n\n## 4) How can we improve performance further based on the above error analysis?\n- I would like to hear your opinions on regarding this point!"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport time\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom tqdm import tqdm\nimport math\nfrom sklearn.model_selection import train_test_split\nfrom sklearn import metrics\n\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.layers import Dense, Input, LSTM, Embedding, Dropout, Activation, CuDNNGRU, Conv1D\nfrom keras.layers import Bidirectional, GlobalMaxPool1D\nfrom keras.models import Model\nfrom keras import initializers, regularizers, constraints, optimizers, layers","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4813a5aff7caff5a9a29a212a9be24e77f58d845"},"cell_type":"code","source":"# credit : https://www.kaggle.com/applecer/use-f1-to-select-model-lstm-based\nfrom keras import backend as K\n\ndef f1(y_true, y_pred):\n    def recall(y_true, y_pred):\n        \"\"\"Recall metric.\n\n        Only computes a batch-wise average of recall.\n\n        Computes the recall, a metric for multi-label classification of\n        how many relevant items are selected.\n        \"\"\"\n        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n        possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n        recall = true_positives / (possible_positives + K.epsilon())\n        return recall\n\n    def precision(y_true, y_pred):\n        \"\"\"Precision metric.\n\n        Only computes a batch-wise average of precision.\n\n        Computes the precision, a metric for multi-label classification of\n        how many selected items are relevant.\n        \"\"\"\n        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n        predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n        precision = true_positives / (predicted_positives + K.epsilon())\n        return precision\n    precision = precision(y_true, y_pred)\n    recall = recall(y_true, y_pred)\n    return 2*((precision*recall)/(precision+recall+K.epsilon()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c2be415d25f2716dab2611f8cfb1eb5232ec5f9d"},"cell_type":"code","source":"import tensorflow as tf\n\n\ndef f1_loss(y_true, y_pred):\n    \n    tp = K.sum(K.cast(y_true*y_pred, 'float'), axis=0)\n    fp = K.sum(K.cast((1-y_true)*y_pred, 'float'), axis=0)\n    fn = K.sum(K.cast(y_true*(1-y_pred), 'float'), axis=0)\n\n    p = tp / (tp + fp + K.epsilon())\n    r = tp / (tp + fn + K.epsilon())\n\n    f1 = 2*p*r / (p+r+K.epsilon())\n    f1 = tf.where(tf.is_nan(f1), tf.zeros_like(f1), f1)\n    return 1 - K.mean(f1)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/train.csv\")\ntest_df = pd.read_csv(\"../input/test.csv\")\nprint(\"Train shape : \",train_df.shape)\nprint(\"Test shape : \",test_df.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ad1e9fdf97f3291a7d1797b25f0c7a0c5d1f1edd"},"cell_type":"markdown","source":"We do exactly the same step as SRK's :\n * Split the training dataset into train and val sample. Cross validation is a time consuming process and so let us do simple train val split.\n * Fill up the missing values in the text column with '_na_'\n * Tokenize the text column and convert them to vector sequences\n * Pad the sequence as needed - if the number of words in the text is greater than 'max_len' trunacate them to 'max_len' or if the number of words in the text is lesser than 'max_len' add zeros for remaining values."},{"metadata":{"trusted":true,"_uuid":"ba5a1b8109dee2c9fbc628d5da4a7c3447d42fb8"},"cell_type":"code","source":"from numpy.random import seed\nseed(1)\nfrom tensorflow import set_random_seed\nset_random_seed(2)\n\n## split to train and val\n\ntrain_df, val_df = train_test_split(train_df, test_size=0.1, random_state=2018)\n\n## some config values \nembed_size = 300 # how big is each word vector\nmax_features = 50000 # how many unique words to use (i.e num rows in embedding vector)\nmaxlen = 100 # max number of words in a question to use\n\n## fill up the missing values\ntrain_X = train_df[\"question_text\"].fillna(\"_na_\").values\nval_X = val_df[\"question_text\"].fillna(\"_na_\").values\ntest_X = test_df[\"question_text\"].fillna(\"_na_\").values\n\n## Tokenize the sentences\ntokenizer = Tokenizer(num_words=max_features)\ntokenizer.fit_on_texts(list(train_X))\ntrain_X = tokenizer.texts_to_sequences(train_X)\nval_X = tokenizer.texts_to_sequences(val_X)\ntest_X = tokenizer.texts_to_sequences(test_X)\n\n## Pad the sentences \ntrain_X = pad_sequences(train_X, maxlen=maxlen,padding='post',truncating='post')\nval_X = pad_sequences(val_X, maxlen=maxlen,padding='post',truncating='post')\ntest_X = pad_sequences(test_X, maxlen=maxlen,padding='post',truncating='post')\n\n## Get the target values\ntrain_y = train_df['target'].values\nval_y = val_df['target'].values","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7c0010e518288bc7f588776c58610949140a139a"},"cell_type":"markdown","source":"Next, loading the glove embedding matrix."},{"metadata":{"trusted":true,"_uuid":"23f130e80159bb1701e449e2e91199dbfff1f1d4"},"cell_type":"code","source":"EMBEDDING_FILE = '../input/embeddings/glove.840B.300d/glove.840B.300d.txt'\ndef get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\nembeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE))\n\nall_embs = np.stack(embeddings_index.values())\nemb_mean,emb_std = all_embs.mean(), all_embs.std()\nembed_size = all_embs.shape[1]\n\nword_index = tokenizer.word_index\nnb_words = min(max_features, len(word_index))\nembedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\nfor word, i in word_index.items():\n    if i >= max_features: continue\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n        \n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"87d5edff35453dd649eeb444d0f24ffb3c1d820d"},"cell_type":"markdown","source":"## 1) Is this architecture's capacity overfit or underfit the problem? \n\nFirst of all, we need to have a benchmark. We would like to understand the best performance this network can give us. \nIn SRK's original notebook, we run the network for only 2 epochs and get good result. So, can we improve the performance by running more and more epochs?\nOr the network will more and more overfit the data?  \n\n![](https://raw.githubusercontent.com/alexeygrigorev/wiki-figures/master/ufrt/kddm/overfitting-logreg-ex.png)\n\nUnderstanding this topic is central of how we can make a justification of how to improve the network. In the case of **overfitting**, it means that **the network has too much capacity, so it is not so useful to increase the number of parameters** such as increasing more layers or increase the RNN's latent dimension. In this case we say that the network have much variance so we should reduce it by using more training data or applying special techniques such as increasing dropout rate (or add more dropout layer) or using ensemble (More detailed below). \n\nOn the other hand, in the case of **underfitting**, i.e. **the network cannot accurately fit the training data**, we have to do a usual routine : add more dimensions or add more layers.\n\nYou can learn more about bias/variance and overfitting/underfitting from [Andrew Ng's lectures](https://www.youtube.com/watch?v=dFX8k1kXhOw&list=PLkDaE6sCZn6E7jZ9sN_xHwSHOdjUxUW_b)\n\nSo let us start by creating the Glove+GRU network and train it for 50 epochs. Note that the purpose of this notebook is not to make a high score submission; we want *insights*. Therefore, we do not care about the 2-hours time limit. We will try to make the network learn from data as much as possible."},{"metadata":{"trusted":true,"_uuid":"004e53c72e906cc8cd7db1a4ebae70dc1d9f13e4","scrolled":true},"cell_type":"code","source":"inp = Input(shape=(maxlen,))\nx = Embedding(max_features, embed_size, weights=[embedding_matrix], trainable=False)(inp)\nx = Bidirectional(CuDNNGRU(64, return_sequences=True))(x)\nx = GlobalMaxPool1D()(x)\nx = Dense(16, activation=\"relu\")(x)\nx = Dropout(0.1)(x)\nx = Dense(1, activation=\"sigmoid\")(x)\nmodel = Model(inputs=inp, outputs=x)\n# model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy',f1])\nmodel.compile(loss=f1_loss, optimizer='adam', metrics=['accuracy',f1])\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a560ab0dbab9cf6fdbdae6721ec030e300f19d78"},"cell_type":"code","source":"history = model.fit(train_X, train_y, batch_size=512, epochs=10, validation_data=(val_X, val_y))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"252f8d2f5b4f66a5e8ccad86a2b42e256692896f"},"cell_type":"code","source":"history = model.fit(train_X, train_y, batch_size=512, epochs=5, validation_data=(val_X, val_y))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"80a99705e9147ad7e1475b62a05e4612a266281d"},"cell_type":"markdown","source":"After a while, we finally get the training and validation results! Let us plot their loss and accuracy."},{"metadata":{"trusted":true,"_uuid":"5bfd2a05ac7a06238be323d51442aa41a0b7b1ba"},"cell_type":"code","source":"#codes from machinelearningmastery.com\nimport matplotlib.pyplot as plt\n# list all data in history\nprint(history.history.keys())\n# summarize history for accuracy\n# codes from machinelearningmastery.com\ndef print_hist(history):\n    plt.plot(history.history['f1'])\n    plt.plot(history.history['val_f1'])\n    plt.title('model af1')\n    plt.ylabel('f1')\n    plt.xlabel('epoch')\n    plt.legend(['train', 'val'], loc='upper left')\n    plt.show()\n\n    # summarize history for accuracy\n    plt.plot(history.history['acc'])\n    plt.plot(history.history['val_acc'])\n    plt.title('model accuracy')\n    plt.ylabel('accuracy')\n    plt.xlabel('epoch')\n    plt.legend(['train', 'val'], loc='upper left')\n    plt.show()\n    \n    # summarize history for loss\n    plt.plot(history.history['loss'])\n    plt.plot(history.history['val_loss'])\n    plt.title('model loss')\n    plt.ylabel('loss')\n    plt.xlabel('epoch')\n    plt.legend(['train', 'val'], loc='upper left')\n    plt.show()\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d6c1d0a42da8c9cf3ac00c525f0c138ee1fcc35b"},"cell_type":"code","source":"print_hist(history)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"36d72748c3bbb56703c08c1fe8d38f63d26ace02"},"cell_type":"markdown","source":"### What we learn from the #1 result\n\n- The result is quite surprising (at least to me) : even though the network is considerably small (1 layer of GRU with only 64 dimensions of latent space) we **overfit** the data very quickly! Only 5 epochs are enough to get the best result. So roughly the 2-hour limitation makes sense as you do not need 6 hours to fit the data.\n\n- Therefore, **there should be no clear advantage of building a deeper network**. what is most promising is how can we reduce its variance. This explains why using ensemble method works well here. \n\n### What can we do to fight overfitting ?\n\n![](https://cdn-images-1.medium.com/max/1600/1*XWh6hd8BgI3RKhd8bkrbuw.png)\n(picture credit : https://hackernoon.com/memorizing-is-not-learning-6-tricks-to-prevent-overfitting-in-machine-learning-820b091dc42)\n\n- **Ensemble** : this combines a lot of classifier to stabilize the prediction. This method was already implemented by SRK.\n\n- **Dropout** : this is effectively similar to ensembling. In each training batch, the network will drop different set of nodes, so we can think that we have one unique classifier for each subset of selected nodes. But at test time, it will use all the nodes to combine all there predictions. Therefore, it can be view as a version of ensemble. Standard dropout layer is easy to implement in Keras, and is already done in SRK's code. Nevertheless, Keras has many types of dropout implementation e.g. spatial dropout, and maybe some of these may suit this problem.\n\n(**Note** that unfortunately, CuDNNLSTM / CuDNNGRU do not support *recurrent_dropout*, and standard LSTM/GRU may be too time consuming to try in the limited 2 hours) \n\n-**Classic regularization** : we can use classic regularization such as L1 or L2 for each neural network layer. This could work, but you have to find appropriate values of regularization parameters. See [Keras manual](https://keras.io/regularizers/) for details.\n\n-**Dimensionality reduction** : Beside reduce overfitting, dimensionality reduction can also help speed up the entire learning process.  PCA is a standard approach to reduce dimensionality. Other non-linear dimensionality reductions  which is popular in Deep Learning era is auto-encoder. In Deep Learning practice, however, the most simple but effective method is to simply apply 'Dense' layer to the input layer to reduce the dimension.\n\n\n- **Early stopping** : if run too many epochs will overfit the data, we will have to stop the training process early. I think we all have done tuning this hyperparameters :)\n\n- **Adjust batch size** : this can be help also! Indeed, when we create each batch, we are randomly select them from the total data. Compared to the gradient of the whole data, the gradient of each batch can be seen as an added noisy direction.  This randomness can sometimes regularize our training process, not to overfit the training set. See [more excellent explanation here](https://www.quora.com/Intuitively-how-does-mini-batch-size-affect-the-performance-of-stochastic-gradient-descent)\n\n- **Increase more training data** : If possible this will let your network understand more about the nature of the problem, and will help it to generalize better. Unfortunately, this competition doesn't allow external dataset. Another method is to create artificial data (which have the same characteristic as the real data), the so-called **Data Augmentation** method. There is a discussion on Data Augmentation [here](https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/71083), but I still cannot think of ways to apply it. If you have any ideas please share :) \n\n- **Use of unlabeled data** : what if we cannot find more *labeled* training data?? Perhaps *unlabeled data* can also help! The method of employing unlabeled data with labeled data can be broadly called as *semi-supervised learning*. At least three methods are possible and exploit successfully in literatures. \n\n(1) *transductive learning* : Here, perhaps we try to exploit the test data itself which is unlabeled. Even though, we have no label of them, some of their information (data manifold) can be useful for our learners. See this [discussion](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/52557) as one example of how to employ test data (they called it *pseudo labelling*) \n\n(2) *language model pretrained and finetuning* : This is a well-known approach which is very successful in academic literatures, i.e. we use all (potentially infinite) unlabeled data to train an auxiliary task first (here, language model training). And then we use the pretrained network to fine-tune to our classification problem at hand. Most state-of-the-art results such as ELMO, BERT, ULMFiT and OpenAI's LM+Transformer all employ this methodology. Nevertheless, with the 2-hour constraints of this competition, it is challenging to find a way to employ this approach.\n\n(3) *multi-task learning* : similar to (2), but instead of pretraining and fine-tuning, we can learn both the auxiliary task and the main classification task simultaneously!. See [Ruder's blog](http://ruder.io/multi-task-learning-nlp/) post for much more comprehensive details.\n\n- **BONUS** : cleaning up original data. This may not be overfitting fighting, but it will definitely help! See [Dieter's excellent kernel here](https://www.kaggle.com/christofhenkel/how-to-preprocessing-when-using-embeddings).\n\nIs there any more methods I missed, if you have more ideas, please share!"},{"metadata":{"_uuid":"4a08c56bf537940d918b86ce32b96cd8f0a3151e"},"cell_type":"markdown","source":"## 2) At its best, what kind of predictions do the network trying to make?\n\nSince the 'insincere' class has only a small number of data, our network has to be careful when it will predict 'class 1'.\n- Does it try to make only a sure prediction? (try a small number of class-1 prediction, but each one of them is precisely correct)\n- OR, Does it make class-1 prediction a lot (in order to cover most class 1 data), but hopefully not make too much mistakes to predict class 0 as class 1. \n\nTo answer this question, let us build the network at its best, i.e. fitting the training data for 5 epochs."},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"b72851ee62c02e068925df29af3628bc11fca434"},"cell_type":"code","source":"inp = Input(shape=(maxlen,))\nx = Embedding(max_features, embed_size, weights=[embedding_matrix], trainable=False)(inp)\nx = Bidirectional(CuDNNGRU(64, return_sequences=True))(x)\nx = GlobalMaxPool1D()(x)\nx = Dense(16, activation=\"relu\")(x)\nx = Dropout(0.1)(x)\nx = Dense(1, activation=\"sigmoid\")(x)\nmodel = Model(inputs=inp, outputs=x)\nmodel.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy',f1])\nprint(model.summary())\n\nhistory = model.fit(train_X, train_y, batch_size=512, epochs=5, validation_data=(val_X, val_y))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f1bd547bd06c07ea5e872d968f34f9dbb3891f1e"},"cell_type":"code","source":"print_hist(history)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"33cfea1334ca66d7a70280b5587a20734fbfdeae"},"cell_type":"markdown","source":"Finish!\n\nThen, with little modification of SRK's code again : let us see the best threshold which makes the best prediction for training set and validation set. Not surprisingly, F1 score on training set is greater than F1 score on validation set."},{"metadata":{"trusted":true,"_uuid":"bd68449a79d053f1476f8d420465bdb7264fd010"},"cell_type":"code","source":"pred_glove_train_y = model.predict([train_X], batch_size=1024, verbose=1)\nscores = []\nthresholds = np.arange(0.1, 0.501, 0.01)\nfor thresh in thresholds:\n    thresh = np.round(thresh, 2)\n    scores.append(metrics.f1_score(train_y, (pred_glove_train_y>thresh).astype(int)))\n    print(\"F1 score at threshold {0} is {1}\".format(thresh, metrics.f1_score(train_y, (pred_glove_train_y>thresh).astype(int))))\nidx = np.argmax(np.array(scores))\nthresh_train = thresholds[idx]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"41dc4d6c3d95564a1a740da62ce112b4c5eaec24"},"cell_type":"code","source":"print(thresh_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ff43855164472de035a5a1d80b3db4838684701a","scrolled":false},"cell_type":"code","source":"pred_glove_val_y = model.predict([val_X], batch_size=1024, verbose=1)\nscores = []\nthresholds = np.arange(0.1, 0.501, 0.01)\nfor thresh in thresholds:\n    thresh = np.round(thresh, 2)\n    scores.append(metrics.f1_score(val_y, (pred_glove_val_y>thresh).astype(int)))\n    print(\"F1 score at threshold {0} is {1}\".format(thresh, metrics.f1_score(val_y, (pred_glove_val_y>thresh).astype(int))))\nidx = np.argmax(np.array(scores))\nthresh_valid = thresholds[idx]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7afcfb0022815c1c35a892d07cbdcdf494234a99"},"cell_type":"code","source":"print(thresh_valid)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5e7dc031ed7f059f22d39df00c8d53f38c77b505"},"cell_type":"markdown","source":"We can see that the best threshold for training data is around 0.35 - 0.4 for validation data (there is some randomness from our environment). Thaaat is only 35% -40% confidence is enough to classify the topic as insincere. Note that paradoxically **we don't need more than 51% confidence to say that the topic is insincere.**\n\nWe can understand its predictions better by looking at its accuray and confusion matrix."},{"metadata":{"_uuid":"d2a33c252f31fddcc65896053184226128562776","trusted":true},"cell_type":"code","source":"from sklearn.metrics import accuracy_score, confusion_matrix\n\nprint('total valid data is ',train_y.shape[0], 'where', np.sum(train_y==1),'is insincere' )\nprint(accuracy_score(train_y, pred_glove_train_y>thresh_train))\nconfusion_matrix(train_y, pred_glove_train_y>thresh_train, labels=[0, 1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2917dce7ecd4656ff855f7ad5ecbcedf68efabb1"},"cell_type":"code","source":"\nprint('total valid data is ',val_y.shape[0], 'where', np.sum(val_y==1),'is insincere' )\nprint(accuracy_score(val_y, pred_glove_val_y>thresh_valid))\nconfusion_matrix(val_y, pred_glove_val_y>thresh_valid, labels=[0, 1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"86e22b5c41fbda5e8cb454b4e327942d130e132f"},"cell_type":"code","source":"print('This is the precision I got from validation set : ', 5712/(5712+3027))\nprint('This is the recall I got from validation set : ', 5712/(5712+2413))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"52827042c9c64d95bb3d28ea45861f874e8743da"},"cell_type":"markdown","source":"At 5 epochs, the network doesn't overfit the data as we can see from the fact that training accuracy and validation accuracy are quite similar.\n\nLook at validation performance, in my run, we can see that : (note that you can get slightly different results due to randomness)\n - Precision is aroud 65.X%\n - Recall is also around = 70.X%\n Since F1 is somewhat an average of the two score, that is why we get F1 validatin score around 67%.\n \n **Interpretation :**  we can see that the network balances its *insincere* prediction quite well. It indeed has a good precision (65% versus 6% of randomly guessing -- a 10X boost!). On the other hand, it doesn't \nmake a small number of class-1 prediction because it covers around 70% of the true insincere class. Our network indeed do a good job.\n\nSo far so good. But can we get more insights on its prediction behavior? Let us explore more on this on the next section."},{"metadata":{"trusted":true,"_uuid":"d51ff8ed6a87b488fec3ac84ca50df661d7c8193"},"cell_type":"markdown","source":"## 3) At its best, what kind of errors do the network make?\n\nContinue from the previous section, now we will try to dig deeper by going through error analysis as taught by [Andrew Ng](https://www.youtube.com/watch?v=JoAxZsdw_3w&list=PLkDaE6sCZn6E7jZ9sN_xHwSHOdjUxUW_b&index=13) and [Fast.ai](https://www.kaggle.com/hortonhearsafoo/fast-ai-lesson-1)\n\n![](https://upload.wikimedia.org/wikipedia/commons/9/94/Contingency_table.png)\n\nWe intend to understand the following :\n- what are *insincere topics* where the network strongly believes to be *sincere* ?\n- what are *sincere topics* where the network strongly believes to be *insincere* ?\n- what are *insincere topics* where the network are most uncertain how to classify ?\n- what are *sincere topics* where the network are most uncertain how to classify?\n- (Not error, but good to see) what are *insincere topics* where the network strongly believes *correctly* ?\n\nBy understanding this kind of errors, perhaps we will get new ideas of how to improve the performance."},{"metadata":{"_uuid":"b789e6d86f7a685743f7a75282ebbf557e78d943"},"cell_type":"markdown","source":"First, let us define these functions which will do the jobs: "},{"metadata":{"trusted":true,"_uuid":"39d4fedab4ac170863a0ee1ca3aa9be1ee58fe02"},"cell_type":"code","source":"def most_false_negative(pred_y,true_y,thresh, N ):\n    pred_y_round = (pred_y>thresh)\n    \n    flag = (pred_y_round == 0) & (true_y == 1)[:,np.newaxis]\n    \n    idxs = np.where(flag)[0] # ignore the np.newaxis dimension\n    newidx = np.argsort(pred_y[idxs,0])\n\n    return idxs[newidx[:N]], pred_y[idxs[newidx[:N]]]\n\ndef most_uncertain_negative(pred_y,true_y,thresh, N ):\n    pred_y_round = (pred_y>thresh)\n    flag = (pred_y_round == 0) & (true_y == 1)[:,np.newaxis]\n    \n    idxs = np.where(flag)[0] # ignore the np.newaxis dimension\n    newidx = np.argsort(pred_y[idxs,0])\n    return idxs[newidx[-N:]], pred_y[idxs[newidx[-N:]]]\n\ndef most_uncertain_positive(pred_y,true_y,thresh, N ):\n    pred_y_round = (pred_y>thresh)\n    \n    flag = (pred_y_round == 1) & (true_y == 0)[:,np.newaxis]\n    \n    idxs = np.where(flag)[0] # ignore the np.newaxis dimension\n    newidx = np.argsort(pred_y[idxs,0])\n\n    return idxs[newidx[:N]], pred_y[idxs[newidx[:N]]]\n\ndef most_false_positive(pred_y,true_y,thresh, N ):\n    pred_y_round = (pred_y>thresh)\n    flag = (pred_y_round == 1) & (true_y == 0)[:,np.newaxis]\n    \n    idxs = np.where(flag)[0] # ignore the np.newaxis dimension\n    newidx = np.argsort(pred_y[idxs,0])\n    return idxs[newidx[-N:]], pred_y[idxs[newidx[-N:]]]\n\ndef most_true_positive(pred_y,true_y,thresh, N ):\n    pred_y_round = (pred_y>thresh)\n    \n    flag = (pred_y_round == 1) & (true_y == 1)[:,np.newaxis]\n    \n    idxs = np.where(flag)[0] # ignore the np.newaxis dimension\n    newidx = np.argsort(pred_y[idxs,0])\n\n    return idxs[newidx[-N:]], pred_y[idxs[newidx[-N:]]]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ba11ca278bd0242abcc91974c36b781d0c53bb88"},"cell_type":"markdown","source":"###  what are *insincere topics* where the network strongly believe to be *sincere* ?\nThis kind of predictions  is the most errorneous that our network made, so let see which kinds of topics are they. You can change the constant *NN* below in order to see more examples. "},{"metadata":{"trusted":true,"_uuid":"d708cdc194495fee77d3848b008f71378e9b3179"},"cell_type":"code","source":"NN = 20\nidx, prob = most_false_positive( pred_glove_val_y, val_y,thresh_valid,NN)\nfor i in range(NN):\n    print('class',val_y[idx[i]],prob[i],val_df[\"question_text\"].values[idx[i]])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aad0e025e0a0a17e2a9896773a06f7b9358bb28f"},"cell_type":"markdown","source":"The results are unexpected! As you can see our network strongly believes the above topics are insincere with high probabilities ( >90%), but the true class is sincere. Nevertheless, by looking at topics it can be seen that these topics should be class 1 instead!  So this should be the mislabels in the dataset. In fact, in the dataset page, quora does say that\n\n> The ground-truth labels contain some amount of noise: they are not guaranteed to be perfect.\n\nThe mislabeling make the problem more difficult since our network are doing good, but is forced to overfit on these kind of topics. "},{"metadata":{"_uuid":"5d4b2c86e16ac1adf252017a074ab3128364aae1"},"cell_type":"markdown","source":"### what are *sincere topics* where the network strongly believe to be *insincere* ?\nThis is similar to the above subsection, but in the opposite direction."},{"metadata":{"trusted":true,"_uuid":"b3979c27b5ada2f9b0cec8109e18366c76274043"},"cell_type":"code","source":"idx, prob = most_false_negative( pred_glove_val_y, val_y,thresh_valid,NN)\nfor i in range(NN):\n    print('class',val_y[idx[i]],prob[i],val_df[\"question_text\"].values[idx[i]])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8ebb4bdd792a985f16109e1e72ff07d99b18de97"},"cell_type":"markdown","source":"It seems that we get the noise again! According to our human knowledge, these topics look very sincere (and our network also strongly believes that), but the labels say that they are *insincere*. Again, this supports the evidence in #1 that our network will eventually overfit the data.\n\nNext, let us consider the following two groups together.\n\n### what are *insincere topics* where the network are most uncertain how to classify ?, and\n### what are *sincere topics* where the network are most uncertain how to classify?\n\nRemember that the best threshold for validation data is :"},{"metadata":{"trusted":true,"_uuid":"3e067d512a5d3c1c629bfbb39672e925f827c58b"},"cell_type":"code","source":"print('our best threshold is',thresh_valid)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4fdbeffc0f84643d2832eec49234bd9d6c6e216b","trusted":true},"cell_type":"code","source":"idx, prob = most_uncertain_positive( pred_glove_val_y, val_y,thresh_valid,NN)\nfor i in range(NN):\n    print('class',val_y[idx[i]],prob[i],val_df[\"question_text\"].values[idx[i]])\nprint('\\n')\nprint('\\n')\nidx, prob = most_uncertain_negative( pred_glove_val_y, val_y,thresh_valid,NN)\nfor i in range(NN):\n    print('class',val_y[idx[i]],prob[i],val_df[\"question_text\"].values[idx[i]])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a63086e81344c113c40ee7b6654ba199bbb0d87d"},"cell_type":"markdown","source":"What do you think of these topics where our network feels super unsure. To me, except for a few,  these topics are very difficult to judge whether they are sincere or not. Therefore, even the human baseline might not do a good job on this dataset.\n\n### Bonus : what are *insincere topics* where the network strongly believes *correctly* ?\nIt is interesting to see what kind of topics both of our network and the labeller agree with."},{"metadata":{"trusted":true,"_uuid":"2d4679a9f5a06c19e6edda2f427a4f834ba612f0"},"cell_type":"code","source":"idx, prob = most_true_positive( pred_glove_val_y, val_y,thresh_valid,NN)\nfor i in range(NN):\n    print('class',val_y[idx[i]],prob[i],val_df[\"question_text\"].values[idx[i]])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"58a2f78ac243d33d0c127a13af202429a7f16c1e"},"cell_type":"markdown","source":"It is clear here! These topics are unacceptable and very easy to classify due to their uses of impolite words.\n\n\nThat's all for now! In fact, for the following last section, I would like to hear your great ideas! I hope that this kernel can be helpful."},{"metadata":{"trusted":true,"_uuid":"f6797ab73bdd5bdb8c8f6d80ec361c50a2b0f56f"},"cell_type":"markdown","source":"\n\n## 4) How can we improve performance further based on the above error analysis?\nPlease discuss :)"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}