{"cells":[{"metadata":{"collapsed":true,"_uuid":"48e7d0b17674d65f7ca1d5994325e7ffe485af69"},"cell_type":"markdown","source":"# ConvNet Training\nThis notebook can be used to train a CNN for binary text classification and generate predictions for the Kaggle competition found [here](https://www.kaggle.com/c/quora-insincere-questions-classification). \n\nThe notebook utilizes Keras and GloVe for preprocessing using word embeddings. Then, Keras with Tensorflow backend is used for training a deep CNN. Feel free to fork!\n\n### Acknowledgements\n* Richard Liao's [blog post](https://richliao.github.io/supervised/classification/2016/11/26/textclassifier-convolutional/) for starter code for the cnn\n* Vladimir Demidov's [notebook](https://www.kaggle.com/yekenot/2dcnn-textclassifier) for the F1 Score calculation\n* This great [blog post](http://debajyotidatta.github.io/nlp/deep/learning/word-embeddings/2016/11/27/Understanding-Convolutions-In-Text/) for understanding convolution in text classification using convolution. Great visuals!"},{"metadata":{"trusted":true,"_uuid":"31651e5a58fc9229e683afac2cbc7b0948a5f240"},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport os\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import f1_score\n\nfrom keras.callbacks import Callback\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.layers import Input, Dense, Embedding, Dropout\nfrom keras.layers import Conv1D, MaxPool1D, Flatten, Concatenate\nfrom keras.models import Model\nfrom keras.utils.vis_utils import plot_model\n\n%load_ext autoreload\n%autoreload 2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9dc94191884a37933e64d2d822040a9779a621fd"},"cell_type":"code","source":"# Load in training and testing data\ntrain_df = pd.read_csv('../input/train.csv')\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"43ebbfd90e0bd2f9f81bf890382dbce6c4a39cbb"},"cell_type":"markdown","source":"# 1. Data Preparation\nThis section of the notebook is devoted to preprocessing the raw data into a form that the neural network can understand."},{"metadata":{"trusted":true,"_uuid":"78488b53255b19d912481adbc91f5bcf0410e689"},"cell_type":"code","source":"# Extract the training data and corresponding labels\ntext = train_df['question_text'].fillna('unk').values\nlabels = train_df['target'].values\n\n# Split into training and validation sets by making use of the scikit-learn\n# function train_test_split\nX_train, X_val, y_train, y_val = train_test_split(text, labels,\\\n                                                  test_size=0.2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"095a8804c4f221331cf6ab4a2480efc62780632d"},"cell_type":"markdown","source":"## 1.1 Create Word Embedding Matrix\nThe code in this section will identify the most commonly occurring words in the dataset. Then, it will extract the vectors for each one of these words from the GloVe pretrained word embedding and place them in an embedding layer matrix. This embedding layer will serve as the first layer of the neural network. \n\nRead more about GloVe word embeddings [here](https://nlp.stanford.edu/projects/glove/).\n\nNote that other word embeddings are also available for this competition, however glove was chosen for this notebook. "},{"metadata":{"trusted":true,"_uuid":"e00b006b3a940240da8d4d8b2bfda5765a9f7a40"},"cell_type":"code","source":"embed_size = 300 # Size of each word vector\nmax_words = 50000 # how many unique words to use (i.e num rows in embedding vector)\nmaxlen = 100 # max number of words in a question to use","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3d68827dba2948ee2ef7dd4be10a1fa4dd629e4a"},"cell_type":"code","source":"## Tokenize the sentences\ntokenizer = Tokenizer(num_words=max_words)\ntokenizer.fit_on_texts(list(X_train))\n\n# The tokenizer will assign an integer value to each word in the dictionary\n# and then convert each string of words into a list of integer values\nX_train = tokenizer.texts_to_sequences(X_train)\nX_val = tokenizer.texts_to_sequences(X_val)\n\nword_index = tokenizer.word_index\nprint('The word index consists of {} unique tokens.'.format(len(word_index)))\n\n## Pad the sentences to the maximum length\nX_train = pad_sequences(X_train, maxlen=maxlen)\nX_val = pad_sequences(X_val, maxlen=maxlen)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"96e773c808de88d2054a64b34f060067dd7e65d3"},"cell_type":"code","source":"# Create the embedding dictionary from the word embedding file\nembedding_dict = {}\nfilename = os.path.join('../input/embeddings/', 'glove.840B.300d/glove.840B.300d.txt')\nwith open(filename) as f:\n    for line in f:\n        line = line.split()\n        token = line[0]\n        try:\n            coefs = np.asarray(line[1:], dtype='float32')\n            embedding_dict[token] = coefs\n        except:\n            pass\nprint('The embedding dictionary has {} items'.format(len(embedding_dict)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8d673151e5ccf77cfdb94aed6e2e64883e490dac"},"cell_type":"code","source":"# Create the embedding layer weight matrix\nembed_mat = np.zeros(shape=[max_words, embed_size])\nfor word, idx in word_index.items():\n    # Word index is ordered from most frequent to least frequent\n    # Ignore words that occur less frequently\n    if idx >= max_words: continue\n    vector = embedding_dict.get(word)\n    if vector is not None:\n        embed_mat[idx] = vector","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"69520f641e945eae0fa41f4dbc8d0f5d85e6f744"},"cell_type":"markdown","source":"# 2. Neural Network Training\nThis section contains the code for designing and training the neural network"},{"metadata":{"_uuid":"152c74706927d3a1e57bb1e621ad6df5fbe9e9cb"},"cell_type":"markdown","source":"## 2.1 Neural Network Architecture\n\nThe following is a summary of the convolutional network:\n* This network configuration uses the pretrained GloVe embedding layer as the first layer of the network. The user can choose to make the word embedding weights trainable or not. \n* A series of Conv1D-MaxPool1D pairs, each with varying paramaters depending on the users input. These pairs all operate in parallel.\n    - The user can choose filter sizes and strides for each pair\n* The Pooling layers are all concatenated and flattened\n* A dropout layer is added with a default dropout of 0.2. Change dropout to 0 to effectively remove this layer\n* Finally, there are 2 dense layers leading to the final prediction. Sigmoid is used rather than softmax because we are performing binary classification\n\nFeel free to modify network parameters and architecture. This is merely a starting point that provides adequate results. "},{"metadata":{"trusted":true,"_uuid":"1e4ec3f078dc5ca43669238813d5ebcadf6ab39d"},"cell_type":"code","source":"def create_cnn(filter_sizes, strides, num_filters, embed_train=False, dropout=0.2, plot=False):\n    # The first layer will be the word embedding layer\n    # by default, the embedding layer will not be trainable as it adds a great deal of complexity\n    sequence_input = Input(shape=(maxlen,), dtype='int32')\n    x = Embedding(max_words, embed_size, weights=[embed_mat], trainable=embed_train)(sequence_input)\n    \n    # Convolutional and maxpool layers for each filter size and stride size\n    # Convolution is 1D and occurs at different stride lengths.\n    # eg. a filter size of 3 and stride of 2 will examine 3 words at a time\n    # in the order 0,1,2 - 2,3,4 - 4,5,6 - etc\n    conv_layers = []\n    maxpool_layers = []\n    for i in range(len(filter_sizes)):\n        conv_layers.append(Conv1D(num_filters, strides=strides[i], padding='same', kernel_size=(filter_sizes[i]),\n                                 kernel_initializer='he_normal', activation='relu')(x))\n        # pool_size calculation: (Width - (Filter_size * 2*Padding))/Stride\n        pool_size = int((maxlen-(filter_sizes[i]*2))/strides[i])\n        maxpool_layers.append(MaxPool1D(pool_size=pool_size, strides=strides[i])(conv_layers[i]))\n\n    # Concatenate pooling layers outputs\n    if len(maxpool_layers)==1:\n        z = maxpool_layers[0]\n    else:\n        z = Concatenate(axis=1)(maxpool_layers)\n    \n    # Finish network with flattened layer, dropout, and fully connected layer\n    z = Flatten()(z)\n    z = Dropout(dropout)(z)\n    z = Dense(64, activation='relu')(z)\n    preds = Dense(1, activation='sigmoid')(z) # Sigmoid for binary classification\n\n    model = Model(sequence_input, preds)\n    model.compile(loss='binary_crossentropy',\n                  optimizer='adam',\n                  metrics=['acc'])\n    model.summary()\n    \n    if plot:\n        plot_model(model, to_file='./ims/ConvNet1D.png', show_shapes=True, show_layer_names=True)\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f62aef993683dfd4a459ebf1d3ffbbc00a01b3b2"},"cell_type":"markdown","source":"## 2.2 Evaluation\nBelow is code for the callback f1 evaluation function, which will be called at the end of each training iteration.\n\nCode for this callback function was grabbed from this [notebook](https://www.kaggle.com/yekenot/2dcnn-textclassifier)."},{"metadata":{"trusted":true,"_uuid":"9aeb979e4a74bb9d6d269ea717b3beeddf2c2509"},"cell_type":"code","source":"threshold = 0.35 # Experimentally found to be the best threshold\nclass F1Evaluation(Callback):\n    def __init__(self, validation_data=(), interval=1):\n        super(Callback, self).__init__()\n        self.interval = interval\n        self.X_val, self.y_val = validation_data\n\n    def on_epoch_end(self, epoch, logs={}):\n        if epoch % self.interval == 0:\n            y_pred = self.model.predict(self.X_val, verbose=0)\n            y_pred = (y_pred > threshold).astype(int)\n            score = f1_score(self.y_val, y_pred)\n            print(\"\\n F1 Score - epoch: %d - score: %.6f \\n\" % (epoch+1, score))\n            \nF1_Score = F1Evaluation(validation_data=(X_val, y_val), interval=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4715bfd726bc55ee45d0379cd5ea7a692836441e"},"cell_type":"markdown","source":"## 2.3 Training\nFeel free to change any of the parameters to improve the model. Feedback is welcome!"},{"metadata":{"trusted":true,"_uuid":"738691c29ce46a05923efc18e3fada0c308e611e"},"cell_type":"code","source":"# A few parameters to define for the network. Feel free to experiment\n# Note that filter_sizes and strides must have the same length\nfilter_sizes = [1,3,3,5]\nstrides = [1,1,2,3]\nnum_filters = 48\ndropout = 0.2\nembed_train = False\n\nepochs = 3\nbatch_size = 1024\n\n# Create and train network\ncnn = create_cnn(filter_sizes, strides, num_filters, embed_train=embed_train, dropout=dropout)\nhistory = cnn.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=epochs,\n                  batch_size=batch_size, callbacks=[F1_Score])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"431bff6d3460db502bddea0023b5f3fa5c567942"},"cell_type":"markdown","source":" ## 2.4 Threshold\nRather than simply rounding the network outputs to the nearest integer {0,1}, the predictions can be made more or less conservative by altering the threshold. For example, lowering the threshold to 0.35 would predict all values above 0.35 as insincere. This model would be more aggressive than a model that used a threshold of 0.5. \n\nThis section of code experimentally finds the best threshold to use for final predictions. "},{"metadata":{"trusted":true,"_uuid":"06d3f429c185984a6361a343ec67f14a17da0f5d"},"cell_type":"code","source":"# The best results are seen with a threshold between 0.1 and 0.5\nthresholds = np.arange(0.15, 0.5, 0.05)\n\nbest_thresh = None\nbest_score = 0.\n\n# Make predictions and evaluate f1 score for each threshold value\nfor thresh in thresholds:\n    y_pred = cnn.predict(X_val, verbose=0)\n    y_pred = (y_pred>thresh).astype(int)\n    score = f1_score(y_val, y_pred)\n    print('F1 Score for threshold {:0.1f}: {:0.3f}'.format(thresh, score))\n    \n    # Store best threshold for later use in predictions\n    if not best_thresh or score>best_score:\n        best_thresh = thresh","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"61aeb24e48c578594d26d0631a46aad49a215c4e"},"cell_type":"markdown","source":"# 3. Predictions\nThe remainder of this notebok will generate predictions from the test set and write them to a submission csv file for the kaggle competition."},{"metadata":{"trusted":true,"_uuid":"91e9da1a1d551932c34405c3a7ed185632f12260"},"cell_type":"code","source":"test_df = pd.read_csv('../input/test.csv')\nX_test = test_df['question_text'].values\n\n# Perform the same preprocessing as was done on the training set\nX_test = tokenizer.texts_to_sequences(X_test)\nX_test = pad_sequences(X_test, maxlen=maxlen)\n\n# Make predictions, ensure that predictions are in integer form\n# Use best threshold from previous section\npreds = np.rint(cnn.predict([X_test], batch_size=1024, verbose=1))\ny_pred = (preds>best_thresh).astype(int)\ntest_df['prediction'] = y_pred","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"72f90edd0b8a0eb4c644e917f7c02068378ad161"},"cell_type":"markdown","source":"Let's examine a few examples of sincere predictions and insincere predictions. It appears that our network is making meaningful predictions."},{"metadata":{"trusted":true,"_uuid":"5b7bd3f417dfa01973f20f12fe967ec61ff3dc69"},"cell_type":"code","source":"n=5\nsin_sample = test_df.loc[test_df['prediction'] == 0]['question_text'].head(n)\nprint('Sincere Samples:')\nfor idx, row in enumerate(sin_sample):\n    print('{}'.format(idx+1), row)\n\nprint('\\n')\nprint('Insincere Samples:')\ninsin_sample = test_df.loc[test_df['prediction'] == 1]['question_text'].head(n)\nfor idx, row in enumerate(insin_sample):\n    print('{}'.format(idx+1), row)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"988fa1a03853289b65bb4c1845c5b7c333868cb8"},"cell_type":"code","source":"# Drop the question text from the dataframe leaving only question ID and preds\n# Then write to submission csv for competition\ntest_df = test_df.drop('question_text', axis=1)\ntest_df.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}