{"cells":[{"metadata":{"trusted":true,"_uuid":"8462a3f84df60518683dd1c1f7e3080ccc3e5699"},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns; sns.set()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9c3a6ae0464550cdbe0220a1c9068dd0efc53da6"},"cell_type":"markdown","source":"# 0. Description\nThis notebook is a first-pass attempt as solving the problem posed in the Quora Insincere Questions competition (and is also a work in progress!):\n\nThe description of the dataset, taken from the competition page, is here:\n\n>In this competition you will be predicting whether a question asked on Quora is sincere or not.\n\n>An insincere question is defined as a question intended to make a statement rather than look for helpful answers. Some characteristics that can signify that a question is insincere:\n\n> * Has a non-neutral tone\n> * Has an exaggerated tone to underscore a point about a group of people\n> * Is rhetorical and meant to imply a statement about a group of people\n> * Is disparaging or inflammatory\n> * Suggests a discriminatory idea against a protected class of people, or seeks confirmation of a stereotype\n> * Makes disparaging attacks/insults against a specific person or group of people\n> * Based on an outlandish premise about a group of people\n> * Disparages against a characteristic that is not fixable and not measurable\n> * Isn't grounded in reality\n> * Based on false information, or contains absurd assumptions\n> * Uses sexual content (incest, bestiality, pedophilia) for shock value, and not to seek genuine answers\n>\n>The training data includes the question that was asked, and whether it was identified as insincere (target = 1). The ground-truth labels contain some amount of noise: they are not guaranteed to be perfect.\n\nMany of the notebooks on the competition page are using neural networks and the provided embeddings. This notebook is my attempt at solving the problem using a neural network architecture."},{"metadata":{"_uuid":"8f306a2cf727240053f3ac4b2b1fa4c2de6cc351"},"cell_type":"markdown","source":"# 0.1 Approach"},{"metadata":{"_uuid":"b59110db5ec296ccd8a86f82b2b1fc554e331396"},"cell_type":"markdown","source":"This approach is based off of the EDA and approach that I've taken in my main [kernel here](https://github.com/dhruvmonga/MachineLearningProjects/blob/master/QuoraClassifications/kernel.ipynb). My main approach was attempting to use model stacking and feature engineering to get good performance. However, I couldn't get much of an improvement in results from my initial modeling using simple features and models. So this kernel is my attempt as solving the problem using a neural network architecture, which is what it appears that most people in the competition are also using."},{"metadata":{"_uuid":"b71f4cc4bf602a09369e7c41b6aed240dede3744"},"cell_type":"markdown","source":"# 1. Dataset reading and exploration"},{"metadata":{"trusted":true,"_uuid":"e8c86621e3fce914e461f9516d0e59439321fca3"},"cell_type":"code","source":"df = pd.read_csv('../input/train.csv',index_col=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fe052f8a4f7581017ea85db89e80860762b4e079"},"cell_type":"code","source":"target = 'target'\nX = df.drop(target,axis=1)\ny = df[target]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c978aaad230e0074570a4ecabe94785e92b3e382"},"cell_type":"markdown","source":"See my [main kernel](https://github.com/dhruvmonga/MachineLearningProjects/blob/master/QuoraClassifications/kernel.ipynb) for a more complete EDA."},{"metadata":{"_uuid":"8f01594aff28b166e8ccf9d1f18b40951badbf2c"},"cell_type":"markdown","source":"# 2. Modelling"},{"metadata":{"_uuid":"907c998667399d1f23f05409ea885488d0cb872e"},"cell_type":"markdown","source":"## 2.1 Dataset preparation"},{"metadata":{"trusted":true,"_uuid":"d76ebbfe278f5da3c7e8c3452d61ac50eb8af3e7"},"cell_type":"code","source":"from sklearn.model_selection import train_test_split, GridSearchCV, StratifiedKFold\nimport xgboost as xgb\nfrom sklearn.linear_model import LogisticRegression","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cb76a20959af153b6e91e40b88732bd9e5d77475"},"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c1455b420d29b984e83ad318883e6f929373930e"},"cell_type":"markdown","source":"## 2.2 Pretraining feature creators on train data to avoid leakage"},{"metadata":{"_uuid":"3bc902dc85c7df00d7c18dce0741923988834afb"},"cell_type":"markdown","source":"### 2.2.1 Feature Creation Function"},{"metadata":{"_uuid":"abd80115b6e9845c0f200c11c4092aaf57b24930"},"cell_type":"markdown","source":"I will use a simple tokenizer, along with the provided embeddings, to implement the features - I want the network architecture to be able to learn all its features. I used the [work done here](https://www.kaggle.com/artgor/eda-and-lstm-cnn) as a basis for the featurizing function and as a starting point to implement the models."},{"metadata":{"trusted":true,"_uuid":"1052ad0efa8a905fb42b8f473cbd3bfd0268782c"},"cell_type":"code","source":"from keras.preprocessing.sequence import pad_sequences\nfrom keras.preprocessing.text import Tokenizer\nfrom sklearn.preprocessing import OneHotEncoder\n\nfrom progressbar import ProgressBar\npbar = ProgressBar()\n\n# len of lstm, chosen to be 200 based on the length of the questions derived from the eda in my main kernel\nmax_len = 200\n# number of features\nmax_features = 50000\n# size of embedding vectors\nembed_size = 300\n## preprocess\ndef preprocess(X):\n    # first we tokenize using the tokenizer\n    print(\"tokenizing\")\n    tk = Tokenizer(lower = True, filters='', num_words=max_features)\n    full_text = list(X['question_text'].values)\n    tk.fit_on_texts(full_text)\n    \n    # then we load up the embeddings\n    embedding_path = \"../input/embeddings/glove.840B.300d/glove.840B.300d.txt\"\n    \n    # and now we'll retrieve the map from the embeddings\n    print(\"getting embeddings\")\n    def get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\n    embedding_index = dict(get_coefs(*o.strip().split(\" \")) for o in open(embedding_path, encoding='utf-8', errors='ignore'))\n    \n    # build up the embedding matrix that maps words to vectors\n    print(\"building embedding matrix\")\n    word_index = tk.word_index\n    nb_words = min(max_features, len(word_index))\n    embedding_matrix = np.zeros((nb_words + 1, embed_size))\n    for word, i in word_index.items():\n        if i >= max_features: continue\n        embedding_vector = embedding_index.get(word)\n        if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n    \n    return tk, embedding_matrix\n\n## featurize X\ndef featurize(X):\n    # tokenize the text\n    X_tokenized = tk.texts_to_sequences(X['question_text'].fillna('missing'))\n    \n    # remove sequences longer than max_len and pad the rest\n    X_pad = pad_sequences(X_tokenized, maxlen = max_len)\n    \n    return X_pad","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f919d81e37fbb46bafcfe07f8b03cb447a71ef70"},"cell_type":"code","source":"# first preprocess on all the available data\n# we need to be careful with what we do here - want to avoid any data\n# leakage when doing the testing later one\ntk, embedding_matrix = preprocess(X)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7cd2c744f7697b243d57e7103c7a828937c5c48e"},"cell_type":"markdown","source":"## 2.3 Create Model"},{"metadata":{"trusted":true,"_uuid":"af45b515fc15981a3fa2f0e45c40ae9080336d06"},"cell_type":"code","source":"# attention layer from https://www.kaggle.com/shujian/different-embeddings-with-attention-fork-fork\nfrom keras.models import Model\nfrom keras.layers import Input, Embedding, SpatialDropout1D, Bidirectional, CuDNNLSTM, Dense, Flatten, Conv2D, Conv1D\nfrom keras.layers import concatenate, GlobalAveragePooling1D, GlobalMaxPooling1D, BatchNormalization, Dropout\nfrom keras.layers import AveragePooling1D, MaxPooling1D, TimeDistributed\nfrom keras.layers import Layer\nfrom keras import initializers, regularizers, constraints\nfrom keras import backend as K\n\nclass Attention(Layer):\n    def __init__(self, step_dim,\n                 W_regularizer=None, b_regularizer=None,\n                 W_constraint=None, b_constraint=None,\n                 bias=True, **kwargs):\n        self.supports_masking = True\n        self.init = initializers.get('glorot_uniform')\n\n        self.W_regularizer = regularizers.get(W_regularizer)\n        self.b_regularizer = regularizers.get(b_regularizer)\n\n        self.W_constraint = constraints.get(W_constraint)\n        self.b_constraint = constraints.get(b_constraint)\n\n        self.bias = bias\n        self.step_dim = step_dim\n        self.features_dim = 0\n        super(Attention, self).__init__(**kwargs)\n\n    def build(self, input_shape):\n        assert len(input_shape) == 3\n\n        self.W = self.add_weight((input_shape[-1],),\n                                 initializer=self.init,\n                                 name='{}_W'.format(self.name),\n                                 regularizer=self.W_regularizer,\n                                 constraint=self.W_constraint)\n        self.features_dim = input_shape[-1]\n\n        if self.bias:\n            self.b = self.add_weight((input_shape[1],),\n                                     initializer='zero',\n                                     name='{}_b'.format(self.name),\n                                     regularizer=self.b_regularizer,\n                                     constraint=self.b_constraint)\n        else:\n            self.b = None\n\n        self.built = True\n\n    def compute_mask(self, input, input_mask=None):\n        return None\n\n    def call(self, x, mask=None):\n        features_dim = self.features_dim\n        step_dim = self.step_dim\n\n        eij = K.reshape(K.dot(K.reshape(x, (-1, features_dim)),\n                        K.reshape(self.W, (features_dim, 1))), (-1, step_dim))\n\n        if self.bias:\n            eij += self.b\n\n        eij = K.tanh(eij)\n\n        a = K.exp(eij)\n\n        if mask is not None:\n            a *= K.cast(mask, K.floatx())\n\n        a /= K.cast(K.sum(a, axis=1, keepdims=True) + K.epsilon(), K.floatx())\n\n        a = K.expand_dims(a)\n        weighted_input = x * a\n        return K.sum(weighted_input, axis=1)\n\n    def compute_output_shape(self, input_shape):\n        return input_shape[0],  self.features_dim","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2919b1ba05ef0c52e1ccabc42762e4acefadde40"},"cell_type":"code","source":"# this model uses my main kernel as inspiration\nfrom keras.models import Model\nfrom keras.layers import Input, Embedding, SpatialDropout1D, Bidirectional, CuDNNLSTM, Dense, Flatten, Conv1D\nfrom keras.layers import concatenate, GlobalAveragePooling1D, GlobalMaxPooling1D, BatchNormalization, Dropout\nfrom keras.layers import AveragePooling1D, MaxPooling1D\ndef createModel1():\n    # input layer\n    inp = Input(shape = (max_len,))\n    # embedding layer\n    x = Embedding(max_features + 1, embed_size, weights = [embedding_matrix], trainable = False)(inp)\n    # spatial dropout to add robustness with respect to feature maps\n    x1 = SpatialDropout1D(0.3)(x)\n    \n    # here I use my original kernel (link above) as inspiration to develop an architecture\n    # in my original kernel, I tried to use feature engineering in terms of topic modeling to encapsulate the interaction terms between features before doing classification\n    # here, I'll follow the same basic approach\n    # first create a convolutional layer that attempts to find the \"interactive\" relationships between words\n    # I'll use a kernel of 3 to encapsulate a \"3-gram\"\n    x_conv1 = Conv1D(32, kernel_size=3, padding='same', kernel_initializer='he_normal')(x1)\n    x_conv1 = MaxPooling1D(pool_size=3)(x_conv1)\n    # I'll also create an lstm that processes the words directly, followed by a convolutional layer to accumulate the results\n    x_lstm1 = Bidirectional(CuDNNLSTM(32, return_sequences = True))(x1)\n    x_lstm1 = Conv1D(32, kernel_size=3, padding='same', kernel_initializer='he_normal')(x_lstm1)\n    x_lstm1 = MaxPooling1D(pool_size=3)(x_lstm1)\n    \n    # create an x2 layer that combines the information from first layers\n    x2 = concatenate([x_conv1, x_lstm1])\n    \n    # create a second layer similar to the first layer, except we also introduce a skip connection to the original layer\n    # this is to emulate the second level stacked model\n    # implement the convolutional layer\n    x_conv2 = Conv1D(32, kernel_size=3, padding='same', kernel_initializer='he_normal')(x2)\n    x_conv2 = MaxPooling1D(pool_size=3)(x_conv2)\n    # lstm layer\n    x_lstm2 = Bidirectional(CuDNNLSTM(32, return_sequences = True))(x2)\n    x_lstm2 = Conv1D(32, kernel_size=3, padding='same', kernel_initializer='he_normal')(x_lstm2)\n    x_lstm2 = MaxPooling1D(pool_size=3)(x_lstm2)\n    \n    # create the final layer that will be used as an input to the classifier unit, including the skip connection to the first layer\n    x3 = concatenate([x_conv2, x_lstm2])\n    x3 = Dense(64, activation='relu')(x2)\n    \n    # create the output classifier unit\n    out = Flatten()(x3)\n    out = BatchNormalization()(out)\n    out = Dropout(0.5)(out)\n    out = Dense(16,activation='relu')(out)\n    out = BatchNormalization()(out)\n    out = Dropout(0.5)(out)\n    out = Dense(1, activation='sigmoid')(out)\n    \n    model = Model(inputs=inp, output=out)\n\n    model.compile(loss='binary_crossentropy',\n                  optimizer='adam',\n                  metrics=['accuracy'])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2919b1ba05ef0c52e1ccabc42762e4acefadde40"},"cell_type":"code","source":"# this is an ensemble network\n# https://www.kaggle.com/yekenot/2dcnn-textclassifier\n# https://www.kaggle.com/ashishsinhaiitr/different-embeddings-with-attention-fork-fork\nfrom keras.models import Model\nfrom keras.layers import Input, Embedding, SpatialDropout1D, Bidirectional, CuDNNLSTM, Dense, Flatten, Conv1D\nfrom keras.layers import concatenate, GlobalAveragePooling1D, GlobalMaxPooling1D, BatchNormalization, Dropout\nfrom keras.layers import AveragePooling1D, MaxPooling1D, TimeDistributed, Reshape\nfrom keras.layers import AveragePooling2D, MaxPooling2D\nfrom keras.layers import CuDNNGRU\ndef createModel2():\n    # input layer\n    inp = Input(shape = (max_len,))\n    # embedding layer\n    x = Embedding(max_features + 1, embed_size, weights = [embedding_matrix], trainable = False)(inp)\n    # spatial dropout to add robustness with respect to feature maps\n    x1 = SpatialDropout1D(0.3)(x)\n    \n    x_lstm1 = Bidirectional(CuDNNLSTM(64, return_sequences = True))(x1)\n    x_lstm2 = Bidirectional(CuDNNLSTM(32, return_sequences = True))(x_lstm1)\n    x_avg1 = GlobalAveragePooling1D()(x_lstm1)\n    x_max1 = GlobalMaxPooling1D()(x_lstm1)\n    x_avg2 = GlobalAveragePooling1D()(x_lstm2)\n    x_max2 = GlobalMaxPooling1D()(x_lstm2)\n    \n    x_gru1 = Bidirectional(CuDNNGRU(64, return_sequences = True))(x1)\n    x_gru2 = Bidirectional(CuDNNGRU(32, return_sequences = True))(x_gru1)\n    x_avg3 = GlobalAveragePooling1D()(x_gru1)\n    x_max3 = GlobalMaxPooling1D()(x_gru1)\n    x_avg4 = GlobalAveragePooling1D()(x_gru2)\n    x_max4 = GlobalMaxPooling1D()(x_gru2)\n    \n    x_lstm3 = Bidirectional(CuDNNLSTM(64, return_sequences = True))(x1)\n    x_lstm4 = Bidirectional(CuDNNLSTM(32, return_sequences = True))(x_lstm3)\n    x_att_1 = Attention(max_len)(x_lstm3)\n    x_att_2 = Attention(max_len)(x_lstm4)\n    \n    x_gru3 = Bidirectional(CuDNNGRU(64, return_sequences = True))(x1)\n    x_gru4 = Bidirectional(CuDNNGRU(32, return_sequences = True))(x_gru3)\n    x_att_3 = Attention(max_len)(x_gru3)\n    x_att_4 = Attention(max_len)(x_gru4)\n    \n    filters = [1, 3, 5, 7]\n\n    x_conv = Reshape((max_len, embed_size, 1))(x1)\n    conv_maxpool = []\n    conv_avgpool = []\n    for filt_size in filters:\n        conv = Conv2D(36, kernel_size=(filt_size, embed_size), kernel_initializer='glorot_uniform', activation='elu')(x_conv)\n        conv_maxpool.append(MaxPooling2D(pool_size=(max_len - filt_size + 1, 1))(conv))\n        conv_avgpool.append(AveragePooling2D(pool_size=(max_len - filt_size + 1, 1))(conv))\n    \n    x_avg5 = Flatten()(concatenate(conv_avgpool))\n    x_max5 = Flatten()(concatenate(conv_maxpool))\n    \n    out = concatenate([x_avg1,x_max1, x_avg2, x_max2,\n                       x_avg3,x_max3, x_avg4, x_max4,\n                       x_avg5, x_max5,\n                       x_att_1, x_att_2, x_att_3, x_att_4])\n    \n    # create the output classifier unit\n#     out = Flatten()(x_lstm1)\n    out = BatchNormalization()(out)\n    out = Dropout(0.5)(out)\n    out = Dense(256,activation='relu')(out)\n    out = BatchNormalization()(out)\n    out = Dropout(0.5)(out)\n    out = Dense(1, activation='sigmoid')(out)\n    \n    model = Model(inputs=inp, output=out)\n\n    model.compile(loss='binary_crossentropy',\n                  optimizer='adam',\n                  metrics=['accuracy'])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"466027a3ba49e16cdf6ceb77da060189a89d2058"},"cell_type":"code","source":"# this model uses skip connections to create a deeper network\nfrom keras.models import Model\nfrom keras.layers import Input, Embedding, SpatialDropout1D, Bidirectional, CuDNNLSTM, Dense, Flatten, Conv1D\nfrom keras.layers import concatenate, GlobalAveragePooling1D, GlobalMaxPooling1D, BatchNormalization, Dropout\nfrom keras.layers import AveragePooling1D, MaxPooling1D\ndef createModel3():\n    # input layer\n    inp = Input(shape = (max_len,))\n    # embedding layer\n    x = Embedding(max_features + 1, embed_size, weights = [embedding_matrix], trainable = False)(inp)\n    # spatial dropout to add robustness with respect to feature maps\n    x1 = SpatialDropout1D(0.3)(x)\n    \n    # I'm going to several layers of lstm-cnn blocks, while introducing skip connections every other block\n    x_lstm1 = Bidirectional(CuDNNLSTM(32, return_sequences = True))(x1)\n    x_conv1 = Conv1D(32, kernel_size=3, padding='same', kernel_initializer='he_normal')(x_lstm1)\n    x_max1 = MaxPooling1D(pool_size=3)(x_conv1)\n    \n    x_lstm2 = Bidirectional(CuDNNLSTM(32, return_sequences = True))(x_conv1)\n    x_conv2 = Conv1D(32, kernel_size=3, padding='same', kernel_initializer='he_normal')(x_lstm2)\n    x_max2 = MaxPooling1D(pool_size=3)(x_conv2)\n    \n    x_max12 = concatenate([x_max1,x_max2])\n    \n    x_lstm3 = Bidirectional(CuDNNLSTM(32, return_sequences = True))(x_max12)\n    x_conv3 = Conv1D(32, kernel_size=3, padding='same', kernel_initializer='he_normal')(x_lstm3)\n    x_max3 = MaxPooling1D(pool_size=3)(x_conv3)\n    \n    x_lstm4 = Bidirectional(CuDNNLSTM(32, return_sequences = True))(x_conv3)\n    x_conv4 = Conv1D(32, kernel_size=3, padding='same', kernel_initializer='he_normal')(x_lstm4)\n    x_max4 = MaxPooling1D(pool_size=3)(x_conv4)\n    \n    x_max34 = concatenate([x_max3, x_max4])\n    \n    x_lstm5= Bidirectional(CuDNNLSTM(32, return_sequences = True))(x_max34)\n    x_conv5= Conv1D(32, kernel_size=3, padding='same', kernel_initializer='he_normal')(x_lstm5)\n    x_max5 = MaxPooling1D(pool_size=3)(x_conv5)\n                                                                                       \n    x_lstm6= Bidirectional(CuDNNLSTM(32, return_sequences = True))(x_conv5)\n    x_conv6 = Conv1D(32, kernel_size=3, padding='same', kernel_initializer='he_normal')(x_lstm6)\n    x_max6 = MaxPooling1D(pool_size=3)(x_conv6)\n    \n    out = concatenate([x_max5, x_max6])\n    out = Flatten()(out)\n    \n    # create the output classifier unit\n    out = BatchNormalization()(out)\n    out = Dropout(0.5)(out)\n    out = Dense(16,activation='relu')(out)\n    out = BatchNormalization()(out)\n    out = Dropout(0.5)(out)\n    out = Dense(1, activation='sigmoid')(out)\n    \n    model = Model(inputs=inp, output=out)\n\n    model.compile(loss='binary_crossentropy',\n                  optimizer='adam',\n                  metrics=['accuracy'])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cca49e751a560cb363552bd8b185d327054e82b7"},"cell_type":"code","source":"# simple 2 layer model\nfrom keras.models import Model\nfrom keras.layers import Input, Embedding, SpatialDropout1D, Bidirectional, CuDNNLSTM, Dense, Flatten, Conv1D\nfrom keras.layers import concatenate, GlobalAveragePooling1D, GlobalMaxPooling1D, BatchNormalization, Dropout\nfrom keras.layers import AveragePooling1D, MaxPooling1D, TimeDistributed\ndef createModel4():\n    # input layer\n    inp = Input(shape = (max_len,))\n    # embedding layer\n    x = Embedding(max_features + 1, embed_size, weights = [embedding_matrix], trainable = False)(inp)\n    # spatial dropout to add robustness with respect to feature maps\n    x1 = SpatialDropout1D(0.3)(x)\n    \n    x_lstm = Bidirectional(CuDNNLSTM(32, return_sequences = True))(x1)\n    x_lstm = TimeDistributed(Dense(32))(x_lstm)\n    x_lstm = Conv1D(32, kernel_size=3, padding='same', kernel_initializer='he_normal')(x_lstm)\n    x_lstm = Bidirectional(CuDNNLSTM(32, return_sequences = True))(x_lstm)\n    x_lstm = TimeDistributed(Dense(32))(x_lstm)\n    x_lstm = Conv1D(32, kernel_size=3, padding='same', kernel_initializer='he_normal')(x_lstm)\n    x_lstm = BatchNormalization()(x_lstm)\n    x_lstm = Dropout(0.3)(x_lstm)\n    \n    out = Flatten()(x_lstm)\n    \n    out = Dense(1, activation='sigmoid')(out)\n    \n    model = Model(inputs=inp, output=out)\n\n    model.compile(loss='binary_crossentropy',\n                  optimizer='adam',\n                  metrics=['accuracy'])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cca49e751a560cb363552bd8b185d327054e82b7"},"cell_type":"code","source":"# simple model with attention layer from https://www.kaggle.com/shujian/different-embeddings-with-attention-fork-fork\nfrom keras.models import Model\nfrom keras.layers import Input, Embedding, SpatialDropout1D, Bidirectional, CuDNNLSTM, Dense, Flatten, Conv2D, Conv1D\nfrom keras.layers import concatenate, GlobalAveragePooling1D, GlobalMaxPooling1D, BatchNormalization, Dropout\nfrom keras.layers import AveragePooling1D, MaxPooling1D, TimeDistributed\nfrom keras.layers import Layer\nfrom keras import initializers, regularizers, constraints\nfrom keras import backend as K\n    \ndef createModel5():\n    # input layer\n    inp = Input(shape = (max_len,))\n    # embedding layer\n    x = Embedding(max_features + 1, embed_size, weights = [embedding_matrix], trainable = False)(inp)\n    # spatial dropout to add robustness with respect to feature maps\n    x = SpatialDropout1D(0.1)(x)\n    \n    x_lstm = Bidirectional(CuDNNLSTM(128, return_sequences = True))(x)\n    x_lstm = Bidirectional(CuDNNLSTM(64, return_sequences = True))(x_lstm)\n    x_lstm = Attention(max_len)(x_lstm)\n    \n    out = Dense(64, activation='relu')(x_lstm)\n    out = BatchNormalization()(out)\n    out = Dropout(0.1)(out)\n    out = Dense(1, activation='sigmoid')(out)\n    \n    model = Model(inputs=inp, output=out)\n\n    model.compile(loss='binary_crossentropy',\n                  optimizer='adam',\n                  metrics=['accuracy'])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5627de09512d7141a9a19479121eb4ec0c58bb8c"},"cell_type":"code","source":"model = createModel2()\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"61e994e168ca1d7c01e060dc166b4be93b4b17bc"},"cell_type":"code","source":"from IPython.display import SVG\nfrom keras.utils.vis_utils import model_to_dot\n\nSVG(model_to_dot(model).create(prog='dot', format='svg'))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"29526defa95138c5903997019aa0d8b2e2ce079e"},"cell_type":"markdown","source":"## 2.4 Train model"},{"metadata":{"trusted":true,"_uuid":"1d38a5f0fe7ae721a31a2ca35b8cf416b5a74331","_kg_hide-output":false},"cell_type":"code","source":"from keras.callbacks import EarlyStopping\nearly_stop = EarlyStopping(monitor = \"val_loss\", mode = \"min\", patience = 3)\nfrom sklearn.utils import class_weight\nclass_weights = class_weight.compute_class_weight('balanced',\n                                                 np.unique(y_train),\n                                                 y_train)\nhistory = model.fit(featurize(X_train),y_train,\n          epochs=3,\n          batch_size=512,\n          validation_split=0.1,\n          verbose=1,\n          callbacks=[early_stop],\n          class_weight=class_weights\n         )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0b9f06e825edc580ceeb252b82a34b6ff6021163"},"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n# Plot training & validation accuracy values\nplt.plot(history.history['acc'])\nplt.plot(history.history['val_acc'])\nplt.title('Model accuracy')\nplt.ylabel('Accuracy')\nplt.xlabel('Epoch')\nplt.legend(['Train', 'Test'], loc='upper left')\nplt.show()\n\n# Plot training & validation loss values\nplt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.title('Model loss')\nplt.ylabel('Loss')\nplt.xlabel('Epoch')\nplt.legend(['Train', 'Test'], loc='upper left')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eda36af8165bd50d06c712a15db2c96afbd2f6ae"},"cell_type":"markdown","source":"# 3. Testing and Results"},{"metadata":{"_uuid":"4e8bd338387c625f37708c26cde189a05ccc7438"},"cell_type":"markdown","source":"let's plot the test set classification report"},{"metadata":{"trusted":true,"_uuid":"8022ad28eb198d9dc29615b72a8d5323282741ca"},"cell_type":"code","source":"pred = model.predict(featurize(X_test), batch_size=1024, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1f853e2a66b1783ba5f6b97bb989c139a95c847e"},"cell_type":"code","source":"y_pred = pd.DataFrame(pred)\ny_pred.index=X_test.index\ny_pred.iloc[:,0].head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"31b9f2aa161f2697a519c36398770b99c5d6a94a","_kg_hide-output":true},"cell_type":"code","source":"from sklearn import metrics\nbest_thresh = None\nbest_score = None\nfor thresh in np.arange(0.001, 0.51, 0.01):\n    thresh = np.round(thresh, 2)\n    score = metrics.f1_score(y_test, (y_pred.iloc[:,0]>thresh).astype(int))\n    print(\"F1 score at threshold {0:.5f} is {1:.5f}\".format(thresh, score))\n    if best_score == None:\n        best_thresh = thresh\n        best_score = score\n    else:\n        if score > best_score:\n            best_thresh = thresh\n            best_score = score\nprint(\"Best F1 score at {0:0.3f} with threshold {1:0.3f}\".format(best_score,best_thresh))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e3a96ec86b79a29eed66d516b3a08286e90e625f"},"cell_type":"code","source":"from sklearn.metrics import classification_report\nprint(classification_report(y_test,y_pred.iloc[:,0]>best_thresh))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9eaa87d3bac6474b764101473d58e6ea467611fc"},"cell_type":"code","source":"df_test = pd.read_csv('../input/test.csv', index_col=0)\ntest_pred = model.predict(featurize(df_test), batch_size=1024, verbose=1)\ntest_pred = pd.DataFrame(test_pred)\ntest_pred.index=df_test.index\ny_test_pred = (test_pred.iloc[:,0]>best_thresh).astype(int)\nsubmit_df = pd.DataFrame({\"qid\": df_test.index.to_series(), \"prediction\": y_test_pred})\nsubmit_df.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}