{"cells":[{"metadata":{"_uuid":"0fc0998666050d889bc354233091563ca8022d3c"},"cell_type":"markdown","source":"# Overview\nQuora is a platform that empowers people to learn from each other. On Quora, people can ask questions and connect with others who contribute unique insights and quality answers. A key challenge is to weed out insincere questions -- those founded upon false premises, or that intend to make a statement rather than look for helpful answers.\n\n* **Target**: In this competition, we will develop models that identify and flag insincere questions.\n* **Data**: The training data includes the question that was asked, and whether it was identified as insincere (target = 1). For each qid in the test set, we must predict whether the corresponding question_text is insincere (1) or not (0). \n* **Evaluation metrics**:Submissions are evaluated on F1 Score between the predicted and the observed targets."},{"metadata":{"_uuid":"4ae26255952f50ce1ffda8fe95aad6c088132958"},"cell_type":"markdown","source":"# Import and Clean the data\n There are 1,306,122 entries in the training set and 375,806 entries in the test set. In the training data the insincere quetions  accout for 1/10 of all the questions. "},{"metadata":{"trusted":true,"_uuid":"a392690be3c8ef700188a356daa1ef19eb086a27"},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\ntrain_df=pd.read_csv(\"../input/quora-insincere-questions-classification/train.csv\")\ntest_df=pd.read_csv(\"../input/quora-insincere-questions-classification/test.csv\")\ntrain_df.sample(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bc99cce9d746d1ca02756cb78d3c7d79466c5da5"},"cell_type":"code","source":"train_df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5682de87d9090f356c4a1cadd149b6accb4780dd"},"cell_type":"code","source":"import matplotlib.pyplot as plt\nplt.hist(train_df.target)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8dce6997a80537c66515e6baace7a7a0f622cb01"},"cell_type":"code","source":"test_df.info()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f31910afa4085c816b208090a35463ce8b16870b"},"cell_type":"markdown","source":"Replace the sepecial symbols by space, convert the string into lowercase and fill the missing values with \"_na_\""},{"metadata":{"trusted":true,"_uuid":"ea19dd8f2b13e39cbe1948840f23dbc993121bf5"},"cell_type":"code","source":"puncts = [',', '.', '\"', ':', ')', '(', '-', '!', '?', '|', ';', \"'\", '$', '&', '/', '[', ']', '>', '%', '=', '#', '*', '+', '\\\\', '•',  '~', '@', '£', \n '·', '_', '{', '}', '©', '^', '®', '`',  '<', '→', '°', '€', '™', '›',  '♥', '←', '×', '§', '″', '′', 'Â', '█', '½', 'à', '…', \n '“', '★', '”', '–', '●', 'â', '►', '−', '¢', '²', '¬', '░', '¶', '↑', '±', '¿', '▾', '═', '¦', '║', '―', '¥', '▓', '—', '‹', '─', \n '▒', '：', '¼', '⊕', '▼', '▪', '†', '■', '’', '▀', '¨', '▄', '♫', '☆', 'é', '¯', '♦', '¤', '▲', 'è', '¸', '¾', 'Ã', '⋅', '‘', '∞', \n '∙', '）', '↓', '、', '│', '（', '»', '，', '♪', '╩', '╚', '³', '・', '╦', '╣', '╔', '╗', '▬', '❤', 'ï', 'Ø', '¹', '≤', '‡', '√', ]\ndef clean_text(x):\n    x = str(x)\n    for punct in puncts:\n        x = x.replace(punct, f' ')\n    return x\n\ndef preprocess(df,col):\n    df[col] = df[col].str.lower()\n    df[col] = df[col].apply(lambda x: clean_text(x))\n    df[col] = df[col].fillna(\"_na_\")\n    \npreprocess(train_df,\"question_text\")\npreprocess(test_df,\"question_text\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4f377b126df21443aebfd2854a9a7347da07a774"},"cell_type":"markdown","source":"Split the training set into train and validate set."},{"metadata":{"trusted":true,"_uuid":"bccf363e39f7b9d6618efd3ba7896462116537a2"},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\ntrain_df, val_df = train_test_split(train_df, test_size=0.1, random_state=2008)\nx_train = train_df[\"question_text\"].values\nx_val = val_df[\"question_text\"].values\nx_test = test_df[\"question_text\"].values","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d5a50d0b599f0f25da704265bba3577df8c24cc1"},"cell_type":"markdown","source":"Most of the questons have less than 200 words and the median of the length is around 70 words. Let us set the max length of the question as 80. "},{"metadata":{"trusted":true,"_uuid":"53dd3bf8dd7873f3781d41c57e230821a21f288e"},"cell_type":"code","source":"import matplotlib.pyplot as plt\nplt.hist([len(item) for item in x_train],bins=100)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f2438c2488b4013529d86034895295de34274eff"},"cell_type":"markdown","source":"# Tokenize the text\n\nUse  Keras to tokenzie the text,  then convert the lists of tokens into sequece. There are 185,320 unique tokens in the train text. I use the top 100,000 words as features by setting MAX_WORDS as 100,000. Since all the pretrained embeddings have  300 dimensions, I set the embedding size as 300."},{"metadata":{"trusted":true,"_uuid":"c4f33e785c80ff62fa099eafb3f0adc6e0fe04c7"},"cell_type":"code","source":"from keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\n\nMAXLEN=80 # cuts off the text after 100 words\nMAX_WORDS=100000 # consider only the top 100,000 words as features\nEMBEDDING_DIM=300 # The dimension of the embeddings\n\ntokenizer=Tokenizer(num_words=MAX_WORDS)\ntokenizer.fit_on_texts(x_train)\nword_index=tokenizer.word_index\nprint(f'Found {len(word_index)} unique tokens.')\n\nsequences_train=tokenizer.texts_to_sequences(x_train)\nx_train_2d=pad_sequences(sequences_train,maxlen=MAXLEN)\n\nsequences_val=tokenizer.texts_to_sequences(x_val)\nx_val_2d=pad_sequences(sequences_val,maxlen=MAXLEN)\n\nsequences_test=tokenizer.texts_to_sequences(x_test)\nx_test_2d=pad_sequences(sequences_test,maxlen=MAXLEN)\n\ny_train=train_df['target'].values\ny_val=val_df['target'].values","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2a6c224b7dd6e806ad983b52e4b17f6fb38a035c"},"cell_type":"markdown","source":"#  Load pretrained embeddings\nFour pretrianed embeddings are available to use in this competition."},{"metadata":{"trusted":true,"_uuid":"5d59b5a0c90b5f473f984b83c11d4c9c37801957"},"cell_type":"code","source":"!ls ../input/quora-insincere-questions-classification/embeddings/","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2c45ea1b0bd8cc8d4bc09bd37b1704aa017cdf0f"},"cell_type":"markdown","source":"Import Glove, Fasttext and Para embeddings. "},{"metadata":{"trusted":true,"_uuid":"8579efa28dbe7d607803c4c6c548adbd1869cd76"},"cell_type":"code","source":"EMBEDDING_FILE_Glove='../input/quora-insincere-questions-classification/embeddings/glove.840B.300d/glove.840B.300d.txt'\nEMBEDDING_FILE_Fasttext='../input/quora-insincere-questions-classification/embeddings/wiki-news-300d-1M/wiki-news-300d-1M.vec'\nEMBEDDING_FILE_Para='../input/quora-insincere-questions-classification/embeddings/paragram_300_sl999/paragram_300_sl999.txt'\n\ndef load_embeddings(EMBEDDING_FILE,word_index):\n\n    def get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')    \n    \n    if EMBEDDING_FILE==EMBEDDING_FILE_Glove:\n        embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE))\n    elif EMBEDDING_FILE==EMBEDDING_FILE_Fasttext:\n        embeddings_index = dict(get_coefs(*o.split(' ')) for o in open(EMBEDDING_FILE) if len(o)>100)\n    else:\n        embeddings_index = dict(get_coefs(*o.split(' ')) for o in open(EMBEDDING_FILE, encoding='utf8', errors='ignore') if len(o)>100)\n    print ('Found %s word vectors' % len(embeddings_index))\n\n    all_embs = np.stack(embeddings_index.values())\n    emb_mean,emb_std = all_embs.mean(), all_embs.std()\n    embeddings_size = all_embs.shape[1]\n\n    embedding_matrix = np.random.normal(emb_mean, emb_std, (MAX_WORDS,embeddings_size))#generate the embedding max with normal distribution \n    for word, i in word_index.items():\n        if i >= MAX_WORDS: continue\n        embedding_vector = embeddings_index.get(word)\n        if embedding_vector is not None: embedding_matrix[i] = embedding_vector# words not found in the embedding index will remain the initial values\n    \n    return embedding_matrix\n\nembeddings_matrix_glove=load_embeddings(EMBEDDING_FILE_Glove,word_index)\nembeddings_matrix_fasttext=load_embeddings(EMBEDDING_FILE_Fasttext,word_index)\nembeddings_matrix_para=load_embeddings(EMBEDDING_FILE_Para,word_index)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"121164238a220c11dcf924d3f9cca18d18aa0ec4"},"cell_type":"markdown","source":"# Prepare for the model\n**The attention mechanism** to overcome the limitation that allows the network to learn where to pay attention in the input sequence for each item in the output sequence."},{"metadata":{"trusted":true,"_uuid":"4fb3768ea6b9e946968721d9e38d715a060754dc"},"cell_type":"code","source":"from keras.engine.topology import Layer\nclass Attention(Layer):\n    def __init__(self, step_dim,\n                 W_regularizer=None, b_regularizer=None,\n                 W_constraint=None, b_constraint=None,\n                 bias=True, **kwargs):\n        self.supports_masking = True\n        self.init = initializers.get('glorot_uniform')\n\n        self.W_regularizer = regularizers.get(W_regularizer)\n        self.b_regularizer = regularizers.get(b_regularizer)\n\n        self.W_constraint = constraints.get(W_constraint)\n        self.b_constraint = constraints.get(b_constraint)\n\n        self.bias = bias\n        self.step_dim = step_dim\n        self.features_dim = 0\n        super(Attention, self).__init__(**kwargs)\n    \n    def build(self, input_shape):\n        assert len(input_shape) == 3\n\n        self.W = self.add_weight((input_shape[-1],),\n                                 initializer=self.init,\n                                 name='{}_W'.format(self.name),\n                                 regularizer=self.W_regularizer,\n                                 constraint=self.W_constraint)\n        self.features_dim = input_shape[-1]\n\n        if self.bias:\n            self.b = self.add_weight((input_shape[1],),\n                                     initializer='zero',\n                                     name='{}_b'.format(self.name),\n                                     regularizer=self.b_regularizer,\n                                     constraint=self.b_constraint)\n        else:\n            self.b = None\n\n        self.built = True\n\n    def compute_mask(self, input, input_mask=None):\n        return None\n    def call(self, x, mask=None):\n        features_dim = self.features_dim\n        step_dim = self.step_dim\n\n        eij = K.reshape(K.dot(K.reshape(x, (-1, features_dim)),\n                        K.reshape(self.W, (features_dim, 1))), (-1, step_dim))\n\n        if self.bias:\n            eij += self.b\n\n        eij = K.tanh(eij)\n\n        a = K.exp(eij)\n\n        if mask is not None:\n            a *= K.cast(mask, K.floatx())\n\n        a /= K.cast(K.sum(a, axis=1, keepdims=True) + K.epsilon(), K.floatx())\n\n        a = K.expand_dims(a)\n        weighted_input = x * a\n        return K.sum(weighted_input, axis=1)\n    \n    def compute_output_shape(self, input_shape):\n        return input_shape[0],  self.features_dim\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"84a3f47f15a1f50fdceb7af566d238e8144b4810"},"cell_type":"markdown","source":"## Build the model with LSTM and Attention\nSince we don't care the direction of the quesitons, we will use the bidrectional LSTM and Attention layes to build the model. The construction of my NN model is as following."},{"metadata":{"trusted":true,"_uuid":"027825aa3b77c0a2a488d3a9c74c41a8a926c0b7"},"cell_type":"code","source":"from IPython.display import Image\nImage(\"../input/picture/NN.png\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e11f28b7e438d5a25469b8d02531c5b68e2a1641"},"cell_type":"code","source":"from keras.layers import Dense, Input, CuDNNLSTM, Embedding, Dropout, Activation, CuDNNGRU, Conv1D\nfrom keras.layers import Bidirectional, GlobalMaxPool1D, GlobalMaxPooling1D, GlobalAveragePooling1D\nfrom keras.layers import Input, Embedding, Dense, Conv2D, MaxPool2D, concatenate, Lambda\nfrom keras.layers import Reshape, Flatten, Concatenate, Dropout, SpatialDropout1D\nfrom keras.optimizers import Adam\nfrom keras.models import Model\nfrom keras import backend as K\nfrom keras.engine.topology import Layer\nfrom keras import initializers, regularizers, constraints, optimizers, layers\nfrom keras.layers import concatenate\n\ndef model_lstm_atten(embedding_matrix):\n    inp = Input(shape=(MAXLEN,))\n    x = Embedding(MAX_WORDS, EMBEDDING_DIM, weights=[embedding_matrix], trainable=False)(inp)\n    x = SpatialDropout1D(0.1)(x)\n    x = Bidirectional(CuDNNLSTM(40, return_sequences=True))(x)\n    y = Bidirectional(CuDNNGRU(40, return_sequences=True))(x)\n    \n    atten_1 = Attention(MAXLEN)(x)\n    atten_2 = Attention(MAXLEN)(y)\n    avg_pool = GlobalAveragePooling1D()(y)\n    max_pool = GlobalMaxPooling1D()(y)\n    \n    conc = concatenate([atten_1, atten_2, avg_pool, max_pool])\n    conc = Dense(16, activation='relu')(conc)\n    conc = Dropout(0.1)(conc)\n    outp = Dense(1, activation='sigmoid')(conc)    \n\n    model = Model(inputs=inp, outputs=outp)\n    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['acc'])\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f11238d265691eeaca1f9d663d7502cd94770766"},"cell_type":"markdown","source":"# Fit and predict"},{"metadata":{"_uuid":"13dcac7a1a39c099006c974f67fdfd2fe01a6431"},"cell_type":"markdown","source":"Fit the model with glove embeddings and predict the validation set. Print f1 score corrsponding to thresh from 0.1 to 0.501. "},{"metadata":{"trusted":true,"_uuid":"ef47cbbaf092f458b6ab5a05f2f7c108584eb84a"},"cell_type":"code","source":"from sklearn import metrics\nmodel_glove=model_lstm_atten(embeddings_matrix_glove)\nmodel_glove.fit(x_train_2d,y_train,\n                   epochs=3,\n                   batch_size=512,\n                   validation_data=(x_val_2d, y_val))\npred_val_glove = model_glove.predict([x_val_2d], batch_size=1024, verbose=1)\nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    print(\"F1 score at threshold {0} is {1}\".format(thresh, metrics.f1_score(y_val, (pred_val_glove>thresh).astype(int))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4bb2a96aaee6e6a272dad37d0c82337fcf5d9e2b"},"cell_type":"code","source":"pred_test_glove = model_glove.predict([x_test_2d], batch_size=1024, verbose=1)\ndel embeddings_matrix_glove, model_glove\nimport gc; gc.collect()\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"06fdb8893bb5e7b0e019f4e95610f15853e691eb"},"cell_type":"markdown","source":"Fit the model with para embeddings and predict the validation set. Print f1 score corresponding to thresh from 0.1 to 0.501. "},{"metadata":{"_uuid":"b9d7bd6c39f61b7297acfc76a9068e26b1d1381d","trusted":true},"cell_type":"code","source":"model_para=model_lstm_atten(embeddings_matrix_para)\nmodel_para.fit(x_train_2d,y_train,\n                   epochs=3,\n                   batch_size=512,\n                   validation_data=(x_val_2d, y_val))\npred_val_para = model_para.predict([x_val_2d], batch_size=1024, verbose=1)\nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    print(\"F1 score at threshold {0} is {1}\".format(thresh, metrics.f1_score(y_val, (pred_val_para>thresh).astype(int))))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c5a7aa062ed031e1cc1bbca60b820b4f6773513f"},"cell_type":"code","source":"pred_test_para = model_para.predict([x_test_2d], batch_size=1024, verbose=1)\ndel embeddings_matrix_para, model_para\nimport gc; gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"01c06d1ba433aa01d0c37825ce6c260e58e051f9"},"cell_type":"markdown","source":"According to the printed f1-thresh data, use 0.35 as the thresh to turn the probabilities of output into binary output. Ensemble the outputs of the above two trained models as the final output to submit."},{"metadata":{"_uuid":"c778857c78d77f75611ab0bebcf1b5e383b0495b","trusted":true},"cell_type":"code","source":"pred_test=0.6*pred_test_glove+0.4*pred_test_para\npred_test=(pred_test>0.35).astype(int)\nplt.hist(pred_test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"24e0eab33ed8e9f0d4e6dab090c268200a46c168","trusted":true},"cell_type":"code","source":"test_df['prediction']=pred_test\ntest_df=test_df.drop(['question_text'],axis=1)\ntest_df.to_csv(\"submission.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8fefd9302a97048306a65aacccfe4ec0434e5e5b"},"cell_type":"markdown","source":"# Further thoughts\nAfter I learned some top rank solutions, I think some skills like the followings could be applied to imporve the performance:<br>\n1) Simplify the RNN model<br>\n2) Keep all the tokens from raw text, that should sacrifice the running speed<br>\n3) Ensemble the embeddings by concatenate them instead of averaging them or weighting them. In another words, increase the embedding size.<br>\n4) Use variety of stemmer to clean the text<br>"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}