{"cells":[{"metadata":{"_uuid":"59f69e153cbdc849cf4269dc6027d7237ddfb170"},"cell_type":"markdown","source":"## Quora Insincere Questions Classification"},{"metadata":{"_uuid":"23ef1f3d08c0ace170883dc43e3862146fa731dd"},"cell_type":"markdown","source":"#### Marco Gancitano\n#### 27 November 2018"},{"metadata":{"_uuid":"94dd123c7e92372f6a08bef144bf23a5a6d93cc3"},"cell_type":"markdown","source":"### Load Packages"},{"metadata":{"trusted":true,"_uuid":"1bbcbce09237f72e66ea4396c8642cac63b69e9c"},"cell_type":"code","source":"import datetime\nimport pandas as pd\nimport numpy as np\nimport random\nimport time\n\n# Data Viz\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nimport os\nfrom yellowbrick.text import TSNEVisualizer\n\n# Hide Warnings\nWarning = True\nif Warning is False:\n    import warnings\n    warnings.filterwarnings(action='ignore')\n    warnings.filterwarnings(action='ignore', category=DeprecationWarning)\n    warnings.filterwarnings(action='ignore', category=FutureWarning)\n\n#Modeling \nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import cross_validate\nfrom sklearn import metrics\nfrom sklearn.linear_model import LogisticRegressionCV\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn import preprocessing\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.discriminant_analysis import LinearDiscriminantAnalysis\n\n# Tf-Idf\nfrom sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.pipeline import FeatureUnion\nfrom scipy.sparse import hstack, csr_matrix\nfrom keras.preprocessing import text, sequence\n    \n\nnp.random.seed(2018)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aa084d344cebe096704ab93abc7b84d0364d4b61"},"cell_type":"markdown","source":"### Read in Data"},{"metadata":{"trusted":true,"_uuid":"a2450f1cd80c5a0880d85f742998d85e93f22855"},"cell_type":"code","source":"train = pd.read_csv(\"../input/train.csv\", index_col= 'qid')#.sample(50000)\ntest = pd.read_csv(\"../input/test.csv\", index_col= 'qid')#.sample(5000)\ntestdex = test.index\n\ntarget_names = [\"Sincere\",\"Insincere\"]\ny = train['target'].copy()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e480d0d0966307048eedcffb0af5302f0c3727bb"},"cell_type":"markdown","source":"#### Take a quick look at the data"},{"metadata":{"trusted":true,"_uuid":"1f3754eaf711819bf7f5e0de59dbf37274dce1a7"},"cell_type":"code","source":"print(train.shape)\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"269707d132e0429f9d05e2c76c7f2b4b32c97f4a"},"cell_type":"code","source":"print(\"Distribution of Classes:\")\ntrain.target.value_counts(normalize=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4aae0b10caa2b7adb9769e9f1b9c8342b7e5ebc6"},"cell_type":"markdown","source":"As we can see most questions are sincere with only <b>6.2% of questions being insincere</b>"},{"metadata":{"_uuid":"3b86f238a8b6cc124b23bff8309a527b4bbccde5"},"cell_type":"markdown","source":"<hr>"},{"metadata":{"_uuid":"e53127a18b296f18cef04a798800f7a35bb29ef6"},"cell_type":"markdown","source":"### Build word vectors and Visualize Results"},{"metadata":{"_uuid":"2c90ef1440f10ac66e4d12688d31cd9491e34e90"},"cell_type":"markdown","source":"#### TF-IDF"},{"metadata":{"_uuid":"4174572ba32a2b7f9b6db3e3a269fa344f6ac18a"},"cell_type":"markdown","source":"[Term Frequency Inverse Document Frequency](https://www.tfidf.com) Technique commonly used when dealing with words in documents, the words that appear most often across documents are usually not that helpful because they don't distinguish documents. TF-IDF is a way to weight a word higher for appearing more often within an article but decrease the weighting for a word appearing more often between articles.)"},{"metadata":{"_uuid":"f0fcbeda2865c81b9f55c863b4bc51fbe8450933"},"cell_type":"markdown","source":"I use this technique to create a word vector for my models that is more robust than typical one-hot encoding"},{"metadata":{"trusted":true,"_uuid":"9176970c31a6a439f63ab224a310bd18c1cf68e1"},"cell_type":"code","source":"all_text = pd.concat([train['question_text'],test['question_text']], axis =0)\n\nword_vect = TfidfVectorizer(\n            sublinear_tf=True,\n            strip_accents='unicode',\n            analyzer='word',\n            token_pattern=r'\\w{1,}',\n            stop_words='english',\n            ngram_range=(1, 2),\n            max_features=20000)\nword_vect.fit(all_text)\nX  = word_vect.transform(train['question_text'])\ntesting  = word_vect.transform(test['question_text'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"20d1a67a9379081fd1bcb7a22764d2f51194b906"},"cell_type":"code","source":"X_train, X_valid, y_train, y_valid = train_test_split(\n        X, y, test_size=0.20, random_state=123, stratify=y)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4067542dc37f00709299ed93b86482528c308e53"},"cell_type":"markdown","source":"#### t-SNE - Visual Cluster Plot"},{"metadata":{"_uuid":"2b6e02786277ffeec57175a6a0239ade2befdaf2"},"cell_type":"markdown","source":"t-SNE ( [t-Distributed Stochastic Neighbor Embedding](https://lvdmaaten.github.io/tsne/)) is a technique for dimensionality reduction suited well for visulizating high-dimensional datasets."},{"metadata":{"trusted":true,"_uuid":"fd05145e62cf35e6b86ffb860ae21a560e9432c0"},"cell_type":"code","source":"# Create the visualizer and draw the vectors\nplt.figure(figsize = [15,9])\ntsne = TSNEVisualizer()\nn = 20000\ntsne.fit(X_train[:n], train.target[:n].map({1: target_names[1],0:target_names[0]}))\ntsne.poof()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a756fff6b9d8b7d81a63448bca5261a25837488f"},"cell_type":"markdown","source":"We can see here that the data isn't very seperable by traditional means. A non-parametric model will most likely do the best (Gradient Boosting or Neural Networks)"},{"metadata":{"_uuid":"dbe6db1e865de2f08862314e71b08cb17cbf1700"},"cell_type":"markdown","source":"<hr>"},{"metadata":{"_uuid":"9103ff17c98418ae75950c68534c678eca68d697"},"cell_type":"markdown","source":"### Modeling "},{"metadata":{"_uuid":"5eb50e4a000a4729d7ec3c1506673d2382e7c211"},"cell_type":"markdown","source":"#### Create function "},{"metadata":{"_uuid":"6ab487cfe479a8023f0fc43dabddf48d7bb34dec"},"cell_type":"markdown","source":"The Quora Kaggle competition is evaluated off of F1-Score so that is the score used for testing models."},{"metadata":{"trusted":true,"_uuid":"babae2f8aa6d28b624f31d57c84259182df7f27f"},"cell_type":"code","source":"def model_fit(model,X_train,X_val,y_train,y_val):\n    model.fit(X_train, y_train)\n    \n    # Predict\n    valid_logistic_pred = model.predict(X_val)\n    train_logistic_pred = model.predict(X_train)\n    \n    print(\"Train Set F1 Score: {:.3f}\".format(metrics.f1_score(train_logistic_pred, y_train)))\n    print(\"Validation Set F1 Score: {:.3f}\".format(metrics.f1_score(valid_logistic_pred, y_val)))\n\n    # Confusion Matrix\n    C = metrics.confusion_matrix(valid_logistic_pred, y_val)/len(y_val)\n    sns.heatmap(C, annot=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d478c8246e97ec7d258c07117dff1261e374b4b2"},"cell_type":"markdown","source":"#### Logistic Regression"},{"metadata":{"trusted":false,"_uuid":"77242a1405e2708fedb07525eff30ec21c56fa80"},"cell_type":"code","source":"# Fit Model\nmodel_fit(LogisticRegression(solver = 'sag'),X_train,X_valid,y_train,y_valid)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a80f817cfc579725cfeb3396441b1caef38ef93b"},"cell_type":"markdown","source":"#### Build F1-score function for keras use"},{"metadata":{"trusted":true,"_uuid":"5fd591ad170570810ae9054a022a36550c44df87"},"cell_type":"code","source":"from keras import backend as K\n\ndef f1(y_true, y_pred):\n    def recall(y_true, y_pred):\n        \"\"\"Recall metric.\n\n        Only computes a batch-wise average of recall.\n\n        Computes the recall, a metric for multi-label classification of\n        how many relevant items are selected.\n        \"\"\"\n        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n        possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n        recall = true_positives / (possible_positives + K.epsilon())\n        return recall\n\n    def precision(y_true, y_pred):\n        \"\"\"Precision metric.\n\n        Only computes a batch-wise average of precision.\n\n        Computes the precision, a metric for multi-label classification of\n        how many selected items are relevant.\n        \"\"\"\n        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n        predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n        precision = true_positives / (predicted_positives + K.epsilon())\n        return precision\n    precision = precision(y_true, y_pred)\n    recall = recall(y_true, y_pred)\n    return 2*((precision*recall)/(precision+recall+K.epsilon()))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"87c52fd42950327e6989363ac8a5db04f9ad51c7"},"cell_type":"markdown","source":"#### Build model validation function to test parameters"},{"metadata":{"trusted":true,"_uuid":"8567a504aa9cd793466262cbf8e03eabe265072a"},"cell_type":"code","source":"def model_val(model,X_valid,y_valid):\n    preds = model.predict([X_valid],batch_size=1024,verbose = True)\n    for i in np.arange(.1,.6,.025):\n        i = np.round(i, 3)\n        score = metrics.f1_score(y_valid,(preds > i))\n        print(\"F1 score at threshold {0} is {1}\".format(i, score))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2d167ab42ad29f977f8a94cc5636a1520609b7ff"},"cell_type":"markdown","source":"#### Neural Networks"},{"metadata":{"trusted":true,"_uuid":"037abe38acdf134ebd0aed6b3c28a79b48da2b75"},"cell_type":"code","source":"from keras.models import Model, Sequential\nfrom keras.layers import CuDNNGRU,CuDNNLSTM,Input, Dense, Embedding,Dropout, concatenate, Bidirectional,Flatten,GlobalAveragePooling1D, GlobalMaxPool1D","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6d377a27f3867af241d2a35fc688e012c87aa8e3"},"cell_type":"code","source":"maxlen = 100\nmax_features = 20000\n\ninp = Input((max_features,))\nhidden1 = Dense(units = maxlen)(inp)\nhidden2 = Dense(units = maxlen)(hidden1)\n\nfinal = Dense(units = 1,activation = 'sigmoid')(hidden2)\n\nmodel = Model(inp,final)\n\nmodel.compile(loss='binary_crossentropy',\n              optimizer='adam',\n              metrics=[f1])\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"8fc02db4240b57838790cc20e4d50995a0ff90a8"},"cell_type":"code","source":"model.fit(X_train,y_train,batch_size = 1024,epochs = 2,validation_data = (X_valid,y_valid),verbose = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"ff1aecbc2abfaca2c6ca102f4b4bfb10b2231049"},"cell_type":"code","source":"model_val(model,X_valid,y_valid)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a3feed592009122a1196d9e328f5b490bead4bda"},"cell_type":"markdown","source":"### Add Neural Network Embeddings"},{"metadata":{"_uuid":"f9d8bee38dad6bcc524c8c2ff257a5e96e2c2021"},"cell_type":"markdown","source":"For this part I lessen the power the TF-IDF has, I still use it to tokenize the word vectors but make them smaller with only 100 words per question. I also pad them so that all sentences have the proper words aligned instead of having misaligned rows. I then add an embedding layer to the neural network so it can learn how the questions are related, this is a more robust technique for finding \"distances\" between sentences."},{"metadata":{"trusted":true,"_uuid":"ca368d636ebfe41fef42de3af558f306fe3df16a"},"cell_type":"code","source":"## some config values \nembed_size = 300 # how big is each word vector\nmax_features = 50000 # how many unique words to use (i.e num rows in embedding vector)\nmaxlen = 100 # max number of words in a question to use\n\n## split to train and val\ntrain_df, val_df = train_test_split(train, test_size=0.1, random_state=2018)\n\n## fill up the missing values\ntrain_X = train_df[\"question_text\"].fillna(\"_na_\").values\nval_X = val_df[\"question_text\"].fillna(\"_na_\").values\ntest_X = test[\"question_text\"].fillna(\"_na_\").values\n\n## Tokenize the sentences\ntokenizer = text.Tokenizer(num_words=max_features)\ntokenizer.fit_on_texts(list(train_X))\nX_train = tokenizer.texts_to_sequences(train_X)\nX_valid = tokenizer.texts_to_sequences(val_X)\nX_test = tokenizer.texts_to_sequences(test_X)\n\n## Pad the sentences \nX_train = sequence.pad_sequences(X_train, maxlen=maxlen)\nX_valid = sequence.pad_sequences(X_valid, maxlen=maxlen)\nX_test = sequence.pad_sequences(X_test, maxlen=maxlen)\n\n## Get the target values\ny_train = train_df['target'].values\ny_valid = val_df['target'].values","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"da22e2591a362f68d4cc6b2d67aa3cc1839ffd14"},"cell_type":"markdown","source":"I've added an Embedding layer in the beginning of the neural network so embeddings are made before going into the network. Additionally, i added a flatten layer so the data goes from 2D (a matrix with each word having it's own embedding array) to 1D which is better for the Dense layers."},{"metadata":{"trusted":false,"_uuid":"fe805f449079db17609196438433333b5772f8d0"},"cell_type":"code","source":"inp = Input(shape=(maxlen,))\nemb = Embedding(max_features, embed_size)(inp)\n\nhidden1 = Dense(units = 64)(emb)\nflat = Flatten()(hidden1)\nhidden2 = Dense(units = 16,activation = 'relu')(flat)\n\nfinal = Dense(units = 1,activation = 'sigmoid')(hidden2)\n\nmodel = Model(inp,final)\n\nmodel.compile(loss='binary_crossentropy',\n              optimizer='adam',\n              metrics=[f1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"b10b2f8b18e5c62d29b612b55d2b55937c131186"},"cell_type":"code","source":"model.fit(X_train, y_train, batch_size=1024, epochs=2, validation_data=(X_valid, y_valid),verbose = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"153c7560c1be44a2ce6f41e9ef749ce246c2e2a7"},"cell_type":"code","source":"model_val(model,X_valid,y_valid)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"18a4e04c431a63b0121f792f58745f8be735eff7"},"cell_type":"markdown","source":"The flattening layer leads to a very wide layer of the neural network so instead of use a pooling layer. This layer takes the maximum and average value from each word vector and uses that as the single value in the neuron."},{"metadata":{"trusted":false,"_uuid":"326730cf84519354b92618242646bf277e34a6d7"},"cell_type":"code","source":"inp = Input(shape=(maxlen,))\nemb = Embedding(max_features, embed_size)(inp)\n\nhidden1 = Dense(units = 64)(emb)\nmax_pool = GlobalMaxPool1D()(hidden1)\navg_pool = GlobalAveragePooling1D()(hidden1)\nconc = concatenate([max_pool,avg_pool])\n\nhidden2 = Dense(units = 16,activation = 'relu')(conc)\n\nfinal = Dense(units = 1,activation = 'sigmoid')(hidden2)\n\nmodel = Model(inp,final)\n\nmodel.compile(loss='binary_crossentropy',\n              optimizer='adam',\n              metrics=[f1])","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"5fcac54507b79d9a0582622346bd381e72909064"},"cell_type":"code","source":"model.fit(X_train, y_train, batch_size=1024, epochs=2, validation_data=(X_valid, y_valid),verbose = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"c995116e62b731aec51a90b8bd69bc0d49b51834"},"cell_type":"code","source":"model_val(model,X_valid,y_valid)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"297d74e84439c42c7e8d11bd906b8f2161a780b9"},"cell_type":"markdown","source":"The last piece of the puzzle for creating a robust neural network for NLP is adding some memory to the system. Long-short term memory neural networks are a type of Recurrent Neural Network which have the benefit of using previous runs outputs as input into the next run which has been proven to work well for classifiying speech. In addition, the layer is bidirectional which means the first half of the layer are the inputs normally and the second half is them reversed. This allows for a more robust understanding of the language used."},{"metadata":{"trusted":true,"_uuid":"fe702475177c8863b04100404e71a678d2f62b62"},"cell_type":"code","source":"inp = Input(shape=(maxlen,))\nemb = Embedding(max_features, embed_size)(inp)\n\nhidden1 = Bidirectional(CuDNNLSTM(units = 64,return_sequences = True))(emb)\nmax_pool = GlobalMaxPool1D()(hidden1)\navg_pool = GlobalAveragePooling1D()(hidden1)\nconc = concatenate([max_pool,avg_pool])\n\nhidden2 = Dense(units = 16,activation = 'relu')(conc)\n\nfinal = Dense(units = 1,activation = 'sigmoid')(hidden2)\n\nmodel = Model(inp,final)\n\nmodel.compile(loss='binary_crossentropy',\n              optimizer='adam',\n              metrics=[f1])\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"90194c947787bbe43950695a33eb8bcb3ac932b5"},"cell_type":"code","source":"model.fit(X_train, y_train, batch_size=1024, epochs=2, validation_data=(X_valid, y_valid),verbose = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"afe5eb4218dcd8efce23052749cb830b08a5a1db"},"cell_type":"code","source":"model_val(model,X_valid,y_valid)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ea50c25effa6446f51dc3fe6df41aa629579450e"},"cell_type":"markdown","source":"### Add external embeddings"},{"metadata":{"trusted":true,"_uuid":"6d2f82c947dfb4710621094a2a91d56e8392c704"},"cell_type":"code","source":"EMBEDDING_FILE = '../input/embeddings/glove.840B.300d/glove.840B.300d.txt'\ndef get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\nembeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE))\n\nall_embs = np.stack(embeddings_index.values())\nemb_mean,emb_std = all_embs.mean(), all_embs.std()\nembed_size = all_embs.shape[1]\n\nword_index = tokenizer.word_index\nnb_words = min(max_features, len(word_index))\nembedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\nfor word, i in word_index.items():\n    if i >= max_features: continue\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None: embedding_matrix[i] = embedding_vector","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d853e4c0a7eee0a834630e4635fe0c3ffca8b33e"},"cell_type":"code","source":"inp = Input(shape=(maxlen,))\nemb = Embedding(max_features, embed_size, weights=[embedding_matrix])(inp)\n\nhidden1 = Bidirectional(CuDNNLSTM(units = 64,return_sequences = True))(emb)\nmax_pool = GlobalMaxPool1D()(hidden1)\navg_pool = GlobalAveragePooling1D()(hidden1)\nconc = concatenate([max_pool,avg_pool])\n\nhidden2 = Dense(units = 16,activation = 'relu')(conc)\ndrop = Dropout(0.1)(hidden2)\nfinal = Dense(units = 1,activation = 'sigmoid')(drop)\n\nmodel = Model(inp,final)\n\nmodel.compile(loss='binary_crossentropy',\n              optimizer='adam',\n              metrics=[f1])\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"05e5a169f462ab6d03340a1a0d3fa0f5534d7060"},"cell_type":"code","source":"model.fit(X_train, y_train, batch_size=1024, epochs=2, validation_data=(X_valid, y_valid),verbose = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8a319ca29df44d3d15ac69d96d42dfa43c67544b"},"cell_type":"code","source":"model_val(model,X_valid,y_valid)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ac043d04c7312196b2fa76574f3fdfba2f391aaa"},"cell_type":"markdown","source":"### Make Predictions"},{"metadata":{"trusted":true,"_uuid":"a083c20dbc8a9481a0f23eff06d5e8483d6ce997"},"cell_type":"code","source":"pred_test_y = model.predict(X_test,batch_size = 1024,verbose = True)\npred_test_y = (pred_test_y>0.35).astype(int)\nout_df = pd.DataFrame({\"qid\":test.index.values})\nout_df['prediction'] = pred_test_y\nout_df.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}