{"cells":[{"metadata":{"collapsed":true,"_uuid":"48e7d0b17674d65f7ca1d5994325e7ffe485af69"},"cell_type":"markdown","source":"# LSTM Training\nThis notebook can be used to train an LSTM for text classification and generate predictions for the kaggle competition found [here](https://www.kaggle.com/c/quora-insincere-questions-classification). \n\nThe notebook utilizes Keras and GloVe for preprocessing using word embeddings. Then, Keras with Tensorflow backend is used for training a deep LSTM. Feel free to fork!"},{"metadata":{"trusted":true,"_uuid":"31651e5a58fc9229e683afac2cbc7b0948a5f240"},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport os\n\nfrom sklearn.model_selection import train_test_split\n\n\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.layers import Input, Dense, Embedding, CuDNNLSTM\nfrom keras.layers import Bidirectional, GlobalMaxPool1D, Dropout\nfrom keras.models import Model\n\n%load_ext autoreload\n%autoreload 2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9dc94191884a37933e64d2d822040a9779a621fd"},"cell_type":"code","source":"# Load in training and testing data\ntrain_df = pd.read_csv('../input/train.csv')\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c5f86071af4160a6d9370a06dea2fb6e2f95533e"},"cell_type":"markdown","source":"# 1. EDA\nThe following code will perform basic EDA to gain an understanding of the dataset and perhaps inspire the design of the neural network.\n\nNote: Section 1 is not required for the rest of the notebook and may be skipped if desired. "},{"metadata":{"_uuid":"105a3a836d32cdfad2a54ec005d9f1aff389a032"},"cell_type":"markdown","source":"## 1.1 Sentiment Analysis\nThis subsection analyzes each question using the NLTK sentiment analyzer. The goal is to determine whether the sincere and insincere questions have a significant difference in sentiment. \n\nNote: The sentiment analyis takes about 3 minutes to run on my machine. Tips are welcome on how to increase the speed of this operation!"},{"metadata":{"trusted":true,"_uuid":"bdbf35d1f29cf895323223d028a515c0bbbd4541"},"cell_type":"code","source":"# Sentiment analysis requires the nltk package\nimport nltk\nnltk.download('vader_lexicon')\nfrom nltk.sentiment.vader import SentimentIntensityAnalyzer\n\n# Initialize the sentiment analyzer and define the sentiment extraction function\nsid = SentimentIntensityAnalyzer()\ndef extract_sentiment(text):\n    scores = sid.polarity_scores(text)\n    return scores['compound']\n\ntrain_df['sent'] = train_df['question_text'].apply(lambda x: extract_sentiment(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ea35bcaeed73403f2b866b357b7ba27b549b5b47"},"cell_type":"code","source":"print('The average sentiment score for sincere questions is {:0.4f}'.\\\n          format(np.mean(train_df[train_df['target']==0]['sent'])))\nprint('The average sentiment score for insincere questions is {:0.4f}'. \\\n          format(np.mean(train_df[train_df['target']==1]['sent'])))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3a1f9db0fe8858795559928381f1e12e4c04ee14"},"cell_type":"markdown","source":"From the density plot below, we can see that a large number of questions are classified as exactly neutral, with a sentiment score of 0.0. It would appear that there is no strong significant difference in sentiment between sincere and insincere questions"},{"metadata":{"trusted":true,"_uuid":"5c4ca2b7af0006db02ed2f2e477a5d45a88e2a4c"},"cell_type":"code","source":"sent_sinc = train_df[train_df['target']==0]['sent'].values\nsent_insinc = train_df[train_df['target']==1]['sent'].values\n\nsns.kdeplot(sent_sinc, shade=1, color=\"skyblue\", label=\"Sincere\")\nsns.kdeplot(sent_insinc, shade=1, color=\"red\", label=\"Insincere\")\n\n# Plot formatting\nplt.legend(prop={'size': 12}, title = 'Sincerity')\nplt.title('Sentiment Density Plot')\nplt.xlabel('Sentiment')\nplt.ylabel('Density')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fc6c4998c331c811194c4c9ed3206c889e032f67"},"cell_type":"markdown","source":"## 1.2 Word Count\nThis code will analyze word count characteristics of the dataset and determine whether there is a distinction in word count between the sincere and insincere questions. "},{"metadata":{"trusted":true,"_uuid":"18ba5b3b859d75bc6ba9e2b9dc7956e50f5e54d5"},"cell_type":"code","source":"train_df['word_count'] = train_df['question_text'].apply(lambda x: len(x.split()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1e5c03f5d075dfe8f48ce5edd696c380804e01fa"},"cell_type":"code","source":"print('The average word length of sincere questions in the training set is {0:.1f}.'\\\n          .format(np.mean(train_df[train_df['target']==0]['word_count'])))\nprint('The average word length of insincere questions in the training set is {0:.1f}.' \\\n          .format(np.mean(train_df[train_df['target']==1]['word_count'])))\nprint('The maximum word length for a question in the training set is {0:.0f}.'\\\n          .format(np.max(train_df['word_count'])))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8faba7fa22ab26b799626f6ae350f12154bac555"},"cell_type":"markdown","source":"From the plot below, we can see that the word count distribution for sincere questions has a much sharper peak. It appears that insincere questions tend to be a bit longer than sincere questions. "},{"metadata":{"trusted":true,"_uuid":"d3edc422056baf9acd45e36e6d86bfbed6b16838"},"cell_type":"code","source":"wc_sinc = train_df[train_df['target']==0]['word_count'].values\nwc_insinc = train_df[train_df['target']==1]['word_count'].values\n\nsns.kdeplot(wc_sinc, shade=1, color=\"skyblue\", label=\"Sincere\")\nsns.kdeplot(wc_insinc, shade=1, color=\"red\", label=\"Insincere\")\n\n# Plot formatting\nplt.legend(prop={'size': 12}, title = 'Sincerity')\nplt.title('Word Count Density Plot')\nplt.xlabel('Word Count')\nplt.ylabel('Density')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e56b45eca68b4061b2c36d6d6a4399bd389fe613"},"cell_type":"markdown","source":"## 1.3 Word Cloud\nBecause word clouds look cool.\n\nThe word clouds do present some interesting information. Specifically, the insincere questions have a much higher proportion of words relating to identities and politics, such as 'Muslim', 'Trump', 'Men', 'Women', 'Liberal', 'American', etc. Conversely, the sincere questions have more constructive words that indicate the user is trying to solve a problem: 'Will', 'Good', 'Use', 'Think', etc."},{"metadata":{"trusted":true,"_uuid":"ed99e577ffb6d57d4342be7b8ddc82eb0c8420cc"},"cell_type":"code","source":"from wordcloud import WordCloud,STOPWORDS\n\ndef wordcloud_draw(data, title, color='black'):\n    words = ' '.join(data)\n    wordcloud = WordCloud(stopwords=STOPWORDS,\n                      background_color=color,\n                      width=2500,\n                      height=2000\n                     ).generate(words)\n    plt.figure(1,figsize=(13, 13))\n    plt.imshow(wordcloud)\n    plt.title(title, fontdict={'fontsize':18})\n    plt.axis('off')\n    plt.show()\n\ntitle = \"Sincere words\"\nwordcloud_draw(train_df[train_df['target']==0]['question_text'], title, 'white')\ntitle = \"Insincere Words\"\nwordcloud_draw(train_df[train_df['target']==1]['question_text'], title, 'black')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"da5d0f6e01ebc85a0588ecf83c55086dfa144b84"},"cell_type":"markdown","source":"# 2. Data Preparation\nThis section of the notebook is devoted to preprocessing the raw data into a form that the neural network can understand."},{"metadata":{"trusted":true,"_uuid":"78488b53255b19d912481adbc91f5bcf0410e689"},"cell_type":"code","source":"# Extract the training data and corresponding labels\ntext = train_df['question_text'].fillna('unk').values\nlabels = train_df['target'].values\n\n# Split into training and validation sets by making use of the scikit-learn\n# function train_test_split\nX_train, X_val, y_train, y_val = train_test_split(text, labels,\\\n                                                  test_size=0.2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"571063c924939109f5f08c5cb9ab2d5435901777"},"cell_type":"markdown","source":"## 2.1 Create Word Embedding Matrix\nThe code in this section will identify the most commonly occurring words in the dataset. Then, it will extract the vectors for each one of these words from the GloVe pretrained word embedding and place them in an embedding layer matrix. This embedding layer will serve as the first layer of the neural network. \n\nRead more about GloVe word embeddings [here](https://nlp.stanford.edu/projects/glove/).\n\nNote that other word embeddings are also available for this competition, however glove was chosen for this notebook. "},{"metadata":{"trusted":true,"_uuid":"e00b006b3a940240da8d4d8b2bfda5765a9f7a40"},"cell_type":"code","source":"embed_size = 300 # Size of each word vector\nmax_words = 50000 # how many unique words to use (i.e num rows in embedding vector)\nmaxlen = 100 # max number of words in a question to use","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3d68827dba2948ee2ef7dd4be10a1fa4dd629e4a"},"cell_type":"code","source":"## Tokenize the sentences\ntokenizer = Tokenizer(num_words=max_words)\ntokenizer.fit_on_texts(list(X_train))\n\n# The tokenizer will assign an integer value to each word in the dictionary\n# and then convert each string of words into a list of integer values\nX_train = tokenizer.texts_to_sequences(X_train)\nX_val = tokenizer.texts_to_sequences(X_val)\n\nword_index = tokenizer.word_index\nprint('The word index consists of {} unique tokens.'.format(len(word_index)))\n\n## Pad the sentences \nX_train = pad_sequences(X_train, maxlen=maxlen)\nX_val = pad_sequences(X_val, maxlen=maxlen)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"55461170dae0671d226504bd23aec3fa43f31cb3"},"cell_type":"code","source":"# Create the embedding dictionary from the word embedding file\nembedding_dict = {}\nfilename = os.path.join('../input/embeddings/', 'glove.840B.300d/glove.840B.300d.txt')\nwith open(filename) as f:\n    for line in f:\n        line = line.split()\n        token = line[0]\n        try:\n            coefs = np.asarray(line[1:], dtype='float32')\n            embedding_dict[token] = coefs\n        except:\n            pass\nprint('The embedding dictionary has {} items'.format(len(embedding_dict)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0fc9a47b228f234e5162d596091b72a69e8c5a7f"},"cell_type":"code","source":"# Create the embedding layer weight matrix\nembed_mat = np.zeros(shape=[max_words, embed_size])\nfor word, idx in word_index.items():\n    # Word index is ordered from most frequent to least frequent\n    # Ignore words that occur less frequently\n    if idx >= max_words: continue\n    vector = embedding_dict.get(word)\n    if vector is not None:\n        embed_mat[idx] = vector","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a6ca879360293235af25b97d7a3dc75aac866707"},"cell_type":"markdown","source":"# 3. Neural Network Training\nThis network configuration uses the pretrained GloVe embedding layer as the first layer of the network. This is followed by a bidirectional LSTM and then pooling layer. Finally, there are 2 dense layers leading to the final prediction. Feel free to modify network parameters and architecture. This is merely a starting point that provides adequate results. "},{"metadata":{"trusted":true,"_uuid":"1e4ec3f078dc5ca43669238813d5ebcadf6ab39d"},"cell_type":"code","source":"def create_lstm():\n    input = Input(shape=(maxlen,))\n    \n    # Embedding layer has fixed weights, so set 'trainable' to False\n    x = Embedding(max_words, embed_size, weights=[embed_mat], trainable=False)(input)\n    x = Bidirectional(CuDNNLSTM(64, return_sequences=True))(x)\n    x = GlobalMaxPool1D()(x)\n    x = Dense(16, activation=\"relu\")(x)\n    x = Dropout(0.2)(x)\n    x = Dense(1, activation=\"sigmoid\")(x)\n    model = Model(inputs=input, outputs=x)\n    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n    print(model.summary())\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"738691c29ce46a05923efc18e3fada0c308e611e"},"cell_type":"code","source":"# Create and train network\nlstm = create_lstm()\nlstm.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=3, batch_size=512)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"61aeb24e48c578594d26d0631a46aad49a215c4e"},"cell_type":"markdown","source":"# 4. Predictions\nThe remainder of this notebok will generate predictions from the test set and write them to a submission csv file for the kaggle competition."},{"metadata":{"trusted":true,"_uuid":"91e9da1a1d551932c34405c3a7ed185632f12260"},"cell_type":"code","source":"test_df = pd.read_csv('../input/test.csv')\nX_test = test_df['question_text'].values\n\n# Perform the same preprocessing as was done on the training set\nX_test = tokenizer.texts_to_sequences(X_test)\nX_test = pad_sequences(X_test, maxlen=maxlen)\n\n# Make predictions, ensure that predictions are in integer form\npreds = np.rint(lstm.predict([X_test], batch_size=1024, verbose=1)).astype('int')\ntest_df['prediction'] = preds","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"72f90edd0b8a0eb4c644e917f7c02068378ad161"},"cell_type":"markdown","source":"Let's examine a few examples of sincere predictions and insincere predictions. It appears that our network is making meaningful predictions."},{"metadata":{"trusted":true,"_uuid":"5b7bd3f417dfa01973f20f12fe967ec61ff3dc69"},"cell_type":"code","source":"n=5\nsin_sample = test_df.loc[test_df['prediction'] == 0]['question_text'].head(n)\nprint('Sincere Samples:')\nfor idx, row in enumerate(sin_sample):\n    print('{}'.format(idx+1), row)\n\nprint('\\n')\nprint('Insincere Samples:')\ninsin_sample = test_df.loc[test_df['prediction'] == 1]['question_text'].head(n)\nfor idx, row in enumerate(insin_sample):\n    print('{}'.format(idx+1), row)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"988fa1a03853289b65bb4c1845c5b7c333868cb8"},"cell_type":"code","source":"test_df = test_df.drop('question_text', axis=1)\ntest_df.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}