{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Quora Questions\n\nThe goal of this project is to identify insincere questions in a dataset of around 1 300 000 questions from Quora. The questions are about various topics and vary in length.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"## Imports\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nimport pandas as pd\nimport numpy as np\nimport pickle\nimport string\nimport random\n\nfrom keras.preprocessing.text import Tokenizer\nimport keras.preprocessing.sequence as sq\n\nfrom sklearn.model_selection import train_test_split\n\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.metrics import precision_score\nfrom sklearn.metrics import recall_score\nfrom sklearn.metrics import f1_score\nfrom sklearn.metrics import confusion_matrix\nimport seaborn as sn\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\nplt.rcParams['figure.figsize'] = 20, 10\nplt.rcParams.update({'font.size': 25})\n\nfrom IPython.display import clear_output","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1. Data exploration\n## [Header Name](#Header Name1)\n\nThere are over 1 300 000 sentences in the dataset. Let's import the labelled data (training)."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load data and embedding\npath_in = '../input/quora-insincere-questions-classification/'\n\ntrain = pd.read_csv(path_in + 'train.csv')\n\n## Limit data to prevent kernel from crashing in the from scratch part\ntrain = train[:50000]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> ### 1.1 Example sentences\n\nThe sentences are in `question_text` and the label is in `target`, where `1` is a positive (insincere) and `0`a negative. Here is an example sentence along with the label:"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Example sentence: \\\"{}\\\"\\nLabel: {}\".format(train['question_text'][8], train['target'][8]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see how many positives and negatives we have in the data:"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Ratio of positives to total: {:.2f}%\".format(len(train.loc[train['target'] == 1])*100 / len(train)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That means our model can achieve an accuracy of around 94% purley by guessing. This shows that the accuracy score is not a good metric to quanitify a model when having strongly unbalanced data and is known as the *class imbalance problem* [2]. "},{"metadata":{},"cell_type":"markdown","source":"### 1.2 Sentence structures and ratio\n\nTo get an idea about the data we are dealing with, we see how many characters are in sentence. Characters include both words and punctuation."},{"metadata":{"trusted":true},"cell_type":"code","source":"def getNumberofWordsFound():\n    ## Generate random\n    population = list(np.arange(0, len(train)))\n    population = random.sample(population, 100)\n    \n    number_of_words = []\n    \n    for sample in population:\n        sentence = train['question_text'].iloc[int(sample)]\n\n        ## Convert to lowercase and split sentence\n        sentence = sentence.lower()\n\n        ## Sepearate punctuation\n        chs = string.punctuation\n        for ch in chs:\n            idx = sentence.find(ch)\n\n            if idx != -1:\n                sentence = sentence.replace(ch, \" \" + ch)\n\n        sentence = sentence.split(' ') \n        number_of_words.append(len(sentence))\n        \n    return number_of_words\n\n\nnum_of_words = getNumberofWordsFound()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist(num_of_words, range=(0, 30), color=\"green\")\nplt.title(\"Number of characters per sentence\")\nplt.xlabel(\"# characters\")\nplt.ylabel(\"# sentences\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2. Data Preparation\nWe cannot just feed sentences to our network. Instead, we perform the following steps:\n\n* Generate a vocabulary with all words/characters in the data\n\n* Tokenize this vocabulary\n\n* Use tokens to express individual sentences as vectors\n\n* Pad/truncate the vectors so that all have the same length\n\n* Get weights from embedding\n\nThis steps can be done with libraries like in *Keras*. For illustrative purposes we first implement the steps fom scratch. Afterwards we use the *Keras* library to achieve the goal more efficiently."},{"metadata":{},"cell_type":"markdown","source":"### Define embedding loader\nBefore we tokenize the vocabulary we define an embedding loader, since it is used in 2.1 and 2.2.\n\nWe use the GloVe 300d 840B embedding. It was trained on 840 billion tokens of web data [1]. We want to get a tensor which includes the 300 dimensional weights for all words in our vocabulary. This `embedding_matrix` is used later by the embedding layer."},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_embedding(embedding_path, embedding_dim, word_index):\n    print('Loading word embeddings...')\n    \n    vocab_size = len(word_index)    \n    embedding_index = pickle.load(open(embedding_path, 'rb'))\n    words = embedding_index.keys()\n    coef = embedding_index.values()\n    \n    embedding_matrix = np.zeros((vocab_size+1, embedding_dim));\n    for word, i in word_index.items():\n        embedding_vector = embedding_index.get(word);\n        if embedding_vector is not None:\n            embedding_matrix[i] = embedding_vector;  \n            \n    print('Finished loading word embeddings. {:.0f} words loaded.'.format(len(embedding_matrix)))\n    return embedding_matrix","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.1 The hard way\n\nFirst we get a vocabulary from the data. Punctuation is isolated.\n\nLater we split the dataset into train, validation and testing. We also utilize the *PyTorch* Datalaoder functions to simplify batching."},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_vocabulary(data):\n    '''\n    Input:\n    data: series of sentences for which to build the vocabulary\n    \n    Output:\n    Dictionary of words and their frequency\n    '''\n    vocab = {}\n    for sentence_number in range(len(data)):\n        clear_output(wait=True)\n        print(\"Processing sentence {} / {}\".format(sentence_number+1 ,len(data)))\n        sentence = data[sentence_number]\n                \n        ## Sepearate punctuation\n        chs = string.punctuation\n        \n        for ch in chs:\n            idx = sentence.find(ch)\n\n            if idx != -1:\n                sentence = sentence.replace(ch, \" \" + ch)\n        \n        ## Split into words\n        sentence = sentence.split(' ')   \n        \n        for word in sentence:\n            word = word.lower()\n            try:\n                vocab[word] += 1\n            except KeyError:\n                    vocab[word] = 1\n        \n    print(\"Done\")\n        \n    return vocab\n\nvocab = make_vocabulary(train['question_text'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Tokenization\n\nFor the net to process the input sentences the sentences have to be tokenized. That means each character gets a token assigned. The sentences are then represented as lists of intergers called sequences.\n\nThe sequences get truncated if necessary and post-padded with $0$s."},{"metadata":{"trusted":true},"cell_type":"code","source":"## Take vocabulary\ndef tokenizer_fit(sentences, vocabulary):\n    '''\n    Inputs: \n    sentences: series object\n    vocabulary: dictionary with words as keys\n    \n    Output:\n    word_index: dictionary with words as keys and tokens as columns\n    '''\n    word_index = {}    \n    token = 1\n    \n    for i in range(len(sentences)):\n        \n        sentence = sentences[i]\n        \n        ## Sepearate punctuation\n        chs = string.punctuation\n        \n        for ch in chs:\n            idx = sentence.find(ch)\n\n            if idx != -1:\n                sentence = sentence.replace(ch, \" \" + ch)\n        \n        ## Split into characters\n        sentence = sentence.split(' ')   \n        \n        for word in sentence:            \n            if word not in word_index:\n                word_index[word] = token\n                token += 1\n    \n    return word_index","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def tokenizer_apply(word_index, sentences, max_dim):\n    '''\n    Inputs:\n    word_index: dictionary with tokens\n    sentences: series with sentences\n    \n    Output:\n    sequences: array with post-padded tokens\n    '''\n    ## Initialize array    \n    sequences = np.zeros((len(sentences), max_dim))\n    \n    for i in range(len(sentences)):\n        sentence = sentences[i]\n        \n        ## Sepearate punctuation\n        chs = string.punctuation\n        \n        for ch in chs:\n            idx = sentence.find(ch)\n\n            if idx != -1:\n                sentence = sentence.replace(ch, \" \" + ch)\n        \n        ## Split into characters\n        sentence = sentence.split(' ')\n        \n        ## Truncate at max_dim\n        sentence = sentence[0:max_dim]\n                        \n        for j in range(len(sentence)):\n            word = sentence[j]\n            \n\n            try:\n                sequences[i,j] = int(word_index[word])\n            \n            except KeyError:\n                sequences[i, j] = 0\n                \n    return sequences","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"max_len = 50\n\n# Build the tokenizer dictionary in the tokenizer class\nword_index = tokenizer_fit(train['question_text'], vocab) \n\n# Split train set into train and validation sets\ntrain, validation = train_test_split(train, test_size=0.2, shuffle = True)\ntrain.reset_index(inplace=True)\nvalidation.reset_index(inplace=True)\n\n# Tokenize the questions\ntrain_sequences = tokenizer_apply(word_index, train['question_text'], max_len)\nvalidation_sequences = tokenizer_apply(word_index, validation['question_text'], max_len)\n\n## Convert data to tensors\ntraining_dataset = torch.utils.data.TensorDataset(torch.LongTensor(np.array(train_sequences)), torch.FloatTensor(np.array(train['target'])))\nvalidation_dataset = torch.utils.data.TensorDataset(torch.LongTensor(np.array(validation_sequences)), torch.FloatTensor(np.array(validation['target'])))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Load embedding\nembedding_dim = 300\nembedding_path = \"../input/pickled-glove840b300d-for-10sec-loading/glove.840B.300d.pkl\"\n\n\nembedding_matrix = load_embedding(embedding_path, embedding_dim, word_index)\nweights = torch.FloatTensor(embedding_matrix)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.2 The easy way - Keras tokenizer\n\nAfter defining the functions ourselves, we now use the *Keras* library.\n\nFirst we tokenize all words in the data using the `Tokenizer` from the Keras library. Note that we filter out the characters given to the `filters` attribute.\n\nNext, we split the data into training and validation data.\n\nFinally, using the tokens defined in the first step and the embedding, we get the weight matrix which we can use in the net."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Build the tokenizer dictionary in the tokenizer class\ntokenizer = Tokenizer(filters='!\"#$%&()*+,-./:;<=>?@[\\\\]^_`{|}~\\t\\n')\ntokenizer.fit_on_texts(train['question_text'])\n\n# Split train set into train and validation sets\ntrain, validation = train_test_split(train, test_size=0.1, shuffle = True)\n#train.reset_index(inplace=True)\n#validation.reset_index(inplace=True)\n\n# Tokenize the questions\ntrain_sequences = tokenizer.texts_to_sequences(train['question_text'])\nvalidation_sequences = tokenizer.texts_to_sequences(validation['question_text'])\n\n# Extract the dictionary witht the tokens\nword_index = tokenizer.word_index","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Pad sequences with 0s so that each sequence is of the same length respresented by the max_length\nmax_length = 50\npadding_type = 'pre'\ntrunc_type = 'pre'\n\ntrain_sequences_padded = sq.pad_sequences(train_sequences, maxlen=max_length, padding=padding_type, truncating=trunc_type)\nvalidation_sequences_padded = sq.pad_sequences(validation_sequences, maxlen=max_length, padding=padding_type, truncating=trunc_type)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Convert data to tensors\ntraining_dataset = torch.utils.data.TensorDataset(torch.LongTensor(np.array(train_sequences_padded)), torch.FloatTensor(np.array(train['target'])))\nvalidation_dataset = torch.utils.data.TensorDataset(torch.LongTensor(np.array(validation_sequences_padded)), torch.FloatTensor(np.array(validation['target'])))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Load embedding\nembedding_dim = 300\nembedding_path = \"../input/pickled-glove840b300d-for-10sec-loading/glove.840B.300d.pkl\"\n\n\nembedding_matrix = load_embedding(embedding_path, embedding_dim, word_index)\nweights = torch.FloatTensor(embedding_matrix)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3. Prediction\n\nUsing the weight matrix which is generated in 2. we now train a net."},{"metadata":{},"cell_type":"markdown","source":"### 3.1 Net\nThe net has an embedding layer which feeds into the LSTM layer. The LSTM is connected to linear layers.\n\nFor the LSTM layer we want a many-to-one architecture. That means we want to feed in the sequence (whole sentence) and get one output.\n\nThe LSTM layer has the outputs `lstm_out, (h_n, c_n)`. To get the many-to-one output, you can either take the last element of the sequence `lstm_out`, or `h_n` which is the same. For more info see the documentation [https://pytorch.org/docs/stable/nn.html#lstm].\n\nWe also use dropout to reduce overfitting, as well as initialization layers."},{"metadata":{"trusted":true},"cell_type":"code","source":"class Net(nn.Module):\n    def __init__(self, seq_length, hidden_layer):\n        super(Net, self).__init__()\n        \n        ## Embedding layer\n        self.embd = nn.Embedding.from_pretrained(weights)\n        \n        ## LSTM\n        self.lstm = nn.LSTM(input_size=len(weights[0,:]), hidden_size=hidden_layer, batch_first=True)\n        \n        ## Linear layers\n        self.fc1 = nn.Linear(hidden_layer, hidden_layer)\n        self.dropout1 = nn.Dropout(p=0.4)\n        self.fc2 = nn.Linear(hidden_layer, 1)\n        self.tanh = nn.Tanh()\n        self.out = nn.Sigmoid()\n        \n        self.initialize()\n        \n    def initialize(self):\n        nn.init.xavier_uniform_(self.fc1.weight.data)\n        self.fc1.bias.data.zero_()\n      \n\n    def forward(self, x):\n        embd_out = self.embd(x)       \n        lstm_out, (h_out, _) = self.lstm(embd_out)\n        x = self.tanh(self.dropout1(self.fc1(h_out)))\n        x = self.fc2(x)\n        x = self.out(x)\n        \n        return x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.2 Train the net"},{"metadata":{"trusted":true},"cell_type":"code","source":"### Function for training\nepochs = 10\nlearning_rate = 0.001\n\nseq_length = 50\nhidden_layer_1 = 300\n\nbatch_size = 1000\n\n\n## Define dataloader\ntrain_loader = torch.utils.data.DataLoader(dataset=training_dataset, batch_size=batch_size, shuffle=True)\nvalidation_loader = torch.utils.data.DataLoader(dataset=validation_dataset, batch_size=len(validation_dataset), shuffle=False)\n\n## Initialize net and define loss function and optimizer\nmodel = Net(seq_length, hidden_layer_1)\ncriterion = nn.BCELoss()\noptimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n\n\n##### Iterate through the data\nloss_acc = {'Loss': [], 'Train accuracy': [], 'Validation accuracy':[]}\n\nfor epoch in range(epochs):\n    print(\"Epoch {} / {}\".format(epoch+1, epochs))\n\n    train_correct = 0    \n    model.train()\n    \n    for x, y in train_loader:\n        # Set gradients to zero\n        optimizer.zero_grad()\n        \n        # Forward pass\n        y_hat = model(x)\n        \n        # Evaluate output / compute loss\n        y_hat = y_hat.view(-1)        \n        loss = criterion(y_hat, y)       \n        \n        # Backward pass / optimize\n        loss.backward()\n        \n        # Update weights\n        optimizer.step()\n                \n        ## Evaluate train result\n        y_hat = np.where(y_hat.detach().numpy() > 0.5, 1, 0)\n        train_correct += (y_hat == y.numpy()).sum()\n    \n    train_acc = train_correct / len(train_loader.dataset)\n    loss_acc['Loss'].append(loss.item())\n      \n        \n    # Get validation accuracy\n    val_correct = 0\n    with torch.no_grad():\n        for x, y in validation_loader:\n            y_hat = model(x)\n            y_hat = y_hat.view(-1)\n\n            ## Evaluate validation result\n            y_hat = np.where(y_hat.numpy() > 0.5, 1, 0)\n            val_correct += (y_hat == y.numpy()).sum()\n    \n    val_acc = val_correct / len(validation_loader.dataset)\n    \n    \n    # Append scores to dictionary\n    loss_acc['Train accuracy'].append(train_acc)\n    loss_acc['Validation accuracy'].append(val_acc)\n    \n    print(\"Loss: {:.4f}\".format(loss_acc['Loss'][-1]))\n    print(\"Training accuracy: {:.4f} | Validation accuracy: {:.4f}\".format(loss_acc['Train accuracy'][-1], loss_acc['Validation accuracy'][-1]))\n\n## Save the model\n#torch.save(model.state_dict(), './quora_questions_classifier.pt')\n    \nprint(\"\\nTraining completed!\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Plots\nfig = plt.figure(1)\nplt.plot(loss_acc['Loss'], color=\"red\")\nplt.title(\"Loss\")\nplt.xlabel(\"Iteration\")\nplt.ylabel(\"Loss [-]\")\n\nfig = plt.figure(2)\nplt.plot(loss_acc['Train accuracy'], \"o-\", color=\"black\", label=\"Train\")\nplt.plot(loss_acc['Validation accuracy'], \"^-\",color=\"blue\", label=\"Validation\")\nplt.title(\"Accuracy\")\nplt.xlabel(\"Epoch\")\nplt.legend()\nplt.ylabel(\"Accuracy [%]\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4. Evaluation\n\nFinally we evaluate the performance of our model in more detail. Note that we use the validation data again for this. This is okay in this case, since the model has not been tuned based on this data, even though it has been used during training. However, a cleaner approach would be to have training, validation and testing data.\n\nLet's compute a few metrics. We begin with the accuracy:"},{"metadata":{"trusted":true},"cell_type":"code","source":"## Use trained model for validation\nmodel.eval()\n\nfor x, y in validation_loader:\n    validation_data = x\n    validation_labels = y\n\npred = model(validation_data)\npred = pred.squeeze()\npredicted_labels = np.where(pred > 0.5, 1, 0)\n\naccuracy_score(np.array(validation_labels), predicted_labels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As mentioned earlier, in unbalanced data other metrics can be more useful to assess the model's performance. Let's look at:\n\n*precision* $= \\frac{t_{p}}{t_{p} + f_{p}}$\n\n\n*recall* $= \\frac{t_{p}}{t_{p} + f_{n}}$\n\n\n*F1-Score* $= 2\\cdot \\frac{precision \\cdot recall}{precision + recall}$"},{"metadata":{"trusted":true},"cell_type":"code","source":"def Compute_Metrics():\n    precision = precision_score(validation_labels, predicted_labels)\n    print(\"Precision: {:.4f}\".format(precision))\n    \n    recall = recall_score(validation_labels, predicted_labels)\n    print(\"Recall: {:.4f}\".format(recall))\n    \n    f1 = f1_score(validation_labels, predicted_labels)\n    print(\"F1 Score: {:.4f}\".format(f1))\n    \n    return precision, recall, f1\n\nprecision, recall, f1 = Compute_Metrics()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Confusion matrix\ndef Confusion_Matrix(con_mat):\n    \n    plt.title(\"Confusion Matrix\", fontsize=50)\n    sn.set(font_scale=2.5)\n    sn.heatmap(con_mat, annot=True, fmt='g', annot_kws={\"size\":30}, xticklabels=[\"Negatives\", \"Positives\"], yticklabels=[\"Negatives\", \"Positives\"], \n               cmap=\"Blues\")\n    plt.xlabel(\"Prediction\", fontsize=35)\n    plt.ylabel(\"Truth\", fontsize=35)\n\n    \ncon_mat = confusion_matrix(validation_labels, predicted_labels)\nConfusion_Matrix(con_mat)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Conclusion\n\nUsing only a fraction of the available data a precision of $60$% has been achived. That means from the flagged posts the majority really is considered as divisive. At the same time, the system misses a lot of positives (low recall).\n\nIn order to improve the performance the following steps are recommended: train on all data, optimize data pre-processing, use a larger embedding, optimize hyper-parameters, optimize net."},{"metadata":{},"cell_type":"markdown","source":"## References\n[1] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. *GloVe: Global Vectors for Word Representation*. 2014.\n\n[2] Han, Kamber, Pei. *Data Mining: Concepts and Techniques*. 2011"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.5"}},"nbformat":4,"nbformat_minor":1}