{"cells":[{"metadata":{"_uuid":"5263dc8b35545cfcf180375f42d3e8fe4659674a"},"cell_type":"markdown","source":"# Challenge Kaggle - Quora Insincere Questions Classification - Advanced version"},{"metadata":{"_uuid":"0ac0d08e2ecac89d9d6513fec8583522f405f016"},"cell_type":"markdown","source":"## Introduction\nThis notebook presents a mode advanced solution for the Kaggle Challenge: [Quora Insincere Questions Classification](https://www.kaggle.com/c/quora-insincere-questions-classification).\n\nThe objective here is to implement and improved solution of the notebook [Simple 1-layer GRU](https://www.kaggle.com/andrelmfarias/simple-1-layer-gru) in order to get a better score.\n\n### Improvements and key factors\n\nIn order to obtain a better model, I implemented some ideas from other kernels / solutions and mixed them with some ideas I had. The main factors of my solution are:\n1. Use of pre-trained embeddings. It saves much training time, as we do not need to train new embeddings, and it allows us to use intrisec information contained in these embeddings that come from solid models. At the end I decided to use only Glove and Paragram because the vocabularies of the other two models contain less than 50% of the unique words presents in the training and test datasets.\n2. Use of statistical features such as length of sentence, number of capital letters and number of special characters. This can be usefull because several insincere question presents a special pattern of characters (eg. use of * for bad words). The model use dense layers for these features and then merge with the output from the RNN in order to compute probability of positive target.\n3. Use of Spacy parser and tokenizer\n4. Lemmatization and Stemming of words not found in the embeddings vocabulary in order to increase the proportion of known words\n5. Adaptative length of sequences fed to the model. I used the max length per batch, instead of overall max length, in order to speed up the training.\n6. Use of bi-directional GRU. It can capture more meaning and context for the sentences than a simple GRU.\n7. Application of self-attention on the outputs of GRU layer. The self-attention mechanism allows the model to capture better the whole context of the sentence as it send as output a hidden state obtained by a weighted average of all the hidden states generated by the GRU on the sentence.\n8. Find best threshold for label decision by cross-validation\n\nWith these improvements, my solution is able to achieve **68%** of F1-score, which is a huge improvement in comparison to the simpler solution (with **60%**).\n\n### Other potential improvements\n\nDue to the time and hardware constraints, I was not able to perform better cross-validation using 5-fold or 10-fold CV. Indeed, by using only train_test_split as CV technique, our model does not generalize that well as we tend to overfit on the validation set.\n\nAnother potential improvement would be to train different models with different architectures and/or using the different embeddings, blending the results and the end (Ensemble). It would yield more stable results, as we reduce variance by taking the (weighted) mean of different predictions, and it would probably give us improved results.\n\nTrying to do spelling correction might improve the performance as well because it would probabily increase the quantity of known word vectors and yield better results overall.\n\nOne could also try to train word embeddings using the avalaible corpus (training and test set) in orther to have vector representations for the whole vocabulary."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"## Import of useful libraries"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport re\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import f1_score\n\nimport torch\nfrom torch import nn\nfrom torch import optim\nimport torch.nn.functional as F\nfrom torch.utils.data import TensorDataset, DataLoader\n\nimport spacy\nfrom gensim.models import KeyedVectors\nfrom nltk.stem import PorterStemmer, SnowballStemmer\nfrom nltk.stem.lancaster import LancasterStemmer\n\nimport time\nimport gc\nfrom tqdm import tqdm","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"67b9a011d765cca6d98488e8ca631c2409d5ccc6"},"cell_type":"markdown","source":"## Loading data"},{"metadata":{"trusted":true,"_uuid":"a35b08d905732c6a28a2ca59166011561a0787a3"},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/train.csv\")\ntest_df = pd.read_csv(\"../input/test.csv\")\n\nlabels = np.array(train_df.target, dtype=int)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eb5c2a168243fc1bf342e558e0928bee44128ac0"},"cell_type":"markdown","source":"## Preprocessing"},{"metadata":{"_uuid":"9f73cf3e45a0e93aafd91bd5881e0d689ac45239"},"cell_type":"markdown","source":"### Statistical features"},{"metadata":{"trusted":true,"_uuid":"ce72db7cce107b1f381411ad548f8cf7300a46d6"},"cell_type":"code","source":"# Functions to extract statistical features\n\ndef n_upper(sentence):\n    return len(re.findall(r'[A-Z]',sentence))\n\ndef n_unique_words(sentence):\n    return len(set(sentence.split()))\n\ndef n_question_mark(sentence):\n    return len(re.findall(r'[?]',sentence))\n\ndef n_exclamation_mark(sentence):\n    return len(re.findall(r'[!]',sentence))\n\ndef n_asterisk(sentence):\n    return len(re.findall(r'[*]',sentence))\n\ndef n_parentheses(sentence):\n    return len(re.findall(r'[()]',sentence))\n\ndef n_brackets(sentence):\n    return len(re.findall(r'[\\[\\]]',sentence))\n\ndef n_braces(sentence):\n    return len(re.findall(r'[{}]',sentence))\n\ndef n_quotes(sentence):\n    return len(re.findall(r'[\"]',sentence))\n\ndef n_ampersand(sentence):\n    return len(re.findall(r'[&]',sentence))\n\ndef n_dash(sentence):\n    return len(re.findall(r'[-]',sentence))\n\nn_stats = 11 # Number of statistical features excluding sequence length\n\ndef get_stat(questions_list):\n    ''' \n    Function that builds matrix of statistical features\n    '''\n    stat_feat = np.zeros((len(questions_list), n_stats), dtype=int)\n    for i,question in tqdm(enumerate(questions_list)):\n        stat_feat[i,0] = n_upper(question)\n        stat_feat[i,1] = n_unique_words(question)\n        stat_feat[i,2] = n_question_mark(question)\n        stat_feat[i,3] = n_exclamation_mark(question)\n        stat_feat[i,4] = n_asterisk(question)\n        stat_feat[i,5] = n_parentheses(question)\n        stat_feat[i,6] = n_brackets(question)\n        stat_feat[i,7] = n_braces(question)\n        stat_feat[i,8] = n_quotes(question)\n        stat_feat[i,9] = n_ampersand(question)\n        stat_feat[i,10] = n_dash(question)\n    \n    return stat_feat    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1729841da5b42f1fd962e69116c0c39629782394"},"cell_type":"code","source":"train_stat = get_stat(train_df.question_text)\ntest_stat = get_stat(test_df.question_text)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c61fefab36bd274a693b0ff6ba99a39ee7f10dde"},"cell_type":"markdown","source":"### Tokenization"},{"metadata":{"trusted":true,"_uuid":"f09fdf0e79f48644e5ac30f6b4bbb0f85a7ba8fc"},"cell_type":"code","source":"# Lowering all the text and storing the lists\ntrain_list = list(train_df.question_text.apply(lambda s: s.lower()))\ntest_list = list(test_df.question_text.apply(lambda s: s.lower()))\n\n# Getting all text in both samples\ntrain_text = ' '.join(train_list)\ntest_text = ' '.join(test_list)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1f140cfeb519cf552f175e7f9fae0b3eb6af01f4"},"cell_type":"code","source":"# Using spacy parser and tokenizer\nnlp = spacy.load(\"en\", disable=['tagger','parser','ner','textcat'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4e80221925960c7c553e471bca94f1bd8d1e25a0"},"cell_type":"code","source":"# Creating the vocabulary and tokenizing datasets\nvocab = {}\nlemma_vocab = {} # lemmatizing vocabulary\nword_idx = 1 # start from 1 as we use 0 for padding\n\ntrain_tokens = []\nfor doc in tqdm(nlp.pipe(train_list)):\n    curr_tokens = []\n    for token in doc:\n        if token.text not in vocab:\n            vocab[token.text] = word_idx\n            lemma_vocab[token.text] = token.lemma_\n            word_idx += 1\n        curr_tokens.append(vocab[token.text])\n    train_tokens.append(np.array(curr_tokens, dtype=int))\n\ntest_tokens = []\nfor doc in tqdm(nlp.pipe(test_list)):\n    curr_tokens = []\n    for token in doc:\n        if token.text not in vocab:\n            vocab[token.text] = word_idx\n            lemma_vocab[token.text] = token.lemma_\n            word_idx += 1\n        curr_tokens.append(vocab[token.text])\n    test_tokens.append(np.array(curr_tokens, dtype=int))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"96055b80cbc00c2ecf21870b8600057ff49112d0"},"cell_type":"markdown","source":"#### Padding"},{"metadata":{"trusted":true,"_uuid":"97ca70df86658c12391f61bba1f42c84df994abc"},"cell_type":"code","source":"def pad(questions, seq_length):\n    '''\n    This function pad the questions fed as list of tokens with 0 at left\n    and returns a numpy array, and the length of the sentence at the position 0 of the array.\n    This length will be useful in order to change the sequences length for each batch during \n    training and to be used as statistical feature.\n    '''\n    \n    features = np.zeros((len(questions), seq_length+1), dtype=int)\n    for i, sentence in enumerate(questions):\n        if len(sentence)==0: # dealing with empty sentences\n            continue\n        features[i, 0] = len(sentence)\n        features[i, -len(sentence):] = sentence\n    \n    return features","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"34d9da253b6345cc53e508b4bcf0b40ed2e54342"},"cell_type":"code","source":"# We are going to use a sequence length that does not truncates any of the samples\nseq_length = max(max(map(len, train_tokens)), max(map(len, test_tokens))) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c31cf8d3b89aa75b67c119d12eefc83ea52e2d0a"},"cell_type":"code","source":"# Applying padding\ntrain_tokens = pad(train_tokens, seq_length)\ntest_tokens = pad(test_tokens, seq_length)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b5af498c999bba4d9b9e0f94023f228a3dfde73e"},"cell_type":"markdown","source":"### Embeddings"},{"metadata":{"trusted":true,"_uuid":"8d56d26d377f6c67598c7d189b55964039e5dd2c"},"cell_type":"code","source":"def get_embeddings(file):\n    embeddings = {}\n    with open(file, encoding=\"utf8\", errors='ignore') as f:\n        for line in tqdm(f):\n            line_list = line.split(\" \")\n            if len(line_list) > 100:\n                embeddings[line_list[0]] = np.array(line_list[1:], dtype='float32')\n    return embeddings\n\ndef get_embeddings_matrix(vocab, lemma_vocab, embeddings, keyedVector=False):\n    \n    # Stemmers\n    ps = PorterStemmer()\n    lc = LancasterStemmer()\n    sb = SnowballStemmer(\"english\")\n    \n    n_words = len(vocab)\n    if keyedVector:\n        emb_size = embeddings.vector_size\n    else:\n        emb_size = next(iter(embeddings.values())).shape[0]\n        \n    # If word2vec, convert it to dict for simplicity and compatibility with the others vectors\n    if keyedVector:\n        emb_dict = {}\n        for word in vocab:\n            try:\n                emb_dict[word] = embeddings.get_vector(word)\n            except:\n                continue\n        embeddings = emb_dict\n    \n    embedding_matrix = np.zeros((n_words+1, emb_size), dtype=np.float32)\n    unknown_vec = np.zeros((emb_size,), dtype=np.float32) - 1 # (-1, -1, ..., -1)\n    unknown_words = 0  # unknown words counter  \n    for word in tqdm(vocab):\n        emb_vec = embeddings.get(word)\n        if emb_vec is not None:\n            embedding_matrix[vocab[word]] = emb_vec\n            continue\n            \n        # Lemmatizing\n        emb_vec = embeddings.get(lemma_vocab[word])\n        if emb_vec is not None:\n            embedding_matrix[vocab[word]] = emb_vec\n            continue\n            \n        # Stemming\n        emb_vec = embeddings.get(ps.stem(word))\n        if emb_vec is not None:\n            embedding_matrix[vocab[word]] = emb_vec\n            continue\n        emb_vec = embeddings.get(lc.stem(word))\n        if emb_vec is not None:\n            embedding_matrix[vocab[word]] = emb_vec\n            continue    \n        emb_vec = embeddings.get(sb.stem(word))\n        if emb_vec is not None:\n            embedding_matrix[vocab[word]] = emb_vec\n            continue\n        \n        # If word vector not found\n        embedding_matrix[vocab[word]] = unknown_vec\n        unknown_words += 1\n        \n    print('% known words: {:.2%}'.format(1 - unknown_words/n_words))\n            \n    return embedding_matrix","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5b68aa8cf392d264da175e29f7bfada8f94958f0"},"cell_type":"markdown","source":"#### Glove"},{"metadata":{"trusted":true,"_uuid":"5ab5d88a46d57a746a55c63a32b9eb4609588680"},"cell_type":"code","source":"# Getting embeddings from file\nglove_file = '../input/embeddings/glove.840B.300d/glove.840B.300d.txt'\nglove_emb = get_embeddings(glove_file)\n\nglove_emb_matrix = get_embeddings_matrix(vocab, lemma_vocab, glove_emb)\n\n# Cleaning up memory\ndel glove_emb\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3034b3c4de038f6a8be690ef8cf6930db8164707"},"cell_type":"markdown","source":"#### Fasttext"},{"metadata":{"trusted":true,"_uuid":"06ead262c1c6c4f4753c176bbad6d3f1d889f28d"},"cell_type":"code","source":"# Getting embeddings from file\nfasttext_file = '../input/embeddings/wiki-news-300d-1M/wiki-news-300d-1M.vec'\nfasttext_emb = get_embeddings(fasttext_file)\n\n# Building embedding matrix\nfasttext_emb_matrix = get_embeddings_matrix(vocab, lemma_vocab, fasttext_emb)\n\n# Cleaning up memory\ndel fasttext_emb\ngc.collect()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"70ff05358cafa394de4d56e741c7256e9745dd28"},"cell_type":"markdown","source":"#### Word2vec"},{"metadata":{"trusted":true,"_uuid":"1171657592b8791d8877d50a4ff34ea0482ced73"},"cell_type":"code","source":"# Getting embeddings from file\nword2vec_file = '../input/embeddings/GoogleNews-vectors-negative300/GoogleNews-vectors-negative300.bin'\nword2vec_emb = KeyedVectors.load_word2vec_format(word2vec_file, binary=True)\n\n# Building embedding matrix\nword2vec_emb_matrix = get_embeddings_matrix(vocab, lemma_vocab, word2vec_emb, keyedVector=True)\n\n# Cleaning up memory\ndel word2vec_emb\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"35beaf6deb63eb4b50d035f4261889db34508e3c"},"cell_type":"markdown","source":"#### Paragram"},{"metadata":{"trusted":true,"_uuid":"a04732273555a278f13093c2d69430493f2dbd4f"},"cell_type":"code","source":"# Getting embeddings from file\nparagram_file = '../input/embeddings/paragram_300_sl999/paragram_300_sl999.txt'\nparagram_emb = get_embeddings(paragram_file)\n\n# Building embedding matrix\nparagram_emb_matrix = get_embeddings_matrix(vocab, lemma_vocab, paragram_emb)\n\n# Cleaning up memory\ndel paragram_emb\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"22ce3868fe9f3044aa63527eaba400398656213f"},"cell_type":"markdown","source":"#### Concatenating the embedding matrices"},{"metadata":{"_uuid":"cb3c2e6e85a8ce190fba7cb8163721ceedd54ac1"},"cell_type":"markdown","source":"As we can see above, only Glove and Paragram present and acceptable proportion of known words. Therefore, in order to have as many embeddings as possible, I will use only these both as embeddings. I decided to concatenate the embeddings, one can however implement an average or sum of both embeddings."},{"metadata":{"trusted":true,"_uuid":"4fb7f08aec6a105b5d0c231a7ca5b3bc0f3b55b8"},"cell_type":"code","source":"emb_matrix = np.concatenate((glove_emb_matrix, \n                             paragram_emb_matrix), axis=1)\n\ndel glove_emb_matrix, fasttext_emb_matrix, word2vec_emb_matrix, paragram_emb_matrix\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e34e36241d71540ce63e16794c0af88d46f69428"},"cell_type":"markdown","source":"### Splitting training data and creating dataloaders"},{"metadata":{"trusted":true,"_uuid":"d0bbf0a0b46f3d3d860fee7c3ea1d5a02698d0d3"},"cell_type":"code","source":"# Concatenating tokens and statistical features\ntrain_feat = np.concatenate((train_stat, train_tokens), axis=1)\ntest_feat = np.concatenate((test_stat, test_tokens), axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3c2903f342be15d7b857d7507aa78529d33649ba"},"cell_type":"code","source":"x_train, x_val, label_train, label_val = train_test_split(train_feat, labels, test_size=0.1, random_state=0) \n\n# Create Tensor datasets\ntrain_data = TensorDataset(torch.from_numpy(x_train), torch.from_numpy(label_train))\nvalid_data = TensorDataset(torch.from_numpy(x_val), torch.from_numpy(label_val))\ntest_data = TensorDataset(torch.from_numpy(test_feat))\n\n# Create Dataloaders\nbatch_size = 64\n\ntrain_loader = DataLoader(train_data, shuffle=True, batch_size=batch_size)\nvalid_loader = DataLoader(valid_data, shuffle=True, batch_size=batch_size)\ntest_loader = DataLoader(test_data, shuffle=False, batch_size=batch_size)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a661b720b66f98b477fd6376e8f3e96effafb06b"},"cell_type":"markdown","source":"## The model"},{"metadata":{"trusted":true,"_uuid":"697f53b0f8a61c2fad652e6bcf54d34440f1eb5a"},"cell_type":"markdown","source":"### Training on GPU or CPU"},{"metadata":{"trusted":true,"_uuid":"ed008d68aed01e1af19feae7adcabd31b08ee2dd"},"cell_type":"code","source":"# Checking if GPU is available\ntrain_on_gpu=torch.cuda.is_available()\n\nif train_on_gpu:\n    print('Training on GPU.')\nelse:\n    print('No GPU available, training on CPU.')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0006cfd77debe006cff395b83955225d59b1d380"},"cell_type":"markdown","source":"### The model¶"},{"metadata":{"trusted":true,"_uuid":"f4fe834a50dd60fd006d1530e68c520a3dbb3aa8"},"cell_type":"code","source":"def init_emb_layer(self, embedding_matrix):\n    '''\n    Function to help in the creation of the embedding layer\n    '''\n    embedding_matrix = torch.tensor(embedding_matrix, dtype=torch.float32)\n    num_emb, emb_size = embedding_matrix.size()\n    emb_layer = nn.Embedding.from_pretrained(embedding_matrix)\n    return emb_layer\n\nclass SelfAttention(nn.Module):\n    '''\n    Class that implements a Self-Attention module that will be applied on the outputs of the GRU layer\n    '''\n    \n    def __init__(self, attention_size, batch_first=False, non_linearity=\"tanh\"):\n        super(SelfAttention, self).__init__()\n\n        self.batch_first = batch_first\n        self.attention_weights = nn.Parameter(torch.FloatTensor(attention_size))\n        self.softmax = nn.Softmax(dim=-1)\n\n        if non_linearity == \"relu\":\n            self.non_linearity = nn.ReLU()\n        else:\n            self.non_linearity = nn.Tanh()\n\n        nn.init.uniform(self.attention_weights.data, -0.005, 0.005)\n\n    def get_mask(self, attentions, lengths):\n        \"\"\"\n        Construct mask for padded itemsteps, based on lengths\n        \"\"\"\n        max_len = max(lengths.data)\n        mask = torch.autograd.Variable(torch.ones(attentions.size())).detach()\n\n        if attentions.data.is_cuda:\n            mask = mask.cuda()\n\n        for i, l in enumerate(lengths.data):  # skip the first sentence\n            if l < max_len:\n                mask[i, :-l] = 0\n        return mask\n\n    def forward(self, inputs, lengths):\n\n        # STEP 1 - perform dot product of the attention vector and each hidden state\n        \n        # inputs is a 3D Tensor: batch, len, hidden_size\n        # scores is a 2D Tensor: batch, len\n        scores = self.non_linearity(inputs.matmul(self.attention_weights))\n        scores = self.softmax(scores)\n\n        # Step 2 - Masking\n\n        # construct a mask, based on the sentence lengths\n        mask = self.get_mask(scores, lengths)\n\n        # apply the mask - zero out masked timesteps\n        masked_scores = scores * mask\n\n        # re-normalize the masked scores\n        _sums = masked_scores.sum(-1, keepdim=True)  # sums per row\n        scores = masked_scores.div(_sums)  # divide by row sum\n\n        # Step 3 - Weighted sum of hidden states, by the attention scores\n\n        # multiply each hidden state with the attention weights\n        weighted = torch.mul(inputs, scores.unsqueeze(-1).expand_as(inputs))\n\n        # sum the hidden states\n        representations = weighted.sum(1).squeeze()\n\n        return representations\n\nclass Quora_model(nn.Module):\n    def __init__(self, hidden_layer_dim, embedding_matrix, hidden_dim, gru_layers, stat_layers, drop_prob=0.5):\n        \"\"\"\n        Quora model with bi-directional GRU and self-attention merged with Dense layers\n        \"\"\"\n        super(Quora_model, self).__init__()\n        \n        self.hidden_layer_dim = hidden_layer_dim\n        self.gru_layers = gru_layers\n        self.emb_dim = embedding_matrix.shape[1]\n        self.hidden_dim = hidden_dim   \n        self.stat_layers = stat_layers\n        \n        # Dense layers for statistical features\n        stat_in_dim = n_stats + 1 # including sequence length\n        modules = []\n        for out_dim in self.stat_layers:\n            modules.append(nn.Linear(stat_in_dim, out_dim))\n            modules.append(nn.ReLU())\n            stat_in_dim = out_dim\n        \n        self.stat_dense = nn.Sequential(*modules)\n        \n        # Embedding layer\n        self.embedding = init_emb_layer(self, embedding_matrix)\n        \n        # Bidirectional GRU layer\n        self.gru = nn.GRU(self.emb_dim, self.hidden_dim, self.gru_layers, \n                          batch_first=True, bidirectional=True, dropout = drop_prob)\n        \n        # Attention layer\n        self.attention = SelfAttention(self.hidden_dim*2, batch_first=True)\n        \n        # Final dense --- merger of text and statistical features\n        self.final_dense = nn.Sequential(\n            nn.Dropout(p=drop_prob),\n            nn.Linear(self.hidden_dim*2 + out_dim, self.hidden_layer_dim),\n            nn.ReLU(),\n            nn.Dropout(p=drop_prob),\n            nn.Linear(self.hidden_layer_dim, 1),\n            nn.Sigmoid()\n        )\n        \n    def forward(self, x, hidden):\n        \n        batch_size, _ = x.size()\n        \n        # Deal with cases were the current batch_size is different from general batch_size\n        # It occurrs at the end of iteration with the Dataloaders\n        if hidden.size(1) != batch_size:\n            hidden = hidden[:, :batch_size, :].contiguous()\n            \n        # Lengths of sequences\n        lengths = x[:,n_stats].cpu().numpy().astype(int)\n        \n        # Adapting seq_len for the current batch\n        seq_len = max(lengths) \n        x_text = x[:, -seq_len:] # input to gru layer\n        x_stat = x[:, :n_stats+1].type(torch.FloatTensor) # include sequence length as statistical feature\n        if train_on_gpu:\n            x_stat = x_stat.cuda()\n        \n        # Apply embedding\n        x_text = self.embedding(x_text)\n        \n        # GRU Layer\n        out_gru, _ = self.gru(x_text, hidden)\n        \n        # Apply attention\n        out_att = self.attention(out_gru, lengths)\n        \n        # Dense layer for statistical features\n        out_stat = self.stat_dense(x_stat)\n        \n        # Concatenate output of the RNN with output from statistical features\n        out = torch.cat((out_att, out_stat), dim=1)\n        \n        # Final dense_layer\n        out = self.final_dense(out)\n        \n        return out\n    \n    def init_hidden(self, batch_size):\n        ''' Initializes hidden state '''\n        # Create a new tensor with sizes n_layers x batch_size x hidden_dim,\n        # initialized to zero\n        \n        weight = next(self.parameters()).data\n        \n        if train_on_gpu:\n            hidden = weight.new(self.gru_layers*2, batch_size, self.hidden_dim).zero_().cuda()\n            \n        else:\n            hidden = weight.new(self.gru_layers*2, batch_size, self.hidden_dim).zero_()\n        \n        return hidden","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b7f490caa1f6c4c0ec343071e5202452d9b7f54b"},"cell_type":"markdown","source":"### Hyperparameters and model initiation:"},{"metadata":{"trusted":true,"_uuid":"3fbc4616ae203dde266fd5c8e9592567b28e812e"},"cell_type":"code","source":"hidden_dim = 256\ngru_layers = 1\ndropout = 0.1\nstat_layers_dim = [16, 8] \nhidden_layer_dim = 64\n\n# Initiating the model\nmodel = Quora_model(hidden_layer_dim, emb_matrix, hidden_dim, gru_layers, stat_layers_dim, dropout)\nmodel","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"88bc2087d24123d958fc2cbfcd8d905e7ce2eee5"},"cell_type":"markdown","source":"## Training"},{"metadata":{"trusted":true,"_uuid":"2649b99df6d6dbcef1f6118e67435aa568e71339"},"cell_type":"code","source":"# Training parameters\n\nepochs = 4\n\nprint_every = 1000\nearly_stop = 20\nclip = 5 # gradient clipping - to avoid gradient explosion\n\nlr=0.001\n\n# Defining loss and optimization functions\n\ncriterion = nn.BCELoss()\noptimizer = torch.optim.Adam(model.parameters(), lr=lr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"605fe8b5b005dd16a5e88427adcc70ded5cc0425"},"cell_type":"code","source":"def train_model(model, train_loader, valid_loader, batch_size, epochs, \n                optimizer, criterion, clip, print_every, early_stop):\n    \n    # move model to GPU, if available\n    if(train_on_gpu):\n        model.cuda()\n    \n    counter = 0\n    \n    # Model in training mode\n    model.train()\n    breaker = False\n    for e in range(epochs):\n\n        # Batch loop\n        for inputs, labels in train_loader:\n            counter += 1\n\n            # move data to GPU, if available\n            if(train_on_gpu):\n                inputs, labels = inputs.cuda(), labels.cuda()\n\n            # Initialize hidden state\n            h = model.init_hidden(batch_size)\n\n            # Setting accumulated gradients to zero before backward step\n            model.zero_grad()\n\n            # Output from the model\n            output = model(inputs, h)\n\n            # Calculate the loss and do backprop step\n            loss = criterion(output.squeeze(), labels.float())\n            loss.backward()\n\n            # Clipping the gradient to avoid explosion\n            nn.utils.clip_grad_norm_(model.parameters(), clip)\n\n            # Backpropagation step\n            optimizer.step()\n\n            # Validation stats\n            if counter % print_every == 0:\n\n                with torch.no_grad():\n\n                    # Get validation loss and F1-score on validation set\n\n                    val_losses = []\n                    all_val_labels = []\n                    all_val_preds = []\n                    all_val_probs = []\n\n                    # Model in evaluation mode\n                    model.eval()\n                    for inputs, labels in valid_loader:\n\n                        all_val_labels += list(labels)\n\n                        # Sending data to GPU\n                        if(train_on_gpu):\n                            inputs, labels = inputs.cuda(), labels.cuda()\n\n                        # Initiating hidden state for the validation set\n                        val_h = model.init_hidden(batch_size)\n\n                        output = model(inputs, val_h)\n\n                        # Computing validation loss\n                        val_loss = criterion(output.squeeze(), labels.float())\n\n                        val_losses.append(val_loss.item())\n\n                        # Computing validation F1-score for threshold 0.5\n\n                        preds = torch.round(output.squeeze())  # 1 if output probability >= 0.5\n                        preds = np.squeeze(preds.cpu().numpy())\n                        all_val_preds += list(preds)\n                        \n                        output = np.squeeze(output.cpu().detach().numpy())\n                        all_val_probs += list(output)\n\n                current_loss = np.mean(val_losses)\n                \n                print(\"Epoch: {}/{}...\".format(e+1, epochs),\n                      \"Step: {}...\".format(counter),\n                      \"Loss: {:.6f}...\".format(loss.item()),\n                      \"Val Loss: {:.6f}...\".format(current_loss),\n                      \"F1-score (threshold=0.5): {:.3%}\".format(f1_score(all_val_labels, all_val_preds)))\n                \n                # Saving the best model and stopping if there is no improvement after \"early_stop\" evaluations\n                    \n                if  counter == print_every or current_loss < best_loss: # first evaluation or improvement\n                    best_loss = current_loss\n                    best_val_labels = all_val_labels\n                    best_probs = all_val_probs\n                    torch.save(model.state_dict(), 'checkpoint.pth')\n                    counter_eval = 0 \n                    \n                counter_eval += 1\n                if counter_eval == early_stop:\n                    breaker = True\n                    break\n\n                # Put model back to training mode\n                model.train()\n        \n        # breaking outer loop on epochs\n        if breaker:\n            break\n    \n    # Loading best model\n    state_dict = torch.load('checkpoint.pth')\n    model.load_state_dict(state_dict)\n    \n    return best_probs, best_val_labels","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e56baf82b2683c16b91212ff59e0b94dd01dd680"},"cell_type":"code","source":"t0 = time.time()\nall_val_probs, all_val_labels = train_model(model, train_loader, valid_loader, batch_size, epochs, \n                                            optimizer, criterion, clip, print_every, early_stop)\ntf = time.time()\nprint(\"\\nExecution time: {:.2f}min\".format((tf-t0)/60))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dd61b7987c0660330e6851d868e43e100a6a7681"},"cell_type":"markdown","source":"### Best threshold"},{"metadata":{"trusted":true,"_uuid":"089154b09041012c614990045482b4a972026359"},"cell_type":"code","source":"best_score = 0\nfor thr in np.arange(0.0, 0.5, 0.005):\n    pred = np.array(all_val_probs > thr, dtype=int)\n    score = f1_score(all_val_labels, pred)\n    print(\"Threshold: {:.3f}... F1-score {:.3%}\".format(thr, score))\n    if score > best_score:\n        best_score = score\n        best_thr = thr\nprint(\"\\nBest threshold: {:.3f}... F1-score {:.3%}\".format(best_thr, best_score))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"da3093bdfce8690c1619affcf33b896ebc8e7054"},"cell_type":"markdown","source":"## Predictions on test set"},{"metadata":{"trusted":true,"_uuid":"0f8265bbc979bd10bcde2ac9c6ce3dbfee96508b"},"cell_type":"code","source":"# Model in evaluation mode\nmodel.eval()\n\nwith torch.no_grad():\n    all_test_preds = []\n\n    for inputs in test_loader:\n        inputs = inputs[0]\n        \n        # Sending data to GPU\n        if(train_on_gpu):\n            inputs = inputs.cuda()\n            \n        test_h = model.init_hidden(batch_size)\n        \n        output = model(inputs, test_h)\n        \n        preds = (output.squeeze() > best_thr).type(torch.IntTensor)\n        preds = np.squeeze(preds.cpu().numpy())\n        all_test_preds += list(preds.astype(int))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"033dc0bfba3afa9b72e30c710fbdb3ca2911203e"},"cell_type":"code","source":"sub = pd.DataFrame({\n    'qid': test_df.qid,\n    'prediction': all_test_preds\n})\n\n# Make sure the columns are in the correct order\nsub = sub[['qid', 'prediction']]\n\nsub.to_csv('submission.csv', index=False, sep=',')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}