{"cells":[{"metadata":{"_uuid":"eb38ad12f994c7985d02bb2217b4c5d4fc76936e"},"cell_type":"markdown","source":"# Update\n__There was an issue with the random seed from `os.random` and the seed when creating the embedding matrix in version 1 - 3. Version 2 got a lucky seed and scored 0.694 for that reason. The problem is fixed since version 4.__\n\nIf you look at the version history you can see that the validation and training loss for version 4 and 5 are exactly the same so it is 100% reproducible now. But it dropped to 0.690 because the seed is less lucky on the Leaderboard. The CV score is even slightly better though."},{"metadata":{"_uuid":"b2c273d7e843b34d84610e1729d1d1bf615cc2bb"},"cell_type":"markdown","source":"# Preface"},{"metadata":{"_uuid":"9fef14285bc783a3499ce962ccbc08b83746ee80"},"cell_type":"markdown","source":"There have been many problems with reproducibility of neural networks in this competition. See for example [this post in the discussion forum](https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/73341).\n\nThere, it is recommended to \n\n> Rerun the experiment 10 - 15 times and average the scores to get the progress of your model.\n\nThat's obviously not a good solution because when optimizing the architecture of our NN we would of course like to be as fast as possible while still being sure that our model actually improves. Deviations of up to 0.01 in the F1 score are too large to be even remotely sure of that.\n\nThe problem lies within CuDNN. CuDNN's implementation of GRU and LSTM is [much faster](https://chainer.org/general/2017/03/15/Performance-of-LSTM-Using-CuDNN-v5.html) than the regular implementation but they do not run deterministically in TensorFlow and Keras. In this competition were speed is essential you can not afford to keep determinism  by using the regular implementation of GRU and LSTM.\n\n## PyTorch to the rescue!\n\nIn PyTorch, CuDNN determinism is a one-liner: `torch.backends.cudnn.deterministic = True`. This already solves the problem everyone has had so far with Keras. But that's not the only advantage of PyTorch. PyTorch is:\n\n- significantly faster than Keras and TensorFlow. Again, speed is important in this competition so this is great.\n- has a more pythonic API. I hate working with TensorFlow because there are seemingly tens of thousands of ways to do simple things. PyTorch has (in most cases) one obvious way and is by far not as convoluted as TensorFlow.\n- is executed eagerly. There is no such thing as an execution graph in PyTorch. That makes it much easier to try new things and interact with PyTorch in a notebook.\n\nKeras solves some of these problems with TensorFlow but it has a high-level API. I think that when doing research, it is often preferable to be able to interact with the model on a low-level. And you will see that the lower level API still doesn't make it complicated to work with PyTorch."},{"metadata":{"_uuid":"9d5f1b9360b808d5941d6f37ba2ba20cf3d7f869"},"cell_type":"markdown","source":"# Imports"},{"metadata":{"trusted":true,"_uuid":"6a1856073e217798a8c5ddd17107a8a72855b665"},"cell_type":"code","source":"# standard imports\nimport time\nimport random\nimport os\nfrom IPython.display import display\nimport numpy as np\nimport pandas as pd\n\n# pytorch imports\nimport torch\nimport torch.nn as nn\nimport torch.utils.data\n\n# imports for preprocessing the questions\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\n\n# cross validation and metrics\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import f1_score\n\n# progress bars\nfrom tqdm import tqdm\ntqdm.pandas()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9898133bb6210b97bead2ed9c46e8280defb305e"},"cell_type":"markdown","source":"# Loading the data"},{"metadata":{"trusted":false,"_uuid":"eaa71cd7a40cf4f647b3af9a5b0fdcd903fefeee"},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/train.csv\")\ntest_df = pd.read_csv(\"../input/test.csv\")\nprint('Train data dimension: ', train_df.shape)\ndisplay(train_df.head())\nprint('Test data dimension: ', test_df.shape)\ndisplay(test_df.head())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"301805519ef9c9a7d59eb6c6945480d824d6e160"},"cell_type":"markdown","source":"# Utility functions"},{"metadata":{"_uuid":"559530e8000c8d71eb0c8c6028b1a7209ff88bd9"},"cell_type":"markdown","source":"`seed_torch` sets the seed for numpy and torch to make sure functions with a random component behave deterministically. `torch.backends.cudnn.deterministic = true` sets the CuDNN to deterministic mode. \n\nThis function allows us to run experiments 100% deterministically."},{"metadata":{"trusted":true,"_uuid":"d35808558a9a2d4288ff7993c76f542e87edd62f"},"cell_type":"code","source":"def seed_everything(seed=1234):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\nseed_everything()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d9d33803fd8fa88f8150eb08b7dea0054bce8d9c"},"cell_type":"markdown","source":"Function to search for best threshold regarding the F1 score given labels and predictions from the network."},{"metadata":{"trusted":false,"_uuid":"6c91fec794f4f77aa09a69ca5eee3e7314a99f33"},"cell_type":"code","source":"def threshold_search(y_true, y_proba):\n    best_threshold = 0\n    best_score = 0\n    for threshold in tqdm([i * 0.01 for i in range(100)]):\n        score = f1_score(y_true=y_true, y_pred=y_proba > threshold)\n        if score > best_score:\n            best_threshold = threshold\n            best_score = score\n    search_result = {'threshold': best_threshold, 'f1': best_score}\n    return search_result","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5e86e673bf0ee05cdea28e4a22faf96dd33bcd30"},"cell_type":"markdown","source":"Sigmoid function in plain numpy."},{"metadata":{"trusted":false,"_uuid":"4ea13f2e32ffa7c050af2c2dd9423ed2aa2592f0"},"cell_type":"code","source":"def sigmoid(x):\n    return 1 / (1 + np.exp(-x))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"93f521adfd488bc3472b3184a3dcff92da0eb3c9"},"cell_type":"markdown","source":"# Processing input"},{"metadata":{"_uuid":"7db82f95b66478fb16ee62172ddc895963e688b5"},"cell_type":"markdown","source":"Standard preprocessing procedure. This is not the point of this kernel so I have copied it from [this great kernel](https://www.kaggle.com/gmhost/gru-capsule)."},{"metadata":{"trusted":false,"_uuid":"9a4874c7936cd1e527a82aa779e24fc3de3e23a1"},"cell_type":"code","source":"embed_size = 300 # how big is each word vector\nmax_features = 95000 # how many unique words to use (i.e num rows in embedding vector)\nmaxlen = 70 # max number of words in a question to use","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"bfcd85be032654aac572b3f107b0b4184d34a8c9"},"cell_type":"code","source":"puncts = [',', '.', '\"', ':', ')', '(', '-', '!', '?', '|', ';', \"'\", '$', '&', '/', '[', ']', '>', '%', '=', '#', '*', '+', '\\\\', '•',  '~', '@', '£', \n '·', '_', '{', '}', '©', '^', '®', '`',  '<', '→', '°', '€', '™', '›',  '♥', '←', '×', '§', '″', '′', 'Â', '█', '½', 'à', '…', \n '“', '★', '”', '–', '●', 'â', '►', '−', '¢', '²', '¬', '░', '¶', '↑', '±', '¿', '▾', '═', '¦', '║', '―', '¥', '▓', '—', '‹', '─', \n '▒', '：', '¼', '⊕', '▼', '▪', '†', '■', '’', '▀', '¨', '▄', '♫', '☆', 'é', '¯', '♦', '¤', '▲', 'è', '¸', '¾', 'Ã', '⋅', '‘', '∞', \n '∙', '）', '↓', '、', '│', '（', '»', '，', '♪', '╩', '╚', '³', '・', '╦', '╣', '╔', '╗', '▬', '❤', 'ï', 'Ø', '¹', '≤', '‡', '√', ]\ndef clean_text(x):\n    x = str(x)\n    for punct in puncts:\n        x = x.replace(punct, f' {punct} ')\n    return x","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"c0f3a9ecd216eddc7621915276b65a9fe5e63ad2"},"cell_type":"code","source":"train_df[\"question_text\"] = train_df[\"question_text\"].str.lower()\ntest_df[\"question_text\"] = test_df[\"question_text\"].str.lower()\n\ntrain_df[\"question_text\"] = train_df[\"question_text\"].apply(lambda x: clean_text(x))\ntest_df[\"question_text\"] = test_df[\"question_text\"].apply(lambda x: clean_text(x))\n\n# fill up the missing values\nx_train = train_df[\"question_text\"].fillna(\"_##_\").values\nx_test = test_df[\"question_text\"].fillna(\"_##_\").values\n\n# Tokenize the sentences\ntokenizer = Tokenizer(num_words=max_features)\ntokenizer.fit_on_texts(list(x_train))\nx_train = tokenizer.texts_to_sequences(x_train)\nx_test = tokenizer.texts_to_sequences(x_test)\n\n# Pad the sentences \nx_train = pad_sequences(x_train, maxlen=maxlen)\nx_test = pad_sequences(x_test, maxlen=maxlen)\n\n# Get the target values\ny_train = train_df['target'].values","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8607fe327a4d7f2af0dfad02cb7fae0a14b0f2b4"},"cell_type":"markdown","source":"# Creating the embeddings matrix"},{"metadata":{"_uuid":"60b5b32ac2bf20f76e2156a3db5d20492d773c91"},"cell_type":"markdown","source":"Another step that many others have already done. Again, the same progress as in the kernel from above."},{"metadata":{"trusted":false,"_uuid":"e5df8649c4db0a6e5bbf05088614c8cde4985081"},"cell_type":"code","source":"def load_glove(word_index):\n    EMBEDDING_FILE = '../input/embeddings/glove.840B.300d/glove.840B.300d.txt'\n    def get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')[:300]\n    embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE))\n    \n    all_embs = np.stack(embeddings_index.values())\n    emb_mean,emb_std = -0.005838499,0.48782197\n    embed_size = all_embs.shape[1]\n\n    # word_index = tokenizer.word_index\n    nb_words = min(max_features, len(word_index))\n    embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n    for word, i in word_index.items():\n        if i >= max_features: continue\n        embedding_vector = embeddings_index.get(word)\n        if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n            \n    return embedding_matrix \n    \ndef load_fasttext(word_index):    \n    EMBEDDING_FILE = '../input/embeddings/wiki-news-300d-1M/wiki-news-300d-1M.vec'\n    def get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\n    embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE) if len(o)>100)\n\n    all_embs = np.stack(embeddings_index.values())\n    emb_mean,emb_std = all_embs.mean(), all_embs.std()\n    embed_size = all_embs.shape[1]\n\n    # word_index = tokenizer.word_index\n    nb_words = min(max_features, len(word_index))\n    embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n    for word, i in word_index.items():\n        if i >= max_features: continue\n        embedding_vector = embeddings_index.get(word)\n        if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n\n    return embedding_matrix\n\ndef load_para(word_index):\n    EMBEDDING_FILE = '../input/embeddings/paragram_300_sl999/paragram_300_sl999.txt'\n    def get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\n    embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE, encoding=\"utf8\", errors='ignore') if len(o)>100)\n\n    all_embs = np.stack(embeddings_index.values())\n    emb_mean,emb_std = -0.0053247833,0.49346462\n    embed_size = all_embs.shape[1]\n\n    # word_index = tokenizer.word_index\n    nb_words = min(max_features, len(word_index))\n    embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n    for word, i in word_index.items():\n        if i >= max_features: continue\n        embedding_vector = embeddings_index.get(word)\n        if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n    \n    return embedding_matrix","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"9f47acd3b1d4bec89a16fda89f3d042e44ea7de3"},"cell_type":"code","source":"# missing entries in the embedding are set using np.random.normal so we have to seed here too\nseed_everything()\n\nglove_embeddings = load_glove(tokenizer.word_index)\nparagram_embeddings = load_para(tokenizer.word_index)\n\nembedding_matrix = np.mean([glove_embeddings, paragram_embeddings], axis=0)\nnp.shape(embedding_matrix)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bd3804f9bf44762f7715e23099d42f606a4c281b"},"cell_type":"markdown","source":"# Defining the model"},{"metadata":{"_uuid":"427b23e6db0ebb401ecc144aff9be5b5c4e61407"},"cell_type":"markdown","source":"First, define 5-Fold cross-validation. The `random_state` here is important to make sure this is deterministic too."},{"metadata":{"trusted":false,"_uuid":"a3e084b4fe81baf1bc392d36e2d43b2f6e032331"},"cell_type":"code","source":"splits = list(StratifiedKFold(n_splits=5, shuffle=True, random_state=10).split(x_train, y_train))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cef471e538fc7a7225a26d24b331d0911625d94e"},"cell_type":"markdown","source":"Now it gets interesting. First, I ported the Attention mechanism many others have used in this competition to PyTorch. I am not sure where the Keras snippet originated from, so I am going to give credit to the [kernel where I have first seen it](https://www.kaggle.com/shujian/single-rnn-with-4-folds-clr)."},{"metadata":{"trusted":false,"_uuid":"c78fe9f2abf0db1b3aaf39b97d9303bf03c24d18"},"cell_type":"code","source":"class Attention(nn.Module):\n    def __init__(self, feature_dim, step_dim, bias=True, **kwargs):\n        super(Attention, self).__init__(**kwargs)\n        \n        self.supports_masking = True\n\n        self.bias = bias\n        self.feature_dim = feature_dim\n        self.step_dim = step_dim\n        self.features_dim = 0\n        \n        weight = torch.zeros(feature_dim, 1)\n        nn.init.xavier_uniform_(weight)\n        self.weight = nn.Parameter(weight)\n        \n        if bias:\n            self.b = nn.Parameter(torch.zeros(step_dim))\n        \n    def forward(self, x, mask=None):\n        feature_dim = self.feature_dim\n        step_dim = self.step_dim\n\n        eij = torch.mm(\n            x.contiguous().view(-1, feature_dim), \n            self.weight\n        ).view(-1, step_dim)\n        \n        if self.bias:\n            eij = eij + self.b\n            \n        eij = torch.tanh(eij)\n        a = torch.exp(eij)\n        \n        if mask is not None:\n            a = a * mask\n\n        a = a / torch.sum(a, 1, keepdim=True) + 1e-10\n\n        weighted_input = x * torch.unsqueeze(a, -1)\n        return torch.sum(weighted_input, 1)\n    \nclass SpatialDropout(nn.Dropout2d):\n    def forward(self, x):\n        x = x.unsqueeze(2)    # (N, T, 1, K)\n        x = x.permute(0, 3, 2, 1)  # (N, K, 1, T)\n        x = super(SpatialDropout, self).forward(x)  # (N, K, 1, T), some features are masked\n        x = x.permute(0, 3, 2, 1)  # (N, T, 1, K)\n        x = x.squeeze(2)  # (N, T, K)\n        return x","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d03b225c099f9ebc2f71ec9b71caac69ffd3a9fe"},"cell_type":"markdown","source":"Now define the neural network. Defining a neural network in PyTorch is done by defining a class. This is almost as intuitive as Keras. The main difference is that you have one function (`__init__`) where it is defined which layers there are in the network and another function (`forward`) which defines the flow of data through the net.\n\nI replicated the architecture used in [@Shujian Liu's kernel](https://www.kaggle.com/shujian/single-rnn-with-4-folds-clr) in the network."},{"metadata":{"trusted":false,"_uuid":"313b59ffab2f076409a4bd4e680e3b49e9c923a3"},"cell_type":"code","source":"class NeuralNet(nn.Module):\n    def __init__(self):\n        super(NeuralNet, self).__init__()\n        \n        hidden_size = 40\n        \n        self.embedding = nn.Embedding(max_features, embed_size)\n        self.embedding.weight = nn.Parameter(torch.tensor(embedding_matrix, dtype=torch.float32))\n        self.embedding.weight.requires_grad = False\n        \n        self.embedding_dropout = SpatialDropout(0.1)\n        self.lstm = nn.LSTM(embed_size, hidden_size, bidirectional=True, batch_first=True)\n        self.gru = nn.GRU(hidden_size * 2, hidden_size, bidirectional=True, batch_first=True)\n        \n        self.lstm_attention = Attention(hidden_size * 2, maxlen)\n        self.gru_attention = Attention(hidden_size * 2, maxlen)\n        \n        self.linear = nn.Linear(320, 16)\n        self.relu = nn.ReLU()\n        self.dropout = nn.Dropout(0.1)\n        self.out = nn.Linear(16, 1)\n    \n    def forward(self, x):\n        h_embedding = self.embedding(x)\n        h_embedding = self.embedding_dropout(h_embedding)\n        \n        h_lstm, _ = self.lstm(h_embedding)\n        h_gru, _ = self.gru(h_lstm)\n        \n        h_lstm_atten = self.lstm_attention(h_lstm)\n        h_gru_atten = self.gru_attention(h_gru)\n        \n        # global average pooling\n        avg_pool = torch.mean(h_gru, 1)\n        # global max pooling\n        max_pool, _ = torch.max(h_gru, 1)\n        \n        conc = torch.cat((h_lstm_atten, h_gru_atten, avg_pool, max_pool), 1)\n        conc = self.relu(self.linear(conc))\n        conc = self.dropout(conc)\n        out = self.out(conc)\n        \n        return out","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a5445e83f96ca47abded343f42ed702ad5ecaca7"},"cell_type":"markdown","source":"# Training"},{"metadata":{"trusted":false,"_uuid":"51453ae35fce480b307c64c4e8a82828e785001f"},"cell_type":"code","source":"batch_size = 512 # how many samples to process at once\nn_epochs = 6 # how many times to iterate over all samples","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0939642af666800847c284bf924759c693ea1e83"},"cell_type":"markdown","source":"Now we can already train the network. Unfortunately, we do not have an API as high-level as keras's `.fit` in PyTorch. However, the code is still not too complicated and I have added comments where necessary."},{"metadata":{"trusted":false,"_uuid":"b7ad1398071649e90a0f08c85535c03f8e23aa0c"},"cell_type":"code","source":"# matrix for the out-of-fold predictions\ntrain_preds = np.zeros((len(train_df)))\n# matrix for the predictions on the test set\ntest_preds = np.zeros((len(test_df)))\n\n# always call this before training for deterministic results\nseed_everything()\n\nx_test_cuda = torch.tensor(x_test, dtype=torch.long).cuda()\ntest = torch.utils.data.TensorDataset(x_test_cuda)\ntest_loader = torch.utils.data.DataLoader(test, batch_size=batch_size, shuffle=False)\n\nfor i, (train_idx, valid_idx) in enumerate(splits):    \n    # split data in train / validation according to the KFold indeces\n    # also, convert them to a torch tensor and store them on the GPU (done with .cuda())\n    x_train_fold = torch.tensor(x_train[train_idx], dtype=torch.long).cuda()\n    y_train_fold = torch.tensor(y_train[train_idx, np.newaxis], dtype=torch.float32).cuda()\n    x_val_fold = torch.tensor(x_train[valid_idx], dtype=torch.long).cuda()\n    y_val_fold = torch.tensor(y_train[valid_idx, np.newaxis], dtype=torch.float32).cuda()\n    \n    model = NeuralNet()\n    # make sure everything in the model is running on the GPU\n    model.cuda()\n\n    # define binary cross entropy loss\n    # note that the model returns logit to take advantage of the log-sum-exp trick \n    # for numerical stability in the loss\n    loss_fn = torch.nn.BCEWithLogitsLoss(reduction='mean')\n    optimizer = torch.optim.Adam(model.parameters())\n\n    train = torch.utils.data.TensorDataset(x_train_fold, y_train_fold)\n    valid = torch.utils.data.TensorDataset(x_val_fold, y_val_fold)\n    \n    train_loader = torch.utils.data.DataLoader(train, batch_size=batch_size, shuffle=True)\n    valid_loader = torch.utils.data.DataLoader(valid, batch_size=batch_size, shuffle=False)\n    \n    print(f'Fold {i + 1}')\n    \n    for epoch in range(n_epochs):\n        # set train mode of the model. This enables operations which are only applied during training like dropout\n        start_time = time.time()\n        model.train()\n        avg_loss = 0.  \n        for x_batch, y_batch in tqdm(train_loader, disable=True):\n            # Forward pass: compute predicted y by passing x to the model.\n            y_pred = model(x_batch)\n\n            # Compute and print loss.\n            loss = loss_fn(y_pred, y_batch)\n\n            # Before the backward pass, use the optimizer object to zero all of the\n            # gradients for the Tensors it will update (which are the learnable weights\n            # of the model)\n            optimizer.zero_grad()\n\n            # Backward pass: compute gradient of the loss with respect to model parameters\n            loss.backward()\n\n            # Calling the step function on an Optimizer makes an update to its parameters\n            optimizer.step()\n            avg_loss += loss.item() / len(train_loader)\n            \n        # set evaluation mode of the model. This disabled operations which are only applied during training like dropout\n        model.eval()\n        \n        # predict all the samples in y_val_fold batch per batch\n        valid_preds_fold = np.zeros((x_val_fold.size(0)))\n        test_preds_fold = np.zeros((len(test_df)))\n        \n        avg_val_loss = 0.\n        for i, (x_batch, y_batch) in enumerate(valid_loader):\n            y_pred = model(x_batch).detach()\n            \n            avg_val_loss += loss_fn(y_pred, y_batch).item() / len(valid_loader)\n            valid_preds_fold[i * batch_size:(i+1) * batch_size] = sigmoid(y_pred.cpu().numpy())[:, 0]\n        \n        elapsed_time = time.time() - start_time \n        print('Epoch {}/{} \\t loss={:.4f} \\t val_loss={:.4f} \\t time={:.2f}s'.format(\n            epoch + 1, n_epochs, avg_loss, avg_val_loss, elapsed_time))\n        \n    # predict all samples in the test set batch per batch\n    for i, (x_batch,) in enumerate(test_loader):\n        y_pred = model(x_batch).detach()\n\n        test_preds_fold[i * batch_size:(i+1) * batch_size] = sigmoid(y_pred.cpu().numpy())[:, 0]\n\n    train_preds[valid_idx] = valid_preds_fold\n    test_preds += test_preds_fold / len(splits)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4f647692ec05df5f637f5cb4b4429037ea6d0001"},"cell_type":"markdown","source":"# Evaluation"},{"metadata":{"_uuid":"642cde2d37fc6768916ed10379e25ad275c415b3"},"cell_type":"markdown","source":"First, search for the best threshold:"},{"metadata":{"trusted":false,"_uuid":"2346d3937a2f82bc4512036315e7d32e5509d150"},"cell_type":"code","source":"search_result = threshold_search(y_train, train_preds)\nsearch_result","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"154ffa32687fd0cd396a36297a900729502a25fb"},"cell_type":"markdown","source":"That seems inline with the score from the replicated Keras kernel!"},{"metadata":{"_uuid":"8955d3ef4006e7cb358d637c4190715b418e18ef"},"cell_type":"markdown","source":"Finally submit the predictions with the threshold we have just found."},{"metadata":{"trusted":false,"_uuid":"e87e9557c51794f9744e01abc27c6ff380422845"},"cell_type":"code","source":"submission = test_df[['qid']].copy()\nsubmission['prediction'] = test_preds > search_result['threshold']\nsubmission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c0c294e6d4a9767edc5960bf9af5e7b6d7f17ae6"},"cell_type":"markdown","source":"## Ways to improve this kernel"},{"metadata":{"_uuid":"a7ea513e91a4fef9d0bfa5c80ca345271919dc23"},"cell_type":"markdown","source":"This kernel is intended as a demonstration of PyTorch. I did not spend any time tuning anything, I just ported the models to PyTorch. So, ways to improve this kernel are:\n\n- Tune the architecture! Now that training is deterministic it should be much less frustrating\n- Increase the number of folds in K-Fold cross-validation. Now that training is faster we can fit more folds into the kernel.\n- Or, keep the folds the same and increase the number of epochs / decrease the learning rate to improve the model at the cost of more time to train.\n- Load the weights with the best validation score after training (implement the equivalent of `ModelCheckpoint` in PyTorch). I am not sure if this will improve the score because you might overfit to the validation data.\n- Use a PyTorch implementation of CLR (cyclic learning rate). That seemed to make the model converge faster in some other kernels."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}