{"cells":[{"metadata":{"_uuid":"1b0fcb66f3814a4c83dff403c5b309e0159ff936"},"cell_type":"markdown","source":"There have been some issues regarding the correlation between CV and leaderboard scores in this competition. Every top-scoring public kernel has a much lower CV score than leaderboard score. It has also been very frustrating to tune a model to optimal CV score only to discover that the score on the Leaderboard is abysmal.\n\nIn this kernel I am going to address this issue and propose a framework for robust local validation. The preprocessing and model architecture have stayed mostly the same as in [my previous kernel](https://www.kaggle.com/bminixhofer/deterministic-neural-networks-using-pytorch).\n\nI'll also write about the impact of seeds on the score.\n\nAgain, we'll start with standard imports."},{"metadata":{"_uuid":"9d5f1b9360b808d5941d6f37ba2ba20cf3d7f869"},"cell_type":"markdown","source":"# Imports"},{"metadata":{"_uuid":"6a1856073e217798a8c5ddd17107a8a72855b665","trusted":true},"cell_type":"code","source":"# standard imports\nimport time\nimport random\nimport os\nfrom IPython.display import display\nimport numpy as np\nimport pandas as pd\nfrom scipy import stats\nimport warnings\n\n# pytorch imports\nimport torch\nimport torch.nn as nn\nimport torch.utils.data\n\n# imports for preprocessing the questions\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\n\n# cross validation and metrics\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import f1_score\n\n# progress bars\nfrom tqdm import tqdm\ntqdm.pandas()\n\n# visualization\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nwarnings.filterwarnings(\"ignore\", message=\"F-score is ill-defined and being set to 0.0 due to no predicted samples.\")\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9898133bb6210b97bead2ed9c46e8280defb305e"},"cell_type":"markdown","source":"# Loading the data"},{"metadata":{"_uuid":"eaa71cd7a40cf4f647b3af9a5b0fdcd903fefeee","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/train.csv\")\ntest_df = pd.read_csv(\"../input/test.csv\")\nprint('Train data dimension: ', train_df.shape)\ndisplay(train_df.head())\nprint('Test data dimension: ', test_df.shape)\ndisplay(test_df.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"68dcf13011432b865b124ca0b4e8262b9516bc70"},"cell_type":"code","source":"enable_local_test = True\nif enable_local_test:\n    n_test = len(test_df)\n    train_df, local_test_df = (train_df.iloc[:-n_test].reset_index(drop=True), \n                               train_df.iloc[-n_test:].reset_index(drop=True))\nelse:\n    local_test_df = pd.DataFrame([[None, None, 0], [None, None, 0]], columns=['qid', 'question_text', 'target'])\n    n_test = 2","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b708f65338817989da79b07c99f3e86e5acd62df"},"cell_type":"markdown","source":"Here, we create a dataframe I call `local_test_df`. We will pretend that this dataframe is the actual test dataframe. The only difference: We know the labels for this one! So we do not have to blindly submit our model and pray for a good score, but can instead tune the score we achieve on this test dataframe. I have set the size of the local test dataframe to 4 times the size of the public test dataframe. That is a reasonable size (~200k rows) but more or less arbitrary.\n\nSo we are going to test our model on two sets now: First, the regular test set, and second, a local test set which we know the labels for. Overall, the procedure is:\n\n- split the data in a train and local test set\n- perform CV on the train set\n- when tuning the model:\n    - evaluate the predictions on the local test set\n- when submitting:\n    - predict the samples in the true test set\n    - make the size of the local test set 0 for best performance (set `enable_local_test` to False)\n    \n    \nWe actually perform CV on the train side of a regular train / test split. It would be ideal to wrap the whole thing in another K-Fold cross validation procedure. That is, however, not feasible regarding computing power on my local machine and in the kaggle kernels."},{"metadata":{"_uuid":"301805519ef9c9a7d59eb6c6945480d824d6e160"},"cell_type":"markdown","source":"# Utility functions"},{"metadata":{"_uuid":"d35808558a9a2d4288ff7993c76f542e87edd62f","trusted":true},"cell_type":"code","source":"def seed_everything(seed=1234):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\nseed_everything()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6c91fec794f4f77aa09a69ca5eee3e7314a99f33","trusted":true},"cell_type":"code","source":"def threshold_search(y_true, y_proba):\n    best_threshold = 0\n    best_score = 0\n    for threshold in tqdm([i * 0.01 for i in range(100)], disable=True):\n        score = f1_score(y_true=y_true, y_pred=y_proba > threshold)\n        if score > best_score:\n            best_threshold = threshold\n            best_score = score\n    search_result = {'threshold': best_threshold, 'f1': best_score}\n    return search_result","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4ea13f2e32ffa7c050af2c2dd9423ed2aa2592f0","trusted":true},"cell_type":"code","source":"def sigmoid(x):\n    return 1 / (1 + np.exp(-x))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"93f521adfd488bc3472b3184a3dcff92da0eb3c9"},"cell_type":"markdown","source":"# Processing input"},{"metadata":{"_uuid":"9a4874c7936cd1e527a82aa779e24fc3de3e23a1","trusted":true},"cell_type":"code","source":"embed_size = 300\nmax_features = 95000\nmaxlen = 70","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bfcd85be032654aac572b3f107b0b4184d34a8c9","trusted":true},"cell_type":"code","source":"puncts = [',', '.', '\"', ':', ')', '(', '-', '!', '?', '|', ';', \"'\", '$', '&', '/', '[', ']', '>', '%', '=', '#', '*', '+', '\\\\', '•',  '~', '@', '£', \n '·', '_', '{', '}', '©', '^', '®', '`',  '<', '→', '°', '€', '™', '›',  '♥', '←', '×', '§', '″', '′', 'Â', '█', '½', 'à', '…', \n '“', '★', '”', '–', '●', 'â', '►', '−', '¢', '²', '¬', '░', '¶', '↑', '±', '¿', '▾', '═', '¦', '║', '―', '¥', '▓', '—', '‹', '─', \n '▒', '：', '¼', '⊕', '▼', '▪', '†', '■', '’', '▀', '¨', '▄', '♫', '☆', 'é', '¯', '♦', '¤', '▲', 'è', '¸', '¾', 'Ã', '⋅', '‘', '∞', \n '∙', '）', '↓', '、', '│', '（', '»', '，', '♪', '╩', '╚', '³', '・', '╦', '╣', '╔', '╗', '▬', '❤', 'ï', 'Ø', '¹', '≤', '‡', '√', ]\ndef clean_text(x):\n    x = str(x)\n    for punct in puncts:\n        x = x.replace(punct, f' {punct} ')\n    return x","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c0f3a9ecd216eddc7621915276b65a9fe5e63ad2","trusted":true},"cell_type":"code","source":"for df in [train_df, test_df, local_test_df]:\n    df[\"question_text\"] = df[\"question_text\"].str.lower()\n    df[\"question_text\"] = df[\"question_text\"].apply(lambda x: clean_text(x))\n    df[\"question_text\"].fillna(\"_##_\", inplace=True)\n    \nx_train = train_df[\"question_text\"].values\nx_test = test_df[\"question_text\"].values\nx_test_local = local_test_df[\"question_text\"].values\n\ntokenizer = Tokenizer(num_words=max_features)\ntokenizer.fit_on_texts(list(x_train) + list(x_test_local))\nx_train = tokenizer.texts_to_sequences(x_train)\nx_test = tokenizer.texts_to_sequences(x_test)\nx_test_local = tokenizer.texts_to_sequences(x_test_local)\n\nx_train = pad_sequences(x_train, maxlen=maxlen)\nx_test = pad_sequences(x_test, maxlen=maxlen)\nx_test_local = pad_sequences(x_test_local, maxlen=maxlen)\n\ny_train = train_df['target'].values\ny_test = local_test_df['target'].values","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8607fe327a4d7f2af0dfad02cb7fae0a14b0f2b4"},"cell_type":"markdown","source":"# Creating the embeddings matrix"},{"metadata":{"_uuid":"e5df8649c4db0a6e5bbf05088614c8cde4985081","trusted":true},"cell_type":"code","source":"def load_glove(word_index, max_features):\n    EMBEDDING_FILE = '../input/embeddings/glove.840B.300d/glove.840B.300d.txt'\n    def get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')[:300]\n    \n    embeddings_index = []\n    for o in tqdm(open(EMBEDDING_FILE)):\n        try:\n            embeddings_index.append(get_coefs(*o.split(\" \")))\n        except Exception as e:\n            print(e)\n    \n    embeddings_index = dict(embeddings_index)\n            \n    all_embs = np.stack(embeddings_index.values())\n    emb_mean,emb_std = all_embs.mean(), all_embs.std()\n    embed_size = all_embs.shape[1]\n\n    # word_index = tokenizer.word_index\n    nb_words = min(max_features, len(word_index) + 1)\n    embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n    \n    for word, i in word_index.items():\n        if i >= max_features: continue\n        embedding_vector = embeddings_index.get(word)\n        \n        if embedding_vector is not None: \n            embedding_matrix[i] = embedding_vector\n\n    return embedding_matrix\n\ndef load_fasttext(word_index, max_features):    \n    EMBEDDING_FILE = '../input/embeddings/wiki-news-300d-1M/wiki-news-300d-1M.vec'\n    def get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')[:300]\n    \n    embeddings_index = []\n    for o in tqdm(open(EMBEDDING_FILE)):\n        if len(o) <= 100:\n            continue\n        \n        try:\n            coefs = get_coefs(*o.split(\" \"))\n            assert len(coefs[1]) == 300\n            embeddings_index.append(coefs)\n        except Exception as e:\n            print(e)\n\n    embeddings_index = dict(embeddings_index)\n    \n    all_embs = np.stack(embeddings_index.values())\n    emb_mean,emb_std = all_embs.mean(), all_embs.std()\n    embed_size = all_embs.shape[1]\n    \n    nb_words = min(max_features, len(word_index) + 1)\n    embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n    \n    for word, i in word_index.items():\n        if i >= max_features: continue\n        embedding_vector = embeddings_index.get(word)\n        \n        if embedding_vector is not None: \n            embedding_matrix[i] = embedding_vector\n    \n    return embedding_matrix\n\ndef load_para(word_index, max_features):\n    EMBEDDING_FILE = '../input/embeddings/paragram_300_sl999/paragram_300_sl999.txt'\n    def get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\n    \n    embeddings_index = []\n    for o in tqdm(open(EMBEDDING_FILE, encoding=\"utf8\", errors='ignore')):\n        if len(o) <= 100:\n            continue\n        try:\n            coefs = get_coefs(*o.split(\" \"))\n            assert len(coefs[1]) == 300\n            embeddings_index.append(coefs)\n        except Exception as e:\n            print(e)\n    \n    embeddings_index = dict(embeddings_index)\n    \n    all_embs = np.stack(embeddings_index.values())\n    emb_mean,emb_std = all_embs.mean(), all_embs.std()\n    embed_size = all_embs.shape[1]\n    \n    # word_index = tokenizer.word_index\n    nb_words = min(max_features, len(word_index) + 1)\n    embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n    \n    for word, i in word_index.items():\n        if i >= max_features: continue\n        embedding_vector = embeddings_index.get(word)\n        \n        if embedding_vector is not None: \n            embedding_matrix[i] = embedding_vector\n    \n    return embedding_matrix","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9f47acd3b1d4bec89a16fda89f3d042e44ea7de3","trusted":false},"cell_type":"code","source":"seed_everything()\n\nglove_embeddings = load_glove(tokenizer.word_index, max_features)\nparagram_embeddings = load_para(tokenizer.word_index, max_features)\n\nembedding_matrix = np.mean([glove_embeddings, paragram_embeddings], axis=0)\nnp.shape(embedding_matrix)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bd3804f9bf44762f7715e23099d42f606a4c281b"},"cell_type":"markdown","source":"# Defining the model"},{"metadata":{"_uuid":"eb9092a72441fede922a687fe5b1c464aed8a973"},"cell_type":"markdown","source":"The only thing I changed about the model is the size of the LSTM and GRU. They had 40 hidden units previously and 60 now. I also changed the number of K-Fold splits to 4."},{"metadata":{"_uuid":"a3e084b4fe81baf1bc392d36e2d43b2f6e032331","trusted":true},"cell_type":"code","source":"splits = list(StratifiedKFold(n_splits=4, shuffle=True, random_state=10).split(x_train, y_train))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c78fe9f2abf0db1b3aaf39b97d9303bf03c24d18","trusted":false},"cell_type":"code","source":"class Attention(nn.Module):\n    def __init__(self, feature_dim, step_dim, bias=True, **kwargs):\n        super(Attention, self).__init__(**kwargs)\n        \n        self.supports_masking = True\n\n        self.bias = bias\n        self.feature_dim = feature_dim\n        self.step_dim = step_dim\n        self.features_dim = 0\n        \n        weight = torch.zeros(feature_dim, 1)\n        nn.init.xavier_uniform_(weight)\n        self.weight = nn.Parameter(weight)\n        \n        if bias:\n            self.b = nn.Parameter(torch.zeros(step_dim))\n        \n    def forward(self, x, mask=None):\n        feature_dim = self.feature_dim\n        step_dim = self.step_dim\n\n        eij = torch.mm(\n            x.contiguous().view(-1, feature_dim), \n            self.weight\n        ).view(-1, step_dim)\n        \n        if self.bias:\n            eij = eij + self.b\n            \n        eij = torch.tanh(eij)\n        a = torch.exp(eij)\n        \n        if mask is not None:\n            a = a * mask\n\n        a = a / torch.sum(a, 1, keepdim=True) + 1e-10\n\n        weighted_input = x * torch.unsqueeze(a, -1)\n        return torch.sum(weighted_input, 1)\n    \nclass SpatialDropout(nn.Dropout2d):\n    def forward(self, x):\n        x = x.unsqueeze(2)    # (N, T, 1, K)\n        x = x.permute(0, 3, 2, 1)  # (N, K, 1, T)\n        x = super(SpatialDropout, self).forward(x)  # (N, K, 1, T), some features are masked\n        x = x.permute(0, 3, 2, 1)  # (N, T, 1, K)\n        x = x.squeeze(2)  # (N, T, K)\n        return x","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"313b59ffab2f076409a4bd4e680e3b49e9c923a3","trusted":false},"cell_type":"code","source":"class NeuralNet(nn.Module):\n    def __init__(self):\n        super(NeuralNet, self).__init__()\n        \n        hidden_size = 60\n        \n        self.embedding = nn.Embedding(max_features, embed_size)\n        self.embedding.weight = nn.Parameter(torch.tensor(embedding_matrix, dtype=torch.float32))\n        self.embedding.weight.requires_grad = False\n        \n        self.embedding_dropout = SpatialDropout(0.1)\n        self.lstm = nn.LSTM(embed_size, hidden_size, bidirectional=True, batch_first=True)\n        self.gru = nn.GRU(hidden_size * 2, hidden_size, bidirectional=True, batch_first=True)\n        \n        self.lstm_attention = Attention(hidden_size * 2, maxlen)\n        self.gru_attention = Attention(hidden_size * 2, maxlen)\n        \n        self.linear = nn.Linear(480, 16)\n        self.relu = nn.ReLU()\n        self.dropout = nn.Dropout(0.1)\n        self.out = nn.Linear(16, 1)\n    \n    def forward(self, x):\n        h_embedding = self.embedding(x)\n        h_embedding = self.embedding_dropout(h_embedding)\n        \n        h_lstm, _ = self.lstm(h_embedding)\n        h_gru, _ = self.gru(h_lstm)\n        \n        h_lstm_atten = self.lstm_attention(h_lstm)\n        h_gru_atten = self.gru_attention(h_gru)\n        \n        avg_pool = torch.mean(h_gru, 1)\n        max_pool, _ = torch.max(h_gru, 1)\n        \n        conc = torch.cat((h_lstm_atten, h_gru_atten, avg_pool, max_pool), 1)\n        conc = self.relu(self.linear(conc))\n        conc = self.dropout(conc)\n        out = self.out(conc)\n        \n        return out","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a5445e83f96ca47abded343f42ed702ad5ecaca7"},"cell_type":"markdown","source":"# Training"},{"metadata":{"_uuid":"49ab78365628601cece06c57a426e2512061966d"},"cell_type":"markdown","source":"Regarding the training procedure, we use Cyclic LR with 5 epochs. I also made a separate function (`train_model`) to train the model because we are going to use it multiple times."},{"metadata":{"_uuid":"51453ae35fce480b307c64c4e8a82828e785001f","trusted":false},"cell_type":"code","source":"batch_size = 512\nn_epochs = 5","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"1ff4129c342ea6f6de6fd35377a9d426798409d3"},"cell_type":"code","source":"class CyclicLR(object):\n    def __init__(self, optimizer, base_lr=1e-3, max_lr=6e-3,\n                 step_size=2000, factor=0.6, min_lr=1e-4, mode='triangular', gamma=1.,\n                 scale_fn=None, scale_mode='cycle', last_batch_iteration=-1):\n\n        if not isinstance(optimizer, torch.optim.Optimizer):\n            raise TypeError('{} is not an Optimizer'.format(\n                type(optimizer).__name__))\n        self.optimizer = optimizer\n\n        if isinstance(base_lr, list) or isinstance(base_lr, tuple):\n            if len(base_lr) != len(optimizer.param_groups):\n                raise ValueError(\"expected {} base_lr, got {}\".format(\n                    len(optimizer.param_groups), len(base_lr)))\n            self.base_lrs = list(base_lr)\n        else:\n            self.base_lrs = [base_lr] * len(optimizer.param_groups)\n\n        if isinstance(max_lr, list) or isinstance(max_lr, tuple):\n            if len(max_lr) != len(optimizer.param_groups):\n                raise ValueError(\"expected {} max_lr, got {}\".format(\n                    len(optimizer.param_groups), len(max_lr)))\n            self.max_lrs = list(max_lr)\n        else:\n            self.max_lrs = [max_lr] * len(optimizer.param_groups)\n\n        self.step_size = step_size\n\n        if mode not in ['triangular', 'triangular2', 'exp_range'] \\\n                and scale_fn is None:\n            raise ValueError('mode is invalid and scale_fn is None')\n\n        self.mode = mode\n        self.gamma = gamma\n\n        if scale_fn is None:\n            if self.mode == 'triangular':\n                self.scale_fn = self._triangular_scale_fn\n                self.scale_mode = 'cycle'\n            elif self.mode == 'triangular2':\n                self.scale_fn = self._triangular2_scale_fn\n                self.scale_mode = 'cycle'\n            elif self.mode == 'exp_range':\n                self.scale_fn = self._exp_range_scale_fn\n                self.scale_mode = 'iterations'\n        else:\n            self.scale_fn = scale_fn\n            self.scale_mode = scale_mode\n\n        self.batch_step(last_batch_iteration + 1)\n        self.last_batch_iteration = last_batch_iteration\n        \n        self.last_loss = np.inf\n        self.min_lr = min_lr\n        self.factor = factor\n        \n    def batch_step(self, batch_iteration=None):\n        if batch_iteration is None:\n            batch_iteration = self.last_batch_iteration + 1\n        self.last_batch_iteration = batch_iteration\n        for param_group, lr in zip(self.optimizer.param_groups, self.get_lr()):\n            param_group['lr'] = lr\n\n    def step(self, loss):\n        if loss > self.last_loss:\n            self.base_lrs = [max(lr * self.factor, self.min_lr) for lr in self.base_lrs]\n            self.max_lrs = [max(lr * self.factor, self.min_lr) for lr in self.max_lrs]\n            \n    def _triangular_scale_fn(self, x):\n        return 1.\n\n    def _triangular2_scale_fn(self, x):\n        return 1 / (2. ** (x - 1))\n\n    def _exp_range_scale_fn(self, x):\n        return self.gamma**(x)\n\n    def get_lr(self):\n        step_size = float(self.step_size)\n        cycle = np.floor(1 + self.last_batch_iteration / (2 * step_size))\n        x = np.abs(self.last_batch_iteration / step_size - 2 * cycle + 1)\n\n        lrs = []\n        param_lrs = zip(self.optimizer.param_groups, self.base_lrs, self.max_lrs)\n        for param_group, base_lr, max_lr in param_lrs:\n            base_height = (max_lr - base_lr) * np.maximum(0, (1 - x))\n            if self.scale_mode == 'cycle':\n                lr = base_lr + base_height * self.scale_fn(cycle)\n            else:\n                lr = base_lr + base_height * self.scale_fn(self.last_batch_iteration)\n            lrs.append(lr)\n        return lrs","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"9e7f8e30974db91d3681bcba28296e7ef9855e60"},"cell_type":"code","source":"def train_model(model, x_train, y_train, x_val, y_val, validate=True):\n    optimizer = torch.optim.Adam(model.parameters())\n\n    step_size = 300\n    scheduler = CyclicLR(optimizer, base_lr=0.001, max_lr=0.003,\n                         step_size=step_size, mode='exp_range',\n                         gamma=0.99994)\n    \n    train = torch.utils.data.TensorDataset(x_train, y_train)\n    valid = torch.utils.data.TensorDataset(x_val, y_val)\n    \n    train_loader = torch.utils.data.DataLoader(train, batch_size=batch_size, shuffle=True)\n    valid_loader = torch.utils.data.DataLoader(valid, batch_size=batch_size, shuffle=False)\n  \n    loss_fn = torch.nn.BCEWithLogitsLoss(reduction='mean').cuda()\n    best_score = -np.inf\n    \n    for epoch in range(n_epochs):\n        start_time = time.time()\n        model.train()\n        avg_loss = 0.\n        \n        for x_batch, y_batch in tqdm(train_loader, disable=True):\n            y_pred = model(x_batch)\n            scheduler.batch_step()\n            \n            loss = loss_fn(y_pred, y_batch)\n\n            optimizer.zero_grad()\n\n            loss.backward()\n\n            optimizer.step()\n            avg_loss += loss.item() / len(train_loader)\n            \n        model.eval()\n        \n        valid_preds = np.zeros((x_val_fold.size(0)))\n        \n        if validate:\n            avg_val_loss = 0.\n            for i, (x_batch, y_batch) in enumerate(valid_loader):\n                y_pred = model(x_batch).detach()\n\n                avg_val_loss += loss_fn(y_pred, y_batch).item() / len(valid_loader)\n                valid_preds[i * batch_size:(i+1) * batch_size] = sigmoid(y_pred.cpu().numpy())[:, 0]\n            search_result = threshold_search(y_val.cpu().numpy(), valid_preds)\n\n            val_f1, val_threshold = search_result['f1'], search_result['threshold']\n            elapsed_time = time.time() - start_time\n            print('Epoch {}/{} \\t loss={:.4f} \\t val_loss={:.4f} \\t val_f1={:.4f} best_t={:.2f} \\t time={:.2f}s'.format(\n                epoch + 1, n_epochs, avg_loss, avg_val_loss, val_f1, val_threshold, elapsed_time))\n        else:\n            elapsed_time = time.time() - start_time\n            print('Epoch {}/{} \\t loss={:.4f} \\t time={:.2f}s'.format(\n                epoch + 1, n_epochs, avg_loss, elapsed_time))\n   \n    valid_preds = np.zeros((x_val_fold.size(0)))\n    \n    avg_val_loss = 0.\n    for i, (x_batch, y_batch) in enumerate(valid_loader):\n        y_pred = model(x_batch).detach()\n\n        avg_val_loss += loss_fn(y_pred, y_batch).item() / len(valid_loader)\n        valid_preds[i * batch_size:(i+1) * batch_size] = sigmoid(y_pred.cpu().numpy())[:, 0]\n\n    print('Validation loss: ', avg_val_loss)\n\n    test_preds = np.zeros((len(test_loader.dataset)))\n    \n    for i, (x_batch,) in enumerate(test_loader):\n        y_pred = model(x_batch).detach()\n\n        test_preds[i * batch_size:(i+1) * batch_size] = sigmoid(y_pred.cpu().numpy())[:, 0]\n\n    \n    test_preds_local = np.zeros((len(test_local_loader.dataset)))\n    \n    for i, (x_batch,) in enumerate(test_local_loader):\n        y_pred = model(x_batch).detach()\n\n        test_preds_local[i * batch_size:(i+1) * batch_size] = sigmoid(y_pred.cpu().numpy())[:, 0]\n\n    return valid_preds, test_preds, test_preds_local","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"a65460e7b5b1ede06660ced9ea4a7eeae97ec496"},"cell_type":"code","source":"seed = 6017","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b7ad1398071649e90a0f08c85535c03f8e23aa0c","trusted":false},"cell_type":"code","source":"x_test_cuda = torch.tensor(x_test, dtype=torch.long).cuda()\ntest = torch.utils.data.TensorDataset(x_test_cuda)\ntest_loader = torch.utils.data.DataLoader(test, batch_size=batch_size, shuffle=False)\n\nx_test_local_cuda = torch.tensor(x_test_local, dtype=torch.long).cuda()\ntest_local = torch.utils.data.TensorDataset(x_test_local_cuda)\ntest_local_loader = torch.utils.data.DataLoader(test_local, batch_size=batch_size, shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b7ad1398071649e90a0f08c85535c03f8e23aa0c","trusted":false},"cell_type":"code","source":"train_preds = np.zeros(len(train_df))\ntest_preds = np.zeros((len(test_df), len(splits)))\ntest_preds_local = np.zeros((n_test, len(splits)))\n\nfor i, (train_idx, valid_idx) in enumerate(splits):    \n    x_train_fold = torch.tensor(x_train[train_idx], dtype=torch.long).cuda()\n    y_train_fold = torch.tensor(y_train[train_idx, np.newaxis], dtype=torch.float32).cuda()\n    x_val_fold = torch.tensor(x_train[valid_idx], dtype=torch.long).cuda()\n    y_val_fold = torch.tensor(y_train[valid_idx, np.newaxis], dtype=torch.float32).cuda()\n    \n    train = torch.utils.data.TensorDataset(x_train_fold, y_train_fold)\n    valid = torch.utils.data.TensorDataset(x_val_fold, y_val_fold)\n    \n    train_loader = torch.utils.data.DataLoader(train, batch_size=batch_size, shuffle=True)\n    valid_loader = torch.utils.data.DataLoader(valid, batch_size=batch_size, shuffle=False)\n    \n    print(f'Fold {i + 1}')\n    \n    seed_everything(seed + i)\n    model = NeuralNet()\n    model.cuda()\n\n    valid_preds_fold, test_preds_fold, test_preds_local_fold = train_model(model,\n                                                                           x_train_fold, \n                                                                           y_train_fold, \n                                                                           x_val_fold, \n                                                                           y_val_fold, validate=True)\n\n    train_preds[valid_idx] = valid_preds_fold\n    test_preds[:, i] = test_preds_fold\n    test_preds_local[:, i] = test_preds_local_fold","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4f647692ec05df5f637f5cb4b4429037ea6d0001"},"cell_type":"markdown","source":"# Evaluation"},{"metadata":{"_uuid":"2346d3937a2f82bc4512036315e7d32e5509d150","trusted":false},"cell_type":"code","source":"search_result = threshold_search(y_train, train_preds)\nsearch_result","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"429d800ca5ab80e2044349ddc3e2c933b8b2030e"},"cell_type":"markdown","source":"Here we see our very low CV f1 score. But another metric that I have not seen in public kernels yet is the correlation between the test predictions of each fold (in this case the predictions of the local test set)."},{"metadata":{"trusted":false,"_uuid":"67ed4c346322597ffc27ac55864af321cbbb7c72"},"cell_type":"code","source":"pd.DataFrame(test_preds_local).corr()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ff8342d54769338955799d74b5b3a10739e2555e"},"cell_type":"markdown","source":"That is astonishingly low! I am used to seeing correlations of > 99% here. Remember that the model architecture of each of these predictions is exactly the same! The only difference is some of the training data and the seed used to initialize the parameters.\n\nBecause we have very low correlations it makes sense that, when stacking the predictions of each fold, the score gets much higher."},{"metadata":{"trusted":false,"_uuid":"3829e5fc69a3ec57a1a302ac51664ce0bbd56062"},"cell_type":"code","source":"f1_score(y_test, test_preds_local.mean(axis=1) > search_result['threshold'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4ff3a64f7e4dd8a466aea5c214cd7b6cb019b276"},"cell_type":"markdown","source":"And now we see a score that is about the same as what could be expected on the leaderboard. So the reason why models that have a high CV score often score badly on the leaderboard is that they seem to have a higher correlation between folds than models with a lower CV score.\nSo the challenge we face with neural networks in this competition is __finding the perfect tradeoff between CV score and correlation between the predictions of each fold__. \n\nSo when tuning a model, it makes little sense to only track the change in CV score. We have to tune models on a local test set in order to get a valid estimate of how well it will perform on the leaderboard."},{"metadata":{"_uuid":"e87e9557c51794f9744e01abc27c6ff380422845","trusted":false},"cell_type":"code","source":"submission = test_df[['qid']].copy()\nsubmission['prediction'] = test_preds.mean(axis=1) > search_result['threshold']\nsubmission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3311612bd120713b89c05f56bb56d5def0723b3f"},"cell_type":"markdown","source":"# A note on seeds"},{"metadata":{"_uuid":"2c9e3d86ae4058066bb019a2e278a334b86c97eb"},"cell_type":"markdown","source":"You might have noticed the line declaring the random seed to a cryptic value of 6017 above. That is because I hyperparameter-tuned the random seed. That might sound horrifying but, in my opinion, it makes sense in this competition.\n\nThe problem when tuning the seed without a local test set is that you are bound to overfit to the public test set which will be exchanged in stage 2. However, if we tune the seed on a cross-validation of local test sets, we do not have this risk. And the seed does make a huge difference. Not only on the public leaderboard but also on the local test set that is close to the size of the test set used in stage 2.\n\nI said in my first kernel that evaluating the model multiple times will not be necessary anymore because PyTorch behaves deterministically. But I have to correct that statement: PyTorch does behave deterministically, but that only means that we can run the model with one fixed seed and get the same result. That solves the problem of reproducability. But it does not change impact of the seed on the score. If you change some parameter of the model and have an unlucky seed, you might believe that the change was bad. But it could just have been the seed.\n\nThe code below wraps the regular K-Fold CV in a K-Fold CV for the local test set. Seeds are selected randomly. Every seed takes about 1 hour to evaluate on my machine (GTX 1080 TI)."},{"metadata":{"trusted":false,"_uuid":"ccc1223006544e86c8de69698384266e6a922c08"},"cell_type":"markdown","source":"```python\nif enable_local_test:\n    x_train_full = np.concatenate([x_train, x_test_local])\n    y_train_full = np.concatenate((y_train, y_test))\nelse:\n    x_train_full = x_train\n    y_train_full = y_train\n```"},{"metadata":{"trusted":false,"_uuid":"885679bc214fc5c3e3d7970202c1a225f90d5bd8"},"cell_type":"markdown","source":"```python\nseed_splits = list(StratifiedKFold(n_splits=4, shuffle=True, random_state=11).split(x_train, y_train))\nseeds = list(np.random.randint(0, 100000, 8))\ntrain_scores = []\ntest_scores = []\n\nfor seed_i, seed in enumerate(seeds):\n    seed_test_scores = []\n    seed_train_scores = []\n\n    for i, (train_index, val_index) in enumerate(seed_splits):\n        seed_x_train = x_train_full[train_index]\n        seed_y_train = y_train_full[train_index]\n        \n        seed_x_test = x_train_full[val_index]\n        seed_y_test = y_train_full[val_index]\n        \n        splits = list(StratifiedKFold(n_splits=4, shuffle=True, random_state=10).split(seed_x_train, seed_y_train))\n        \n        train_preds = np.zeros(len(seed_x_train))\n        test_preds_local = np.zeros((len(seed_x_test), len(splits)))\n        \n        x_test_local_cuda = torch.tensor(seed_x_test, dtype=torch.long).cuda()\n        test_local = torch.utils.data.TensorDataset(x_test_local_cuda)\n        test_local_loader = torch.utils.data.DataLoader(test_local, batch_size=batch_size, shuffle=False)\n        \n        print(f'Seed Fold {i + 1}\\n')\n        \n        for i, (train_idx, valid_idx) in enumerate(splits):\n            x_train_fold = torch.tensor(seed_x_train[train_idx], dtype=torch.long).cuda()\n            y_train_fold = torch.tensor(seed_y_train[train_idx, np.newaxis], dtype=torch.float32).cuda()\n            x_val_fold = torch.tensor(seed_x_train[valid_idx], dtype=torch.long).cuda()\n            y_val_fold = torch.tensor(seed_y_train[valid_idx, np.newaxis], dtype=torch.float32).cuda()\n\n            train = torch.utils.data.TensorDataset(x_train_fold, y_train_fold)\n            valid = torch.utils.data.TensorDataset(x_val_fold, y_val_fold)\n\n            train_loader = torch.utils.data.DataLoader(train, batch_size=batch_size, shuffle=True)\n            valid_loader = torch.utils.data.DataLoader(valid, batch_size=batch_size, shuffle=False)\n\n            print(f'Fold {i + 1}')\n\n            seed_everything(seed + i)\n            model = NeuralNet()\n            model.cuda()\n\n            valid_preds_fold, test_preds_fold, test_preds_local_fold = train_model(model,\n                                                                                   x_train_fold, \n                                                                                   y_train_fold, \n                                                                                   x_val_fold, \n                                                                                   y_val_fold, validate=True)\n\n            train_preds[valid_idx] = valid_preds_fold\n            test_preds_local[:, i] = test_preds_local_fold\n\n        train_search_result = threshold_search(seed_y_train, train_preds)\n        seed_train_scores.append(train_search_result['f1'])\n\n        test_score = f1_score(seed_y_test, test_preds_local.mean(axis=1) > train_search_result['threshold'])\n        seed_test_scores.append(test_score)\n    \n    train_score = np.mean(seed_train_scores)\n    test_score = np.mean(seed_test_scores)\n    \n    train_scores.append(train_score)\n    test_scores.append(test_score)\n    \n    print('\\ni={} \\t seed={} \\t score={}'.format(seed_i, seed, test_score))\n```"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true,"_uuid":"7ea303bb8ba8dcfe49c4b7f3a8267cf8dcff974b"},"cell_type":"code","source":"# loading the results from my local machine here\n# you have to trust me on this ;)\ntest_scores = [0.6894145809793863, 0.6904706309470233, 0.6905915253597362, 0.6908101789878276, 0.6910334464526553, 0.6916507797390641, 0.6903868185698696, 0.6908830283890897]\ntrain_scores = [0.669555770620476, 0.6708382008438574, 0.6700974173065081, 0.6701065866112219, 0.6704778141088164, 0.6708436318389969, 0.6705310002773053, 0.6710429366071224]\nseeds = [42853, 73399, 21152, 58237, 25688, 6017, 29547, 65803]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"81a8fdad23c38e6c05a84808cbff03aaad06e818"},"cell_type":"markdown","source":"Because seed tuning would exceed the runtime of kernels, I copied the results into this kernel. Now we can evaluate it."},{"metadata":{"trusted":true,"_uuid":"05c6005c5497625e1e663c86d17f0f8e539ae7b9"},"cell_type":"code","source":"eval_df = pd.DataFrame()\neval_df['cv_score'] = train_scores\neval_df['local_test_score'] = test_scores\neval_df['seed'] = seeds\neval_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f5782e9c2e667e36140a53f630abd764eff13519"},"cell_type":"code","source":"eval_df.loc[[eval_df['local_test_score'].idxmax()]]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bb5e3a3712b02f07e483b2261d7a3e432f18152e"},"cell_type":"code","source":"plt.figure(figsize=(14, 14))\nsns.violinplot(x='level_0', y=0, data=eval_df[['cv_score', 'local_test_score']].unstack().reset_index())\nplt.title('Distribution of scores for different random seeds')\nplt.ylabel('')\nplt.xlabel('')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"91bb49b92fb594901728451f0df57692a04044cf"},"cell_type":"code","source":"eval_df['local_test_score'].describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f93a3e93eca048e542a4021c05fb8c860dfea4a0"},"cell_type":"markdown","source":"The seed really does have a huge influence on the score. And the influence shown here is evalulated on the whole training set, it is surely even stronger on the ~ 50k rows in the public test set. And keep in mind that the statistic where calculated on a small sample of only 8 seeds."},{"metadata":{"_uuid":"cdc3e1523aa925905c404f5574281e0daaa119fd"},"cell_type":"markdown","source":"# Possible Shortcomings"},{"metadata":{"_uuid":"79b2103d29165a36489f042bbc94ced92d6c6a0e"},"cell_type":"markdown","source":"- The validation technique shown in this kernel only evaluates the model on a subset of the data. It would be ideal to wrap the procedure in another K-Fold cross-validation, but that is computationally hardly feasible (except for seed tuning where it is absolutely necessary).\n- The model behaves differently when the data in the local test set is added to the training data (e. g. batches are shuffled differently). Thus, it is impossible to tune the seed on a model that is exactly the same as the one used for submitting.\n- When tuning the model using the shown technique, you will tune it so that it behaves ideally with ~1M training samples. The best architecture for the model also changes when more training data is added (e. g. less need for regularization), so it might again not behave ideally when submitting."},{"metadata":{"_uuid":"04e73b2d6d7cfb1222742d481e45efbec00867df"},"cell_type":"markdown","source":"# Takeaway"},{"metadata":{"_uuid":"e3a5d4cb877d28429e7678b646b93db1e7d7b05e"},"cell_type":"markdown","source":"- The much higher scores on the leaderboard compared to CV scores are caused by a low correlation between folds of K-Fold CV.\n- When tuning a model, you have to find the best tradeoff between CV score and correlation between folds.\n- The seed is a valid hyperparameter to tune when not tuning it to the public LB.\n- Because the seed has a huge influence on the score, the LB score of top public kernels is not a good indicator on how good the model architecture is.\n- Although this is a kernels-only competition, local compute does matter a lot because you will likely not be able to achieve a good score on the leaderboard without tuning the seed."},{"metadata":{"_uuid":"aca20e18311bb580203bdb778e28c43d733bfb48"},"cell_type":"markdown","source":"All of the points above are my current beliefs. I might be wrong about some of them. I'm looking forward to discussion in the comments. Thanks for reading!"},{"metadata":{"_uuid":"3857e00ce0c83c1ab52cbbb6fbf7a600be0b5a03"},"cell_type":"markdown","source":"__Note: Version 1 is the one scoring 0.696 on the Leaderboard. The only difference is that `enable_local_test` is set to `False`. Runtime is also only 70 Minutes.__"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}