{"cells":[{"metadata":{"_uuid":"5bccd432da51aed0489e337f2ca024261124a62d"},"cell_type":"markdown","source":"![add all the randomness](https://www.memecreator.org/static/images/memes/4987185.jpg)\n\n<h1><center>or how to increase you ensemble score when your models suck</center></h1>"},{"metadata":{"_uuid":"748d2a1dd64355f9183818ff1fc557a2b0a689bb"},"cell_type":"markdown","source":"Hi guys! This kernel doesn't have a lot of fancy ideas on how to train good models... In fact, at some point a few days before the deadline, I gave up on increasing my CV score...\n\nHowever, CV score is not equal to the score you get after ensembling the predictions together! Here, another variable comes into play: Correlation! The lower the predictions are correlated, the higher the ensemble score.\n\nThat means, the more randomness you can squeeze into the predictions without hurting the CV score, the better!\n\nSo I spent a lot of time thinking of ways to add randomness to my single models. Here are some of the tricks I came up with:\n\n- For each model, average the three embedding matrixes with different weights. I found good and diverse ones through random search and luck\n- Randomized sample weights for each model. That way, each model focuses on different examples and should reach different solutions\n- Re-initialize the random embedding matrix between runs. Since a lot of words don't have embeddings and thus use random vectors, this also helps diversify\n- Add some random features to the embedding, different for each model. The models can overfit very slightly to those, also leading to increased diversification.\n- Replace some random embedding features with a random vector for each model\n- Overall, train longer than I would do otherwise. A slight overfit always seemed to help my ensemble F1.\n- Each model was trained on a different subset of the data and with different layer sizes, dropout values, loss functions, etc.\n\nTo measure how this helps, I added optional `ADD_DEV_SET` and `MEASURE_CORRELATION` flags.\n\n\n### Other things: Cyclic LR\n\nI realized almost every kernel was using Cyclic LR, but not finetuned so the schedule ends in the lowest LR... I took the time to do this right."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import time\nkernel_start_time = time.time()\nimport random\nimport pandas as pd\nimport numpy as np\nimport re\nimport torch\n\nimport torch.nn as nn\nimport torch.optim as optim\nimport torch.nn.functional as F\nfrom torch.utils.data import Dataset, DataLoader\nimport os\nimport math\n\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\n\n# cross validation and metrics\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import f1_score\nfrom sklearn.decomposition import PCA\nfrom torch.optim.optimizer import Optimizer","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"22c2d5de9ec31d103ae8641a3684bc5879534b98"},"cell_type":"markdown","source":"# IMPORTANT\n\n\n- For submissions, set `MEASURE_CORRELATION` and `USE_DEV_SET` to `False`.\n- For local experiments, set them to `True`!"},{"metadata":{"trusted":true,"_uuid":"58d201b61af88fc192ceb5bc21c432af275e13d4"},"cell_type":"code","source":"embed_size = 300\nmax_features = 120000\nmaxlen = 80\nbatch_size = 1024\nn_splits = 5\nMEASURE_CORRELATION = False # Set this to True for you local experiments\nNORMALIZE_EMBEDDINGS = True # Whether the embedding matrix should be normalized\nUSE_DEV_SET = False # Set this to True for you local experiments\nTRY_CAPITALS = True # Whether the embedding lookup should try a capitalized version\nTRY_UPPER = True # Whether the embedding lookup should try an uppercased version\n\nmodel_input_size = embed_size\n\nSEED = 666666","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"253bdf2c0ac28179c8d4342ec18fcfb29af901a1"},"cell_type":"code","source":"def seed_everything(seed=0):\n    seed = SEED + seed\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\nseed_everything()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"aa91baf7ce48737536a0a101f46a27b33adc0f8f"},"cell_type":"code","source":"df_train = pd.read_csv(\"../input/train.csv\")\ndf_test = pd.read_csv(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3970ede56fd9b51ee462020f9fc25adfce7b9b0d"},"cell_type":"code","source":"puncts = [\n    '½', '¿', 'ï', '¸', '-', ',', '/', '\"', '¨', '²', 'è', '×', '❤', '，', '↓', '▾', '↑',\n    'Ã', '±', ']', '·', '_', '<', '?', '⋅', '™', '~', '→', '′', '>', '≤', '€', '¥', '¼',\n    '¶', '@', '√', '®', '\\\\', '…', '、', '¹', '$', '•', '!', '¯', '&', '†', ')', '・', '^',\n    '—', '+', '#', '（', '³', '£', '″', '−', '[', '¬', '¦', '）', '–', '”', '¢', '%', '©',\n    '»', '}', '¾', '§', '=', '{', '‘', '∞', 'Ø', '°', '|', '：', '▒', 'â', 'à', ':', '(',\n    ';', '`', '│', 'é', '*', '’', '.', '\\'', '“',\n]\n\ndef clean_text(x):\n    for punct in puncts:\n        if punct in x:\n            x = x.replace(punct, f' {punct} ')\n    return x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"277d394c9ee4b39603c6f6b597ec7e0f6a612479"},"cell_type":"code","source":"from sklearn.preprocessing import StandardScaler\n\ndef add_features(df):\n    df['total_length'] = df['question_text'].apply(len)\n    df['capitals'] = df['question_text'].apply(lambda comment: sum(1 for c in comment if c.isupper()))\n    df['caps_vs_length'] = df.apply(lambda row: float(row['capitals'])/float(row['total_length']), axis=1)\n    df['num_words'] = df.question_text.str.count('\\S+') + 1\n    df['num_unique_words'] = df['question_text'].apply(lambda comment: len(set(w for w in comment.split())))\n    df['words_vs_unique'] = df['num_unique_words'] / df['num_words']\n\n    return df\n\nFEATURE_NAMES = ['caps_vs_length', 'words_vs_unique', 'total_length', 'num_words']\n\ndef load_and_prec():\n    train_df = df_train\n    test_df = df_test\n    print(\"Train shape : \",train_df.shape)\n    print(\"Test shape : \",test_df.shape)\n\n    # Clean the text\n    train_df[\"question_text\"] = train_df[\"question_text\"].apply(clean_text)\n    test_df[\"question_text\"] = test_df[\"question_text\"].apply(clean_text)\n\n    ## fill up the missing values\n    train_X = train_df[\"question_text\"].fillna(\"_##_\").values\n    test_X = test_df[\"question_text\"].fillna(\"_##_\").values\n\n    ###################### Add Features ###############################\n    #  https://github.com/wongchunghang/toxic-comment-challenge-lstm/blob/master/toxic_comment_9872_model.ipynb\n    train = add_features(train_df)\n    test = add_features(test_df)\n\n    features = train[FEATURE_NAMES].fillna(0)\n    test_features = test[FEATURE_NAMES].fillna(0)\n\n    ss = StandardScaler()\n    ss.fit(np.vstack((features, test_features)))\n    features = ss.transform(features)\n    test_features = ss.transform(test_features)\n    ###########################################################################\n\n    filters = '#&*+<>@\\\\^_`{|}~\\n'\n\n    ## Tokenize the sentences\n    tokenizer = Tokenizer(num_words=max_features, filters=filters)\n\n    tokenizer.fit_on_texts(list(train_X))\n    train_X = tokenizer.texts_to_sequences(train_X)\n    test_X = tokenizer.texts_to_sequences(test_X)\n\n    ## Pad the sentences\n    train_X = pad_sequences(train_X, maxlen=maxlen)\n    test_X = pad_sequences(test_X, maxlen=maxlen)\n\n    ## Get the target values\n    train_y = train_df['target'].values\n\n    return train_X, test_X, train_y, features, test_features, tokenizer.word_index","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"135a5becccdd4953544c3988aa52458e72a23c46"},"cell_type":"code","source":"def add_dev_set(train_X, train_y, train_features, size=100000):\n    dev_X = train_X[:size]\n    dev_y = train_y[:size]\n    dev_features = train_features[:size]\n\n    train_X = train_X[size:]\n    train_y = train_y[size:]\n    train_features = train_features[size:]\n\n    return train_X, train_y, train_features, dev_X, dev_y, dev_features\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5aa878a4f79ee86d3ec5607edf657fb3004c39e7","scrolled":true},"cell_type":"code","source":"x_train, x_test, y_train, train_features, test_features, word_index = load_and_prec()\nprint('Time elapsed:', time.time() - kernel_start_time)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"667e655a0cbdca0c1a6bfc4a13771d54bbba5edc"},"cell_type":"code","source":"if USE_DEV_SET:\n    x_train, y_train, train_features, x_dev, y_dev, dev_features = add_dev_set(x_train, y_train, train_features)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3f1f97a1a618e83bd39755c3ec4cfb7f5d24d7bd"},"cell_type":"code","source":"embedding_data = {\n    'glove': {\n        'file': '../input/embeddings/glove.840B.300d/glove.840B.300d.txt',\n        'emb_mean': -0.005838499,\n        'emb_std': 0.48782197,\n        'file_kwargs': {\n            'encoding': 'utf8',\n        },\n    },\n    'fasttext': {\n        'file': '../input/embeddings/wiki-news-300d-1M/wiki-news-300d-1M.vec',\n        'emb_mean': -0.0033469985,\n        'emb_std': 0.109855495,\n    },\n    'paragram': {\n        'file': '../input/embeddings/paragram_300_sl999/paragram_300_sl999.txt',\n        'emb_mean': -0.0053247833,\n        'emb_std': 0.49346462,\n        'file_kwargs': {\n            'encoding': 'utf8',\n            'errors': 'ignore',\n        },\n    },\n}\n\ndef load_embedding(\n    type, word_index,\n    normalize=NORMALIZE_EMBEDDINGS,\n    try_capitals=TRY_CAPITALS, try_upper=TRY_UPPER,\n):\n    assert type in embedding_data\n    data = embedding_data.get(type)\n\n    emb_mean, emb_std = data['emb_mean'], data['emb_std']\n    nb_words = min(max_features, len(word_index))\n\n    embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, model_input_size))\n\n    all_words = set(k for k, i in word_index.items() if i <= max_features)\n    oov_candidates = {}\n\n    def set_row(i, vecstring):\n        extra_features = []\n\n        embedding_vector = np.asarray(vec.split(' ') + extra_features, dtype='float32')[:model_input_size]\n        if len(embedding_vector) == model_input_size:\n            embedding_matrix[i] = embedding_vector\n            return True\n\n    with open(data['file'], 'r', **data.get('file_kwargs', {})) as f:\n        for line in f:\n            word, vec = line.split(' ', 1)\n            i = word_index.get(word)\n\n            if i is None and try_capitals:\n                oov_candidates[word] = vec\n\n            if i is None or i >= max_features:\n                continue\n\n            all_words.discard(word)\n\n            set_row(i, vec)\n\n    oov_words = [(w, i) for w, i in word_index.items() if w in all_words]\n\n    if try_capitals or TRY_UPPER:\n        n_added = 0\n        for word, i in oov_words:\n            capitalized = oov_candidates.get(word[0].upper() + word[1:]) if try_capitals else None\n            upper = TRY_UPPER and (capitalized == None) and (len(word) < 7) and oov_candidates.get(word.upper())\n            replacement = capitalized or upper\n            if replacement:\n                if set_row(i, replacement):\n                    n_added += 1\n                    continue\n\n        print(n_added, 'added')\n\n    if normalize:\n        embedding_matrix = embedding_matrix / emb_std\n\n    return embedding_matrix, oov_words","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9373008cfa9fe7ed26ee195fd9a61a9cb0b56ea9"},"cell_type":"code","source":"def load_all_embeddings(seed, load_embeddings=['glove', 'paragram', 'fasttext']):\n    seed_everything(seed)\n\n    embeddings = []\n\n    for embedding_name in load_embeddings:\n        matrix, oov = load_embedding(embedding_name, word_index)\n        embeddings.append(matrix)\n\n        print(embedding_name, 'OOV', len(oov))\n\n    return embeddings\n\nembeddings = load_all_embeddings(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4a0b4903bdf172fcc56c3fe2717e5753d01343dd"},"cell_type":"code","source":"print('Time elapsed:', time.time() - kernel_start_time)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9be23980194ef11c80fb6c3078901f5f2835a160"},"cell_type":"code","source":"# code inspired from: https://github.com/anandsaha/pytorch.cyclic.learning.rate/blob/master/cls.py\nclass CyclicLR(object):\n    def __init__(self, optimizer, base_lr=1e-3, max_lr=6e-3,\n                 step_size=2000, mode='triangular', gamma=1.,\n                 scale_fn=None, scale_mode='cycle', last_batch_iteration=-1):\n\n        if not isinstance(optimizer, Optimizer):\n            raise TypeError('{} is not an Optimizer'.format(\n                type(optimizer).__name__))\n        self.optimizer = optimizer\n\n        if isinstance(base_lr, list) or isinstance(base_lr, tuple):\n            if len(base_lr) != len(optimizer.param_groups):\n                raise ValueError(\"expected {} base_lr, got {}\".format(\n                    len(optimizer.param_groups), len(base_lr)))\n            self.base_lrs = list(base_lr)\n        else:\n            self.base_lrs = [base_lr] * len(optimizer.param_groups)\n\n        if isinstance(max_lr, list) or isinstance(max_lr, tuple):\n            if len(max_lr) != len(optimizer.param_groups):\n                raise ValueError(\"expected {} max_lr, got {}\".format(\n                    len(optimizer.param_groups), len(max_lr)))\n            self.max_lrs = list(max_lr)\n        else:\n            self.max_lrs = [max_lr] * len(optimizer.param_groups)\n\n        self.step_size = step_size\n\n        if mode not in ['triangular', 'triangular2', 'exp_range'] \\\n                and scale_fn is None:\n            raise ValueError('mode is invalid and scale_fn is None')\n\n        self.mode = mode\n        self.gamma = gamma\n\n        if scale_fn is None:\n            if self.mode == 'triangular':\n                self.scale_fn = self._triangular_scale_fn\n                self.scale_mode = 'cycle'\n            elif self.mode == 'triangular2':\n                self.scale_fn = self._triangular2_scale_fn\n                self.scale_mode = 'cycle'\n            elif self.mode == 'exp_range':\n                self.scale_fn = self._exp_range_scale_fn\n                self.scale_mode = 'iterations'\n        else:\n            self.scale_fn = scale_fn\n            self.scale_mode = scale_mode\n\n        self.in_final_stage = False\n        self.batch_step(last_batch_iteration + 1)\n        self.last_batch_iteration = last_batch_iteration\n\n    def batch_step(self, batch_iteration=None):\n        if batch_iteration is None:\n            batch_iteration = self.last_batch_iteration + 1\n        self.last_batch_iteration = batch_iteration\n        for param_group, lr in zip(self.optimizer.param_groups, self.get_lr()):\n            param_group['lr'] = lr\n\n    def enter_final_stage(self, remaining_iterations):\n        # Call to linearly decrease down to final lr over the remaining iterations\n        self.in_final_stage = True\n        self.remaining_iterations = remaining_iterations\n        self.final_lr = self.base_lrs[0] / 100\n        self.final_iterations = 0\n\n    def _triangular_scale_fn(self, x):\n        return 1.\n\n    def _triangular2_scale_fn(self, x):\n        return 1 / (2. ** (x - 1))\n\n    def _exp_range_scale_fn(self, x):\n        return self.gamma**(x)\n\n    def get_lr(self):\n        if self.in_final_stage:\n            lrs = []\n            param_lrs = zip(self.optimizer.param_groups, self.base_lrs, self.max_lrs)\n            for param_group, base_lr, max_lr in param_lrs:\n                step = (base_lr - self.final_lr) / self.remaining_iterations\n                lr = base_lr - step * self.final_iterations\n\n                if self.final_iterations < self.remaining_iterations:\n                    self.final_iterations += 1\n\n                lrs.append(lr)\n\n            return lrs\n\n\n        step_size = float(self.step_size)\n        cycle = np.floor(1 + self.last_batch_iteration / (2 * step_size))\n        x = np.abs(self.last_batch_iteration / step_size - 2 * cycle + 1)\n\n        lrs = []\n        param_lrs = zip(self.optimizer.param_groups, self.base_lrs, self.max_lrs)\n        for param_group, base_lr, max_lr in param_lrs:\n            base_height = (max_lr - base_lr) * np.maximum(0, (1 - x))\n            if self.scale_mode == 'cycle':\n                lr = base_lr + base_height * self.scale_fn(cycle)\n            else:\n                lr = base_lr + base_height * self.scale_fn(self.last_batch_iteration)\n            lrs.append(lr)\n\n        return lrs","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1b06855332c17f878401899e25d193a34493699e"},"cell_type":"code","source":"class Attention(nn.Module):\n    def __init__(self, feature_dim, step_dim, bias=True, **kwargs):\n        super(Attention, self).__init__(**kwargs)\n\n        self.supports_masking = True\n\n        self.bias = bias\n        self.feature_dim = feature_dim\n        self.step_dim = step_dim\n        self.features_dim = 0\n\n        weight = torch.zeros(feature_dim, 1)\n        nn.init.xavier_uniform_(weight)\n        self.weight = nn.Parameter(weight)\n\n        if bias:\n            self.b = nn.Parameter(torch.zeros(step_dim))\n\n    def forward(self, x, mask=None):\n        feature_dim = self.feature_dim\n        step_dim = self.step_dim\n\n        eij = torch.mm(\n            x.contiguous().view(-1, feature_dim),\n            self.weight\n        ).view(-1, step_dim)\n\n        if self.bias:\n            eij = eij + self.b\n\n        eij = torch.tanh(eij)\n        a = torch.exp(eij)\n\n        if mask is not None:\n            a = a * mask\n\n        a = a / torch.sum(a, 1, keepdim=True) + 1e-10\n\n        weighted_input = x * torch.unsqueeze(a, -1)\n        return torch.sum(weighted_input, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5ad19aec647076bdc0713afcc45403ec743b3c43"},"cell_type":"code","source":"class NeuralNetBig(nn.Module):\n    def __init__(self, embedding=embeddings[0], config={}):\n        super(NeuralNetBig, self).__init__()\n        self.config = config = {\n            'hidden_size_1': 132,\n            'hidden_size_2': 100,\n            'embedding_dropout': 0.1,\n            'dropout': 0.1,\n            'fc_size': 50,\n            **config\n        }\n        model_input_size = embedding.shape[1]\n        self.embedding = nn.Embedding(max_features, model_input_size)\n        self.embedding.weight = nn.Parameter(torch.tensor(embedding, dtype=torch.float32))\n        self.embedding.weight.requires_grad = False\n        self.embedding_dropout = nn.Dropout2d(config['embedding_dropout'])\n\n        self.lstm = nn.LSTM(model_input_size, config['hidden_size_1'], bidirectional=True, batch_first=True)\n        self.gru = nn.GRU(config['hidden_size_1'] * 2, config['hidden_size_2'], bidirectional=True, batch_first=True)\n\n        self.lstm_attention = Attention(config['hidden_size_1'] * 2, maxlen)\n        self.gru_attention = Attention(config['hidden_size_2'] * 2, maxlen)\n        f_size = config['hidden_size_1'] * 2 + config['hidden_size_2'] * 6 + len(FEATURE_NAMES)\n        self.linear = nn.Linear(f_size, config['fc_size'])\n        self.bn = nn.BatchNorm1d(config['fc_size'], momentum=0.5)\n        self.relu = nn.ReLU()\n        self.dropout = nn.Dropout(config['dropout'])\n        self.out = nn.Linear(config['fc_size'], 1)\n\n    def forward(self, x):\n        x_embed, x_features = x\n\n        h_embedding = self.embedding(x_embed)\n        h_embedding = torch.squeeze(self.embedding_dropout(torch.unsqueeze(h_embedding, 0)))\n\n        h_lstm, _ = self.lstm(h_embedding)\n        h_gru, _ = self.gru(h_lstm)\n\n        h_lstm_atten = self.lstm_attention(h_lstm)\n        h_gru_atten = self.gru_attention(h_gru)\n\n        avg_pool = torch.mean(h_gru, 1)\n        max_pool, _ = torch.max(h_gru, 1)\n\n        f = torch.tensor(x_features, dtype=torch.float).cuda()\n\n        conc = torch.cat((h_lstm_atten, h_gru_atten, avg_pool, max_pool, f), 1)\n        conc = self.dropout(conc)\n        conc = self.relu(self.linear(conc))\n        conc = self.bn(conc)\n        out = self.out(conc)\n\n        return out","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a863b7c58ac84aa87853fb69fc4e85d14bdb2e85"},"cell_type":"code","source":"class NeuralNetGRU(nn.Module):\n    def __init__(self, embedding=embeddings[0], config={}):\n        super(NeuralNetGRU, self).__init__()\n        self.config = config = {\n            'hidden_size': [132, 100, 50],\n            'dropout': 0.05,\n            **config\n        }\n\n        self.hidden_size = hidden_size = config['hidden_size']\n\n        model_input_size = embedding.shape[1]\n        self.embedding = nn.Embedding(max_features, model_input_size)\n        self.embedding.weight = nn.Parameter(torch.tensor(embedding, dtype=torch.float32))\n        self.embedding.weight.requires_grad = False\n\n        self.gru_1 = nn.GRU(model_input_size, hidden_size[0], bidirectional=True, batch_first=True)\n        self.gru_2 = nn.GRU(2 * hidden_size[0], hidden_size[1], bidirectional=True, batch_first=True)\n\n        f_size = hidden_size[0] * 2 + hidden_size[1] * 2 + len(FEATURE_NAMES)\n\n        self.fc_size = config.get('fc_size')\n\n        if config.get('fc_size'):\n            self.linear = nn.Linear(f_size, config['fc_size'])\n            self.bn = nn.BatchNorm1d(config['fc_size'])\n            self.relu = nn.ReLU()\n            self.out = nn.Linear(config['fc_size'], 1)\n\n        else:\n            self.bn = nn.BatchNorm1d(f_size)\n            self.out = nn.Linear(f_size, 1)\n\n    def forward(self, x):\n        x_embed, x_features = x\n\n        h_embedding = self.embedding(x_embed)\n\n        h_gru_1, _ = self.gru_1(h_embedding)\n        h_gru_2, _ = self.gru_2(h_gru_1)\n\n        h_max1, _ = torch.max(h_gru_1, 1)\n        h_max2, _ = torch.max(h_gru_2, 1)\n\n        f = torch.tensor(x_features, dtype=torch.float).cuda()\n        out = torch.cat((h_max1, h_max2, f), 1)\n\n        if self.fc_size:\n            out = self.linear(out)\n            out = self.bn(out)\n            out = self.relu(out)\n\n        out = self.out(out)\n\n        return out","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"40d482ba2771e9f070b8ad6c7aebfaae6b4b6d5f"},"cell_type":"code","source":"models = {\n    'gru': NeuralNetGRU,\n    'big': NeuralNetBig,\n}\n\nclass MyDataset(Dataset):\n    def __init__(self,dataset):\n        self.dataset = dataset\n\n    def __getitem__(self, index):\n        data, target = self.dataset[index]\n\n        return data, target, index\n    def __len__(self):\n        return len(self.dataset)\n\ndef sigmoid(x):\n    return 1 / (1 + np.exp(-x))\n\nfrom sklearn.metrics import roc_curve, precision_recall_curve\ndef best_thresshold(y_true, y_proba):\n    precision, recall, thresholds = precision_recall_curve(y_true, y_proba)\n    thresholds = np.append(thresholds, 1.001) \n    F = 2 / (1/precision + 1/recall)\n    best_score = np.max(F)\n    best_th = thresholds[np.argmax(F)]\n    return best_th, best_score ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3f710ac4beb4e49594969a93010b5041c5208891"},"cell_type":"code","source":"x_test_cuda = torch.tensor(x_test, dtype=torch.long).cuda()\ntest = torch.utils.data.TensorDataset(x_test_cuda)\n\nif USE_DEV_SET:\n    x_dev_cuda = torch.tensor(x_dev, dtype=torch.long).cuda()\n    dev = torch.utils.data.TensorDataset(x_dev_cuda)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"109dd8e00583a34a0ca2ec17754c92345ffca975"},"cell_type":"code","source":"# Creates a weighted average or concat of embeddings, specified in the settings\ndef ensemble_embeddings(settings, split_i):\n    global embeddings\n\n    mode = settings.get('embedding_mode', 'avg')\n    weights = settings.get('embedding_weights', [1, 1, 1, 0])\n\n    assert mode in ['avg', 'sample', 'concat']\n\n    if mode == 'avg':\n        embedding = np.average(embeddings, axis=0, weights=weights)\n    elif mode == 'sample':\n        sample_embedding_size = settings['embedding_sample']\n        weight_sum = np.array(weights).sum()\n        emb_list = []\n\n        for e, w in zip(embeddings, weights):\n            count = int(round((w / weight_sum) * sample_embedding_size))\n            emb_list.append(\n                e[:, np.random.randint(e.shape[1], size=count)]\n            )\n\n        embedding = np.concatenate(emb_list, axis=1)\n    elif mode == 'concat':\n        embedding = np.concatenate(embeddings, axis=1)\n\n    if settings.get('pca', 0):\n        pca = PCA(n_components=settings['pca'])\n        embedding = pca.fit_transform(embedding)\n\n    if settings.get('set_random_columns', 0):\n        n_random_cols = settings['set_random_columns']\n        random_cols = np.random.normal(0, 1, (len(embedding), n_random_cols))\n        embedding[:, np.random.randint(embedding.shape[1], size=n_random_cols)] = random_cols\n\n    if settings.get('add_random_columns', 0):\n        n_random_cols = settings['add_random_columns']\n        random_cols = np.random.normal(0, 1, (len(embedding), n_random_cols))\n        embedding = np.concatenate([embedding, random_cols], -1)\n\n    if settings.get('final_sample'):\n        embedding = embedding[:, np.random.randint(embedding.shape[1], size=settings['final_sample'])]\n\n    return embedding","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b04d87eee69450484ce79301f2d2c353f9c261a3"},"cell_type":"code","source":"def training(n_splits, schedule, base_seed=1):\n    global embeddings\n\n    training_start = time.time()\n    base_seed += SEED\n\n    # matrix for the out-of-fold predictions\n    train_preds = np.zeros((len(x_train)))\n    # matrix for the predictions on the test set\n    test_preds = np.zeros((len(df_test)))\n\n    if USE_DEV_SET:\n        dev_preds = np.zeros((len(x_dev)))\n\n    # always call this before training for deterministic results\n    seed_everything(base_seed)\n\n    avg_losses_f = []\n    avg_val_losses_f = []\n\n    splits = StratifiedKFold(n_splits=n_splits, shuffle=True, random_state=SEED+base_seed-1).split(x_train, y_train)\n\n    while len(schedule) < n_splits:\n        schedule *= 2\n\n    if MEASURE_CORRELATION:\n        all_test_preds = np.zeros((len(x_test), n_splits))\n\n    for i, (train_idx, valid_idx) in enumerate(splits):\n        settings = {\n            'gamma': 0.99994,\n            **schedule[i]\n        }\n        init_seed = settings.get('init_seed', base_seed + i * 100)\n        seed_everything(init_seed)\n        model_class = models[settings['model']]\n\n        batch_size = settings['batch_size']\n        n_epochs = settings['n_epochs']\n\n        test_loader = torch.utils.data.DataLoader(test, batch_size=batch_size, shuffle=False)\n        if USE_DEV_SET:\n            dev_loader = torch.utils.data.DataLoader(dev, batch_size=batch_size, shuffle=False)\n\n        x_train_fold = torch.tensor(x_train[train_idx.astype(int)], dtype=torch.long).cuda()\n        y_train_fold = torch.tensor(y_train[train_idx.astype(int), np.newaxis], dtype=torch.float32).cuda()\n        x_val_fold = torch.tensor(x_train[valid_idx.astype(int)], dtype=torch.long).cuda()\n        y_val_fold = torch.tensor(y_train[valid_idx.astype(int), np.newaxis], dtype=torch.float32).cuda()\n        kfold_X_features = train_features[train_idx.astype(int)]\n        kfold_X_valid_features = train_features[valid_idx.astype(int)]\n\n        random_sample_weights_range = settings.get('random_sample_weights_range', 0)\n        if settings.get('add_sample_weights') and random_sample_weights_range != 0:\n            # Here come the random sample weights for each fold:\n            kfold_weights = np.random.uniform(\n                1 - random_sample_weights_range / 2,\n                1 + random_sample_weights_range / 2,\n                (len(x_train_fold),),\n            )\n\n        embedding = ensemble_embeddings(settings, i)\n        model = model_class(embedding=embedding, config=settings['nn_config'])\n\n        # make sure everything in the model is running on the GPU\n        model.cuda()\n\n        loss_factor = 1\n        loss_fn = torch.nn.BCEWithLogitsLoss(reduction='none')\n\n        train = torch.utils.data.TensorDataset(x_train_fold, y_train_fold)\n        train = MyDataset(train)\n        train_loader = torch.utils.data.DataLoader(train, batch_size=batch_size, shuffle=True)\n\n        updates_per_epoch = math.ceil(len(train_idx) / batch_size)\n\n        step_size = round(updates_per_epoch * ((n_epochs * settings['n_cycles']) / 2))\n        base_lr, max_lr = settings['base_lr'], settings['max_lr']\n        optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=base_lr)\n\n        scheduler = CyclicLR(optimizer, base_lr=base_lr, max_lr=max_lr,\n                   step_size=step_size, mode='exp_range',\n                   gamma=settings['gamma'])\n\n        valid = torch.utils.data.TensorDataset(x_val_fold, y_val_fold)\n        valid = MyDataset(valid)\n        valid_loader = torch.utils.data.DataLoader(valid, batch_size=batch_size, shuffle=False)\n\n        print(f'Fold {i + 1}')\n        for epoch in range(n_epochs + (1 if settings.get('add_final_stage', 0) > 0 else 0)):\n            seed_everything(init_seed + epoch)\n            # set train mode of the model. This enables operations which are only applied during training like dropout\n            start_time = time.time()\n            model.train()\n\n            is_final_stage = False\n            steps_per_epoch = len(train_loader)\n\n            if settings.get('add_final_stage') and epoch == n_epochs:\n                is_final_stage = True\n                steps_per_epoch = settings['add_final_stage']\n                scheduler.enter_final_stage(settings['add_final_stage'])\n\n            avg_loss = 0.\n            for batch_i, (x_batch, y_batch, index) in enumerate(train_loader):\n                # Forward pass: compute predicted y by passing x to the model.\n                f = kfold_X_features[index]\n                y_pred = model([x_batch, f])\n\n                if scheduler:\n                    scheduler.batch_step()\n\n                if settings['add_sample_weights']:\n                    loss_weights = torch.tensor(kfold_weights[index], dtype=torch.float).unsqueeze(1).cuda()\n                    losses = loss_fn(y_pred, y_batch)\n                    losses *= loss_weights\n                else:\n                    losses = loss_fn(y_pred, y_batch)\n\n                loss = losses.sum()\n\n                # Before the backward pass, use the optimizer object to zero all of the\n                # gradients for the Tensors it will update (which are the learnable weights\n                # of the model)\n                optimizer.zero_grad()\n\n                # Backward pass: compute gradient of the loss with respect to model parameters\n                loss.backward()\n\n                # Calling the step function on an Optimizer makes an update to its parameters\n                optimizer.step()\n\n                avg_loss += ((loss.item() / steps_per_epoch) / batch_size) * loss_factor\n\n                if is_final_stage and batch_i >= settings['add_final_stage']:\n                    break\n\n            # set evaluation mode of the model. This disabled operations which are only applied during training like dropout\n            model.eval()\n\n            # predict all the samples in y_val_fold batch per batch\n            test_preds_fold = np.zeros((len(df_test)))\n            avg_val_loss = 0\n            val_f1 = 0\n\n            valid_preds_fold = np.zeros((x_val_fold.size(0)))\n            avg_val_loss = 0.\n            for bi, (x_batch, y_batch, index) in enumerate(valid_loader):\n                f = kfold_X_valid_features[index]\n                y_pred = model([x_batch,f]).detach()\n\n                avg_val_loss += ((loss_fn(y_pred, y_batch).sum().item() / len(valid_loader)) / batch_size) * loss_factor\n                valid_preds_fold[bi * batch_size:(bi+1) * batch_size] = sigmoid(y_pred.cpu().numpy())[:, 0]\n\n            threshold, val_f1 = best_thresshold(y_val_fold.cpu().numpy(), valid_preds_fold)\n\n            elapsed_time = time.time() - start_time\n            print('Epoch {}/{} \\t loss={:.4f} \\t val_loss={:.4f} \\t val_f1={:.4f} \\t thresh={:.4f} \\t time={:.2f}s'.format(\n                epoch + 1, n_epochs, avg_loss, avg_val_loss, val_f1, threshold, elapsed_time))\n\n        ensemble_weight = settings.get('ensemble_weight', 1 / n_splits)\n\n        avg_losses_f.append(avg_loss)\n        avg_val_losses_f.append(avg_val_loss)\n        # predict all samples in the test set batch per batch\n        for bi, (x_batch,) in enumerate(test_loader):\n            f = test_features[bi * batch_size:(bi+1) * batch_size]\n            y_pred = model([x_batch,f]).detach()\n\n            test_preds_fold[bi * batch_size:(bi+1) * batch_size] = sigmoid(y_pred.cpu().numpy())[:, 0]\n\n        if USE_DEV_SET:\n            dev_preds_fold = np.zeros((len(x_dev)))\n\n            for bi, (x_batch,) in enumerate(dev_loader):\n                f = dev_features[bi * batch_size:(bi+1) * batch_size]\n                y_pred = model([x_batch, f]).detach()\n\n                dev_preds_fold[bi * batch_size:(bi+1) * batch_size] = sigmoid(y_pred.cpu().numpy())[:, 0]\n\n            dev_preds += dev_preds_fold * ensemble_weight\n\n        if MEASURE_CORRELATION:\n            all_test_preds[:, i] = test_preds_fold\n\n        train_preds[valid_idx] = valid_preds_fold\n\n        test_preds += test_preds_fold  * ensemble_weight\n\n    correlation = 0\n    if MEASURE_CORRELATION:\n        correlation_matrix = pd.DataFrame(all_test_preds).corr()\n        print('\\ncorrelation matrix')\n        print(correlation_matrix)\n        print()\n        correlation = correlation_matrix.values[np.triu_indices_from(correlation_matrix.values, 1)].mean()\n\n    threshold, val_f1 = best_thresshold(y_train, train_preds)\n    average_val_loss = np.average(avg_val_losses_f)\n    overfit = average_val_loss / np.average(avg_losses_f)\n    training_time = time.time() - training_start\n    dev_threshold, dev_f1 = 0, 0\n\n    if USE_DEV_SET:\n        dev_threshold, dev_f1 = best_thresshold(y_dev, dev_preds)\n\n    return test_preds, val_f1, average_val_loss, correlation, training_time, overfit, threshold, dev_f1, dev_threshold","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a1a64dd4a988e93959917b051f2c38c3855b95c3"},"cell_type":"code","source":"# This is my training schedule... It consists of training options for each model in my ensemble\nschedule = [\n    {\n        \"embedding_weights\": [\n            0.44439510338875615,\n            0.3859913694202824,\n            0.1696135271909615,\n        ],\n        \"model\": \"big\",\n        \"base_lr\": 0.001,\n        \"max_lr\": 0.006,\n        \"n_epochs\": 4,\n        \"n_cycles\": 1,\n        \"batch_size\": 1024,\n        \"add_final_stage\": 5,\n        \"nn_config\": {\n            \"hidden_size_1\": 123,\n            \"hidden_size_2\": 73,\n            \"embedding_dropout\": 0.04055337160498571,\n            \"dropout\": 0.15174998980548882,\n            \"fc_size\": 77\n        },\n        \"add_sample_weights\": True,\n        'set_random_columns': 5,\n        'add_random_columns': 5,\n        'random_sample_weights_range': 0.25,\n    },\n    {\n        \"embedding_weights\": [\n            0.6303467699905284,\n            0.09360214396255329,\n            0.27605108604691825,\n        ],\n        \"model\": \"big\",\n        \"base_lr\": 0.0006013507213308578,\n        \"max_lr\": 0.006102420734804078,\n        \"n_epochs\": 4,\n        \"n_cycles\": 1,\n        \"batch_size\": 1024,\n        \"add_final_stage\": 0,\n        \"nn_config\": {\n            \"hidden_size_1\": 103,\n            \"hidden_size_2\": 140,\n            \"embedding_dropout\": 0.0728942043640125,\n            \"dropout\": 0.1812402857338244,\n            \"fc_size\": 19\n        },\n        \"add_sample_weights\": True,\n        'set_random_columns': 5,\n        'add_random_columns': 5,\n        'random_sample_weights_range': 0.25,\n    },\n    {\n        \"embedding_weights\": [\n            0.5497754648703559,\n            0.28986474829626435,\n            0.16035978683337976,\n        ],\n        \"model\": \"big\",\n        \"base_lr\": 0.0007647926420277793,\n        \"max_lr\": 0.006060292392943227,\n        \"n_epochs\": 4,\n        \"n_cycles\": 1,\n        \"batch_size\": 1024,\n        \"add_final_stage\": 10,\n        \"nn_config\": {\n            \"hidden_size_1\": 89,\n            \"hidden_size_2\": 74,\n            \"embedding_dropout\": 0.041249174666613736,\n            \"dropout\": 0.19393867662442568,\n            \"fc_size\": 87\n        },\n        \"add_sample_weights\": True,\n        'set_random_columns': 5,\n        'add_random_columns': 5,\n        'random_sample_weights_range': 0.25,\n    },\n    {\n        \"embedding_weights\": [\n            0.38162653503923166,\n            0.34041829363892023,\n            0.27795517132184816,\n        ],\n        \"model\": \"gru\",\n        \"base_lr\": 0.000765667042217169,\n        \"max_lr\": 0.006078156994754822,\n        \"n_epochs\": 3,\n        \"n_cycles\": 1,\n        \"batch_size\": 1024,\n        \"nn_config\": {\n            \"hidden_size\": [\n                154,\n                81\n            ],\n            \"fc_size\": 0\n        },\n        \"add_sample_weights\": True,\n        'set_random_columns': 5,\n        'add_random_columns': 5,\n        'random_sample_weights_range': 0.25,\n        \"add_final_stage\": 10,\n        \"reload_embeddings\": True,\n    },\n    {\n        \"embedding_weights\": [\n            0.09924183199312638,\n            0.3664584186963535,\n            0.53429974931052,\n        ],\n        \"model\": \"gru\",\n        \"base_lr\": 0.0012809083457068925,\n        \"max_lr\": 0.006110334179517582,\n        \"n_epochs\": 4,\n        \"n_cycles\": 1,\n        \"batch_size\": 1024,\n        \"nn_config\": {\n            \"hidden_size\": [\n                157,\n                60\n            ],\n            \"fc_size\": 0\n        },\n        \"add_sample_weights\": True,\n        'set_random_columns': 5,\n        'add_random_columns': 5,\n        'random_sample_weights_range': 0.25,\n        \"add_final_stage\": 0\n    },\n]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d3df9eeb1235ccba2e83f806d5a0f6ad47c61d87"},"cell_type":"code","source":"test_preds, val_f1, average_val_loss, correlation, training_time, overfit, threshold, dev_f1, dev_threshold = training(n_splits, schedule)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9ef058e568cb86f5d370424692b6c48c56e3b010","scrolled":true},"cell_type":"code","source":"print('val_f1', val_f1)\nprint('average_val_loss', average_val_loss)\nprint('overfit', overfit)\nprint('threshold', threshold)\nprint('correlation', correlation)\nprint('dev f1', dev_f1)\nprint('dev threshold', dev_threshold)\nprint('training_time', training_time)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bf754b126f7a0ea9040c14d9804b7256edc77a98"},"cell_type":"code","source":"if USE_DEV_SET:\n    threshold = (threshold + dev_threshold) / 2\n\nsubmission = df_test[['qid']]\nsubmission['prediction'] = (test_preds > threshold).astype(int)\nsubmission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"print('Time elapsed:', time.time() - kernel_start_time)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}