{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"This is my first baseline linear NN for this competition implemented in Pytorch. Since there are no Pytorch pipelines publicly available yet, I thought I would share mine. Inference runs without problems and the score seems reasonable to me. Still, I am not 100 % sure that the pipeline is bug-free. So, please do check yourself if you want to build off of this baseline.\n\n**Preprocessing:** You can find the code to preprocess the train files in my other notebook: https://www.kaggle.com/nicohrubec/unnest-train?scriptVersionId=66371803\n\n**Model:** Simple 3-layer DNN with 100 neurons each.\n\n**Validation:** 5-Fold TimeSeriesSplit.\n\n**Feature Engineering:** For now very basic, the model gets fed with some time features (weekday, week, month) + some target means (overall player target mean and player target mean for each year). I do not use any pandas for the target means but use a loop in which I build up dictionaries row-wise that store stats for each player. I then use these dictionaries to calculate the features. That way there is no leakage and it's quite straightforward to update the stats with new incoming test data.\n\nI'd definitely appreciate any feedback. If I did miss to give any credits, please write in the comments below. Feel free to ask any questions, if you do not understand any part of the code. Also pls upvote, if you find it useful.\n\n\n**Changelog:**\n\n**v2:**\n- Fix the validation setup. Feature dictionary now get computed within the fold loop to prevent leakage. \n- Clip predictions.\n\n**v3:**\n- Remove year mean features.\n\n**v4:**\n- Fix inference so that test is predicted with all 5 fold models.\n\n**v6:**\n- Fix another mild leak in the validation procedure. Before the update feature dicts were updated with the current row before obtaining the feature values for the current row. This is not a real issue with the current feature sets, since only very broad means are used, but would become a more problematic source of leakage as soon as more granular features are introduced.\n\n**v8:**\n- Add targets stats as shown in this notebook: https://www.kaggle.com/mlconsult/1-38-lb-lightgbm-with-target-statistics/notebook. Note that the CV score gets slightly inflated by that.\n- Cleaning up some vars that are no longer needed.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom tqdm import tqdm\nimport os\nfrom sklearn.model_selection import StratifiedKFold, TimeSeriesSplit\n\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom torch.utils.data import DataLoader, Dataset\nimport torch.nn.functional as F\n\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"execution":{"iopub.status.busy":"2021-06-24T12:45:33.204606Z","iopub.execute_input":"2021-06-24T12:45:33.205126Z","iopub.status.idle":"2021-06-24T12:45:33.792192Z","shell.execute_reply.started":"2021-06-24T12:45:33.205049Z","shell.execute_reply":"2021-06-24T12:45:33.791337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv('../input/unnest-train/train_nextDayPlayerEngagement.csv')\ntarget_stats = pd.read_csv('../input/player-target-stats/player_target_stats.csv')","metadata":{"execution":{"iopub.status.busy":"2021-06-24T12:45:33.799660Z","iopub.execute_input":"2021-06-24T12:45:33.800039Z","iopub.status.idle":"2021-06-24T12:45:38.218034Z","shell.execute_reply.started":"2021-06-24T12:45:33.800000Z","shell.execute_reply":"2021-06-24T12:45:38.217171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class CFG:\n    nfolds = 5\n    batch_size = 1024\n    val_batch_size = batch_size * 8\n    nepochs = 10\n    lr = 0.001\n    device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n    seed = 42\n    targets = ['target1', 'target2', 'target3', 'target4']\n    engineered_cols = ['player_t1_mean', 'player_t2_mean', 'player_t3_mean', 'player_t4_mean']\n    cols = ['month', 'week', 'weekday'] + engineered_cols + [col for col in target_stats.columns if col not in ['playerId']]\n    debug = False","metadata":{"execution":{"iopub.status.busy":"2021-06-24T12:45:38.219686Z","iopub.execute_input":"2021-06-24T12:45:38.220044Z","iopub.status.idle":"2021-06-24T12:45:38.245030Z","shell.execute_reply.started":"2021-06-24T12:45:38.220003Z","shell.execute_reply":"2021-06-24T12:45:38.243590Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train","metadata":{"execution":{"iopub.status.busy":"2021-06-24T12:45:38.246286Z","iopub.execute_input":"2021-06-24T12:45:38.246718Z","iopub.status.idle":"2021-06-24T12:45:38.367859Z","shell.execute_reply.started":"2021-06-24T12:45:38.246678Z","shell.execute_reply":"2021-06-24T12:45:38.366985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if CFG.debug:\n    train = train[:10000]","metadata":{"execution":{"iopub.status.busy":"2021-06-24T12:45:38.369278Z","iopub.execute_input":"2021-06-24T12:45:38.369679Z","iopub.status.idle":"2021-06-24T12:45:38.374122Z","shell.execute_reply.started":"2021-06-24T12:45:38.369635Z","shell.execute_reply":"2021-06-24T12:45:38.373116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class MLBDataset(Dataset):\n    def __init__(self, df, targets=None, mode='train'):\n        self.mode = mode\n        self.data = df\n        if mode == 'train':\n            self.targets = targets\n        \n    def __len__(self):\n        return len(self.data)\n    \n    def __getitem__(self, idx):\n        if self.mode == 'train':\n            x = self.data[idx]\n            y = np.array(self.targets[idx])\n            return torch.from_numpy(x).float(), torch.from_numpy(y).float()\n        elif self.mode == 'test':\n            return torch.from_numpy(self.data[idx]).float()","metadata":{"execution":{"iopub.status.busy":"2021-06-24T12:45:38.375597Z","iopub.execute_input":"2021-06-24T12:45:38.375987Z","iopub.status.idle":"2021-06-24T12:45:38.386769Z","shell.execute_reply.started":"2021-06-24T12:45:38.375939Z","shell.execute_reply":"2021-06-24T12:45:38.385884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class MLBModel(nn.Module):\n    def __init__(self, num_cols):\n        super(MLBModel, self).__init__()\n        self.dense1 = nn.Linear(num_cols, 100)\n        self.dense2 = nn.Linear(100, 100)\n        self.dense3 = nn.Linear(100, len(CFG.targets))\n\n    def forward(self, x):\n        x = F.relu(self.dense1(x))\n        x = F.relu(self.dense2(x))\n        x = self.dense3(x).squeeze()\n\n        return x","metadata":{"execution":{"iopub.status.busy":"2021-06-24T12:45:38.388182Z","iopub.execute_input":"2021-06-24T12:45:38.388574Z","iopub.status.idle":"2021-06-24T12:45:38.398549Z","shell.execute_reply.started":"2021-06-24T12:45:38.388534Z","shell.execute_reply":"2021-06-24T12:45:38.397642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def set_seed(seed):\n    torch.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = False\n    np.random.seed(seed)","metadata":{"execution":{"iopub.status.busy":"2021-06-24T12:45:38.402029Z","iopub.execute_input":"2021-06-24T12:45:38.402424Z","iopub.status.idle":"2021-06-24T12:45:38.409128Z","shell.execute_reply.started":"2021-06-24T12:45:38.402390Z","shell.execute_reply":"2021-06-24T12:45:38.408285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# takes an unnested df and does a feature lookup for each player from the given feature dictionary\ndef add_player_features(row, player_dict, n_feats, id_col, nan_value=-1):\n    features = np.full((n_feats), nan_value, dtype=np.float32)\n    \n    pid = row[id_col]\n    \n    if pid in player_dict:\n        # overall player means\n        p_count = player_dict[pid]['count']\n        features[0] = player_dict[pid]['mean_t1'] / p_count\n        features[1] = player_dict[pid]['mean_t2'] / p_count\n        features[2] = player_dict[pid]['mean_t3'] / p_count\n        features[3] = player_dict[pid]['mean_t4'] / p_count\n    \n    return features","metadata":{"execution":{"iopub.status.busy":"2021-06-24T12:45:38.411111Z","iopub.execute_input":"2021-06-24T12:45:38.411639Z","iopub.status.idle":"2021-06-24T12:45:38.419435Z","shell.execute_reply.started":"2021-06-24T12:45:38.411599Z","shell.execute_reply":"2021-06-24T12:45:38.418179Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# takes a sample and uses it to update the features in the player_dict\n# if a dict is given further calculations will be based on it, else an empty one is created\ndef update_player_features(row, player_dict, nan_value=-1):\n    assert player_dict is not None\n    \n    pid = row[2]\n    t1, t2, t3, t4 = row[3], row[4], row[5], row[6]\n    \n    if pid in player_dict:\n        # update target lags\n        player_dict[pid]['mean_t1'] += t1\n        player_dict[pid]['mean_t2'] += t2\n        player_dict[pid]['mean_t3'] += t3\n        player_dict[pid]['mean_t4'] += t4\n        player_dict[pid]['count'] += 1\n    else:\n        # init the feature dict for this player\n        player_dict[pid] = {'mean_t1': t1, 'mean_t2': t2, 'mean_t3': t3, 'mean_t4': t4, 'count': 1}\n    \n    return player_dict","metadata":{"execution":{"iopub.status.busy":"2021-06-24T12:45:38.420851Z","iopub.execute_input":"2021-06-24T12:45:38.421548Z","iopub.status.idle":"2021-06-24T12:45:38.429679Z","shell.execute_reply.started":"2021-06-24T12:45:38.421506Z","shell.execute_reply":"2021-06-24T12:45:38.428568Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def do_feature_engineering(df, id_col=2):\n    player_features = {}\n    nfeats = len(CFG.engineered_cols)\n    features = np.zeros((df.shape[0], nfeats), dtype=np.float32)\n    \n    for idx, row in enumerate(tqdm(df.values)):\n        row_features = add_player_features(row, player_features, nfeats, id_col=id_col)\n        player_features = update_player_features(row, player_features)\n        features[idx] = row_features\n    \n    features = pd.DataFrame(features, columns=CFG.engineered_cols)\n    df = pd.concat([df, features], axis=1)\n    \n    return df, player_features","metadata":{"execution":{"iopub.status.busy":"2021-06-24T12:45:38.431306Z","iopub.execute_input":"2021-06-24T12:45:38.431907Z","iopub.status.idle":"2021-06-24T12:45:38.443488Z","shell.execute_reply.started":"2021-06-24T12:45:38.431858Z","shell.execute_reply":"2021-06-24T12:45:38.442561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def do_feature_engineering_test(df, player_dict, id_col=2):\n    features = np.zeros((df.shape[0], len(CFG.engineered_cols)), dtype=np.float32)\n    \n    for idx, row in enumerate(df.values):\n        row_features = add_player_features(row, player_dict, len(CFG.engineered_cols), id_col=id_col)\n        features[idx] = row_features\n    \n    features = pd.DataFrame(features, columns=CFG.engineered_cols, index=df.index)\n    df = pd.concat([df, features], axis=1)\n    \n    return df","metadata":{"execution":{"iopub.status.busy":"2021-06-24T12:45:38.444878Z","iopub.execute_input":"2021-06-24T12:45:38.445434Z","iopub.status.idle":"2021-06-24T12:45:38.453527Z","shell.execute_reply.started":"2021-06-24T12:45:38.445396Z","shell.execute_reply":"2021-06-24T12:45:38.452348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def train_epoch(model, loader, optimizer, device, criterion):\n    model.train()\n    train_loss = 0.0\n\n    for i, (sample, target) in enumerate(tqdm(loader)):\n        sample, target = sample.to(device), target.to(device)\n        optimizer.zero_grad()\n\n        preds = model(sample)\n\n        loss = criterion(preds, target)\n        loss.backward()\n        optimizer.step()\n\n        train_loss += loss.item() / len(loader)\n\n    return model, train_loss","metadata":{"execution":{"iopub.status.busy":"2021-06-24T12:45:38.455247Z","iopub.execute_input":"2021-06-24T12:45:38.455640Z","iopub.status.idle":"2021-06-24T12:45:38.463250Z","shell.execute_reply.started":"2021-06-24T12:45:38.455603Z","shell.execute_reply":"2021-06-24T12:45:38.462058Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def validate(model, loader, device, criterion):\n    model.eval()\n    val_loss = 0.0\n    val_preds = []\n    val_targets = []\n\n    with torch.no_grad():\n        for i, (sample, target) in enumerate(tqdm(loader)):\n            sample, target = sample.to(device), target.to(device)\n\n            preds = model(sample)\n            loss = criterion(preds, target)\n            val_loss += loss.item() / len(loader)\n\n            val_preds.append(preds.cpu())\n            val_targets.append(target.cpu())\n\n        val_preds = np.concatenate(val_preds)\n        val_targets = np.concatenate(val_targets)\n\n    return val_loss, val_preds","metadata":{"execution":{"iopub.status.busy":"2021-06-24T12:45:38.464881Z","iopub.execute_input":"2021-06-24T12:45:38.465818Z","iopub.status.idle":"2021-06-24T12:45:38.476710Z","shell.execute_reply.started":"2021-06-24T12:45:38.465775Z","shell.execute_reply":"2021-06-24T12:45:38.475865Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class ColL1Loss(nn.Module):\n    def __init__(self, num_targets):\n        super().__init__()\n        self.mae = nn.L1Loss()\n        assert num_targets != 0\n        self.num_targets = num_targets\n    \n    def forward(self, preds, targets):\n        l1 = 0.0\n        \n        for i in range(self.num_targets):\n            l1 += self.mae(preds[:, i], targets[:, i])\n        \n        return l1 / self.num_targets","metadata":{"execution":{"iopub.status.busy":"2021-06-24T12:45:38.478024Z","iopub.execute_input":"2021-06-24T12:45:38.478470Z","iopub.status.idle":"2021-06-24T12:45:38.486436Z","shell.execute_reply.started":"2021-06-24T12:45:38.478427Z","shell.execute_reply":"2021-06-24T12:45:38.485301Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def train_fold(xtrn, ytrn, xval, yval, fold):\n    print(f\"Train fold {fold}\")\n    train_set = MLBDataset(xtrn, ytrn)\n    val_set = MLBDataset(xval, yval)\n    \n    train_loader = DataLoader(train_set, batch_size=CFG.batch_size, shuffle=True)\n    val_loader = DataLoader(val_set, batch_size=CFG.val_batch_size, shuffle=False)\n    \n    model = MLBModel(xtrn.shape[1]).to(CFG.device)\n    criterion = ColL1Loss(len(CFG.targets))\n    optimizer = optim.Adam(model.parameters(), lr=CFG.lr)\n    \n    best_loss = {'train': np.inf, 'val': np.inf}\n    best_val_preds = None\n    \n    for epoch in range(1, CFG.nepochs + 1):\n        print(f\"Train epoch {epoch}\")\n        model, loss = train_epoch(model, train_loader, optimizer, CFG.device, criterion)\n        val_loss, val_preds = validate(model, val_loader, CFG.device, criterion)\n        \n        # save best model\n        if val_loss < best_loss['val']:\n            best_loss = {'train': loss, 'val': val_loss}\n            torch.save(model.state_dict(), f'fold{fold}.pt')\n            \n            # save best oof predictions\n            best_val_preds = val_preds\n            np.save(f'oof_fold{fold}.npy', val_preds)\n        \n        print(\"Train loss: {:5.5f}\".format(loss))\n        print(\"Val loss: {:5.5f}\".format(val_loss))\n    \n    return best_val_preds","metadata":{"execution":{"iopub.status.busy":"2021-06-24T12:45:38.488346Z","iopub.execute_input":"2021-06-24T12:45:38.488923Z","iopub.status.idle":"2021-06-24T12:45:38.500627Z","shell.execute_reply.started":"2021-06-24T12:45:38.488877Z","shell.execute_reply":"2021-06-24T12:45:38.499467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# extract very basic datetime features\ndef extract_date_feats(df):\n    df['year'] = pd.DatetimeIndex(df['engagementMetricsDate']).year\n    df['month'] = pd.DatetimeIndex(df['engagementMetricsDate']).month\n    df['week'] = pd.DatetimeIndex(df['engagementMetricsDate']).week\n    df['weekday'] = pd.DatetimeIndex(df['engagementMetricsDate']).weekday\n    \n    return df","metadata":{"execution":{"iopub.status.busy":"2021-06-24T12:45:38.502165Z","iopub.execute_input":"2021-06-24T12:45:38.502781Z","iopub.status.idle":"2021-06-24T12:45:38.511502Z","shell.execute_reply.started":"2021-06-24T12:45:38.502739Z","shell.execute_reply":"2021-06-24T12:45:38.510640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def train_kfolds(df):\n    set_seed(CFG.seed)\n    ts = TimeSeriesSplit(n_splits=CFG.nfolds)\n    df = extract_date_feats(df)\n    \n    oof_preds = []\n    oof_targets = []\n    \n    for fold_idx, (trn_idx, val_idx) in enumerate(ts.split(df)):\n        trn_df, val_df = df.iloc[trn_idx], df.iloc[val_idx]\n        \n        trn_df, player_dict = do_feature_engineering(trn_df)\n        val_df = do_feature_engineering_test(val_df, player_dict)\n        trn_df = pd.merge(trn_df, target_stats, how='left', on='playerId')\n        val_df = pd.merge(val_df, target_stats, how='left', on='playerId')\n        \n        xtrn, ytrn = trn_df[CFG.cols].values, trn_df[CFG.targets].values\n        xval, yval = val_df[CFG.cols].values, val_df[CFG.targets].values\n        \n        val_preds = train_fold(xtrn, ytrn, xval, yval, fold_idx)\n        \n        oof_preds.append(val_preds)\n        oof_targets.append(yval)\n        del trn_df, val_df, player_dict\n        del xtrn, ytrn, xval, yval\n    \n    oof_preds = torch.from_numpy(np.clip(np.concatenate(oof_preds), 0, 100)).float()\n    oof_targets = torch.from_numpy(np.concatenate(oof_targets)).float()\n    \n    eval_metric = ColL1Loss(len(CFG.targets))\n    oof_loss = eval_metric(oof_preds, oof_targets)\n    df, player_dict = do_feature_engineering(df)\n    del oof_preds, oof_targets\n    del df\n    \n    return oof_loss, player_dict","metadata":{"execution":{"iopub.status.busy":"2021-06-24T12:45:38.512873Z","iopub.execute_input":"2021-06-24T12:45:38.513555Z","iopub.status.idle":"2021-06-24T12:45:38.526293Z","shell.execute_reply.started":"2021-06-24T12:45:38.513512Z","shell.execute_reply":"2021-06-24T12:45:38.525407Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"loss, player_dict = train_kfolds(train)\nprint(\"OOF loss: {:5.5f}\".format(loss))\ndel train","metadata":{"execution":{"iopub.status.busy":"2021-06-24T12:45:38.527774Z","iopub.execute_input":"2021-06-24T12:45:38.528363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def predict_test(test, model, device):\n    model.eval()\n    model = model.to(device)\n    test_preds = []\n    \n    test_set = MLBDataset(xtest, mode='test')\n    test_loader = DataLoader(test_set, batch_size=CFG.val_batch_size, shuffle=False)\n    \n    with torch.no_grad():\n        for i, (sample) in enumerate(test_loader):\n            sample = sample.to(device)\n            preds = model(sample)\n            test_preds.append(preds.cpu())\n        \n        test_preds = np.concatenate(test_preds)\n        \n    return test_preds","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import mlb\nenv = mlb.make_env()\n\nfor (test_df, sub) in env.iter_test():\n    sub_df = sub.copy()\n    sub_df['playerId'] = sub_df[\"date_playerId\"].apply(lambda x: int( x.split(\"_\")[1] ) )\n    sub_df['engagementMetricsDate'] = sub_df[\"date_playerId\"].apply(lambda x: int( x.split(\"_\")[0] ) )\n    sub_df['engagementMetricsDate'] = sub_df['engagementMetricsDate'].apply(lambda x: str(x)[:4] + \"-\" + str(x)[4:6] + \"-\" + str(x)[6:])\n    sub_df = extract_date_feats(sub_df)\n    sub_df = do_feature_engineering_test(sub_df, player_dict, id_col=5)\n    sub_df = pd.merge(sub_df, target_stats, how='left', on='playerId')\n    \n    for fold_idx in range(CFG.nfolds):\n        # init model and load checkpoint\n        model = MLBModel(len(CFG.cols))\n        path = f'fold{fold_idx}.pt'\n        model.load_state_dict(torch.load(path))\n        \n        # predict\n        xtest = sub_df[CFG.cols].values\n        sub[CFG.targets] += np.clip(predict_test(xtest, model, CFG.device), 0, 100) / CFG.nfolds\n    \n    env.predict(sub)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}