{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# [LB 0.694] Event-Aware TConv with Only 4 Features - Training Part\n\n## Introduction\nBefore diving into the manual feature engineering, I'm interested in seeing how far a DL-based model can go with a small set of features. After running experiments day and night, I temporarily can achieve **CV 0.6914** and LB 0.694 with only four features, including:\n\n1. Difference of `elapsed_time`\n2. `event_name`\n3. `name`\n4. `room_fqid`\n\nFirst, `event_name` is combined with `name` to form a new feature `event_comb` (*i.e.*, event combination), which represents the **status** of the corresponding event (*e.g.*, `notebook_click_open`, opening the notebook). As many have pointed out that the difference of `elapsed_time` is a key feature to use, I also treat it as a (and the only) numeric feature, which interacts with the other categorical features. The motivation behind the scene is that I hope the model can capture **event-aware temporal patterns**; that is, each time difference value is **enriched by the event information and where the event take places**. Finally, the model achieves the performance summarized as follows:\n\n| CV (GroupKFold with k=5) | Holdout (Released Old Test Set) | LB    |\n| ------------------------ | ------------------------------- | ----- |\n| 0.6914                   | 0.6911                          | 0.694 |\n\n## About this Notebook\nIn this kernel, I implement a complete pipeline, including data cleaning and processing, simple feature engineering, and model training with the specified CV scheme (*i.e.,* `GroupKFold` with `k=5`). After the process is done, output objects (*e.g.,* best model checkpoints) can be downloaded and used in the inference part.\n\n## Acknowledgements\n* Special thanks to [@chrisqiu](https://www.kaggle.com/chrisqiu)'s sharing [here](https://www.kaggle.com/code/chrisqiu/0-681-pytorch-using-only-1-column), which provides the inspiration for model building.\n* For section 7, all the credits should go to [@cdeotte](https://www.kaggle.com/cdeotte). I use and modify the code snippets in [his amazing starter notebook](https://www.kaggle.com/code/cdeotte/xgboost-baseline-0-680).\n\n## Other Works\n* To explore dangers in data, please see [EDA on Game Progress](https://www.kaggle.com/code/abaojiang/eda-on-game-progress)\n* To run the inference counterpart of this notebook, please see [[LB 0.694] Event-Aware TConv with Only 4 Features](https://www.kaggle.com/code/abaojiang/lb-0-694-event-aware-tconv-with-only-4-features)\n\n<a id=\"toc\"></a>\n## Table of Contents\n* [1. Prepare Data](#prep_data)\n* [2. Define Dataset](#dataset)\n* [3. Define Model Architecture](#model)\n* [4. Define Evaluator](#evaluator)\n* [5. Define Trainer](#trainer)\n* [6. Run Cross-Validation](#cv)\n* [7. Derive Final CV Score](#cv_score)\n\n## Import Packages","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-04-02T09:14:20.872156Z","iopub.execute_input":"2023-04-02T09:14:20.872597Z","iopub.status.idle":"2023-04-02T09:14:22.01591Z","shell.execute_reply.started":"2023-04-02T09:14:20.872543Z","shell.execute_reply":"2023-04-02T09:14:22.014417Z"}}},{"cell_type":"code","source":"import os\nimport gc\nimport pickle\nimport random\nimport warnings\nwarnings.simplefilter(\"ignore\")\nfrom abc import abstractmethod\nfrom copy import deepcopy\nfrom typing import Any, Callable, Dict, List, Optional, Tuple, Type, Union\nfrom tqdm import tqdm\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom sklearn.metrics import f1_score\nfrom sklearn.model_selection import GroupKFold\nfrom torch import optim, Tensor\nfrom torch.nn import Module\nfrom torch.optim import Optimizer, lr_scheduler\nfrom torch.optim.lr_scheduler import _LRScheduler\nfrom torch.utils.data import Dataset, DataLoader\nfrom torch.nn.modules.loss import _Loss\nfrom torchmetrics import AUROC\n\nN_QNS = 18\nLEVEL = list(range(23))\nLEVEL_GROUP = [\"0-4\", \"5-12\", \"13-22\"]\nLVGP_ORDER = {\"0-4\": 0, \"5-12\": 1, \"13-22\": 2}\nQNS_PER_LV_GP = {\"0-4\": list(range(1, 4)), \"5-12\": list(range(4, 14)), \"13-22\": list(range(14, 19))}\nLV_PER_LV_GP = {\"0-4\": list(range(0, 5)), \"5-12\": list(range(5, 13)), \"13-22\": list(range(13, 23))}\nCAT_FEAT_SIZE = {\n    \"event_comb_code\": 19,\n    \"room_fqid_code\": 19,\n}","metadata":{"execution":{"iopub.status.busy":"2023-04-02T15:04:34.751387Z","iopub.execute_input":"2023-04-02T15:04:34.752348Z","iopub.status.idle":"2023-04-02T15:04:41.073779Z","shell.execute_reply.started":"2023-04-02T15:04:34.752305Z","shell.execute_reply":"2023-04-02T15:04:41.072595Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Define Experiment Configuration ","metadata":{}},{"cell_type":"code","source":"def seed_all(seed: int) -> None:\n    \"\"\"Seed current experiment to guarantee reproducibility.\n\n    Parameters:\n        seed: manually specified seed number\n\n    Return:\n        None\n    \"\"\"\n    np.random.seed(seed)\n    random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.cuda.manual_seed_all(seed)\n    # When running with cudnn backend\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = False\n    # Set a fixed value for the hash seed\n    os.environ[\"PYTHONHASHSEED\"] = str(seed)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-02T15:04:43.182607Z","iopub.execute_input":"2023-04-02T15:04:43.182979Z","iopub.status.idle":"2023-04-02T15:04:43.191178Z","shell.execute_reply.started":"2023-04-02T15:04:43.18294Z","shell.execute_reply":"2023-04-02T15:04:43.190124Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class CFG:\n    # ==Mode==\n    # Specify True to enable model training\n    train = False\n    \n    # ==Data===\n    FEATS = [\"et_diff\", \"event_comb_code\", \"room_fqid_code\"]\n    CAT_FEATS = [\"event_comb_code\", \"room_fqid_code\"]\n    COLS_TO_USE = [\"session_id\", \"level\", \"level_group\", \"elapsed_time\",\n                   \"event_name\", \"name\", \"room_fqid\"]\n    T_WINDOW = 1000\n    \n    # ==Training==\n    SEED = 42\n    DEVICE = \"cuda:0\"\n    EPOCH = 100\n    CKPT_METRIC = \"f1@0.63\"\n\n    # ==DataLoader==\n    BATCH_SIZE = 128\n    NUM_WORKERS: 4\n\n    # ==Solver==\n    LR = 1e-3\n    WEIGHT_DECAY = 1e-4\n\n    # ==Early Stopping==\n    ES_PATIENCE = 0\n\n    # ==Evaluator==\n    EVAL_METRICS = [\"auroc\", \"f1\"]\n\nINPUT_PATH = \"/kaggle/input/jo-old-train/upload/train.parquet\"\nTARGET_PATH = \"/kaggle/input/jo-old-train/upload/train_labels.csv\"\ncfg = CFG()\nseed_all(cfg.SEED)","metadata":{"execution":{"iopub.status.busy":"2023-04-02T15:04:44.300463Z","iopub.execute_input":"2023-04-02T15:04:44.301176Z","iopub.status.idle":"2023-04-02T15:04:44.311862Z","shell.execute_reply.started":"2023-04-02T15:04:44.301137Z","shell.execute_reply":"2023-04-02T15:04:44.310714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"prep_data\"></a>\n## 1. Prepare Data\n[**<span style=\"color:#FEF1FE; background-color:#535d70;border-radius: 5px; padding: 2px\">Go to Table of Content</span>**](#toc)\n\n### *1.1 Load and Clean Data*","metadata":{"execution":{"iopub.status.busy":"2023-04-02T10:49:28.811559Z","iopub.execute_input":"2023-04-02T10:49:28.812633Z","iopub.status.idle":"2023-04-02T10:49:28.822649Z","shell.execute_reply.started":"2023-04-02T10:49:28.812584Z","shell.execute_reply":"2023-04-02T10:49:28.821603Z"}}},{"cell_type":"code","source":"def summarize(\n    df: pd.DataFrame,\n    file_name: Optional[str] = None,\n    n_rows_to_display: Optional[int] = 5,\n) -> None:\n    \"\"\"Summarize DataFrame.\n\n    Parameters:\n        df: input data\n        file_name: name of the input file\n        n_rows_to_display: number of rows to display\n\n    Return:\n        None\n    \"\"\"\n    file_name = \"Data\" if file_name is None else file_name\n\n    # Derive NaN ratio for each column\n    nan_ratio = pd.isna(df).sum() / len(df) * 100\n    nan_ratio.sort_values(ascending=False, inplace=True)\n    nan_ratio = nan_ratio.to_frame(name=\"NaN Ratio\").T\n\n    # Derive zero ratio for each column\n    zero_ratio = (df == 0).sum() / len(df) * 100\n    zero_ratio.sort_values(ascending=False, inplace=True)\n    zero_ratio = zero_ratio.to_frame(name=\"Zero Ratio\").T\n\n    # Print out summarized information\n    print(f\"=====Summary of {file_name}=====\")\n    display(df.head(n_rows_to_display))\n    print(f\"Shape: {df.shape}\")\n    print(\"NaN ratio:\")\n    display(nan_ratio)\n    print(\"Zero ratio:\")\n    display(zero_ratio)\n    \ndef drop_multi_game_naive(\n    df: pd.DataFrame,\n    local: bool = True\n) -> pd.DataFrame:\n    \"\"\"Drop events not occurring at the first game play.\n    \n    Note: `groupby` should be done with `sort=False` for the new\n    training set, because `session_id` isn't sorted by default.\n    \n    Parameters:\n        df: input DataFrame\n        local: if False, the processing step is simplified based on the\n            properties of returned test DataFrame by time series API\n    \n    Return:\n        df: DataFrame with events occurring at the first game play only\n    \"\"\"\n    df = df.copy()\n    if local:\n        df[\"lv_diff\"] = df.groupby(\"session_id\", sort=False).apply(lambda x: x[\"level\"].diff().fillna(0)).values\n    else:\n        df[\"lv_diff\"] = df[\"level\"].diff().fillna(0)\n    reversed_lv_pts = df[\"lv_diff\"] < 0\n    df.loc[~reversed_lv_pts, \"lv_diff\"] = 0\n    if local:\n        df[\"multi_game_flag\"] = df.groupby(\"session_id\", sort=False)[\"lv_diff\"].cumsum().values\n    else:\n        df[\"multi_game_flag\"] = df[\"lv_diff\"].cumsum()\n    multi_game_mask = df[\"multi_game_flag\"] < 0\n    multi_game_rows = df[multi_game_mask].index\n    df = df.drop(multi_game_rows).reset_index(drop=True)\n    \n    # Drop redundant columns\n    df.drop([\"lv_diff\", \"multi_game_flag\"], axis=1, inplace=True)\n    \n    return df\n\ndef map_lvgp_order(lvgp_seq: pd.Series) -> pd.Series:\n    \"\"\"Map level_group sequence to level_group order sequence.\n    \n    Parameters:\n        lvgp_seq: level_group sequence\n    \n    Return:\n        lvgp_order_seq: level_group order sequence\n    \"\"\"\n    lvgp_order_seq = lvgp_seq.map(LVGP_ORDER)\n    \n    return lvgp_order_seq\n\ndef check_multi_game(df: pd.DataFrame) -> bool:\n    \"\"\"Check if multiple game plays exist in any session.\n    \n    Parameters:\n        df: input DataFrame\n    \n    Return:\n        multi_game_exist: if True, multiple game plays exist in at\n            least one of the session\n    \"\"\"\n    multi_game_exist = False\n    for i, (sess_id, gp) in enumerate(df.groupby(\"session_id\", sort=False)):\n        if ((not gp[\"level\"].is_monotonic_increasing)\n            or (not gp[\"lvgp_order\"].is_monotonic_increasing)):\n            multi_game_exist = True\n            break\n            \n    return multi_game_exist","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-02T15:04:45.481794Z","iopub.execute_input":"2023-04-02T15:04:45.482249Z","iopub.status.idle":"2023-04-02T15:04:45.498027Z","shell.execute_reply.started":"2023-04-02T15:04:45.482215Z","shell.execute_reply":"2023-04-02T15:04:45.49694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_parquet(INPUT_PATH, columns=cfg.COLS_TO_USE)\ny = pd.read_csv(TARGET_PATH)\nsummarize(df, \"X\", 2); summarize(y, \"y\", 2)","metadata":{"execution":{"iopub.status.busy":"2023-04-02T15:04:45.954251Z","iopub.execute_input":"2023-04-02T15:04:45.954955Z","iopub.status.idle":"2023-04-02T15:04:56.344738Z","shell.execute_reply.started":"2023-04-02T15:04:45.954917Z","shell.execute_reply":"2023-04-02T15:04:56.343609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = drop_multi_game_naive(df)\ndf[\"lvgp_order\"] = map_lvgp_order(df[\"level_group\"])\nif check_multi_game(df):\n    print(\"There exist multiple game plays in at least one session.\")","metadata":{"execution":{"iopub.status.busy":"2023-04-02T15:04:56.346748Z","iopub.execute_input":"2023-04-02T15:04:56.347222Z","iopub.status.idle":"2023-04-02T15:05:16.60565Z","shell.execute_reply.started":"2023-04-02T15:04:56.347175Z","shell.execute_reply":"2023-04-02T15:05:16.604503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### *1.2 Do Simple Feature Engineering*","metadata":{}},{"cell_type":"code","source":"def get_factorize_map(\n    values: Union[np.ndarray, pd.Series, pd.Index],\n    sort: bool = True\n) -> Dict[str, int]:\n    \"\"\"Factorize array and return numeric representation map.\n\n    Parameters:\n        values: 1-D sequence to factorize\n        sort: whether to sort unique values\n\n    Return:\n        val2code: mapping from value to numeric code\n    \"\"\"\n    if isinstance(values, np.ndarray):\n        values = pd.Series(values)\n\n    vals_uniq = values.unique()\n    if sort:\n        vals_uniq = sorted(vals_uniq)\n    val2code = {val: code for code, val in enumerate(vals_uniq)}\n\n    return val2code","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-02T15:05:16.607368Z","iopub.execute_input":"2023-04-02T15:05:16.607717Z","iopub.status.idle":"2023-04-02T15:05:16.61478Z","shell.execute_reply.started":"2023-04-02T15:05:16.607679Z","shell.execute_reply":"2023-04-02T15:05:16.613633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Generate the only numeric feature\ndf[\"et_diff\"] = df.groupby(\"session_id\", sort=False).apply(lambda x: x[\"elapsed_time\"].diff().fillna(0)).values\ndf[\"et_diff\"] = df[\"et_diff\"].clip(0, 3.6e6)\n\n# Factorize categorical features\ndf[\"event_comb\"] = df[\"event_name\"] + \"_\" + df[\"name\"]\nfor cat_feat in cfg.CAT_FEATS:\n    orig_col = cat_feat[:-5]\n    cat2code = get_factorize_map(df[orig_col])\n    df[cat_feat] = df[orig_col].map(cat2code)\n    \n    with open(f\"./{orig_col}2code.pkl\", \"wb\") as f:\n        pickle.dump(cat2code, f)\n        \nX = df[[\"session_id\", \"level_group\", \"level\"] + cfg.FEATS]\ndel df; _ = gc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-04-02T15:05:16.618334Z","iopub.execute_input":"2023-04-02T15:05:16.618771Z","iopub.status.idle":"2023-04-02T15:05:37.29095Z","shell.execute_reply.started":"2023-04-02T15:05:16.618733Z","shell.execute_reply":"2023-04-02T15:05:37.289859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### *1.3 Process Labels*","metadata":{}},{"cell_type":"code","source":"y[\"q\"] = y[\"session_id\"].apply(lambda x: x.split(\"_q\")[1]).astype(int)\ny[\"session_id\"] = y[\"session_id\"].apply(lambda x: x.split(\"_q\")[0]).astype(int)\ny = y.sort_values([\"session_id\", \"q\"]).reset_index(drop=True)\nqn2lvgp = {qn: lv_gp for lv_gp, qns in QNS_PER_LV_GP.items() for qn in qns}\ny[\"level_group\"] = y[\"q\"].map(qn2lvgp)\ny_lvgp = y.groupby([\"session_id\", \"level_group\"]).apply(lambda x: list(x[\"correct\"])).reset_index()\ny_lvgp[\"lvgp_order\"] = y_lvgp[\"level_group\"].map(LVGP_ORDER)\ny_lvgp = y_lvgp.sort_values([\"session_id\", \"lvgp_order\"]).reset_index(drop=True)\ny_lvgp.rename({0: \"correct\"}, axis=1, inplace=True)\ny_lvgp.drop([\"lvgp_order\"], axis=1, inplace=True)\n\nprint(\"=====Labels Aggregated by `level_group`=====\")\ny_lvgp.head(3)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-02T15:05:37.292394Z","iopub.execute_input":"2023-04-02T15:05:37.292785Z","iopub.status.idle":"2023-04-02T15:05:39.194969Z","shell.execute_reply.started":"2023-04-02T15:05:37.292747Z","shell.execute_reply":"2023-04-02T15:05:39.194041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_all = y_lvgp.groupby(\"session_id\").apply(lambda x: list(x[\"correct\"])).reset_index()\nans_tmp = np.array(list(y_all.apply(lambda x: np.concatenate(x[0]), axis=1).values))\ny_all.drop(0, axis=1, inplace=True)\ny_all[list(range(18))] = ans_tmp\ny_all = y_all.set_index(\"session_id\").sort_index()\n\nprint(\"=====Flattened Labels=====\")\ny_all.head(3)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-02T15:05:39.196567Z","iopub.execute_input":"2023-04-02T15:05:39.196923Z","iopub.status.idle":"2023-04-02T15:05:39.836199Z","shell.execute_reply.started":"2023-04-02T15:05:39.196886Z","shell.execute_reply":"2023-04-02T15:05:39.835115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del y; _ = gc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-04-02T15:05:39.837736Z","iopub.execute_input":"2023-04-02T15:05:39.838377Z","iopub.status.idle":"2023-04-02T15:05:40.007684Z","shell.execute_reply.started":"2023-04-02T15:05:39.838338Z","shell.execute_reply":"2023-04-02T15:05:40.006511Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"dataset\"></a>\n## 2. Define Dataset\n[**<span style=\"color:#FEF1FE; background-color:#535d70;border-radius: 5px; padding: 2px\">Go to Table of Content</span>**](#toc)","metadata":{}},{"cell_type":"code","source":"class LvGpDataset(Dataset):\n    \"\"\"Dataset for `level_group`-wise modeling.\n\n    Input data are all from the same `level_group`. Also, if `t_window`\n    isn't long enough, only raw features from the last level of a\n    `level_group` is retrieved.\n\n    Commonly used notations are summarized as follows:\n    * `N` denotes the number of samples\n    * `P` denotes the length of lookback time window\n    * `Q` denotes the number of questions in the current level group\n    * `C` denotes the number of channels (numeric features)\n        *Note: Currently, only `et_diff` is used as the numeric feature\n    * `M` denotes the number of categorical features\n\n    Each sample is session-specific, which is illustrated as follows:\n\n    Let C=1 (i.e., only `et_diff` is used as the feature) and P=60, we\n    have data appearance like:\n\n    idx  t-59  t-58  ...  t-1  t\n    0     1     2          0   2   -> session_id == 20090312431273200\n    1     2     3          4   1   -> session_id == 20090312433251036\n    .\n    .\n    .\n    N-2\n    N-1\n\n    Parameters:\n        data: input data\n        t_window: length of lookback time window\n    \"\"\"\n\n    n_samples: int\n\n    def __init__(\n        self,\n        data: Tuple[pd.DataFrame, pd.DataFrame],\n        t_window: int,\n        **kwargs: Any,\n    ) -> None:\n        self.X_base, self.y_base = data\n        #         self.y_base[\"correct\"] = self.y_base[\"correct\"].apply(lambda x: ast.literal_eval(x))\n        self.t_window = t_window\n        \n        # Specify features to use\n        self.feats = [feat for feat in cfg.FEATS if feat not in cfg.CAT_FEATS]\n        self.cat_feats = cfg.CAT_FEATS\n\n        # Setup level-related metadata\n        self.lv_gp = self.X_base[\"level_group\"].unique()[0]\n        self.levels = LV_PER_LV_GP[self.lv_gp]\n        self.n_lvs = len(self.levels)\n\n        # Generate data samples\n        self._chunk_X_y()\n        self._set_n_samples()\n\n    def __len__(self) -> int:\n        return self.n_samples\n\n    def __getitem__(self, idx: int) -> Dict[str, Tensor]:\n        X = self.X[idx, ...]\n        y = self.y[idx, ...]\n        X_cat = self.X_cat[idx, ...]\n        \n        data_sample = {\n            \"X\": torch.tensor(X, dtype=torch.float32),\n            \"X_cat\": torch.tensor(X_cat, dtype=torch.int32),\n            \"y\": torch.tensor(y, dtype=torch.float32),\n        }\n\n        return data_sample\n\n    def _set_n_samples(self) -> None:\n        \"\"\"Derive the number of samples.\"\"\"\n        self.n_samples = len(self.X)\n\n    def _chunk_X_y(self) -> None:\n        \"\"\"Chunk data samples.\"\"\"\n        X, y = [], []\n        X_cat = []\n\n        for sess_id in self.y_base[\"session_id\"].values:\n            # Target columns hard-coded temporarily\n            X_sess = self.X_base[self.X_base[\"session_id\"] == sess_id]\n            \n            pad_len = 0\n            x_num = []\n            for i, feat in enumerate(self.feats):\n                x_num_i = X_sess[feat].values[-self.t_window :]\n                if i == 0 and len(x_num_i) < self.t_window:\n                    pad_len = self.t_window - len(x_num_i)\n                \n                if pad_len != 0:\n                    x_num_i = np.pad(x_num_i, (pad_len, 0), \"constant\")\n                    \n                x_num.append(x_num_i)\n            \n            x_cat = X_sess[self.cat_feats].values[-self.t_window :]\n            if pad_len != 0:\n                x_cat = np.pad(x_cat, ((pad_len, 0), (0, 0)), \"constant\", constant_values=-1)  # (P, M)\n\n            x_num = np.stack(x_num, axis=1)   # (P, C)\n            X.append(x_num)\n            X_cat.append(x_cat)\n            y.append(self.y_base[self.y_base[\"session_id\"] == sess_id][\"correct\"].values[0])\n\n        self.X = np.stack(X)  # (N, P, C)\n        self.X_cat = np.stack(X_cat)  # (N, P, M)\n        self.y = np.vstack(y)  # (N, Q)","metadata":{"execution":{"iopub.status.busy":"2023-04-02T15:05:40.010272Z","iopub.execute_input":"2023-04-02T15:05:40.011017Z","iopub.status.idle":"2023-04-02T15:05:40.027845Z","shell.execute_reply.started":"2023-04-02T15:05:40.010976Z","shell.execute_reply":"2023-04-02T15:05:40.026815Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def build_dataloaders(\n    data_tr: Tuple[pd.DataFrame, pd.DataFrame],\n    data_val: Tuple[pd.DataFrame, pd.DataFrame],\n    batch_size: int,\n    **dataset_cfg: Any,\n) -> Tuple[DataLoader, Optional[DataLoader]]:\n    \"\"\"Create and return train and validation dataloaders.\n\n    Parameters:\n        data_tr: training data\n        data_val: validation data\n        dataloader_cfg: hyperparameters of dataloader\n        dataset_cfg: hyperparameters of customized dataset\n\n    Return:\n        train_loader: training dataloader\n        val_loader: validation dataloader\n    \"\"\"\n    if data_tr is not None:\n        train_loader = DataLoader(\n            LvGpDataset(data_tr, **dataset_cfg),\n            batch_size=batch_size,\n            shuffle=True,\n            num_workers=2,\n            collate_fn=None,\n        )\n    else:\n        train_loader = None\n    if data_val is not None:\n        val_loader = DataLoader(\n            LvGpDataset(data_val, **dataset_cfg),\n            batch_size=batch_size,\n            shuffle=False,  # Hard-coded\n            num_workers=2,\n            collate_fn=None,\n        )\n    else:\n        val_loader = None\n\n    return train_loader, val_loader","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-02T15:05:40.030244Z","iopub.execute_input":"2023-04-02T15:05:40.030923Z","iopub.status.idle":"2023-04-02T15:05:40.039437Z","shell.execute_reply.started":"2023-04-02T15:05:40.030887Z","shell.execute_reply":"2023-04-02T15:05:40.038426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"model\"></a>\n## 3. Define Model Architecture\n[**<span style=\"color:#FEF1FE; background-color:#535d70;border-radius: 5px; padding: 2px\">Go to Table of Content</span>**](#toc)\n\nFollowing is the overview of the model architecture,\n\n[![2023-04-02-10-39-43.png](https://i.postimg.cc/2y9chX1n/2023-04-02-10-39-43.png)](https://postimg.cc/wRJQd2cB)","metadata":{}},{"cell_type":"code","source":"class TConvLayer(nn.Module):\n    \"\"\"Dilated temporal convolution layer.\n    \n    Considering the time cost, I currently disable dilation.\n    \"\"\"\n    \n    def __init__(\n        self,\n        in_dim: int,\n        out_dim: int,\n        kernel_size: int,\n        dilation: int,\n        bias: bool = True,\n        act: str = \"relu\",\n        dropout: float = 0.1,\n    ):\n        super(TConvLayer, self).__init__()\n        \n        # Network parameters\n        self.in_dim = in_dim\n        self.out_dim = out_dim\n        self.kernel_size = kernel_size\n        self.dilation = dilation\n        self.bias = bias\n        \n        # Model blocks\n        self.conv = nn.utils.weight_norm(\n            nn.Conv1d(in_dim, out_dim, kernel_size, dilation=dilation, bias=bias),\n            dim=None\n        )\n        self.bn = nn.BatchNorm1d(out_dim)\n        if act == \"relu\":\n            self.act = nn.ReLU()\n        if dropout == 0:\n            self.dropout = None\n        else:\n            self.dropout = nn.Dropout(dropout)\n            \n    def forward(self, x: Tensor) -> Tensor:\n        \"\"\"Forward pass.\n        \n        Shape:\n            x: (B, C', P)\n        \"\"\"\n        x = self.conv(x)\n        x = self.bn(x)\n        x = self.act(x)\n        if self.dropout is not None:\n            x = self.dropout(x)\n            \n        return x\n    \n\nclass EventAwareEncoder(nn.Module):\n    \"\"\"Event-aware encoder based on 1D-Conv.\"\"\"\n    \n    def __init__(\n        self,\n        h_dim: int = 128,\n        out_dim: int = 128,\n        readout: bool = True,\n        cat_feats: List[str] = [\"event_comb_code\", \"room_fqid_code\"]\n    ):\n        super(EventAwareEncoder, self).__init__()\n\n        # Network parameters\n        self.h_dim = h_dim\n        self.out_dim = out_dim\n        self.cat_feats = cat_feats\n\n        # Model blocks\n        # Categorical embeddings\n        self.embs = nn.ModuleList()\n        for cat_feat in cat_feats:\n            self.embs.append(nn.Embedding(CAT_FEAT_SIZE[cat_feat] + 1, 32, padding_idx=0))\n        self.emb_enc = nn.Sequential(\n            nn.Linear(64, 128),\n            nn.ReLU(),\n            nn.Linear(128, 64),\n            nn.ReLU(),\n        )\n        self.dropout = nn.Dropout(0.2)\n        # Feature extractor\n        self.convs = nn.ModuleList()\n        for l, (dilation, kernel_size) in enumerate(zip([2**i for i in range(3)], [7, 7, 5])):\n            self.convs.append(TConvLayer(64, h_dim, kernel_size, dilation=1))   # No dilation\n        # Readout layer\n        if readout:\n            self.readout = nn.Sequential(\n                nn.Linear(2 * (h_dim // 2), out_dim),\n                nn.ReLU(),\n                nn.Dropout(0.2),\n            )\n        else:\n            self.readout = None\n\n    def forward(self, x: Tensor, x_cat: Tensor) -> Tensor:\n        \"\"\"Forward pass.\n\n        Shape:\n            x: (B, P, C)\n            x_cat: (B, P, M)\n        \"\"\"\n\n        # Categorical embeddings\n        x_cat = x_cat + 1\n        x_emb = []\n        for i in range(len(self.cat_feats)):\n            x_emb.append(self.embs[i](x_cat[..., i]))  # (B, P, emb_dim)\n        x_emb = torch.cat(x_emb, dim=-1)  # (B, P, C')\n        x_emb = self.emb_enc(x_emb) + x_emb  # (B, P, C')\n        x = x * x_emb  # (B, P, C')\n        x = self.dropout(x)\n        \n        # Feature extractor\n        x = x.transpose(1, 2)  # (B, C', P)\n        x_skip = []\n        for l in range(3):\n            x_conv = self.convs[l](x)   # (B, C' * 2, P')\n            x_filter, x_gate = torch.split(x_conv, x_conv.size(1) // 2, dim=1)\n            x_conv = F.tanh(x_filter) * F.sigmoid(x_gate)   # (B, C', P')\n            \n            x_conv = self.dropout(x_conv)\n            \n            # Skip connection\n            x_skip.append(x_conv.unsqueeze(dim=1))  # (B, L (1), C', P')\n            \n            x = x_conv\n            \n        # Process skipped latent representation\n        for l in range(3-1):\n            x_skip[l] = x_skip[l][..., -x_skip[-1].size(3) : ]\n        x_skip = torch.cat(x_skip, dim=1)   # (B, L, C', P_truc)\n        x_skip = torch.sum(x_skip, dim=1)   # (B, C', P_truc)\n\n        # Readout layer\n        if self.readout is not None:\n            x_std = torch.std(x_skip, dim=-1)  # Std pooling\n            x_mean = torch.mean(x_skip, dim=-1)  # Mean pooling\n            x = torch.cat([x_std, x_mean], dim=1)\n            x = self.readout(x)  # (B, out_dim)\n\n        return x\n\n\nclass EventConvSimple(nn.Module):\n\n    def __init__(self, n_lvs: int, out_dim: int, **model_cfg: Any):\n        self.name = self.__class__.__name__\n        super(EventConvSimple, self).__init__()\n\n        enc_out_dim = 128\n\n        # Network parameters\n        self.n_lvs = n_lvs\n        self.out_dim = out_dim\n        self.cat_feats = model_cfg[\"cat_feats\"]\n        \n        self.encoder = EventAwareEncoder(h_dim=128, out_dim=enc_out_dim, cat_feats=self.cat_feats)\n        self.clf = nn.Sequential(\n            nn.Linear(enc_out_dim, enc_out_dim // 2),\n            nn.ReLU(),\n            nn.Dropout(0.2),\n            nn.Linear(enc_out_dim // 2, out_dim),\n        )\n\n    def forward(self, x: Tensor, x_cat: Tensor) -> Tensor:\n        \"\"\"Forward pass.\n\n        Shape:\n            x: (B, P, C)\n            x_cat: (B, P, M)\n        \"\"\"\n        x = self.encoder(x, x_cat)\n        x = self.clf(x)\n\n        return x","metadata":{"execution":{"iopub.status.busy":"2023-04-02T15:05:40.044534Z","iopub.execute_input":"2023-04-02T15:05:40.044812Z","iopub.status.idle":"2023-04-02T15:05:40.069114Z","shell.execute_reply.started":"2023-04-02T15:05:40.044787Z","shell.execute_reply":"2023-04-02T15:05:40.067845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"evaluator\"></a>\n## 4. Define Evaluator\n[**<span style=\"color:#FEF1FE; background-color:#535d70;border-radius: 5px; padding: 2px\">Go to Table of Content</span>**](#toc)","metadata":{}},{"cell_type":"code","source":"class Evaluator(object):\n    \"\"\"Evaluator.\n\n    Parameters:\n        metric_names: evaluation metrics\n        n_qns: number of questions\n    \"\"\"\n\n    eval_metrics: Dict[str, Callable[..., Union[float]]]\n\n    def __init__(self, metric_names: List[str], n_qns: int):\n        self.metric_names = metric_names\n        self.n_qns = n_qns\n        self.eval_metrics = {}\n        self._build()\n\n    def evaluate(\n        self,\n        y_true: Tensor,\n        y_pred: Tensor,\n    ) -> Dict[str, float]:\n        \"\"\"Run evaluation using pre-specified metrics.\n\n        Parameters:\n            y_true: groundtruths\n            y_pred: predicting results\n\n        Return:\n            eval_result: evaluation performance report\n        \"\"\"\n        eval_result = {}\n        for metric_name, metric in self.eval_metrics.items():\n            if metric_name == \"f1\":\n                for thres in [0.63]:\n                    eval_result[f\"{metric_name}@{thres}\"] = metric(y_pred, y_true, thres)\n            else:\n                eval_result[metric_name] = metric(y_pred, y_true)\n\n        return eval_result\n\n    def _build(self) -> None:\n        \"\"\"Build evaluation metric instances.\"\"\"\n        for metric_name in self.metric_names:\n            if metric_name == \"auroc\":\n                self.eval_metrics[metric_name] = self._AUROC\n            elif metric_name == \"f1\":\n                self.eval_metrics[metric_name] = self._F1\n\n    def _AUROC(self, y_pred: Tensor, y_true: Tensor) -> float:\n        \"\"\"Area Under the Receiver Operating Characteristic curve.\n\n        Parameters:\n            y_pred: predicting results\n            y_true: groundtruths\n\n        Return:\n            auroc: area under the receiver operating characteristic\n                curve\n        \"\"\"\n        metric = AUROC(task=\"multilabel\", num_labels=self.n_qns)\n        _ = metric(y_pred, y_true.int())\n        auroc = metric.compute().item()\n        metric.reset()\n\n        return auroc\n\n    def _F1(self, y_pred: Tensor, y_true: Tensor, thres: float) -> float:\n        \"\"\"F1 score.\n\n        Parameters:\n            y_pred: predicting results\n            y_true: groundtruths\n            thres: threshold to convert probability to bool\n\n        Return:\n            f1: F1 score\n        \"\"\"\n        y_pred = (y_pred.numpy().reshape(-1, ) > thres).astype(\"int\")\n        y_true = y_true.numpy().reshape(-1, )\n        f1 = f1_score(y_true, y_pred, average=\"macro\")\n\n        return f1","metadata":{"execution":{"iopub.status.busy":"2023-04-02T15:05:40.071153Z","iopub.execute_input":"2023-04-02T15:05:40.071923Z","iopub.status.idle":"2023-04-02T15:05:40.084207Z","shell.execute_reply.started":"2023-04-02T15:05:40.071881Z","shell.execute_reply":"2023-04-02T15:05:40.083446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"trainer\"></a>\n## 5. Define Trainer\n[**<span style=\"color:#FEF1FE; background-color:#535d70;border-radius: 5px; padding: 2px\">Go to Table of Content</span>**](#toc)","metadata":{}},{"cell_type":"code","source":"class BaseTrainer:\n    \"\"\"Base class for all customized trainers.\n\n    Parameters:\n        cfg: experiment configuration\n        model: model instance\n        loss_fn: loss criterion\n        optimizer: optimization algorithm\n        lr_skd: learning rate scheduler\n        evaluator: task-specific evaluator\n    \"\"\"\n\n    def __init__(\n        self,\n        cfg: Type[CFG],\n        model: Module,\n        loss_fn: _Loss,\n        optimizer: Optimizer,\n        lr_skd: _LRScheduler,\n        evaluator: Evaluator,\n    ):\n        self.cfg = cfg\n        self.model = model\n        self.loss_fn = loss_fn\n        self.optimizer = optimizer\n        self.lr_skd = lr_skd\n        self.evaluator = evaluator\n\n        self.device = cfg.DEVICE\n        self.epochs = cfg.EPOCH\n\n        # Model checkpoint\n        self.ckpt_metric = cfg.CKPT_METRIC\n\n        self._iter = 0\n        self._track_best_model = True\n\n    def train_eval(self, proc_id: Union[str, int]) -> Tuple[Module, Dict[str, Tensor]]:\n        \"\"\"Run training and evaluation processes for either one fold or\n        one random seed (commonly used when training on whole dataset).\n\n        Parameters:\n            proc_id: identifier of the current process, indicating\n                current fold number or random seed.\n\n        Return:\n            best_model: model instance with the best monitored\n                objective (e.g., the lowest validation loss)\n            y_preds: inference results on different datasets\n        \"\"\"\n        best_val_loss = 1e18\n        best_epoch = 0\n        try:\n            best_model = deepcopy(self.model)\n        except RuntimeError as e:\n            best_model = None\n            self._track_best_model = False\n            print(\"In-memoey best model tracking is disabled.\")\n\n        for epoch in range(self.epochs):\n            train_loss = self._train_epoch()\n            val_loss, val_result, _ = self._eval_epoch()\n\n            # Adjust learning rate\n            if self.lr_skd is not None:\n                if isinstance(self.lr_skd, lr_scheduler.ReduceLROnPlateau):\n                    self.lr_skd.step(val_loss)\n                else:\n                    self.lr_skd.step()\n\n            # Track and log process result\n            val_metric_msg = \"\"\n            for metric, score in val_result.items():\n                val_metric_msg += f\"{metric.upper()} {round(score, 4)} | \"\n            print(f\"Epoch{epoch} | Training loss {train_loss:.4f} | Validation loss {val_loss:.4f} | {val_metric_msg}\")\n\n            # Record the best checkpoint\n            ckpt_metric_val = val_result[self.ckpt_metric]\n            ckpt_metric_val = -ckpt_metric_val\n            if ckpt_metric_val < best_val_loss:\n                print(f\"Validation performance improves at epoch {epoch}!!\")\n                best_val_loss = ckpt_metric_val\n                if self._track_best_model:\n                    best_model = deepcopy(self.model)\n                else:\n                    self._save_ckpt()\n                best_epoch = epoch\n\n        # Run final evaluation\n        if not self._track_best_model:\n            self._load_best_ckpt()\n            best_model = self.model\n        else:\n            self.model = best_model\n        final_prf_report, y_preds = self._eval_with_best()\n        self._log_best_prf(final_prf_report)\n\n        return best_model, y_preds\n\n    @abstractmethod\n    def _train_epoch(self) -> Union[float, Dict[str, float]]:\n        \"\"\"Run training process for one epoch.\n\n        Return:\n            train_loss_avg: average training loss over batches\n                *Note: If multitask is used, returned object will be\n                    a dictionary containing losses of subtasks and the\n                    total loss.\n        \"\"\"\n        raise NotImplementedError\n\n    @abstractmethod\n    def _eval_epoch(\n        self,\n        return_output: bool = False,\n        test: bool = False,\n    ) -> Tuple[float, Dict[str, float], Optional[Tensor]]:\n        \"\"\"Run evaluation process for one epoch.\n\n        Parameters:\n            return_output: whether to return inference result of model\n            test: if evaluation is run on test set, set it to True\n                *Note: The setting is mainly used to disable DAE doping\n                    during test phase.\n\n        Return:\n            eval_loss_avg: average evaluation loss over batches\n            eval_result: evaluation performance report\n            y_pred: inference result\n        \"\"\"\n        raise NotImplementedError\n\n    def _eval_with_best(self) -> Tuple[Dict[str, Dict[str, float]], Dict[str, Tensor]]:\n        \"\"\"Run final evaluation process with the best checkpoint.\n\n        Return:\n            final_prf_report: performance report of final evaluation\n            y_preds: inference results on different datasets\n        \"\"\"\n        final_prf_report = {}\n        y_preds = {}\n\n        self._disable_shuffle()\n        dataloaders = {\"train\": self.train_loader}\n        if self.eval_loader is not None:\n            dataloaders[\"val\"] = self.eval_loader\n\n        for datatype, dataloader in dataloaders.items():\n            self.eval_loader = dataloader\n            eval_loss, eval_result, y_pred = self._eval_epoch(return_output=True)\n            final_prf_report[datatype] = eval_result\n            y_preds[datatype] = y_pred\n\n        return final_prf_report, y_preds\n\n    def _disable_shuffle(self) -> None:\n        \"\"\"Disable shuffle in train dataloader for final evaluation.\"\"\"\n        self.train_loader = DataLoader(\n            self.train_loader.dataset,\n            batch_size=self.train_loader.batch_size,\n            shuffle=False,  # Reset shuffle to False\n            num_workers=self.train_loader.num_workers,\n            collate_fn=self.train_loader.collate_fn,\n        )\n\n    def _save_ckpt(self, proc_id: int = 0, save_best_only: bool = True) -> None:\n        \"\"\"Save checkpoints.\n\n        Parameters:\n            proc_id: identifier of the current process, indicating\n                current fold number or random seed\n            save_best_only: only checkpoint of the best epoch is saved\n\n        Return:\n            None\n        \"\"\"\n        torch.save(self.model.state_dict(), \"model_tmp.pt\")\n\n    def _load_best_ckpt(self, proc_id: int = 0) -> None:\n        \"\"\"Load the best model checkpoint for final evaluation.\n\n        The best checkpoint is loaded and assigned to `self.model`.\n\n        Parameters:\n            proc_id: identifier of the current process, indicating\n                current fold number or random seed\n\n        Return:\n            None\n        \"\"\"\n        device = torch.device(self.device)\n        self.model.load_state_dict(\n            torch.load(\"model_tmp.pt\", map_location=device)\n        )\n        self.model = self.model.to(device)\n\n    def _log_best_prf(self, prf_report: Dict[str, Any]) -> None:\n        \"\"\"Log performance evaluated with the best model checkpoint.\n\n        Parameters:\n            prf_report: performance report\n\n        Return:\n            None\n        \"\"\"\n        import json\n\n        print(\">>>>> Performance Report - Best Ckpt <<<<<\")\n        print(json.dumps(prf_report, indent=4))","metadata":{"execution":{"iopub.status.busy":"2023-04-02T15:05:40.085569Z","iopub.execute_input":"2023-04-02T15:05:40.086172Z","iopub.status.idle":"2023-04-02T15:05:40.108732Z","shell.execute_reply.started":"2023-04-02T15:05:40.086137Z","shell.execute_reply":"2023-04-02T15:05:40.108002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class MainTrainer(BaseTrainer):\n    \"\"\"Main trainer.\n\n    Parameters:\n        cfg: experiment configuration\n        model: model instance\n        loss_fn: loss criterion\n        optimizer: optimization algorithm\n        lr_scheduler: learning rate scheduler\n        train_loader: training data loader\n        eval_loader: validation data loader\n    \"\"\"\n\n    def __init__(\n        self,\n        cfg: Type[CFG],\n        model: Module,\n        loss_fn: _Loss,\n        optimizer: Optimizer,\n        lr_skd: _LRScheduler,\n        evaluator: Evaluator,\n        train_loader: DataLoader,\n        eval_loader: DataLoader,\n    ):\n        super(MainTrainer, self).__init__(\n            cfg, model, loss_fn, optimizer, lr_skd, evaluator\n        )\n        self.train_loader = train_loader\n        self.eval_loader = eval_loader if eval_loader else train_loader\n\n    def _train_epoch(self) -> float:\n        \"\"\"Run training process for one epoch.\n\n        Return:\n            train_loss_avg: average training loss over batches\n        \"\"\"\n        train_loss_total = 0\n\n        self.model.train()\n        for i, batch_data in enumerate(tqdm(self.train_loader)):\n            self.optimizer.zero_grad(set_to_none=True)\n\n            # Retrieve batched raw data\n            x = batch_data[\"X\"].to(self.device)\n            x_cat = batch_data[\"X_cat\"].to(self.device)\n            y = batch_data[\"y\"].to(self.device)\n\n            # Forward pass\n            output = self.model(x, x_cat)\n            self._iter += 1\n\n            # Derive loss\n            loss = self.loss_fn(output, y)\n\n            # Backpropagation\n            loss.backward()\n            self.optimizer.step()\n\n            train_loss_total += loss.item()\n\n            # Free mem.\n            del x, y, output\n            _ = gc.collect()\n\n        train_loss_avg = train_loss_total / len(self.train_loader)\n\n        return train_loss_avg\n\n    @torch.no_grad()\n    def _eval_epoch(\n        self,\n        return_output: bool = False,\n        test: bool = False,\n    ) -> Tuple[float, Dict[str, float], Optional[Tensor]]:\n        \"\"\"Run evaluation process for one epoch.\n\n        Parameters:\n            return_output: whether to return inference result of model\n            test: always ignored, exists for compatibility\n\n        Return:\n            eval_loss_avg: average evaluation loss over batches\n            eval_result: evaluation performance report\n            y_pred: inference result\n        \"\"\"\n        eval_loss_total = 0\n        y_true = None\n        y_pred = None\n\n        self.model.eval()\n        for i, batch_data in enumerate(self.eval_loader):\n            # Retrieve batched raw data\n            x = batch_data[\"X\"].to(self.device)\n            x_cat = batch_data[\"X_cat\"].to(self.device)\n            y = batch_data[\"y\"].to(self.device)\n\n            # Forward pass\n            output = self.model(x, x_cat)\n\n            # Derive loss\n            loss = self.loss_fn(output, y)\n            eval_loss_total += loss.item()\n\n            # Record batched output\n            if i == 0:\n                y_true = y.detach().cpu()\n                y_pred = output.detach().cpu()\n            else:\n                # Avoid situ ation that batch_size is just equal to 1\n                y_true = torch.cat((y_true, y.detach().cpu()))\n                y_pred = torch.cat((y_pred, output.detach().cpu()))\n\n            del x, y, output\n            _ = gc.collect()\n\n        y_pred = F.sigmoid(y_pred)  # Tmp. workaround (because loss has built-in sigmoid)\n        eval_loss_avg = eval_loss_total / len(self.eval_loader)\n        eval_result = self.evaluator.evaluate(y_true, y_pred)\n\n        if return_output:\n            return eval_loss_avg, eval_result, y_pred\n        else:\n            return eval_loss_avg, eval_result, None","metadata":{"execution":{"iopub.status.busy":"2023-04-02T15:05:40.110015Z","iopub.execute_input":"2023-04-02T15:05:40.110637Z","iopub.status.idle":"2023-04-02T15:05:40.127147Z","shell.execute_reply.started":"2023-04-02T15:05:40.110601Z","shell.execute_reply":"2023-04-02T15:05:40.126174Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"cv\"></a>\n## 6. Run Cross-Validation\n[**<span style=\"color:#FEF1FE; background-color:#535d70;border-radius: 5px; padding: 2px\">Go to Table of Content</span>**](#toc)","metadata":{}},{"cell_type":"code","source":"if cfg.train:\n    sess_id = X[\"session_id\"].unique()\n\n    oof_pred = pd.DataFrame(np.zeros((len(sess_id), N_QNS)), index=sess_id)\n    cv = GroupKFold(n_splits=5)\n    for i, (tr_idx, val_idx) in enumerate(cv.split(X=X, groups=X[\"session_id\"])):\n        print(f\"Training and evaluation process of fold{i} starts...\")\n\n        # Prepare data\n        X_tr, X_val = X.iloc[tr_idx, :], X.iloc[val_idx, :]\n        sess_tr, sess_val = X_tr[\"session_id\"].unique(), X_val[\"session_id\"].unique()\n        y_tr, y_val = y_lvgp[y_lvgp[\"session_id\"].isin(sess_tr)], y_lvgp[y_lvgp[\"session_id\"].isin(sess_val)]\n\n        # Run level_group-wise modeling\n        oof_pred_fold = []\n        for lv_gp in LEVEL_GROUP:\n            print(f\"=====LEVEL GROUP {lv_gp}=====\")\n            qn_idx = QNS_PER_LV_GP[lv_gp]  # Question index\n            lvs = LV_PER_LV_GP[lv_gp]\n            X_tr_, X_val_ = X_tr[X_tr[\"level_group\"] == lv_gp], X_val[X_val[\"level_group\"] == lv_gp]\n            y_tr_, y_val_ = y_tr[y_tr[\"level_group\"] == lv_gp], y_val[y_val[\"level_group\"] == lv_gp]\n\n            # Build dataloader\n            train_loader, val_loader = build_dataloaders((X_tr_, y_tr_), (X_val_, y_val_), cfg.BATCH_SIZE, **{\"t_window\": cfg.T_WINDOW})\n\n            # Build model\n            model = EventConvSimple(len(lvs), len(qn_idx), **{\"cat_feats\": cfg.CAT_FEATS})\n            model.to(cfg.DEVICE)\n\n            # Build criterion\n            loss_fn = nn.BCEWithLogitsLoss()\n\n            # Build solvers\n            optimizer = optim.Adam(list(model.parameters()), lr=cfg.LR, weight_decay=cfg.WEIGHT_DECAY)\n            lr_skd = lr_scheduler.CosineAnnealingWarmRestarts(optimizer=optimizer, T_0=10, eta_min=1e-5, T_mult=1)\n\n            # Build evaluator\n            evaluator = Evaluator(cfg.EVAL_METRICS, len(qn_idx))\n\n            # Build trainer\n            trainer_cfg = {\n                \"cfg\": cfg,\n                \"model\": model,\n                \"loss_fn\": loss_fn,\n                \"optimizer\": optimizer,\n                \"lr_skd\": lr_skd,\n                \"evaluator\": evaluator,\n                \"train_loader\": train_loader,\n                \"eval_loader\": val_loader,\n            }\n            trainer = MainTrainer(**trainer_cfg)\n\n            # Run training and evaluation processes for one fold\n            best_model, best_preds = trainer.train_eval(lv_gp)\n            oof_pred_fold.append(best_preds[\"val\"])\n\n            # Dump output objects of the current fold\n            torch.save(best_model.state_dict(), f\"fold{i}_{lv_gp}\")\n\n            # Free mem.\n            del (X_tr_, X_val_, y_tr_, y_val_, train_loader, val_loader,\n                 model, loss_fn, optimizer, lr_skd, evaluator, trainer)\n            _ = gc.collect()\n\n        # Record oof prediction of the current fold\n        oof_pred.loc[sess_val, :] = torch.cat(oof_pred_fold, dim=1).numpy()\nelse:\n    oof_pred = pd.read_csv(\"/kaggle/input/0330-16-53-13/0330-16_53_13/preds/oof.csv\")\n    oof_pred.set_index(\"session\", inplace=True)\n    oof_pred.rename({\"session\": \"session_id\"}, axis=1, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2023-04-02T15:05:40.129025Z","iopub.execute_input":"2023-04-02T15:05:40.129652Z","iopub.status.idle":"2023-04-02T15:05:40.261249Z","shell.execute_reply.started":"2023-04-02T15:05:40.129617Z","shell.execute_reply":"2023-04-02T15:05:40.260254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"cv_score\"></a>\n## 7. Derive Final CV Score\n[**<span style=\"color:#FEF1FE; background-color:#535d70;border-radius: 5px; padding: 2px\">Go to Table of Content</span>**](#toc)","metadata":{}},{"cell_type":"code","source":" def derive_cv_score(\n     y_true: pd.DataFrame,\n     y_pred: pd.DataFrame,\n     return_traces: bool = False\n ) -> Tuple[float, float, Optional[Tuple[List[float], List[float]]]]:\n        \"\"\"Compute the fianl CV score (i.e., f1 score).\n\n        Parameters:\n            y_true: groundtruths\n            y_pred: predicting results\n            return_traces: if True, f1 scores at different thresholds\n                are returned\n\n        Return:\n            best_f1: final CV score at the best prob threshold\n            best_thres: threshold with the best CV score\n            f1s: f1 scores at different thresholds\n            thresholds: thresholds to convert probability to bool\n        \"\"\"\n        y_pred = y_pred.sort_index()\n        thres_range = np.arange(0.2, 0.81, 0.01)\n        \n        f1s, thresholds = [], []\n        best_f1 = 0.0\n        best_thres = 0.0\n        for thres in thres_range:\n            y_pred_bool = (y_pred.values.reshape(-1, ) > thres).astype(\"int\")\n            f1 = f1_score(y_true.values.reshape(-1, ), y_pred_bool, average=\"macro\")\n            f1s.append(f1)\n            thresholds.append(thres)\n\n            if f1 > best_f1:\n                best_f1 = f1\n                best_thres = thres\n\n        traces = (f1s, thresholds)\n\n        if return_traces:\n            return best_f1, best_thres, traces\n        else:\n            return best_f1, best_thres, None","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-02T15:05:40.262823Z","iopub.execute_input":"2023-04-02T15:05:40.263193Z","iopub.status.idle":"2023-04-02T15:05:40.272707Z","shell.execute_reply.started":"2023-04-02T15:05:40.263155Z","shell.execute_reply":"2023-04-02T15:05:40.271254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"best_f1, best_thres, (f1s, thresholds) = derive_cv_score(y_all, oof_pred, return_traces=True)\nfig, ax = plt.subplots(figsize=(14, 4))\nax.plot(thresholds, f1s, \"-o\", color=\"blue\")\nax.scatter([best_thres], [best_f1], color=\"red\", s=300, alpha=1)\nax.set_title(f\"Threshold vs. F1 with Best F1 = {best_f1:.4f} at Best Threshold = {best_thres:.3}\")\nax.set_xlabel(\"Threshold\", size=14)\nax.set_ylabel(\"CV Score (F1)\", size=14)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-02T15:05:40.274534Z","iopub.execute_input":"2023-04-02T15:05:40.275469Z","iopub.status.idle":"2023-04-02T15:05:44.043025Z","shell.execute_reply.started":"2023-04-02T15:05:40.275431Z","shell.execute_reply":"2023-04-02T15:05:44.042054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}