{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# ⚡⚡ PyTorch Quickstart for the American Express - Default Prediction competition\nThis notebook shows how to define and train a Pytorch LSTM to leverages the time series structure of the data.\n\nI expect Deep Learning models to dominate in this competition, so here's a simple LSTM architecture.\n\nParameters were not really tweaked so the baseline is improvable.\n\n**Please consider upvoting if you find this work helpful. Don't fork without upvoting !**\n\n","metadata":{}},{"cell_type":"markdown","source":"## Why PyTorch Lightning?\n\nLightning is simply organized PyTorch code. There's NO new framework to learn. For more details about Lightning visit the repo:\n\nhttps://github.com/PyTorchLightning/pytorch-lightning\n\nRun on CPU, GPU clusters or TPU, without any code changes\n\n","metadata":{}},{"cell_type":"markdown","source":"# Imports\n","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport gc\nimport numpy as np\n\n# Torch and Sklearn\nimport pytorch_lightning as pl\nimport torch\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.model_selection import train_test_split\nfrom torch.utils.data import DataLoader, TensorDataset\nfrom torchmetrics import Metric\nfrom pytorch_lightning.loggers import TensorBoardLogger\nfrom pytorch_lightning.callbacks import ModelCheckpoint\nfrom pytorch_lightning import Trainer\nimport torch.nn.functional as F\nfrom torch import nn, Tensor\nfrom torchmetrics.utilities import rank_zero_warn\n\n# Typing \nfrom typing import Optional","metadata":{"execution":{"iopub.status.busy":"2022-06-20T12:47:12.175119Z","iopub.execute_input":"2022-06-20T12:47:12.17543Z","iopub.status.idle":"2022-06-20T12:47:16.417577Z","shell.execute_reply.started":"2022-06-20T12:47:12.175398Z","shell.execute_reply":"2022-06-20T12:47:16.416553Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# File system\ntrain_file   = \"../input/amex-data-integer-dtypes-100k-cid-per-chunk/train_chunk_0.parquet\"\ntrain_labels = \"../input/amex-default-prediction/train_labels.csv\"\nmodel_output_folder = './experiment'\n\n# Data\nbatch_size   = 1028\nnum_workers  = 4\nepochs = 5\n\n# Model \nin_features=188\nhidden_dim=128\nnum_layers=2\nlearning_rate=1e-3","metadata":{"execution":{"iopub.status.busy":"2022-06-20T12:47:16.419867Z","iopub.execute_input":"2022-06-20T12:47:16.420245Z","iopub.status.idle":"2022-06-20T12:47:16.426521Z","shell.execute_reply.started":"2022-06-20T12:47:16.420198Z","shell.execute_reply":"2022-06-20T12:47:16.425895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data\n\nReading and preprocessing the data\n\nWe read the data from @raddar's [dataset](http://https://www.kaggle.com/datasets/raddar/amex-data-integer-dtypes-parquet-format) that i splitted into chunks [dataset](https://www.kaggle.com/datasets/what5up/amex-data-integer-dtypes-100k-cid-per-chunk). @raddar has denoised the data so that we can achieve better results with his dataset than with the original competition csv files.\n\nWe also convert the dataframe into a 3D-tensor dataset as highlighted by Chris Deotte [here](https://www.kaggle.com/competitions/amex-default-prediction/discussion/327828) \n\n","metadata":{}},{"cell_type":"code","source":"def load_train_df(train_file, train_labels):\n    train = pd.read_parquet(train_file)\n    train['S_2'] = pd.to_datetime(train['S_2'])\n    tmp = train[['customer_ID','S_2']].groupby('customer_ID').count()\n\n    missing_cids = []\n    for nb_available_rows in range(1, 14):\n        cids = tmp[tmp['S_2'] == nb_available_rows].index.values\n        batch_missing_cids = [cid for cid in cids for _ in range(13 - nb_available_rows)]\n        missing_cids.extend(batch_missing_cids)\n\n    train_part2 = train.iloc[:len(missing_cids)].copy()\n    train_part2.loc[:] = np.nan\n    train_part2['customer_ID'] = missing_cids\n\n    train = pd.concat([train_part2, train])\n    \n    train = train.sort_values('customer_ID')\n    \n    train_labels = pd.read_csv(train_labels)\n    train = pd.merge(train, train_labels, how='inner', on='customer_ID')\n    \n    train = train.sort_values('customer_ID')\n    return train\n\ntrain_df = load_train_df(train_file, train_labels)","metadata":{"execution":{"iopub.status.busy":"2022-06-20T12:47:16.427512Z","iopub.execute_input":"2022-06-20T12:47:16.428214Z","iopub.status.idle":"2022-06-20T12:47:33.582987Z","shell.execute_reply.started":"2022-06-20T12:47:16.428177Z","shell.execute_reply":"2022-06-20T12:47:33.582101Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class DataModule(pl.LightningDataModule):\n\n    def __init__(self, all_data: pd.DataFrame, batch_size: int = batch_size, num_workers: int = num_workers):\n        super().__init__()\n        self.all_data = all_data\n        self.batch_size = batch_size\n        self.num_workers = num_workers\n        self.sc = StandardScaler()\n\n    def prepare_data(self):\n        pass\n\n    def setup(self, stage=None):\n        # All data comumns except customer_ID, target, and S_2 are features\n        features = self.all_data.columns[2:-1]\n        self.all_data[features] = self.sc.fit_transform(self.all_data[features])\n        self.all_data[features] = self.all_data[features].fillna(0)\n        \n        # https://www.kaggle.com/competitions/amex-default-prediction/discussion/327828 !! Many Thanks @Chris Deotte for your sharing\n        all_tensor_x = torch.reshape(torch.tensor(self.all_data[features].to_numpy()), (-1, 13, 188)).float()\n        all_tensor_y = torch.tensor(self.all_data.groupby('customer_ID').first()['target'].to_numpy()).float()\n\n        X_trainval, X_test, y_trainval, y_test = train_test_split(all_tensor_x, all_tensor_y, test_size=0.1, random_state=1)\n        X_train, X_val, y_train, y_val = train_test_split(X_trainval, y_trainval, test_size=0.1, random_state=1)\n\n        # TRAIN\n        self.train_tensor = TensorDataset(X_train, y_train)\n        # VAL\n        self.val_tensor = TensorDataset(X_val, y_val)\n        # TEST\n        self.test_tensor = TensorDataset(X_test, y_test)\n\n    def train_dataloader(self):\n        return DataLoader(self.train_tensor, batch_size=self.batch_size, num_workers=self.num_workers)\n\n    def val_dataloader(self):\n        return DataLoader(self.val_tensor, batch_size=self.batch_size, num_workers=self.num_workers)\n\n    def test_dataloader(self):\n        return DataLoader(self.test_tensor, batch_size=self.batch_size, num_workers=self.num_workers)","metadata":{"execution":{"iopub.status.busy":"2022-06-20T12:47:33.585396Z","iopub.execute_input":"2022-06-20T12:47:33.585754Z","iopub.status.idle":"2022-06-20T12:47:33.59989Z","shell.execute_reply.started":"2022-06-20T12:47:33.585709Z","shell.execute_reply":"2022-06-20T12:47:33.598769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Metrics : [Implementation Source](https://www.kaggle.com/code/rohanrao/amex-competition-metric-implementations)","metadata":{}},{"cell_type":"code","source":"## https://www.kaggle.com/code/rohanrao/amex-competition-metric-implementations\n\nclass AmexMetric(Metric):\n    is_differentiable: Optional[bool] = False\n\n    # Set to True if the metric reaches it optimal value when the metric is maximized.\n    # Set to False if it when the metric is minimized.\n    higher_is_better: Optional[bool] = True\n\n    # Set to True if the metric during 'update' requires access to the global metric\n    # state for its calculations. If not, setting this to False indicates that all\n    # batch states are independent and we will optimize the runtime of 'forward'\n    full_state_update: bool = True\n\n    def __init__(self):\n        super().__init__()\n        \n        self.add_state(\"all_true\", default=[], dist_reduce_fx=\"cat\")\n        self.add_state(\"all_pred\", default=[], dist_reduce_fx=\"cat\")\n\n        rank_zero_warn(\n            \"Metric `Amex` will save all targets and predictions in buffer.\"\n            \" For large datasets this may lead to large memory footprint.\"\n        )\n\n    def update(self, y_pred: torch.Tensor, y_true: torch.Tensor):\n        \n        y_true = y_true.double()\n        y_pred = y_pred.double()\n        \n        self.all_true.append(y_true)\n        self.all_pred.append(y_pred)\n        \n    def compute(self):\n        y_true = torch.cat(self.all_true)\n        y_pred = torch.cat(self.all_pred)\n        # count of positives and negatives\n        n_pos = y_true.sum()\n        n_neg = y_pred.shape[0] - n_pos\n\n        # sorting by descring prediction values\n        indices = torch.argsort(y_pred, dim=0, descending=True)\n        preds, target = y_pred[indices], y_true[indices]\n\n        # filter the top 4% by cumulative row weights\n        weight = 20.0 - target * 19.0\n        cum_norm_weight = (weight / weight.sum()).cumsum(dim=0)\n        four_pct_filter = cum_norm_weight <= 0.04\n\n        # default rate captured at 4%\n        d = target[four_pct_filter].sum() / n_pos\n\n        # weighted gini coefficient\n        lorentz = (target / n_pos).cumsum(dim=0)\n        gini = ((lorentz - cum_norm_weight) * weight).sum()\n\n        # max weighted gini coefficient\n        gini_max = 10 * n_neg * (1 - 19 / (n_pos + 20 * n_neg))\n\n        # normalized weighted gini coefficient\n        g = gini / gini_max\n        \n        return 0.5 * (g + d)\n\n\ndef amex_metric(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n    def top_four_percent_captured(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n        df = (pd.concat([y_true, y_pred], axis='columns')\n              .sort_values('prediction', ascending=False))\n        df['weight'] = df['target'].apply(lambda x: 20 if x == 0 else 1)\n        four_pct_cutoff = int(0.04 * df['weight'].sum())\n        df['weight_cumsum'] = df['weight'].cumsum()\n        df_cutoff = df.loc[df['weight_cumsum'] <= four_pct_cutoff]\n        return (df_cutoff['target'] == 1).sum() / (df['target'] == 1).sum()\n\n    def weighted_gini(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n        df = (pd.concat([y_true, y_pred], axis='columns')\n              .sort_values('prediction', ascending=False))\n        df['weight'] = df['target'].apply(lambda x: 20 if x == 0 else 1)\n        df['random'] = (df['weight'] / df['weight'].sum()).cumsum()\n        total_pos = (df['target'] * df['weight']).sum()\n        df['cum_pos_found'] = (df['target'] * df['weight']).cumsum()\n        df['lorentz'] = df['cum_pos_found'] / total_pos\n        df['gini'] = (df['lorentz'] - df['random']) * df['weight']\n        return df['gini'].sum()\n\n    def normalized_weighted_gini(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n        y_true_pred = y_true.rename(columns={'target': 'prediction'})\n        return weighted_gini(y_true, y_pred) / weighted_gini(y_true, y_true_pred)\n\n    g = normalized_weighted_gini(y_true, y_pred)\n    d = top_four_percent_captured(y_true, y_pred)\n\n    return 0.5 * (g + d)","metadata":{"execution":{"iopub.status.busy":"2022-06-20T12:47:33.601168Z","iopub.execute_input":"2022-06-20T12:47:33.6014Z","iopub.status.idle":"2022-06-20T12:47:33.792869Z","shell.execute_reply.started":"2022-06-20T12:47:33.601373Z","shell.execute_reply":"2022-06-20T12:47:33.792169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model","metadata":{}},{"cell_type":"code","source":"class LSTMClassifier(nn.Module):\n    \"\"\"Very simple implementation of LSTM-based time-series classifier.\"\"\"\n\n    def __init__(self, input_dim, hidden_dim, num_layers, output_dim, device):\n        super().__init__()\n        self.num_layers = num_layers\n        self.hidden_dim = hidden_dim\n        self.rnn = nn.LSTM(input_dim, hidden_dim, num_layers, batch_first=True)\n        self.fc1 = nn.Linear(hidden_dim, 100)\n        self.fc2 = nn.Linear(100, output_dim)\n        self.device = device\n\n    def forward(self, x):\n        h0, c0 = self.init_hidden(x)\n        out, (_, _) = self.rnn(x, (h0, c0))\n        out = F.relu(self.fc1(out[:, -1, :]))\n        out = torch.sigmoid(self.fc2(out))\n        return out\n\n    def init_hidden(self, x):\n        batch_size = x.size(0)\n        h0 = torch.zeros(self.num_layers, batch_size, self.hidden_dim)\n        c0 = torch.zeros(self.num_layers, batch_size, self.hidden_dim)\n        if torch.cuda.is_available():\n            h0, c0 = h0.cuda(), c0.cuda()\n        return h0, c0\n\n\nclass TsLstmLightning(pl.LightningModule):\n    def __init__(self, in_features, hidden_dim, num_layers, learning_rate):\n        super(TsLstmLightning, self).__init__()\n\n        self.learning_rate = learning_rate\n\n        self.train_amex_metric = AmexMetric()\n        self.val_amex_metric   = AmexMetric()\n\n        self.model = LSTMClassifier(in_features, hidden_dim, num_layers, 1, device = self.device)\n\n        self.num_parameters = count_parameters(self.model)\n        \n        print(f\"Trainable params: {self.num_parameters:,}\")\n\n        self.loss_fn = nn.BCELoss(reduction=\"mean\")\n\n    def forward(self, x):\n        res = self.model(x)\n        return res\n\n    def training_step(self, batch, batch_idx):\n        X, target = batch\n        preds = self(X)  # (batch_size, 1)\n        preds = preds.squeeze(1)\n\n        loss = self.loss_fn(preds, target)\n        \n        self.train_amex_metric.update(preds, target) \n\n        self.log_dict({'train_loss': loss, 'train_amex_metric': self.train_amex_metric }, on_step=True, on_epoch=True, prog_bar=True, logger=True)\n\n        return {'loss': loss}\n\n    def validation_step(self, batch, batch_idx):\n        with torch.no_grad():\n            X, target = batch\n            preds = self(X)  \n            preds = preds.squeeze(1)\n\n            loss = self.loss_fn(preds, target)\n            \n            self.val_amex_metric.update(preds, target),\n\n            self.log_dict({'val_loss': loss, 'val_amex_metric': self.val_amex_metric}, on_step=True, on_epoch=True, prog_bar=True, logger=True)\n\n            return {'loss': loss}\n\n\n    def predict_step(self, batch, batch_idx, dataloader_idx=None):\n        with torch.no_grad():\n            X = batch[0]\n            preds = self(X)\n            return preds.detach().cpu()\n\n    def configure_optimizers(self):\n        optimizer = torch.optim.Adam(self.parameters(), lr=self.learning_rate)\n        lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=1)\n        return [optimizer], [lr_scheduler]\n\n\ndef count_parameters(model):\n    return sum(p.numel() for p in model.parameters() if p.requires_grad)","metadata":{"execution":{"iopub.status.busy":"2022-06-20T12:50:14.049043Z","iopub.execute_input":"2022-06-20T12:50:14.049474Z","iopub.status.idle":"2022-06-20T12:50:14.075078Z","shell.execute_reply.started":"2022-06-20T12:50:14.049425Z","shell.execute_reply":"2022-06-20T12:50:14.074129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training Loop ","metadata":{}},{"cell_type":"code","source":"print(f\"Train Shape: {train_df.shape}\")\ndm = DataModule(train_df, batch_size=batch_size)\n\nmodel = TsLstmLightning(in_features=in_features, hidden_dim=hidden_dim, num_layers=num_layers, learning_rate=learning_rate)\n\nlogger = TensorBoardLogger(model_output_folder, name=f\"logs\", default_hp_metric=True)\n\ncheckpoint_callback = ModelCheckpoint(dirpath= \"mycheckpoints\", save_top_k=1, save_weights_only=True, save_last=False, verbose=True,\n                                      monitor='val_loss_epoch', mode='min')\n\ncallbacks = [checkpoint_callback]\n\ntrainer = Trainer(\n    gpus=[0] if torch.cuda.is_available() else None,\n    max_epochs=epochs,\n    benchmark=False,\n    deterministic=True,\n    callbacks=callbacks,\n    logger=logger)\n\ntrainer.fit(model, dm)","metadata":{"execution":{"iopub.status.busy":"2022-06-20T12:50:14.944099Z","iopub.execute_input":"2022-06-20T12:50:14.945111Z","iopub.status.idle":"2022-06-20T12:55:18.397963Z","shell.execute_reply.started":"2022-06-20T12:50:14.945054Z","shell.execute_reply":"2022-06-20T12:55:18.396989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# TODO : TPU SUPPORT --> https://www.kaggle.com/code/justusschock/pytorch-on-tpu-with-pytorch-lightning/notebook\n# !curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py\n# !python pytorch-xla-env-setup.py --version 1.7 --apt-packages libomp5 libopenblas-dev","metadata":{"execution":{"iopub.status.busy":"2022-06-20T00:01:11.331313Z","iopub.execute_input":"2022-06-20T00:01:11.332114Z","iopub.status.idle":"2022-06-20T00:01:11.336949Z","shell.execute_reply.started":"2022-06-20T00:01:11.332065Z","shell.execute_reply":"2022-06-20T00:01:11.336308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train Monitoring\n","metadata":{}},{"cell_type":"code","source":"# Tensorboard temporarly disabled on Kaggle. https://www.kaggle.com/product-feedback/89671 --> Download the data and run it on your computer to follow the metrics\n%tensorboard --logdir model_output_folder","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Possible Nexts steps\n\n\n1. Have a best handling of missiing values in the data ( Ex : Do not drop customers that don't have 13 records,  do not fill N/A with 0, Replace previous -1 values with N/A, ...)\n2. Enhance the model (More LSTM / 1D CNN / Transformers / Param optimisation / ... ) \n3. Have a better understanding of the predictive features = Feature engineering ( feature selection or permutation feature importance, for instance.)\n4. (1. + 2.--> Transformer Unsupervised training with Times Series) https://arxiv.org/abs/2010.02803\n5. Model ensemble with other classification techniques\n6. Enable Cross-validation (Stratified K-Fold, ...)\n","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}