{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# ⚡⚡ PyTorch Quickstart for the American Express - Default Prediction competition\nThis notebook shows how to define and train a Pytorch LSTM to leverages the time series structure of the data.\n\nI expect Deep Learning models to dominate in this competition, so here's a simple LSTM architecture.\n\nParameters were not really tweaked so the baseline is improvable.\n\n**Please consider upvoting if you find this work helpful. Don't fork without upvoting !**\n\n","metadata":{}},{"cell_type":"markdown","source":"## Why PyTorch Lightning?\n\nLightning is simply organized PyTorch code. There's NO new framework to learn. For more details about Lightning visit the repo:\n\nhttps://github.com/PyTorchLightning/pytorch-lightning\n\nRun on CPU, GPU clusters or TPU, without any code changes\n\n","metadata":{}},{"cell_type":"markdown","source":"# Imports\n","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport gc\nimport numpy as np\nfrom tqdm.notebook import tqdm\n\n# Torch and Sklearn\nimport pytorch_lightning as pl\nimport torch\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.model_selection import train_test_split\nfrom torch.utils.data import DataLoader, TensorDataset\nfrom torchmetrics import Metric\nfrom pytorch_lightning.loggers import TensorBoardLogger\nfrom pytorch_lightning.callbacks import ModelCheckpoint\nfrom pytorch_lightning import Trainer\nimport torch.nn.functional as F\nfrom torch import nn, Tensor\nfrom torchmetrics.utilities import rank_zero_warn\n\n# Typing \nfrom typing import Optional","metadata":{"execution":{"iopub.status.busy":"2022-06-20T13:18:58.24329Z","iopub.execute_input":"2022-06-20T13:18:58.243902Z","iopub.status.idle":"2022-06-20T13:19:02.48156Z","shell.execute_reply.started":"2022-06-20T13:18:58.243811Z","shell.execute_reply":"2022-06-20T13:19:02.480593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# File system\ntest_files    = [f\"../input/amex-data-integer-dtypes-100k-cid-per-chunk/test_chunk_{i}.parquet\" for i in range(10)]\nmodel_weights = '../input/simple-amex-lstm-v2/epoch4-step394.ckpt'\n\n# Data\nbatch_size   = 1028\nnum_workers  = 4\n\n# Model \nin_features=188\nhidden_dim=128\nnum_layers=4\nlearning_rate=1e-3","metadata":{"execution":{"iopub.status.busy":"2022-06-20T13:19:02.483267Z","iopub.execute_input":"2022-06-20T13:19:02.483634Z","iopub.status.idle":"2022-06-20T13:19:02.489198Z","shell.execute_reply.started":"2022-06-20T13:19:02.483603Z","shell.execute_reply":"2022-06-20T13:19:02.48803Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data\n\nReading and preprocessing the data\n\nWe read the data from @raddar's [dataset](http://https://www.kaggle.com/datasets/raddar/amex-data-integer-dtypes-parquet-format) that i splitted into chunks [dataset](https://www.kaggle.com/datasets/what5up/amex-data-integer-dtypes-100k-cid-per-chunk). @raddar has denoised the data so that we can achieve better results with his dataset than with the original competition csv files.\n\nWe also convert the dataframe into a 3D-tensor dataset as highlighted by Chris Deotte [here](https://www.kaggle.com/competitions/amex-default-prediction/discussion/327828) \n\n","metadata":{}},{"cell_type":"code","source":"def load_test_df(test_file):\n    test = pd.read_parquet(test_file)\n    test['S_2'] = pd.to_datetime(test['S_2'])\n    tmp = test[['customer_ID','S_2']].groupby('customer_ID').count()\n\n    missing_cids = []\n    for nb_available_rows in range(1, 14):\n        cids = tmp[tmp['S_2'] == nb_available_rows].index.values\n        batch_missing_cids = [cid for cid in cids for _ in range(13 - nb_available_rows)]\n        missing_cids.extend(batch_missing_cids)\n\n    test_part2 = test.iloc[:len(missing_cids)].copy()\n    test_part2.loc[:] = np.nan\n    test_part2['customer_ID'] = missing_cids\n\n    test = pd.concat([test_part2, test])\n    \n    test = test.sort_values('customer_ID')\n    return test","metadata":{"execution":{"iopub.status.busy":"2022-06-20T13:19:02.490221Z","iopub.execute_input":"2022-06-20T13:19:02.490917Z","iopub.status.idle":"2022-06-20T13:19:02.503268Z","shell.execute_reply.started":"2022-06-20T13:19:02.490872Z","shell.execute_reply":"2022-06-20T13:19:02.502352Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class DataModule(pl.LightningDataModule):\n\n    def __init__(self, all_data: pd.DataFrame, batch_size: int = batch_size, num_workers: int = num_workers):\n        super().__init__()\n        self.all_data = all_data\n        self.batch_size = batch_size\n        self.num_workers = num_workers\n        self.sc = StandardScaler()\n\n    def prepare_data(self):\n        pass\n\n    def setup(self, stage=None):\n        # All data comumns except customer_ID, target, and S_2 are features\n        features = self.all_data.columns[2:]\n        self.all_data[features] = self.sc.fit_transform(self.all_data[features])\n        self.all_data[features] = self.all_data[features].fillna(0)\n        \n        # https://www.kaggle.com/competitions/amex-default-prediction/discussion/327828 !! Many Thanks @Chris Deotte for your sharing\n        all_tensor_x   = torch.reshape(torch.tensor(self.all_data[features].to_numpy()), (-1, 13, 188)).float()\n        \n        self.predict_tensor = TensorDataset(all_tensor_x)\n    def predict_dataloader(self):\n        return DataLoader(self.predict_tensor, batch_size=self.batch_size, num_workers=self.num_workers)","metadata":{"execution":{"iopub.status.busy":"2022-06-20T13:19:02.505183Z","iopub.execute_input":"2022-06-20T13:19:02.50598Z","iopub.status.idle":"2022-06-20T13:19:02.517239Z","shell.execute_reply.started":"2022-06-20T13:19:02.505936Z","shell.execute_reply":"2022-06-20T13:19:02.516356Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Metrics : [Implementation Source](https://www.kaggle.com/code/rohanrao/amex-competition-metric-implementations)","metadata":{}},{"cell_type":"code","source":"## https://www.kaggle.com/code/rohanrao/amex-competition-metric-implementations\n\nclass AmexMetric(Metric):\n    is_differentiable: Optional[bool] = False\n\n    # Set to True if the metric reaches it optimal value when the metric is maximized.\n    # Set to False if it when the metric is minimized.\n    higher_is_better: Optional[bool] = True\n\n    # Set to True if the metric during 'update' requires access to the global metric\n    # state for its calculations. If not, setting this to False indicates that all\n    # batch states are independent and we will optimize the runtime of 'forward'\n    full_state_update: bool = True\n\n    def __init__(self):\n        super().__init__()\n        \n        self.add_state(\"all_true\", default=[], dist_reduce_fx=\"cat\")\n        self.add_state(\"all_pred\", default=[], dist_reduce_fx=\"cat\")\n\n        rank_zero_warn(\n            \"Metric `Amex` will save all targets and predictions in buffer.\"\n            \" For large datasets this may lead to large memory footprint.\"\n        )\n\n    def update(self, y_pred: torch.Tensor, y_true: torch.Tensor):\n        \n        y_true = y_true.double()\n        y_pred = y_pred.double()\n        \n        self.all_true.append(y_true)\n        self.all_pred.append(y_pred)\n        \n    def compute(self):\n        y_true = torch.cat(self.all_true)\n        y_pred = torch.cat(self.all_pred)\n        # count of positives and negatives\n        n_pos = y_true.sum()\n        n_neg = y_pred.shape[0] - n_pos\n\n        # sorting by descring prediction values\n        indices = torch.argsort(y_pred, dim=0, descending=True)\n        preds, target = y_pred[indices], y_true[indices]\n\n        # filter the top 4% by cumulative row weights\n        weight = 20.0 - target * 19.0\n        cum_norm_weight = (weight / weight.sum()).cumsum(dim=0)\n        four_pct_filter = cum_norm_weight <= 0.04\n\n        # default rate captured at 4%\n        d = target[four_pct_filter].sum() / n_pos\n\n        # weighted gini coefficient\n        lorentz = (target / n_pos).cumsum(dim=0)\n        gini = ((lorentz - cum_norm_weight) * weight).sum()\n\n        # max weighted gini coefficient\n        gini_max = 10 * n_neg * (1 - 19 / (n_pos + 20 * n_neg))\n\n        # normalized weighted gini coefficient\n        g = gini / gini_max\n        \n        return 0.5 * (g + d)\n\n\ndef amex_metric(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n    def top_four_percent_captured(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n        df = (pd.concat([y_true, y_pred], axis='columns')\n              .sort_values('prediction', ascending=False))\n        df['weight'] = df['target'].apply(lambda x: 20 if x == 0 else 1)\n        four_pct_cutoff = int(0.04 * df['weight'].sum())\n        df['weight_cumsum'] = df['weight'].cumsum()\n        df_cutoff = df.loc[df['weight_cumsum'] <= four_pct_cutoff]\n        return (df_cutoff['target'] == 1).sum() / (df['target'] == 1).sum()\n\n    def weighted_gini(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n        df = (pd.concat([y_true, y_pred], axis='columns')\n              .sort_values('prediction', ascending=False))\n        df['weight'] = df['target'].apply(lambda x: 20 if x == 0 else 1)\n        df['random'] = (df['weight'] / df['weight'].sum()).cumsum()\n        total_pos = (df['target'] * df['weight']).sum()\n        df['cum_pos_found'] = (df['target'] * df['weight']).cumsum()\n        df['lorentz'] = df['cum_pos_found'] / total_pos\n        df['gini'] = (df['lorentz'] - df['random']) * df['weight']\n        return df['gini'].sum()\n\n    def normalized_weighted_gini(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n        y_true_pred = y_true.rename(columns={'target': 'prediction'})\n        return weighted_gini(y_true, y_pred) / weighted_gini(y_true, y_true_pred)\n\n    g = normalized_weighted_gini(y_true, y_pred)\n    d = top_four_percent_captured(y_true, y_pred)\n\n    return 0.5 * (g + d)\n","metadata":{"execution":{"iopub.status.busy":"2022-06-20T13:19:02.908596Z","iopub.execute_input":"2022-06-20T13:19:02.908915Z","iopub.status.idle":"2022-06-20T13:19:02.933825Z","shell.execute_reply.started":"2022-06-20T13:19:02.908879Z","shell.execute_reply":"2022-06-20T13:19:02.932705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model","metadata":{}},{"cell_type":"code","source":"class LSTMClassifier(nn.Module):\n    \"\"\"Very simple implementation of LSTM-based time-series classifier.\"\"\"\n\n    def __init__(self, input_dim, hidden_dim, num_layers, output_dim, device):\n        super().__init__()\n        self.num_layers = num_layers\n        self.hidden_dim = hidden_dim\n        self.rnn = nn.LSTM(input_dim, hidden_dim, num_layers, batch_first=True)\n        self.fc1 = nn.Linear(hidden_dim, 100)\n        self.fc2 = nn.Linear(100, output_dim)\n        self.device = device\n\n    def forward(self, x):\n        h0, c0 = self.init_hidden(x)\n        out, (_, _) = self.rnn(x, (h0, c0))\n        out = F.relu(self.fc1(out[:, -1, :]))\n        out = torch.sigmoid(self.fc2(out))\n        return out\n\n    def init_hidden(self, x):\n        batch_size = x.size(0)\n        h0 = torch.zeros(self.num_layers, batch_size, self.hidden_dim)\n        c0 = torch.zeros(self.num_layers, batch_size, self.hidden_dim)\n        if torch.cuda.is_available():\n            h0, c0 = h0.cuda(), c0.cuda()\n        return h0, c0\n\n\nclass TsLstmLightning(pl.LightningModule):\n    def __init__(self, in_features, hidden_dim, num_layers, learning_rate):\n        super(TsLstmLightning, self).__init__()\n\n        self.learning_rate = learning_rate\n\n        self.train_amex_metric = AmexMetric()\n        self.val_amex_metric   = AmexMetric()\n\n        self.model = LSTMClassifier(in_features, hidden_dim, num_layers, 1, device = self.device)\n\n        self.num_parameters = count_parameters(self.model)\n\n        self.loss_fn = nn.BCELoss(reduction=\"mean\")\n\n    def forward(self, x):\n        res = self.model(x)\n        return res\n\n    def training_step(self, batch, batch_idx):\n        X, target = batch\n        preds = self(X)  # (batch_size, 1)\n        preds = preds.squeeze(1)\n\n        loss = self.loss_fn(preds, target)\n        \n        self.train_amex_metric.update(preds, target)\n\n        self.log_dict({'train_loss': loss, 'train_amex_metric': self.train_amex_metric}, on_step=True, on_epoch=True, prog_bar=True, logger=True)\n\n        return {'loss': loss}\n\n    def validation_step(self, batch, batch_idx):\n        with torch.no_grad():\n            X, target = batch\n            preds = self(X)  \n            preds = preds.squeeze(1)\n\n            loss = self.loss_fn(preds, target)\n            \n            self.val_amex_metric.update(preds, target)\n\n            self.log_dict({'val_loss': loss, 'val_amex_metric': self.val_amex_metric }, on_step=True, on_epoch=True, prog_bar=True, logger=True)\n\n            return {'loss': loss}\n\n\n    def predict_step(self, batch, batch_idx, dataloader_idx=None):\n        with torch.no_grad():\n            X = batch[0]\n            preds = self(X)\n            return preds.detach().cpu()\n\n    def configure_optimizers(self):\n        optimizer = torch.optim.Adam(self.parameters(), lr=self.learning_rate)\n        lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=1)\n        return [optimizer], [lr_scheduler]\n\n\ndef count_parameters(model):\n    return sum(p.numel() for p in model.parameters() if p.requires_grad)","metadata":{"execution":{"iopub.status.busy":"2022-06-20T13:19:04.404576Z","iopub.execute_input":"2022-06-20T13:19:04.404891Z","iopub.status.idle":"2022-06-20T13:19:04.42809Z","shell.execute_reply.started":"2022-06-20T13:19:04.404854Z","shell.execute_reply":"2022-06-20T13:19:04.427334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Inference Loop ","metadata":{}},{"cell_type":"code","source":"model = TsLstmLightning(in_features=in_features, hidden_dim=hidden_dim, num_layers=num_layers, learning_rate=learning_rate)\nmodel = model.load_from_checkpoint(checkpoint_path=model_weights, in_features=in_features, hidden_dim=hidden_dim, num_layers=num_layers, learning_rate=learning_rate)\ntrainer = Trainer(gpus=[0] if torch.cuda.is_available() else None)\nall_ss = []\nfor test_file in test_files:\n    test_df = load_test_df(test_file)\n    print(f\"Test Shape: {test_df.shape}\")\n    dm = DataModule(test_df, batch_size=batch_size)\n    \n    customer_ID = test_df['customer_ID'].unique()\n\n    del test_df \n    gc.collect()\n    \n    prediction = trainer.predict(model, dm)\n    \n    prediction = torch.cat(prediction).detach().numpy().squeeze()\n\n    ss = pd.DataFrame({'customer_ID': customer_ID, 'prediction':prediction})\n    all_ss.append(ss)\n    \n    del dm\n    gc.collect()\n\nss = pd.concat(all_ss)\n\nss.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-06-20T13:19:06.470114Z","iopub.execute_input":"2022-06-20T13:19:06.470863Z","iopub.status.idle":"2022-06-20T13:28:39.17453Z","shell.execute_reply.started":"2022-06-20T13:19:06.470829Z","shell.execute_reply":"2022-06-20T13:28:39.173609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n","metadata":{}},{"cell_type":"markdown","source":"# Possible Nexts steps\n\n1. Have a best handling of missiing values in the data ( Ex : Do not drop customers that don't have 13 records,  do not fill N/A with 0, Replace previous -1 values with N/A, ...)\n2. Enhance the model (More LSTM / 1D CNN / Transformers / Param optimisation / ... ) \n3. Have a better understanding of the predictive features = Feature engineering ( feature selection or permutation feature importance, for instance.)\n4. (1. + 2.--> Transformer Unsupervised training with Times Series) https://arxiv.org/abs/2010.02803\n5. Model ensemble with other classification techniques\n6. Enable Cross-validation (Stratified K-Fold, ...)\n","metadata":{}}]}