{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"---\n# [Spaceship Titanic][1]\n\n- We are challenged to predict which passengers were transported by the anomaly using records recovered from the spaceship’s damaged computer system.\n\n---\n#### **The aim of this notebook is to...**\n- **1. Execute exploratory data analysis (EDA) and Feature Engineering. (Chapter3)**\n- **2. Learn about [TabNet](https://github.com/google-research/google-research/tree/master/tabnet), and use [PyTorch-TabNet](https://github.com/dreamquark-ai/tabnet) library. (Chapter4)**\n- **3. Deepen understanding of TabNet by building the model from scratch. (Chapter5)**\n\n\n---\n**References:** Thanks to previous great codes, blogs and notebooks.\n\n- I refered to [[TabNet in Tensorflow 2.0][2]] when I built TabNet architecture from scratch, and I modified it in many ways.\n\n- If you want to know more details about the TabNet, please refer to the paper: \"[TabNet: Attentive Interpretable Tabular Learning](https://arxiv.org/abs/1908.07442)\".\n\n- Japanese tech-blogs.\n - [【論文解説】TabNetを理解する][3]\n - [【解説+実装】Sparsemax関数を理解する][4]\n - [TabNetとは一体何者なのか？][5]\n\n---\n#### **My Previous Notebooks:**\nPlease note that the EDA part in this notebook is same as my previous notebook below. If you have read it, you can skip over chapter 3.\n- [SpaceshipTitanic: EDA + TabTransformer[TensorFlow]][6]\n\n---\n### **If you find this notebook useful, or when you copy&edit this notebook, please do give me an upvote. It helps me keep up my motivation.**\n\n---\n[1]: https://www.kaggle.com/competitions/spaceship-titanic/overview\n[2]: https://www.kaggle.com/code/marcusgawronsky/tabnet-in-tensorflow-2-0/notebook\n[3]: https://data-analytics.fun/2021/09/04/understanding-tabnet/\n[4]: https://data-analytics.fun/2021/08/17/understanding-sparsemax/\n[5]: https://zenn.dev/sinchir0/articles/9228eccebfbf579bfdf4\n[6]: https://www.kaggle.com/code/masatomurakawamm/spaceshiptitanic-eda-tabtransformer-tensorflow","metadata":{}},{"cell_type":"markdown","source":"

0. TABLE OF CONTENTS

\n\n","metadata":{}},{"cell_type":"markdown","source":"

1. Settings

","metadata":{}},{"cell_type":"code","source":"## Parameters\ndata_config = {'train.csv': '../input/spaceship-titanic/train.csv',\n 'test.csv': '../input/spaceship-titanic/test.csv',\n 'sample_submission.csv': '../input/spaceship-titanic/sample_submission.csv',\n }\n\nexp_config = {'competition_name': 'spaceship-titanic',\n 'n_bins': 10,\n 'batch_size': 512,\n 'n_splits': 5,\n 'num_columns': 13,\n 'train_epochs': 50,\n 'pretrain_epochs': 10,\n 'learning_rate': 2e-2,\n 'pretrain_lr': 1e-2,\n 'finetune_lr': 2e-2,\n 'emb_learning_rate': 5e-3,\n 'emb_finetune_lr': 5e-3,\n 'checkpoint_filepath': './tmp/model/exp.ckpt',\n }\n\nmodel_config = {'emb_dim': 1,\n 'input_dims': 13,\n 'output_dims': 1,\n 'n_steps': 3,\n 'feature_dims': 16,\n 'gamma': 1.3,\n 'epsilon': 1e-8,\n 'lambda_sparsity': 1e-3,\n 'mask_ratio': 0.25,\n 'label_smoothing': 0.01,\n }\n\nprint('Parameters setted!')","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:36:26.249257Z","iopub.execute_input":"2022-07-22T07:36:26.250083Z","iopub.status.idle":"2022-07-22T07:36:26.283080Z","shell.execute_reply.started":"2022-07-22T07:36:26.249975Z","shell.execute_reply":"2022-07-22T07:36:26.282190Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Import dependencies \nimport numpy as np\nimport pandas as pd\nimport scipy as sp\nimport matplotlib.pyplot as plt \n%matplotlib inline\n\nimport seaborn as sns\nimport plotly.express as px\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\n\nimport os\nimport pathlib\nimport gc\nimport sys\nimport re\nimport math \nimport random\nimport time \nimport datetime as dt\nfrom tqdm import tqdm\nfrom typing import Optional, Union, Tuple\n\nimport sklearn\nfrom sklearn.model_selection import StratifiedKFold\n\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nimport tensorflow_addons as tfa\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nprint('import done!')","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:36:26.284499Z","iopub.execute_input":"2022-07-22T07:36:26.285286Z","iopub.status.idle":"2022-07-22T07:36:39.494784Z","shell.execute_reply.started":"2022-07-22T07:36:26.285252Z","shell.execute_reply":"2022-07-22T07:36:39.493716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## For reproducible results \ndef seed_all(s):\n random.seed(s)\n np.random.seed(s)\n tf.random.set_seed(s)\n os.environ['TF_CUDNN_DETERMINISTIC'] = '1'\n os.environ['PYTHONHASHSEED'] = str(s) \n print('Seeds setted!')\n \nglobal_seed = 42\nseed_all(global_seed)\n\n## Limit GPU Memory in TensorFlow\n## Because TensorFlow, by default, allocates the full amount of available GPU memory when it is launched. \nphysical_devices = tf.config.list_physical_devices('GPU')\nif len(physical_devices) > 0:\n for device in physical_devices:\n tf.config.experimental.set_memory_growth(device, True)\n print('{} memory growth: {}'.format(device, tf.config.experimental.get_memory_growth(device)))\nelse:\n print(\"Not enough GPU hardware devices available\")","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:36:39.496265Z","iopub.execute_input":"2022-07-22T07:36:39.497072Z","iopub.status.idle":"2022-07-22T07:36:39.510861Z","shell.execute_reply.started":"2022-07-22T07:36:39.497035Z","shell.execute_reply":"2022-07-22T07:36:39.509702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

2. Data Loading

","metadata":{}},{"cell_type":"markdown","source":"---\n### [File and Data Field Descriptions](https://www.kaggle.com/competitions/spaceship-titanic/data)\n\n- **train.csv** - Personal records for about two-thirds (~8700) of the passengers, to be used as training data.\n - `PassengerId` - A unique Id for each passenger. Each Id takes the form `gggg_pp` where `gggg` indicates a group the passenger is travelling with and `pp` is their number within the group. People in a group are often family members, but not always.\n - `HomePlanet` - The planet the passenger departed from, typically their planet of permanent residence.\n - `CryoSleep` - Indicates whether the passenger elected to be put into suspended animation for the duration of the voyage. Passengers in cryosleep are confined to their cabins.\n - `Cabin` - The cabin number where the passenger is staying. Takes the form `deck/num/side`, where `side` can be either `P` for *Port* or `S` for *Starboard*.\n - `Destination` - The planet the passenger will be debarking to.\n - `Age` - The age of the passenger.\n - `VIP` - Whether the passenger has paid for special VIP service during the voyage.\n - `RoomService`, `FoodCourt`, `ShoppingMall`, `Spa`, `VRDeck` - Amount the passenger has billed at each of the *Spaceship Titanic*'s many luxury amenities.\n - `Name` - The first and last names of the passenger.\n - `Transported` - Whether the passenger was transported to another dimension. This is the target, the column you are trying to predict.\n\n\n- **test.csv** - Personal records for the remaining one-third (~4300) of the passengers, to be used as test data. Your task is to predict the value of `Transported` for the passengers in this set.\n\n\n- **sample_submission.csv** - A submission file in the correct format.\n - `PassengerId` - Id for each passenger in the test set.\n - `Transported` - The target. For each passenger, predict either *True* or *False*.\n\n---\n### [Submission & Evaluation](https://www.kaggle.com/competitions/spaceship-titanic/overview/evaluation)\n\n- Submissions are evaluated based on their classification accuracy, the percentage of predicted labels that are correct.\n\n---","metadata":{}},{"cell_type":"code","source":"## Data Loading\ntrain_df = pd.read_csv(data_config['train.csv'])\ntest_df = pd.read_csv(data_config['test.csv'])\nsubmission_df = pd.read_csv(data_config['sample_submission.csv'])\n\nprint(f'train_length: {len(train_df)}')\nprint(f'test_lenght: {len(test_df)}')\nprint(f'submission_length: {len(submission_df)}')","metadata":{"_kg_hide-output":false,"execution":{"iopub.status.busy":"2022-07-22T07:36:39.513243Z","iopub.execute_input":"2022-07-22T07:36:39.513602Z","iopub.status.idle":"2022-07-22T07:36:39.645879Z","shell.execute_reply.started":"2022-07-22T07:36:39.513543Z","shell.execute_reply":"2022-07-22T07:36:39.644923Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Null Value Check\nprint('train_df.info()'); print(train_df.info(), '\\n')\nprint('test_df.info()'); print(test_df.info(), '\\n')\n\n## train_df Check\ntrain_df.head()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:36:39.646900Z","iopub.execute_input":"2022-07-22T07:36:39.647408Z","iopub.status.idle":"2022-07-22T07:36:39.711515Z","shell.execute_reply.started":"2022-07-22T07:36:39.647377Z","shell.execute_reply":"2022-07-22T07:36:39.710707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

3. Exploratory Data Analysis

","metadata":{}},{"cell_type":"markdown","source":"

3.1 Feature Engineering

","metadata":{}},{"cell_type":"code","source":"## Feature Selection\nnumerical_columns = ['Age', 'RoomService', 'FoodCourt',\n 'ShoppingMall', 'Spa', 'VRDeck']\ncategorical_columns = ['PassengerId', 'HomePlanet', 'CryoSleep',\n 'Cabin', 'Destination', 'VIP', 'Name']\ntarget = 'Transported'\n\n## Number of unique values in each categorical features.\ncategorical_n_unique = {cc: train_df[cc].nunique() for cc in categorical_columns}\ncategorical_n_unique","metadata":{"_kg_hide-output":false,"execution":{"iopub.status.busy":"2022-07-22T07:36:39.712582Z","iopub.execute_input":"2022-07-22T07:36:39.713547Z","iopub.status.idle":"2022-07-22T07:36:39.728234Z","shell.execute_reply.started":"2022-07-22T07:36:39.713513Z","shell.execute_reply":"2022-07-22T07:36:39.727329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def preprocess_df(dataframe):\n df = dataframe.copy()\n \n ## Drop 'Name'\n df = df.drop(['Name'], axis=1)\n \n ## Transform 'Transported' column to 0 or 1.\n if 'Transported' in df.columns:\n df.loc[df['Transported']==True, 'Transported'] = 1.\n df.loc[df['Transported']==False, 'Transported'] = 0.\n df['Transported'] = df['Transported'].astype('int64')\n \n ## Transform True-False features (CryoSleep and VIP) to 'Yes' or 'No'.\n df.loc[df['CryoSleep']==True, 'CryoSleep'] = 'Yes'\n df.loc[df['CryoSleep']==False, 'CryoSleep'] = 'No'\n df['CryoSleep'] = df['CryoSleep'].astype(str)\n \n df.loc[df['VIP']==True, 'VIP'] = 'Yes'\n df.loc[df['VIP']==False, 'VIP'] = 'No'\n df['VIP'] = df['VIP'].astype(str)\n \n ## Transform the dtypes of HomePlanet and Destination to str\n df['HomePlanet'] = df['HomePlanet'].astype(str)\n df['Destination'] = df['Destination'].astype(str)\n \n return df\n\ntrain = preprocess_df(train_df)\ntrain.head()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:36:39.729630Z","iopub.execute_input":"2022-07-22T07:36:39.730467Z","iopub.status.idle":"2022-07-22T07:36:39.772579Z","shell.execute_reply.started":"2022-07-22T07:36:39.730433Z","shell.execute_reply":"2022-07-22T07:36:39.771618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Note: After `astype(str)`, null values (np.nan) are replaced by the string 'nan'.**","metadata":{}},{"cell_type":"code","source":"## Handle 'Cabin' feature\ndef cabin_split(dataframe):\n df = dataframe.copy()\n \n df['Cabin'] = df['Cabin'].astype(str)\n cabins = df['Cabin'].str.split('/', expand=True)\n cabins.columns = ['Cabin_0', 'Cabin_1', 'Cabin_2']\n \n df = pd.concat([df, cabins], axis=1)\n df = df.drop(['Cabin'], axis=1)\n df['Cabin_0'].astype(str)\n df['Cabin_1'] = pd.to_numeric(df['Cabin_1'], errors='coerce')\n df['Cabin_2'].astype(str)\n df['Cabin_2'] = df['Cabin_2'].map(lambda x: 'nan' if x is None else x)\n \n return df\n\ntrain = cabin_split(train)\ntrain.head()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:36:39.773824Z","iopub.execute_input":"2022-07-22T07:36:39.774643Z","iopub.status.idle":"2022-07-22T07:36:39.827603Z","shell.execute_reply.started":"2022-07-22T07:36:39.774608Z","shell.execute_reply":"2022-07-22T07:36:39.826721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

3.2 Target Distribution

","metadata":{}},{"cell_type":"code","source":"categorical_columns = ['HomePlanet', 'CryoSleep',\n 'Destination', 'VIP']\n\ntrain_pos = train.query('Transported==1').reset_index(drop=True)\ntrain_neg = train.query('Transported==0').reset_index(drop=True)\nprint(f'positive samples: {len(train_pos)}, negative samples: {len(train_neg)}')","metadata":{"_kg_hide-output":false,"execution":{"iopub.status.busy":"2022-07-22T07:36:39.828877Z","iopub.execute_input":"2022-07-22T07:36:39.829405Z","iopub.status.idle":"2022-07-22T07:36:39.847293Z","shell.execute_reply.started":"2022-07-22T07:36:39.829374Z","shell.execute_reply":"2022-07-22T07:36:39.846520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Target Distribution\ntarget_count = train.groupby(['Transported'])['PassengerId'].count()\ntarget_percent = target_count / target_count.sum()\n\nfig = go.Figure()\ndata = go.Bar(x=target_count.index.astype(str).values, \n y=target_count.values)\nfig.add_trace(data)\nfig.update_layout(title = dict(text=\"Target distribution\"),\n xaxis = dict(title=\"Transported' values\"),\n yaxis = dict(title='counts'))\nfig.show()","metadata":{"_kg_hide-output":false,"execution":{"iopub.status.busy":"2022-07-22T07:36:39.850496Z","iopub.execute_input":"2022-07-22T07:36:39.851073Z","iopub.status.idle":"2022-07-22T07:36:39.977047Z","shell.execute_reply.started":"2022-07-22T07:36:39.851042Z","shell.execute_reply":"2022-07-22T07:36:39.975885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

3.3 Numerical Features

","metadata":{}},{"cell_type":"markdown","source":"

3.3.1 Statistics of Numerical Features

","metadata":{}},{"cell_type":"code","source":"train.describe().T.style.bar(subset=['mean'],)\\\n .background_gradient(subset=['std'], cmap='coolwarm')\\\n .background_gradient(subset=['50%'], cmap='coolwarm')","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:36:39.978653Z","iopub.execute_input":"2022-07-22T07:36:39.979123Z","iopub.status.idle":"2022-07-22T07:36:40.081200Z","shell.execute_reply.started":"2022-07-22T07:36:39.979073Z","shell.execute_reply":"2022-07-22T07:36:40.080124Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.groupby('Transported').describe().T","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:36:40.084395Z","iopub.execute_input":"2022-07-22T07:36:40.085086Z","iopub.status.idle":"2022-07-22T07:36:40.146713Z","shell.execute_reply.started":"2022-07-22T07:36:40.085043Z","shell.execute_reply":"2022-07-22T07:36:40.145539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"quantiles = [0, 0.9, 0.95, 0.98, 0.99, 1]\ntrain_quantile_values = train[['RoomService', 'FoodCourt', 'ShoppingMall', 'Spa', 'VRDeck']].quantile(quantiles)\ntrain_quantile_values","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:36:40.148138Z","iopub.execute_input":"2022-07-22T07:36:40.148458Z","iopub.status.idle":"2022-07-22T07:36:40.169848Z","shell.execute_reply.started":"2022-07-22T07:36:40.148430Z","shell.execute_reply":"2022-07-22T07:36:40.168636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n#### There seems to be outliers...\n\n---","metadata":{}},{"cell_type":"code","source":"## Clipping outliers on 99% quantile\ndef clipping_quantile(dataframe, quantile_values=None, quantile=0.99):\n df = dataframe.copy()\n if quantile_values is None:\n quantile_values = df[['RoomService', 'FoodCourt', 'ShoppingMall', 'Spa', 'VRDeck']].quantile(quantile)\n \n for num_column in ['RoomService', 'FoodCourt', 'ShoppingMall', 'Spa', 'VRDeck']:\n num_values = df[num_column].values\n threshold = quantile_values[num_column]\n num_values = np.where(num_values > threshold, threshold, num_values)\n df[num_column] = num_values \n return df\n\ntrain = clipping_quantile(train, quantile_values=None, quantile=0.99)\n\ntrain.describe().T.style.bar(subset=['mean'],)\\\n .background_gradient(subset=['std'], cmap='coolwarm')\\\n .background_gradient(subset=['50%'], cmap='coolwarm')","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:36:40.171263Z","iopub.execute_input":"2022-07-22T07:36:40.171676Z","iopub.status.idle":"2022-07-22T07:36:40.222830Z","shell.execute_reply.started":"2022-07-22T07:36:40.171641Z","shell.execute_reply":"2022-07-22T07:36:40.221606Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## After clipping outliers on 99% quantile\ntrain.groupby('Transported').describe().T","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:36:40.224385Z","iopub.execute_input":"2022-07-22T07:36:40.225385Z","iopub.status.idle":"2022-07-22T07:36:40.280664Z","shell.execute_reply.started":"2022-07-22T07:36:40.225348Z","shell.execute_reply":"2022-07-22T07:36:40.279619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## After clipping outliers on 99% quantile\nn_cols = 2\nn_rows = int(np.ceil(len(numerical_columns) / n_cols))\n\nfig, axes = plt.subplots(nrows=n_rows,ncols=n_cols,figsize=(20,15))\n\nbins = 50\nfor i, column in enumerate(numerical_columns):\n q, mod = divmod(i, n_cols)\n sns.histplot(x=column, data=train, hue='Transported', ax=axes[q][mod], bins=bins, stat=\"percent\", legend=True)\n axes[q][mod].set_title(f'Distribution of {numerical_columns[i]}',size=15)\n \nfig.suptitle('Blue: Transported=0, Red: Transported=1', fontsize=20)\nfig.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:36:40.282056Z","iopub.execute_input":"2022-07-22T07:36:40.282373Z","iopub.status.idle":"2022-07-22T07:36:43.002621Z","shell.execute_reply.started":"2022-07-22T07:36:40.282344Z","shell.execute_reply":"2022-07-22T07:36:43.001458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Heat map of Correlation Matrix\nfig = px.imshow(train.corr(),\n color_continuous_scale='RdBu_r',\n color_continuous_midpoint=0, \n aspect='auto')\nfig.update_layout(height=500, \n width=500,\n title = \"Heatmap\", \n showlegend=False)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:36:43.004184Z","iopub.execute_input":"2022-07-22T07:36:43.005090Z","iopub.status.idle":"2022-07-22T07:36:43.986767Z","shell.execute_reply.started":"2022-07-22T07:36:43.005037Z","shell.execute_reply":"2022-07-22T07:36:43.985540Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

3.3.2 Binning for Numerical Features

","metadata":{}},{"cell_type":"markdown","source":"### Binning Method\n- `Age`: 0 to 100 at intervals of 5.\n\n- `other numerical features`: Split into 10 bins.\n - 1. Value=0 is the first bin ( get by (-1, 0] ).\n - 2. Get quantiles at [ 0, 0.9, 0.95, 0.99, 1 ].\n - 3. Split between quantiles_0 and quantiles_0.9 into 6 bins.\n - 4. Use quantiles_0.95, _0.99, _1 for the rest boundary.","metadata":{}},{"cell_type":"code","source":"def bin_split(dataframe, column, n_bins, thresholds=None):\n if thresholds is None:\n if column == 'Age':\n bins = np.array([i*5 for i in range(21)])\n else:\n bins = np.array([-1, ])\n x = dataframe[column]\n x_quantiles = x.quantile([0, 0.9, 0.95, 0.99, 1])\n bins = np.append(bins, [i * ((x_quantiles.iloc[1] - x_quantiles.iloc[0]) / (n_bins-4)) for i in range(n_bins-4)])\n bins = np.append(bins, [x_quantiles.iloc[1], x_quantiles.iloc[2], x_quantiles.iloc[3], x_quantiles.iloc[4]+1])\n else:\n bins = thresholds[column]\n \n splits = pd.cut(dataframe[column], bins=bins, labels=False, right=True)\n return splits, bins\n\ndef binning(dataframe, numerical_columns, n_bins, thresholds=None):\n df = dataframe.copy()\n df_split_bins = {}\n for num_column in numerical_columns:\n splits, bins = bin_split(df, num_column, n_bins, thresholds)\n df[num_column] = splits\n df_split_bins[num_column] = bins \n return df, df_split_bins\n\nn_bins = exp_config['n_bins']\ntrain, train_split_bins = binning(train, numerical_columns, n_bins, thresholds=None)\n\nfor key in train_split_bins:\n print(f'{key} bins: \\n{train_split_bins[key]}\\n\\n')","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:36:43.988154Z","iopub.execute_input":"2022-07-22T07:36:43.988488Z","iopub.status.idle":"2022-07-22T07:36:44.019783Z","shell.execute_reply.started":"2022-07-22T07:36:43.988459Z","shell.execute_reply":"2022-07-22T07:36:44.018508Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## After Binning\nn_cols = 2\nn_rows = int(np.ceil(len(numerical_columns) / n_cols))\n\nfig, axes = plt.subplots(nrows=n_rows,ncols=n_cols,figsize=(20,15))\n\nbins = 50\nfor i, column in enumerate(numerical_columns):\n q, mod = divmod(i, n_cols)\n sns.histplot(x=column, data=train, hue='Transported', ax=axes[q][mod], bins=bins, stat=\"percent\", legend=True)\n axes[q][mod].set_title(f'Distribution of {numerical_columns[i]}',size=15)\n \nfig.suptitle('Blue: Transported=0, Red: Transported=1', fontsize=20)\nfig.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:36:44.021599Z","iopub.execute_input":"2022-07-22T07:36:44.022101Z","iopub.status.idle":"2022-07-22T07:36:46.393193Z","shell.execute_reply.started":"2022-07-22T07:36:44.022053Z","shell.execute_reply":"2022-07-22T07:36:46.391654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

3.4 Categorical Features

","metadata":{}},{"cell_type":"code","source":"## Make Figure object\nfig = make_subplots(rows=2, cols=2,\n subplot_titles=categorical_columns,\n shared_yaxes='all')\n\nfor i in range(2):\n for j in range(2):\n n = i*2 + j\n ## Make trace (graph object)\n data0 = go.Histogram(x=train_neg[categorical_columns[n]],\n marker = dict(color='#0000FF'), ## Blue\n name='Transporetd=0')\n data1 = go.Histogram(x=train_pos[categorical_columns[n]],\n marker = dict(color='#FF0000'), ## Red\n name='Transported=1')\n \n ## Add the trace to the Figure\n fig.add_trace(data0, row=i+1, col=j+1)\n fig.add_trace(data1, row=i+1, col=j+1)\n \n fig.update_traces(opacity=0.75, histnorm='probability')\n #fig.update_layout(barmode='overlay')\n\n## Setting layouts\nfig.update_layout(title = dict(text='Blue: Transported=0, Red: Transported=1'),\n showlegend=False,)\nfig.update_yaxes(title='probability', row=1, col=1)\nfig.update_yaxes(title='probability', row=2, col=1)\n\n## Show the Figure\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:36:46.394927Z","iopub.execute_input":"2022-07-22T07:36:46.395392Z","iopub.status.idle":"2022-07-22T07:36:46.589151Z","shell.execute_reply.started":"2022-07-22T07:36:46.395344Z","shell.execute_reply":"2022-07-22T07:36:46.587871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Cabin Features","metadata":{}},{"cell_type":"code","source":"## 'Cabin_0'\nsns.countplot(x='Cabin_0', data=train, hue='Transported')","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:36:46.590439Z","iopub.execute_input":"2022-07-22T07:36:46.590787Z","iopub.status.idle":"2022-07-22T07:36:46.852097Z","shell.execute_reply.started":"2022-07-22T07:36:46.590754Z","shell.execute_reply":"2022-07-22T07:36:46.851210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## 'Cabin_1'\nsns.histplot(x='Cabin_1', data=train, hue='Transported', kde=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:36:46.853446Z","iopub.execute_input":"2022-07-22T07:36:46.854070Z","iopub.status.idle":"2022-07-22T07:36:47.224679Z","shell.execute_reply.started":"2022-07-22T07:36:46.854027Z","shell.execute_reply":"2022-07-22T07:36:47.223753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## 'Cabin_2'\nsns.countplot(x='Cabin_2', data=train, hue='Transported')","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:36:47.226010Z","iopub.execute_input":"2022-07-22T07:36:47.226600Z","iopub.status.idle":"2022-07-22T07:36:47.420462Z","shell.execute_reply.started":"2022-07-22T07:36:47.226548Z","shell.execute_reply":"2022-07-22T07:36:47.419311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Binning 'Cabin_1'","metadata":{}},{"cell_type":"code","source":"## Histogram of 'Cabin_1' by Plotly (interactive)\nfig = go.Figure()\n\ndata0 = go.Histogram(x=train_neg['Cabin_1'],\n marker = dict(color='#0000FF'), # Blue\n opacity=0.6,\n name='Transporetd=0')\ndata1 = go.Histogram(x=train_pos['Cabin_1'],\n marker = dict(color='#FF0000'), # Red\n opacity=0.6,\n name='Transported=1')\n\nfig.add_trace(data0)\nfig.add_trace(data1)\n\nfig.update_layout(xaxis = dict(title='Cabin_1'),\n yaxis = dict(title='Count'))\nfig.update_layout(barmode='overlay')\n\nfig.show()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:36:47.421546Z","iopub.execute_input":"2022-07-22T07:36:47.421882Z","iopub.status.idle":"2022-07-22T07:36:47.439083Z","shell.execute_reply.started":"2022-07-22T07:36:47.421851Z","shell.execute_reply":"2022-07-22T07:36:47.438253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Binning 'Cabin_1' based on the above graph\ncabin_1_bins = np.array([0, 300, 600, 1150, 1500, 1700, 2000])\ntrain['Cabin_1'] = pd.cut(train['Cabin_1'], bins=cabin_1_bins, labels=False, right=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:36:47.440412Z","iopub.execute_input":"2022-07-22T07:36:47.441456Z","iopub.status.idle":"2022-07-22T07:36:47.460005Z","shell.execute_reply.started":"2022-07-22T07:36:47.441406Z","shell.execute_reply":"2022-07-22T07:36:47.459024Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## 'Cabin_1' after binning\nsns.countplot(x='Cabin_1', data=train, hue='Transported')","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:36:47.461357Z","iopub.execute_input":"2022-07-22T07:36:47.462267Z","iopub.status.idle":"2022-07-22T07:36:47.639079Z","shell.execute_reply.started":"2022-07-22T07:36:47.462232Z","shell.execute_reply":"2022-07-22T07:36:47.638288Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

3.5 Data Processing Complete

","metadata":{}},{"cell_type":"code","source":"numerical_columns_0 = ['Age', 'RoomService', 'FoodCourt',\n 'ShoppingMall', 'Spa', 'VRDeck']\nnumerical_columns_1 = ['Age', 'RoomService', 'FoodCourt',\n 'ShoppingMall', 'Spa', 'VRDeck', 'Cabin_1']\ncategorical_columns_0 = ['PassengerId', 'HomePlanet', 'CryoSleep',\n 'Cabin', 'Destination', 'VIP', 'Name']\ncategorical_columns_1 = ['PassengerId', 'HomePlanet', 'CryoSleep',\n 'Cabin', 'Destination', 'VIP', 'Name',\n 'Cabin_0', 'Cabin_2']","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:36:47.640616Z","iopub.execute_input":"2022-07-22T07:36:47.641265Z","iopub.status.idle":"2022-07-22T07:36:47.648088Z","shell.execute_reply.started":"2022-07-22T07:36:47.641222Z","shell.execute_reply":"2022-07-22T07:36:47.647299Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Before filling null values,　making the string 'nan' (transformed by astype(str) in preprocess_df() function) back to np.nan.\nfor column in ['CryoSleep', 'VIP', 'HomePlanet', 'Destination', 'Cabin_0', 'Cabin_2']:\n train[column] = train[column].map(lambda x: np.nan if x=='nan' else x)\n\n\n## Filling null values with mode\ntrain = train.fillna(train.mode().iloc[0])\n\nfor numerical in numerical_columns_1:\n train[numerical] = train[numerical].astype('int64')\n\ntrain.info()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:36:47.923300Z","iopub.execute_input":"2022-07-22T07:36:47.924184Z","iopub.status.idle":"2022-07-22T07:36:47.987104Z","shell.execute_reply.started":"2022-07-22T07:36:47.924130Z","shell.execute_reply":"2022-07-22T07:36:47.986145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Test Data Processing\ntest = preprocess_df(test_df)\ntest = cabin_split(test)\n\ntest = clipping_quantile(test, quantile_values=train_quantile_values.loc[0.99])\ntest, _ = binning(test, numerical_columns_0, n_bins, thresholds=train_split_bins)\ntest['Cabin_1'] = pd.cut(test['Cabin_1'], bins=cabin_1_bins, labels=False, right=False)\n\nfor column in ['CryoSleep', 'VIP', 'HomePlanet', 'Destination', 'Cabin_0', 'Cabin_2']:\n test[column] = test[column].map(lambda x: np.nan if x=='nan' else x)\n\ntest = test.fillna(train.mode().iloc[0])\n\nfor numerical in numerical_columns_1:\n test[numerical] = test[numerical].astype('int64')\n\ntest.info()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:36:47.988220Z","iopub.execute_input":"2022-07-22T07:36:47.988988Z","iopub.status.idle":"2022-07-22T07:36:48.073993Z","shell.execute_reply.started":"2022-07-22T07:36:47.988951Z","shell.execute_reply":"2022-07-22T07:36:48.072824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

3.6 Validation Split

","metadata":{}},{"cell_type":"code","source":"## Split train samples for cross-validation\nn_splits = exp_config['n_splits']\nskf = StratifiedKFold(n_splits=n_splits)\ntrain['k_folds'] = -1\nfor fold, (train_idx, valid_idx) in enumerate(skf.split(X=train,\n y=train['Transported'])):\n train['k_folds'][valid_idx] = fold\n \n## Check split samples\nfor i in range(n_splits):\n print(f\"fold {i}: {len(train.query('k_folds==@i'))} samples\")","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:36:48.075497Z","iopub.execute_input":"2022-07-22T07:36:48.075960Z","iopub.status.idle":"2022-07-22T07:36:48.112693Z","shell.execute_reply.started":"2022-07-22T07:36:48.075912Z","shell.execute_reply":"2022-07-22T07:36:48.111732Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Hold-out validation\nvalid_fold = train.query(f'k_folds == 0').reset_index(drop=True)\ntrain_fold = train.query(f'k_folds != 0').reset_index(drop=True)\nprint(len(train_fold), len(valid_fold))","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:36:48.113880Z","iopub.execute_input":"2022-07-22T07:36:48.114692Z","iopub.status.idle":"2022-07-22T07:36:48.128602Z","shell.execute_reply.started":"2022-07-22T07:36:48.114658Z","shell.execute_reply":"2022-07-22T07:36:48.127478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

4. TabNet

","metadata":{}},{"cell_type":"markdown","source":"**[TabNet](https://github.com/google-research/google-research/tree/master/tabnet)** is a deep neural network architecture which adapts the decision tree algorithm, and thereby improves its interpretability and performance on the tabular data.\n\nThe figure below shows the simple example of how to achieve decision tree-like calculation by neural networks. The features used are selected by masks at each decision steps: $x_1$ is used at the first step, and $x_2$ at the second step.\n\n\n\nIn the TabNet model, feature selecting-masks are made for each sample from the input data by the attention mechanism. I will explain details in the following section (5. TabNet from Scratch), and now let's look at the example of data-processing flow in TabNet.\n\n\n\nThere is a commonly used repository, **[pytorch-tabnet](https://github.com/dreamquark-ai/tabnet)**. Let's make use of it!","metadata":{}},{"cell_type":"markdown","source":"

4.1 Pytorch_TabNet

","metadata":{}},{"cell_type":"code","source":"## Data Preprocessing for Pytorch_TabNet\nfrom sklearn.preprocessing import LabelEncoder\n\nnumerical_columns = []\ncategorical_columns = ['Age', 'RoomService', 'FoodCourt',\n 'ShoppingMall', 'Spa', 'VRDeck',\n 'HomePlanet', 'CryoSleep',\n 'Destination', 'VIP', \n 'Cabin_0', 'Cabin_1', 'Cabin_2']\nfeature_columns = numerical_columns + categorical_columns\ntarget_column = 'Transported'\n\n\n## Create Datasets\ny_train = train_fold[target_column]\ny_train = y_train.values\ntrain_fold_features = train_fold[feature_columns]\n\ny_valid = valid_fold[target_column]\ny_valid = y_valid.values\nvalid_fold_features = valid_fold[feature_columns]\n\ntest_features = test[feature_columns]\n\ncategorical_dims = {}\nfor col in categorical_columns:\n #print(col, train_fold_features[col].nunique())\n l_enc = LabelEncoder()\n train_fold_features[col] = train_fold_features[col].fillna(\"VV_likely\")\n train_fold_features[col] = l_enc.fit_transform(train_fold_features[col].values)\n valid_fold_features[col] = valid_fold_features[col].fillna('VV_likey')\n valid_fold_features[col] = l_enc.transform(valid_fold_features[col].values) \n test_features[col] = test_features[col].fillna('VV_likey')\n test_features[col] = l_enc.transform(test_features[col].values) \n categorical_dims[col] = len(l_enc.classes_)\n \ncat_idxs = [ i for i, f in enumerate(train_fold_features.columns) if f in categorical_columns]\ncat_dims = [ categorical_dims[f] for i, f in enumerate(train_fold_features.columns) if f in categorical_columns]\n\nX_train = train_fold_features.values\nX_valid = valid_fold_features.values\nX_test = test_features.values","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:36:48.130112Z","iopub.execute_input":"2022-07-22T07:36:48.130956Z","iopub.status.idle":"2022-07-22T07:36:48.199888Z","shell.execute_reply.started":"2022-07-22T07:36:48.130915Z","shell.execute_reply":"2022-07-22T07:36:48.198754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Import dependencies \n!pip install -q -U pytorch_tabnet \n\nimport torch\nfrom pytorch_tabnet.pretraining import TabNetPretrainer\nfrom pytorch_tabnet.tab_model import TabNetRegressor\nfrom pytorch_tabnet.tab_model import TabNetClassifier","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:36:48.201815Z","iopub.execute_input":"2022-07-22T07:36:48.202280Z","iopub.status.idle":"2022-07-22T07:37:02.093332Z","shell.execute_reply.started":"2022-07-22T07:36:48.202220Z","shell.execute_reply":"2022-07-22T07:37:02.092062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Creating Model\ntabnet_params = dict(n_d=8, n_a=8, n_steps=3, gamma=1.3,\n n_independent=2, n_shared=2,\n seed=global_seed, lambda_sparse=1e-3,\n optimizer_fn=torch.optim.Adam,\n optimizer_params=dict(lr=exp_config['learning_rate']),\n mask_type='entmax',\n scheduler_params=dict(mode=\"min\",\n patience=5,\n min_lr=1e-5,\n factor=0.5,),\n scheduler_fn=torch.optim.lr_scheduler.ReduceLROnPlateau,\n verbose=10,\n cat_idxs=cat_idxs, # comment out when Unsupervised\n cat_dims=cat_dims, # comment out when Unsupervised\n cat_emb_dim=1 # comment out when Unsupervised\n )\n\nmodel = TabNetClassifier(**tabnet_params)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:37:02.095119Z","iopub.execute_input":"2022-07-22T07:37:02.095573Z","iopub.status.idle":"2022-07-22T07:37:02.107166Z","shell.execute_reply.started":"2022-07-22T07:37:02.095505Z","shell.execute_reply":"2022-07-22T07:37:02.106299Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Model Training\nbatch_size = exp_config['batch_size']\n\nmodel.fit(\n X_train=X_train,\n y_train=y_train,\n eval_set=[(X_train, y_train), (X_valid, y_valid)],\n eval_name=['train', 'valid'],\n eval_metric = ['auc'],\n max_epochs=100,\n patience=20, batch_size=batch_size,\n virtual_batch_size=128,\n num_workers=0, drop_last=False)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:37:02.108656Z","iopub.execute_input":"2022-07-22T07:37:02.109262Z","iopub.status.idle":"2022-07-22T07:37:37.656517Z","shell.execute_reply.started":"2022-07-22T07:37:02.109215Z","shell.execute_reply":"2022-07-22T07:37:37.655637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(model.history.history.keys())\n\n## Plot Losses\nplt.figure(figsize=(7, 5))\nplt.plot(np.arange(len(model.history['loss'])),\n model.history['loss'], label='Train Loss')\nplt.xlabel('Epoch')\nplt.ylabel('Loss')\nplt.legend()\nplt.title('Loss Plot')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:37:37.657810Z","iopub.execute_input":"2022-07-22T07:37:37.658928Z","iopub.status.idle":"2022-07-22T07:37:37.852732Z","shell.execute_reply.started":"2022-07-22T07:37:37.658889Z","shell.execute_reply":"2022-07-22T07:37:37.851480Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Plot AUCs\nplt.figure(figsize=(7, 5))\nplt.plot(np.arange(len(model.history['train_auc'])),\n model.history['train_auc'], label='Train AUC')\nplt.plot(np.arange(len(model.history['valid_auc'])),\n model.history['valid_auc'], label='Valid AUC')\nplt.xlabel('Epoch')\nplt.ylabel('AUC')\nplt.legend()\nplt.title('AUC Plot')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:37:37.854172Z","iopub.execute_input":"2022-07-22T07:37:37.854521Z","iopub.status.idle":"2022-07-22T07:37:38.043039Z","shell.execute_reply.started":"2022-07-22T07:37:37.854488Z","shell.execute_reply":"2022-07-22T07:37:38.041749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Plot Learning Rates\nplt.figure(figsize=(7, 5))\nplt.plot(np.arange(len(model.history['lr'])),\n model.history['lr'])\nplt.xlabel('Epoch')\nplt.ylabel('Learning Rate')\nplt.legend()\nplt.title('Learning Rate Plot')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:37:38.044811Z","iopub.execute_input":"2022-07-22T07:37:38.045382Z","iopub.status.idle":"2022-07-22T07:37:38.244144Z","shell.execute_reply.started":"2022-07-22T07:37:38.045331Z","shell.execute_reply":"2022-07-22T07:37:38.243035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Plot Feature Importance\nimportance = model.feature_importances_\nimportance_df = pd.DataFrame(importance,\n index=feature_columns,\n columns=['importance'])\nimportance_df.sort_values(by='importance',\n ascending=False,\n inplace=True)\ntop10_importance = np.array(importance_df[:10].values.reshape([-1]))\ntop10_features = importance_df[:10].index\n\n#importance_df[:10].plot.bar()\n\nplt.figure(figsize=(7, 5))\nplt.bar(x=np.arange(10),\n height=top10_importance)\nplt.xticks(np.arange(10), top10_features, rotation='vertical')\nplt.title('Feature Importance')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:37:38.245969Z","iopub.execute_input":"2022-07-22T07:37:38.246358Z","iopub.status.idle":"2022-07-22T07:37:38.459696Z","shell.execute_reply.started":"2022-07-22T07:37:38.246324Z","shell.execute_reply":"2022-07-22T07:37:38.458707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Inference and Submission\nprobas = model.predict(X_test)\nprobas = np.squeeze(probas)\n\npreds = np.where(probas > 0.5, True, False)\n\nsubmission_df['Transported'] = preds\nsubmission_df.to_csv('submission_pt.csv', index=False)\nsubmission_df.head()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:37:38.461275Z","iopub.execute_input":"2022-07-22T07:37:38.461657Z","iopub.status.idle":"2022-07-22T07:37:38.586753Z","shell.execute_reply.started":"2022-07-22T07:37:38.461623Z","shell.execute_reply":"2022-07-22T07:37:38.585608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n#### Public Score: ~ 0.772\n\n---","metadata":{}},{"cell_type":"markdown","source":"

4.2 Pre-training and Fine-tuning

","metadata":{}},{"cell_type":"markdown","source":"TabNet can be pre-trained on the 'self-supervised' (or you can also say 'Unsupervised') tarbular learning. The pre-training task is as follows. Some features in the input data are randomly masked, and encoder-decoder architecture, or autoencoder, model predicts the masked features. Please note that input-mask is different from feature selecting-mask which we saw earlier. Through this task, TabNet encoder learns the relevance between features.\n\n\n\n\nAfter the pre-training, TabNet encoder is subsequently fine-tuned on the supervised learning of aimed task. Please note that TabNet decoder is not used for the fine-tuning.\n\n","metadata":{}},{"cell_type":"code","source":"## Creating Model\npretrain_params = dict(n_d=8, n_a=8, n_steps=3, gamma=1.3,\n n_independent=2, n_shared=2,\n seed=global_seed, lambda_sparse=1e-3,\n optimizer_fn=torch.optim.Adam,\n optimizer_params=dict(lr=exp_config['pretrain_lr']),\n mask_type='entmax',\n scheduler_params=dict(mode=\"min\",\n patience=5,\n min_lr=1e-5,\n factor=0.5,),\n scheduler_fn=torch.optim.lr_scheduler.ReduceLROnPlateau,\n verbose=10,\n )\n\npretrainer = TabNetPretrainer(**pretrain_params)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:37:38.588194Z","iopub.execute_input":"2022-07-22T07:37:38.589294Z","iopub.status.idle":"2022-07-22T07:37:38.598140Z","shell.execute_reply.started":"2022-07-22T07:37:38.589239Z","shell.execute_reply":"2022-07-22T07:37:38.596968Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Model Pre-Training\npretrainer.fit(\n X_train=X_train,\n eval_set=[X_valid],\n max_epochs=200,\n patience=20, batch_size=256, virtual_batch_size=128,\n num_workers=1, drop_last=True)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:37:38.599663Z","iopub.execute_input":"2022-07-22T07:37:38.600428Z","iopub.status.idle":"2022-07-22T07:37:57.647243Z","shell.execute_reply.started":"2022-07-22T07:37:38.600390Z","shell.execute_reply":"2022-07-22T07:37:57.645672Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Create Pre-trained Model\npt_model = TabNetClassifier(**tabnet_params)\nbatch_size = exp_config['batch_size']\n\n## Model Fine-tuning\npt_model.fit(X_train=X_train,\n y_train=y_train,\n eval_set=[(X_train, y_train), (X_valid, y_valid)],\n eval_name=['train', 'valid'],\n eval_metric = ['auc'],\n max_epochs=100,\n patience=20, batch_size=batch_size,\n virtual_batch_size=128,\n num_workers=0, drop_last=False,\n from_unsupervised=pretrainer\n )","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:37:57.649386Z","iopub.execute_input":"2022-07-22T07:37:57.650739Z","iopub.status.idle":"2022-07-22T07:38:17.258025Z","shell.execute_reply.started":"2022-07-22T07:37:57.650689Z","shell.execute_reply":"2022-07-22T07:38:17.256322Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(pt_model.history.history.keys())\n\n## Plot Losses\nplt.figure(figsize=(7, 5))\nplt.plot(np.arange(len(pt_model.history['loss'])),\n pt_model.history['loss'], label='Train Loss')\nplt.xlabel('Epoch')\nplt.ylabel('Loss')\nplt.legend()\nplt.title('Loss Plot')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:38:17.260446Z","iopub.execute_input":"2022-07-22T07:38:17.260995Z","iopub.status.idle":"2022-07-22T07:38:17.486747Z","shell.execute_reply.started":"2022-07-22T07:38:17.260941Z","shell.execute_reply":"2022-07-22T07:38:17.485538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Plot AUCs\nplt.figure(figsize=(7, 5))\nplt.plot(np.arange(len(pt_model.history['train_auc'])),\n pt_model.history['train_auc'], label='Train AUC')\nplt.plot(np.arange(len(pt_model.history['valid_auc'])),\n pt_model.history['valid_auc'], label='Valid AUC')\nplt.xlabel('Epoch')\nplt.ylabel('AUC')\nplt.legend()\nplt.title('AUC Plot')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:38:17.488575Z","iopub.execute_input":"2022-07-22T07:38:17.489290Z","iopub.status.idle":"2022-07-22T07:38:17.716689Z","shell.execute_reply.started":"2022-07-22T07:38:17.489242Z","shell.execute_reply":"2022-07-22T07:38:17.715517Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Plot Learning Rates\nplt.figure(figsize=(7, 5))\nplt.plot(np.arange(len(pt_model.history['lr'])),\n pt_model.history['lr'])\nplt.xlabel('Epoch')\nplt.ylabel('Learning Rate')\nplt.legend()\nplt.title('Learning Rate Plot')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:38:17.718177Z","iopub.execute_input":"2022-07-22T07:38:17.719134Z","iopub.status.idle":"2022-07-22T07:38:17.934444Z","shell.execute_reply.started":"2022-07-22T07:38:17.719083Z","shell.execute_reply":"2022-07-22T07:38:17.933123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Plot Feature Importance\nimportance = pt_model.feature_importances_\nimportance_df = pd.DataFrame(importance,\n index=feature_columns,\n columns=['importance'])\nimportance_df.sort_values(by='importance',\n ascending=False,\n inplace=True)\ntop10_importance = np.array(importance_df[:10].values.reshape([-1]))\ntop10_features = importance_df[:10].index\n\n#importance_df[:10].plot.bar()\n\nplt.figure(figsize=(7, 5))\nplt.bar(x=np.arange(10),\n height=top10_importance)\nplt.xticks(np.arange(10), top10_features, rotation='vertical')\nplt.title('Feature Importance')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:38:17.936407Z","iopub.execute_input":"2022-07-22T07:38:17.936912Z","iopub.status.idle":"2022-07-22T07:38:18.173209Z","shell.execute_reply.started":"2022-07-22T07:38:17.936862Z","shell.execute_reply":"2022-07-22T07:38:18.171363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Inference and Submission\nprobas = pt_model.predict(X_test)\nprobas = np.squeeze(probas)\n\npreds = np.where(probas > 0.5, True, False)\n\nsubmission_df['Transported'] = preds\nsubmission_df.to_csv('submission_pt_pretrain.csv', index=False)\nsubmission_df.head()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:38:18.174792Z","iopub.execute_input":"2022-07-22T07:38:18.175182Z","iopub.status.idle":"2022-07-22T07:38:18.300896Z","shell.execute_reply.started":"2022-07-22T07:38:18.175148Z","shell.execute_reply":"2022-07-22T07:38:18.299515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n#### Public Score: ~ 0.787\n\n---","metadata":{}},{"cell_type":"markdown","source":"

5. TabNet from Scratch

","metadata":{}},{"cell_type":"markdown","source":"Now let's dive into the details of TabNet's architecture. I will construct TabNet with TensorFlow step by step. First of all, I will prepare the input data for the model: Datasets, Input Layers, Preprocessing Layers, and Embedding Layers. Construction of TabNet will start at 5.2 TabNet from Scratch[TensorFlow].","metadata":{}},{"cell_type":"markdown","source":"

5.1 Data Preprocessing for TabNet[TensorFlow]

","metadata":{}},{"cell_type":"markdown","source":"

5.1.1 Datasets

","metadata":{}},{"cell_type":"code","source":"def df_to_dataset(dataframe, num_columns, target=None,\n shuffle=False, repeat=False,\n batch_size=5, drop_remainder=False):\n df = dataframe.copy()\n if target is not None:\n labels = df.pop(target)\n data = {key: value[:, tf.newaxis] for key, value in df.items()}\n data = dict(data)\n ds = tf.data.Dataset.from_tensor_slices((data, labels))\n else:\n data = {key: value[:, tf.newaxis] for key, value in df.items()}\n data = dict(data)\n ds = tf.data.Dataset.from_tensor_slices(data)\n \n if shuffle:\n ds = ds.shuffle(buffer_size=len(df))\n if repeat:\n ds = ds.repeat()\n ds = ds.batch(batch_size, drop_remainder=drop_remainder)\n ds = ds.prefetch(batch_size)\n return ds","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:38:18.302947Z","iopub.execute_input":"2022-07-22T07:38:18.303434Z","iopub.status.idle":"2022-07-22T07:38:18.314173Z","shell.execute_reply.started":"2022-07-22T07:38:18.303372Z","shell.execute_reply":"2022-07-22T07:38:18.312642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Create datasets\nnum_columns = exp_config['num_columns']\nbatch_size = exp_config['batch_size']\n\ntrain_ds = df_to_dataset(train_fold, num_columns,\n target='Transported',\n shuffle=True,\n repeat=False,\n batch_size=batch_size,\n drop_remainder=False)\n\nvalid_ds = df_to_dataset(valid_fold, num_columns,\n target='Transported',\n shuffle=False,\n repeat=False,\n batch_size=batch_size,\n drop_remainder=False)\n\n## Display a batch sample\nexample = next(iter(train_ds))[0]\ninput_dtypes = {}\nfor key in example:\n input_dtypes[key] = example[key].dtype\n print(f'{key}, shape:{example[key].shape}, {example[key].dtype}')","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:38:18.316130Z","iopub.execute_input":"2022-07-22T07:38:18.316902Z","iopub.status.idle":"2022-07-22T07:38:18.575674Z","shell.execute_reply.started":"2022-07-22T07:38:18.316852Z","shell.execute_reply":"2022-07-22T07:38:18.574347Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

5.1.2 Preprocessing Layers

","metadata":{}},{"cell_type":"code","source":"## After binning, all features are categorical.\nnumerical_columns = []\ncategorical_columns = ['Age', 'RoomService', 'FoodCourt',\n 'ShoppingMall', 'Spa', 'VRDeck',\n 'HomePlanet', 'CryoSleep',\n 'Destination', 'VIP', \n 'Cabin_0', 'Cabin_1', 'Cabin_2']","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:38:18.578726Z","iopub.execute_input":"2022-07-22T07:38:18.579994Z","iopub.status.idle":"2022-07-22T07:38:18.584395Z","shell.execute_reply.started":"2022-07-22T07:38:18.579951Z","shell.execute_reply":"2022-07-22T07:38:18.583590Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Preprocessing model inputs\ndef create_preprocess_inputs(numerical, categorical, num_columns, input_dtypes):\n preprocess_inputs = {}\n numerical_inputs = {key: layers.Input(shape=(1,),\n dtype=input_dtypes[key]) for key in numerical}\n categorical_inputs = {key: layers.Input(shape=(1,), \n dtype=input_dtypes[key]) for key in categorical}\n preprocess_inputs.update(**numerical_inputs, **categorical_inputs)\n return preprocess_inputs\n\n\npreprocess_inputs = create_preprocess_inputs(numerical_columns,\n categorical_columns,\n num_columns,\n input_dtypes)\npreprocess_inputs","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:38:18.587805Z","iopub.execute_input":"2022-07-22T07:38:18.588125Z","iopub.status.idle":"2022-07-22T07:38:18.646967Z","shell.execute_reply.started":"2022-07-22T07:38:18.588096Z","shell.execute_reply":"2022-07-22T07:38:18.645757Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Create Preprocessing Model\ndef create_preprocessing_model(numerical, categorical,\n num_columns, input_dtypes, df):\n \n ## Create inputs\n preprocess_inputs = create_preprocess_inputs(numerical,\n categorical,\n num_columns,\n input_dtypes)\n \n ## Preprocessing layers for numerical_features\n normalize_layers = {}\n for nc in numerical:\n normalize_layer = layers.Normalization(mean=df[nc].mean(),\n variance=df[nc].var())\n normalize_layers[nc] = normalize_layer\n \n ## Preprocessing layers for categorical_features\n lookup_layers = {}\n for cc in categorical:\n if input_dtypes[cc] is tf.string:\n lookup_layer = layers.StringLookup(vocabulary=df[cc].unique(),\n output_mode='int')\n elif input_dtypes[cc] is tf.int64:\n lookup_layer = layers.IntegerLookup(vocabulary=df[cc].unique(),\n output_mode='int')\n lookup_layers[cc] = lookup_layer\n \n ## Create outputs\n preprocess_outputs = {}\n for key in preprocess_inputs:\n if key in normalize_layers:\n output = normalize_layers[key](preprocess_intputs[key])\n preprocess_outputs[key] = output\n elif key in lookup_layers:\n output = lookup_layers[key](preprocess_inputs[key])\n preprocess_outputs[key] = output\n \n ## Create model\n preprocessing_model = tf.keras.Model(preprocess_inputs,\n preprocess_outputs)\n \n return preprocessing_model, lookup_layers\n\n\npreprocessing_model, lookup_layers = create_preprocessing_model(numerical_columns,\n categorical_columns,\n num_columns,\n input_dtypes,\n train_fold)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:38:18.650277Z","iopub.execute_input":"2022-07-22T07:38:18.650712Z","iopub.status.idle":"2022-07-22T07:38:18.785534Z","shell.execute_reply.started":"2022-07-22T07:38:18.650677Z","shell.execute_reply":"2022-07-22T07:38:18.784327Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Apply the preprocessing model in tf.data.Dataset.map\ntrain_ds = train_ds.map(lambda x, y: (preprocessing_model(x), y),\n num_parallel_calls=tf.data.AUTOTUNE)\nvalid_ds = valid_ds.map(lambda x, y: (preprocessing_model(x), y),\n num_parallel_calls=tf.data.AUTOTUNE)\n\n## Display a preprocessed input sample\nexample = next(train_ds.take(1).as_numpy_iterator())[0]\nfor key in example:\n print(f'{key}, shape:{example[key].shape}, {example[key].dtype}')","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:38:18.787275Z","iopub.execute_input":"2022-07-22T07:38:18.787665Z","iopub.status.idle":"2022-07-22T07:38:19.245972Z","shell.execute_reply.started":"2022-07-22T07:38:18.787632Z","shell.execute_reply":"2022-07-22T07:38:19.244646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

5.1.3 Embedding Layers

","metadata":{}},{"cell_type":"code","source":"## Training Model Inputs\ndef create_model_inputs(numerical, categorical, input_dtypes):\n model_inputs = {}\n \n normalized_inputs = {key: layers.Input(shape=(1,),\n dtype=input_dtypes[key]) for key in numerical}\n lookup_inputs = {key: layers.Input(shape=(1,),\n dtype='int64') for key in categorical}\n model_inputs.update(**normalized_inputs, **lookup_inputs)\n return model_inputs\n\nmodel_inputs = create_model_inputs(numerical_columns,\n categorical_columns,\n input_dtypes)\nmodel_inputs","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:38:19.247849Z","iopub.execute_input":"2022-07-22T07:38:19.248628Z","iopub.status.idle":"2022-07-22T07:38:19.285862Z","shell.execute_reply.started":"2022-07-22T07:38:19.248553Z","shell.execute_reply":"2022-07-22T07:38:19.284683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Create Embedding Layers\ndef create_embedding_layers(model_inputs, numerical, categorical,\n lookup_layers, emb_dim):\n numerical_feature_list = []\n encoded_categorical_feature_list = []\n \n for key in model_inputs:\n if key in numerical:\n numerical_feature_list.append(model_inputs[key])\n elif key in categorical:\n ## Create Embeddings for categorical features\n embedding = layers.Embedding(input_dim=lookup_layers[key].vocabulary_size(),\n output_dim=emb_dim)\n encoded_categorical_feature = embedding(model_inputs[key])\n encoded_categorical_feature_list.append(encoded_categorical_feature)\n \n if len(numerical_feature_list) != 0:\n numerical_features = tf.concat(numerical_feature_list, axis=1)\n else:\n numerical_features = tf.stack(numerical_feature_list)\n \n encoded_categorical_features = tf.concat(encoded_categorical_feature_list, axis=1)\n return numerical_features, encoded_categorical_features\n\nemb_dim = model_config['emb_dim']\nnumerical_features, encoded_categorical_features = create_embedding_layers(model_inputs,\n numerical_columns,\n categorical_columns,\n lookup_layers,\n emb_dim)\nnumerical_features.shape, encoded_categorical_features.shape","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:38:19.287466Z","iopub.execute_input":"2022-07-22T07:38:19.287948Z","iopub.status.idle":"2022-07-22T07:38:19.408826Z","shell.execute_reply.started":"2022-07-22T07:38:19.287898Z","shell.execute_reply":"2022-07-22T07:38:19.407614Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

5.2 TabNet from Scratch[TensorFlow]

","metadata":{}},{"cell_type":"markdown","source":"As we saw earlier, TabNet consists of encoder and decoder. However, when we don't execute pre-training, we only need the encoder structure to make predictions. So at first, I will construct TabNet Encoder at section 5.2 TabNet from Scratch[TensorFlow]. Then, TabNet Decoder would be explained at section 5.3 Pre-training and Fine-tuning from Scratch.","metadata":{}},{"cell_type":"markdown","source":"

5.2.1 TabNet Encoder from Scratch

","metadata":{}},{"cell_type":"markdown","source":"The figure below shows the data-processing flows in TabNet Encoder. Each steps are called decision steps.\n\n\n\nPlease remember that tree-based calculation is executed by the multiplyng masks by features. And the architecture shows that masks are made by Attentive transformer block. The inside of Attentive transformer block is as follows.\n\n\n\nIn Attentive transformer block, features are transformed by Fully Connected Layer, and normalized by Batch Normalization Layer. Then, the results are clipped from zero to one by sparsemax function. Sparsemax is like sigmoid or softmax function, but a big difference is that sparsemax outputs zero, whereas sigmoid and softmax output above zero (please look at the figure below). Thus, features become sparse and are used as masks.\n\n\n\nThe masks represent the importance of features at each decision steps. If a feature is considered important for the first decision step, then other features should be selected at the following decision steps. Such adjustment is executed by Prior scales. Prior scales are calculated as follows:\n\n$$\nP[i]=\\prod^{i-1}_{j=1}\\left(\\gamma -M[j]\\right).\n$$\n\n$P[i]$ is Prior scales at the $i$th decision step. $M[j]$ is mask at the $j$th decision step. $\\gamma\\ge 1$ is called relaxation parameter. The Prior scales at the first decision step are all ones, and are updated at each steps.\n\nNow, we have understood the architecture of Attentive transformer block, so let's create it.","metadata":{}},{"cell_type":"code","source":"class AttentiveTransformer(layers.Layer):\n def __init__(self,\n units: Optional[int]=None):\n super().__init__()\n self.units = units\n \n def build(self,\n input_shape: tf.TensorShape):\n if self.units is None:\n self.units = input_shape[-1]\n \n self.fc = layers.Dense(self.units,\n use_bias=False)\n self.bn = layers.BatchNormalization()\n \n def call(self,\n inputs: Union[tf.Tensor, np.ndarray],\n priors: Optional[Union[tf.Tensor, np.ndarray]]=None,\n training: Optional[bool]=None) -> tf.Tensor:\n x = self.fc(inputs, training=training)\n x = self.bn(x, training=training)\n \n if priors is None:\n outputs = x\n else:\n outputs = x * priors\n \n return tfa.activations.sparsemax(outputs)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:38:19.410021Z","iopub.execute_input":"2022-07-22T07:38:19.410382Z","iopub.status.idle":"2022-07-22T07:38:19.420762Z","shell.execute_reply.started":"2022-07-22T07:38:19.410350Z","shell.execute_reply":"2022-07-22T07:38:19.419479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The results of multiplying the normalized inputs by the mask made with Attentive transformer block is successively going to be the input of Feature transformer block. Next, let's look at the structure of Feature transformer block.\n\n\n\nFeature transformer block consists of Fully Connected Layers, Batch Normalization Layers, and GLU Layers. GLU means \"Gated Linear Unit\", and calculates the following formula:\n\n$$\n(X*W+b)\\otimes \\sigma(X*V+c)\n$$\n\n$X$ is input, and $(X*W+b)$ is a linear transformation. Because $\\sigma(\\cdot)$ is sigmoid function, $\\sigma(X*V+c)$ works as an information gate: The result of $(X*V+c)$ is clipped from 0 to 1, and indicates the percentage of passing the corresponding value of $(X*W+b)$. You can also find skip (residual) connections.\n\nAs the above figure shows, the parameters of first half of Feature transformer block is shared across decision steps, and the second half of it is not. Then, let's define GLU Layer and Feature transformer block.","metadata":{}},{"cell_type":"code","source":"class GLULayer(layers.Layer):\n def __init__(self,\n units: Optional[int]=None):\n super().__init__()\n self.units = units\n \n def build(self,\n input_shape: tf.TensorShape):\n if self.units is None:\n self.units = input_shape[-1]\n self.fc_output = layers.Dense(self.units)\n self.fc_gate = layers.Dense(self.units)\n \n def call(self, \n inputs: Union[tf.Tensor, np.ndarray],\n training: Optional[bool]=None):\n output = self.fc_output(inputs, training=training)\n gate = self.fc_gate(inputs, training=training)\n return output * tf.keras.activations.sigmoid(gate)\n\n \nclass FCBNGLUBlock(layers.Layer):\n def __init__(self,\n units: Optional[int]=None):\n super().__init__()\n self.units = units\n \n def build(self,\n input_shape: tf.TensorShape):\n if self.units is None:\n self.units = input_shape[-1]\n self.fc = layers.Dense(self.units)\n self.bn = layers.BatchNormalization()\n self.glu = GLULayer(self.units)\n \n def call(self,\n inputs: Union[tf.Tensor, np.ndarray],\n training: Optional[bool]=None):\n x = self.fc(inputs, training=training)\n x = self.bn(x, training=training)\n outputs = self.glu(x, training=training)\n return outputs\n\n\nclass FeatureTransformerBlock(layers.Layer):\n def __init__(self, \n units: Optional[int]=None,\n skip: bool=False):\n super().__init__()\n self.units = units\n self.skip = skip\n \n def build(self,\n input_shape: tf.TensorShape):\n if self.units is None:\n self.units = input_shape[-1]\n \n self.initial = FCBNGLUBlock(units=self.units)\n self.residual = FCBNGLUBlock(units=self.units)\n \n def call(self,\n inputs: Union[tf.Tensor, np.ndarray],\n training: Optional[bool]=None):\n initial = self.initial(inputs, training=training)\n \n if self.skip == True:\n initial = (initial + inputs) * np.sqrt(0.5)\n \n residual = self.residual(initial, training=training)\n outputs = (initial + residual) * np.sqrt(0.5)\n return outputs","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:38:19.422329Z","iopub.execute_input":"2022-07-22T07:38:19.422806Z","iopub.status.idle":"2022-07-22T07:38:19.442123Z","shell.execute_reply.started":"2022-07-22T07:38:19.422772Z","shell.execute_reply":"2022-07-22T07:38:19.441247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, we have reached the stage of constructing the whole TabNet encoder. Let's see again the architecture.\n\n\n\nSplit layer in $i$th decision step divides the result of Feature transformer block into $a[i]\\in \\mathbb{R}^{B\\times N_a}$ and $d[i]\\in\\mathbb{R}^{B\\times N_d}$. $a[i]$ is the input of Attentive transformer block in the next step. $d[i]$ is used for two purposes. One is to calculate the final output of the encoder model, called \"decision embedding\". Decision embedding $d_{out}$, which is used for the final prediciton of TabNet model, is calculated as follows:\n\n$$\nd_{out}=\\sum^{N_{steps}}_{i=1}\\text{ReLU}(d[i])\n$$\n\nAnother purpose is to calculate the importance of features (Feature attributes). Mask in the $i$th step $M[i]\\in \\mathbb{R}^{B\\times N_d}$ simply represents the feature importances in the $i$th decision step. To calculate the feature importances over the whole decision steps $M_{agg}\\in \\mathbb{R}^{B\\times N_d}$, the importance of each decision steps itself $\\eta[i]$ should be considered. $\\eta[i]$ is calculated as follows:\n\n$$\n\\eta_{b}[i]=\\sum_{c=1}^{N_d}\\text{ReLU}(d_{b, c}[i])\n$$\n\nThen, Feature attributes $M_{agg}$, which is the weighted average of $M[i]$, is calculated as follows:\n\n$$\nM_{agg-b,j}=\\frac{\\sum^{N_{steps}}_{i=1}\\eta_b[i]M_{b,j}[i]}{\\sum^D_{j=1}\\sum^{N_{steps}}_{i=1}\\eta_b[i]M_{b,j}[i]}\n$$\n\n---\nIn addition, there are some points that you can't see in the architecture figure.\n\n- **The sparsity loss:** For the loss calculation, the sparsity of the mask is considered like this.\n\n $$\n Loss + \\lambda_{sparse} L_{sparse}\n $$\n\n $\\lambda_{sparse}$ is a hyperparameter that controls the impact of sparsity loss, $L_{sparse}$, which is calculated as follows:\n\n $$\n L_{sparse}=\\sum^{N_{step}}_{i=1}\\sum^{B}_{b=1}\\sum^{D}_{j=1}\\frac{-{\\bf{M}}_{b, j}[i]\\log\\left({\\bf{M}}_{b, j}[i]+\\epsilon\\right)}{N_{steps}\\cdot B}\n $$\n\n The graph of $-x\\log x$ looks like the figure below. It means that when the mask values greatly differ from 0 or 1, the sparsity loss increases. Intermediate values would be the penalty.\n\n \n\n \n- **Ghost Batch Normalization:** In the original implementation, batch normalization in Feature transformer block is executed on the ghost batch units, which size is smaller than mini batch. However, in this TabNet model from scratch, I used normal Batch Normalization Layer.\n\nNow, let's complete building TabNet encoder model!","metadata":{}},{"cell_type":"code","source":"class TabNetEncoder(layers.Layer):\n def __init__(self,\n output_dims: int=1,\n n_steps: int=3,\n feature_dims: int=16,\n gamma: float=1.3,\n epsilon: float=1e-8,\n lambda_sparsity: float=1e-4):\n super().__init__()\n self.output_dims = output_dims\n self.n_steps = n_steps\n self.feature_dims = feature_dims\n self.gamma = gamma\n self.epsilon = epsilon\n self.lambda_sparsity = lambda_sparsity\n \n def build(self,\n input_shape: tf.TensorShape):\n self.bn_initial = layers.BatchNormalization()\n self.shared_ft = FeatureTransformerBlock(units=self.feature_dims*2)\n self.final_decision = layers.Dense(self.output_dims)\n \n self.ft_initial = FeatureTransformerBlock(units=self.feature_dims*2, skip=True)\n self.attentives = [AttentiveTransformer(units=input_shape[-1]) for _ in range(self.n_steps)]\n self.unique_fts = [FeatureTransformerBlock(units=self.feature_dims*2,\n skip=True) for _ in range(self.n_steps)]\n \n def call(self, \n X: Union[tf.Tensor, np.ndarray],\n training: Optional[bool]=None) -> Tuple[tf.Tensor]:\n sparse_loss = 0.\n encoded = []\n final_decision_inputs = 0.\n attributes = tf.zeros_like(X)\n priors = tf.ones_like(X)\n \n B = self.bn_initial(X, training=training)\n shared = self.shared_ft(B, training=training)\n initial_split = self.ft_initial(shared, training=training)\n attn_inputs = initial_split[:, :self.feature_dims]\n \n for i in range(self.n_steps):\n mask = self.attentives[i](attn_inputs, priors, training=training)\n shared = self.shared_ft(mask * B, training=training)\n split = self.unique_fts[i](shared, training=training)\n attn_inputs = split[:, :self.feature_dims]\n encoded_repr = split[:, self.feature_dims:]\n encoded.append(encoded_repr)\n \n step_outputs = tf.keras.activations.relu(encoded_repr)\n final_decision_inputs += step_outputs\n \n step_importance = tf.reduce_sum(step_outputs, axis=1, keepdims=True)\n attributes += step_importance * mask\n \n sparse_loss += tf.reduce_mean(tf.reduce_sum(-mask * tf.math.log(mask + self.epsilon), axis=-1)) / tf.cast(self.n_steps, tf.float32)\n priors *= tf.cast(self.gamma, tf.float32) - mask\n \n self.add_loss(self.lambda_sparsity * sparse_loss)\n outputs = self.final_decision(final_decision_inputs, training=training)\n attr_sum = tf.reduce_sum(attributes, axis=-1, keepdims=True)\n importance = attributes / attr_sum\n encoded = tf.stack(encoded)\n \n return outputs, importance, encoded","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:38:19.443638Z","iopub.execute_input":"2022-07-22T07:38:19.444589Z","iopub.status.idle":"2022-07-22T07:38:19.465483Z","shell.execute_reply.started":"2022-07-22T07:38:19.444539Z","shell.execute_reply":"2022-07-22T07:38:19.464145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class TabNet(tf.keras.Model):\n def __init__(self,\n output_dims: int=1,\n n_steps: int=3,\n feature_dims: int=16,\n gamma: float=1.3,\n epsilon: float=1e-8,\n lambda_sparsity: float=1e-4,\n pretrained_encoder: Optional[layers.Layer]=None):\n \n super().__init__()\n self.output_dims = output_dims\n self.n_steps = n_steps\n self.feature_dims = feature_dims\n self.gamma = gamma\n self.epsilon = epsilon\n self.lambda_sparsity = lambda_sparsity\n self.pretrained_encoder = pretrained_encoder\n \n if self.pretrained_encoder is None:\n self.encoder = TabNetEncoder(output_dims=output_dims,\n n_steps=n_steps,\n feature_dims=feature_dims,\n gamma=gamma,\n epsilon=epsilon,\n lambda_sparsity=lambda_sparsity)\n else:\n self.encoder = pretrained_encoder\n \n def forward(self,\n X: Union[tf.Tensor, np.ndarray],\n training: Optional[bool]=None) -> Tuple[tf.Tensor]:\n outputs, importance, encoded = self.encoder(X, training=training)\n return outputs, importance, encoded\n \n def call(self,\n X: Union[tf.Tensor, np.ndarray],\n training: Optional[bool]=None) -> tf.Tensor:\n outputs, importance, _ = self.forward(X, training=training)\n return outputs, importance\n \n def explain(self,\n X: Union[tf.Tensor, np.ndarray],\n training: Optional[bool]=None) -> tf.Tensor:\n _, importance, _ = self.forward(X, training=training)\n return importance\n \n def transform(self,\n X: Union[tf.Tensor, np.ndarray],\n training: Optional[bool]=None) -> tf.Tensor:\n _, _, encoded = self.forward(X, training=training)\n return encoded","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:38:19.467457Z","iopub.execute_input":"2022-07-22T07:38:19.468231Z","iopub.status.idle":"2022-07-22T07:38:19.486430Z","shell.execute_reply.started":"2022-07-22T07:38:19.468180Z","shell.execute_reply":"2022-07-22T07:38:19.485145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Create TabNet model\ndef create_tabnet(emb_dim,\n numerical_columns,\n categorical_columns,\n num_columns,\n input_dtypes,\n lookup_layers,\n output_dims,\n n_steps,\n feature_dims,\n gamma,\n epsilon,\n lambda_sparsity,\n pretrained_encoder=None):\n \n model_inputs = create_model_inputs(numerical_columns,\n categorical_columns,\n input_dtypes)\n \n numerical_features, encoded_categorical_features = create_embedding_layers(model_inputs,\n numerical_columns,\n categorical_columns,\n lookup_layers,\n emb_dim)\n \n encoded_categorical_features = layers.Flatten(\n )(encoded_categorical_features)\n \n \n ## Numerical features\n if len(numerical_columns) > 0:\n numerical_features = layers.LayerNormalization(\n name=f'numerical_norm', \n epsilon=1e-6\n )(numerical_features)\n \n ## Concatenate categorical features with numerical features\n features = layers.Concatenate()([\n encoded_categorical_features,\n numerical_features\n ])\n else:\n features = encoded_categorical_features\n \n tabnet = TabNet(output_dims=output_dims,\n n_steps=n_steps,\n feature_dims=feature_dims,\n gamma=gamma,\n epsilon=epsilon,\n lambda_sparsity=lambda_sparsity,\n pretrained_encoder=pretrained_encoder)\n \n tab_outputs, tab_importance = tabnet(features)\n model_outputs = tf.keras.activations.sigmoid(tab_outputs)\n \n ## Create model\n training_model = keras.Model(inputs=model_inputs,\n outputs=model_outputs)\n \n return training_model\n \n## Settings for TabNet\nemb_dim = model_config['emb_dim']\ninput_dims = model_config['input_dims']\noutput_dims = model_config['output_dims']\nn_steps = model_config['n_steps']\nfeature_dims = model_config['feature_dims']\ngamma = model_config['gamma']\nepsilon = model_config['epsilon']\nlambda_sparsity = model_config['lambda_sparsity']\n\n## Create TabNet\ntraining_model = create_tabnet(emb_dim,\n numerical_columns,\n categorical_columns,\n num_columns,\n input_dtypes,\n lookup_layers,\n output_dims=output_dims,\n n_steps=n_steps,\n feature_dims=feature_dims,\n gamma=gamma,\n epsilon=epsilon,\n lambda_sparsity=lambda_sparsity)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:38:19.488030Z","iopub.execute_input":"2022-07-22T07:38:19.488860Z","iopub.status.idle":"2022-07-22T07:38:21.488382Z","shell.execute_reply.started":"2022-07-22T07:38:19.488819Z","shell.execute_reply":"2022-07-22T07:38:21.486949Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Model Compile and Build\nbatch_size = exp_config['batch_size']\nlearning_rate = exp_config['learning_rate']\ntrain_epochs = exp_config['train_epochs']\nsteps_per_epoch = len(train_fold) // batch_size\n\nlearning_schedule = tf.keras.optimizers.schedules.CosineDecay(\n initial_learning_rate=learning_rate,\n decay_steps=train_epochs * steps_per_epoch, \n alpha=0.0)\n\noptimizer = keras.optimizers.Adam(learning_rate=learning_schedule)\n\nloss_fn = keras.losses.BinaryCrossentropy(\n from_logits=False, \n label_smoothing=model_config['label_smoothing'])\n\ntraining_model.compile(optimizer=optimizer,\n loss=loss_fn,\n metrics=['accuracy', keras.metrics.AUC()])\n\n#training_model.build(input_shape=(None, input_dims))\ntraining_model.summary()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:38:21.489924Z","iopub.execute_input":"2022-07-22T07:38:21.490290Z","iopub.status.idle":"2022-07-22T07:38:21.532676Z","shell.execute_reply.started":"2022-07-22T07:38:21.490254Z","shell.execute_reply":"2022-07-22T07:38:21.531653Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

5.2.2 Training

","metadata":{}},{"cell_type":"code","source":"## Learning Rate Finder\nclass LRFind(tf.keras.callbacks.Callback):\n def __init__(self, min_lr, max_lr, n_rounds):\n self.min_lr = min_lr\n self.max_lr = max_lr\n self.step_up = tf.constant((max_lr / min_lr) ** (1 / n_rounds))\n self.lrs = []\n self.losses = []\n \n def on_train_begin(self, logs=None):\n self.weights = self.model.get_weights()\n self.model.optimizer.lr = self.min_lr\n \n def on_train_batch_end(self, batch, logs=None):\n self.lrs.append(self.model.optimizer.lr.numpy())\n self.losses.append(logs['loss'])\n self.model.optimizer.lr = self.model.optimizer.lr * self.step_up\n if self.model.optimizer.lr > self.max_lr:\n self.model.stop_training = True \n \n def on_train_end(self, logs=None):\n self.model.set_weights(self.weights)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:38:21.534477Z","iopub.execute_input":"2022-07-22T07:38:21.535118Z","iopub.status.idle":"2022-07-22T07:38:21.544592Z","shell.execute_reply.started":"2022-07-22T07:38:21.535067Z","shell.execute_reply":"2022-07-22T07:38:21.543590Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Execute LR Finding\nmin_lr = 1e-6\nmax_lr = 1e-0\nlr_find_epochs = 1\nlr_find_steps = 100\nlr_find_batch_size = 512\n\nlr_find = LRFind(min_lr, max_lr, lr_find_steps)\nlr_find_ds = df_to_dataset(train_fold, num_columns,\n target='Transported',\n repeat=True,\n batch_size=lr_find_batch_size)\nlr_find_ds = lr_find_ds.map(lambda x, y: (preprocessing_model(x), y),\n num_parallel_calls=tf.data.AUTOTUNE)\n\ntraining_model.fit(lr_find_ds,\n steps_per_epoch=lr_find_steps,\n epochs=lr_find_epochs,\n callbacks=[lr_find])\n\nplt.plot(lr_find.lrs, lr_find.losses)\nplt.xscale('log')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:38:21.546001Z","iopub.execute_input":"2022-07-22T07:38:21.546613Z","iopub.status.idle":"2022-07-22T07:38:38.569447Z","shell.execute_reply.started":"2022-07-22T07:38:21.546546Z","shell.execute_reply":"2022-07-22T07:38:38.568160Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Re-construct the model\ntraining_model = create_tabnet(emb_dim,\n numerical_columns,\n categorical_columns,\n num_columns,\n input_dtypes,\n lookup_layers,\n output_dims=output_dims,\n n_steps=n_steps,\n feature_dims=feature_dims,\n gamma=gamma,\n epsilon=epsilon,\n lambda_sparsity=lambda_sparsity)\n\n## Model Compile and Build\nemb_learning_rate = exp_config['emb_learning_rate']\nlearning_rate = exp_config['learning_rate']\nbatch_size = exp_config['batch_size']\ntrain_epochs = exp_config['train_epochs']\nsteps_per_epoch = len(train_fold) // batch_size\n\nlearning_schedule_1 = tf.keras.optimizers.schedules.CosineDecay(\n initial_learning_rate=emb_learning_rate,\n decay_steps=train_epochs * steps_per_epoch, \n alpha=0.0)\n\nlearning_schedule_2 = tf.keras.optimizers.schedules.CosineDecay(\n initial_learning_rate=learning_rate,\n decay_steps=train_epochs * steps_per_epoch, \n alpha=0.0)\n\noptimizers = [\n tf.keras.optimizers.Adam(learning_rate=learning_schedule_1),\n tf.keras.optimizers.Adam(learning_rate=learning_schedule_2)\n]\noptimizers_and_layers = [(optimizers[0], training_model.layers[:26]),\n (optimizers[1], training_model.layers[26:])]\noptimizer = tfa.optimizers.MultiOptimizer(optimizers_and_layers)\n\nloss_fn = keras.losses.BinaryCrossentropy(\n from_logits=False, \n label_smoothing=model_config['label_smoothing'])\n\ntraining_model.compile(optimizer=optimizer,\n loss=loss_fn,\n metrics=['accuracy', keras.metrics.AUC()])","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:38:38.571033Z","iopub.execute_input":"2022-07-22T07:38:38.571426Z","iopub.status.idle":"2022-07-22T07:38:39.686434Z","shell.execute_reply.started":"2022-07-22T07:38:38.571396Z","shell.execute_reply":"2022-07-22T07:38:39.685323Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Checkpoint Callback\ncheckpoint_filepath = exp_config['checkpoint_filepath']\nmodel_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n filepath=checkpoint_filepath, \n save_weights_only=True, \n monitor='val_loss', \n mode='min', \n save_best_only=True)\n\n## Model Training\nhistory = training_model.fit(train_ds,\n epochs=train_epochs,\n shuffle=True,\n validation_data=valid_ds,\n callbacks=[model_checkpoint_callback])\n\n## Load the Best Parameters\ntraining_model.load_weights(checkpoint_filepath)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:38:39.689755Z","iopub.execute_input":"2022-07-22T07:38:39.690111Z","iopub.status.idle":"2022-07-22T07:39:47.453485Z","shell.execute_reply.started":"2022-07-22T07:38:39.690078Z","shell.execute_reply":"2022-07-22T07:39:47.452254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Plot Losses\ndef plot_history(hist, title=None, valid=True):\n plt.figure(figsize=(7, 5))\n plt.plot(np.array(hist.index), hist['loss'], label='Train Loss')\n if valid:\n plt.plot(np.array(hist.index), hist['val_loss'], label='Valid Loss')\n plt.xlabel('Epoch')\n plt.ylabel('Loss')\n plt.legend()\n plt.title(title)\n plt.show()\n \nhist = pd.DataFrame(history.history)\nplot_history(hist)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:39:47.454756Z","iopub.execute_input":"2022-07-22T07:39:47.455113Z","iopub.status.idle":"2022-07-22T07:39:47.675458Z","shell.execute_reply.started":"2022-07-22T07:39:47.455077Z","shell.execute_reply":"2022-07-22T07:39:47.674385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Plot Feature Importance\nlayer_names = [l.name for l in training_model.layers]\nname = layer_names[-2]\n\nimportance_model = tf.keras.Model(inputs=training_model.input,\n outputs=training_model.get_layer(name).output[1])\n\nimportance_list = []\nfor data, label in iter(train_ds):\n importance = importance_model(data)\n importance = importance.numpy()\n importance_list.append(importance)\n \nimportance = np.concatenate(importance_list, axis=0)\nimportance = np.mean(importance, axis=0)\nimportance_df = pd.DataFrame(importance,\n index=list(training_model.input.keys()),\n columns=['importance'])\nimportance_df.sort_values(by='importance',\n ascending=False,\n inplace=True)\ntop10_importance = np.array(importance_df[:10].values.reshape([-1]))\ntop10_features = importance_df[:10].index\n\n#importance_df[:10].plot.bar()\n\nplt.figure(figsize=(7, 5))\nplt.bar(x=np.arange(10),\n height=top10_importance)\nplt.xticks(np.arange(10), top10_features, rotation='vertical')\nplt.title('Feature Importance')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:39:47.676761Z","iopub.execute_input":"2022-07-22T07:39:47.677621Z","iopub.status.idle":"2022-07-22T07:39:48.812818Z","shell.execute_reply.started":"2022-07-22T07:39:47.677584Z","shell.execute_reply":"2022-07-22T07:39:48.811790Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

5.2.3 Inference

","metadata":{}},{"cell_type":"code","source":"## Inference_model = preprocessing_model + training_model\ninference_inputs = preprocessing_model.input\ninference_outputs = training_model(preprocessing_model(inference_inputs))\ninference_model = tf.keras.Model(inputs=inference_inputs,\n outputs=inference_outputs)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:39:48.814739Z","iopub.execute_input":"2022-07-22T07:39:48.815087Z","iopub.status.idle":"2022-07-22T07:39:49.392418Z","shell.execute_reply.started":"2022-07-22T07:39:48.815055Z","shell.execute_reply":"2022-07-22T07:39:49.390889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Test Dataset\ntest_ds = df_to_dataset(test, num_columns,\n target=None,\n shuffle=False,\n repeat=False,\n batch_size=batch_size,\n drop_remainder=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:39:49.394007Z","iopub.execute_input":"2022-07-22T07:39:49.394634Z","iopub.status.idle":"2022-07-22T07:39:49.408252Z","shell.execute_reply.started":"2022-07-22T07:39:49.394592Z","shell.execute_reply":"2022-07-22T07:39:49.406977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Inference and Submission\nprobas = inference_model.predict(test_ds)\nprobas = np.squeeze(probas)\n\npreds = np.where(probas > 0.5, True, False)\n\nsubmission_df['Transported'] = preds\nsubmission_df.to_csv('submission_tf.csv', index=False)\nsubmission_df.head()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:39:49.409840Z","iopub.execute_input":"2022-07-22T07:39:49.410898Z","iopub.status.idle":"2022-07-22T07:39:51.087434Z","shell.execute_reply.started":"2022-07-22T07:39:49.410833Z","shell.execute_reply":"2022-07-22T07:39:51.086625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n#### Public Score: ~ 0.743\n\n---","metadata":{}},{"cell_type":"markdown","source":"

5.3 Pre-training and Fine-tuning from Scratch

","metadata":{}},{"cell_type":"markdown","source":"In this section, I'm going to build TabNet encoder-decoder (autoencoder) model for the pre-training and fine-tuning as we saw in chapter 4.2.","metadata":{}},{"cell_type":"markdown","source":"

5.3.1 TabNet Encoder-Decoder from Scratch

","metadata":{}},{"cell_type":"markdown","source":"The figure below shows the architecture of TabNet decoder.\n\n\n\nTabNet decoder receives the output of the encoder, which is dimensionally reduced, and reconstructs the input features of the encoder. TabNet decoder consists of Feature transformer blocks and Fully Connected Layers. They have same structures as we saw earlier. So, let's create the decoder.","metadata":{}},{"cell_type":"code","source":"class TabNetDecoder(layers.Layer):\n def __init__(self,\n output_dims: int=1,\n n_steps: int=3,\n feature_dims: int=16):\n super().__init__()\n self.output_dims = output_dims\n self.n_steps = n_steps\n self.feature_dims = feature_dims\n \n def build(self,\n input_shape: tf.TensorShape):\n self.shared_block = FeatureTransformerBlock(units=self.feature_dims)\n self.unique_fts = [FeatureTransformerBlock(units=self.feature_dims,\n skip=True) for _ in range(self.n_steps)]\n self.fcs = [layers.Dense(self.output_dims) for _ in range(self.n_steps)]\n \n def call(self,\n encoded: Union[tf.Tensor, np.ndarray],\n training: Optional[bool]=None) -> tf.Tensor:\n decoded = 0.\n \n for i in range(self.n_steps):\n shared = self.shared_block(encoded[i], training=training)\n feature = self.unique_fts[i](shared, training=training)\n outputs = self.fcs[i](feature, training=training)\n decoded += outputs\n \n return decoded","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:39:51.089154Z","iopub.execute_input":"2022-07-22T07:39:51.090137Z","iopub.status.idle":"2022-07-22T07:39:51.101730Z","shell.execute_reply.started":"2022-07-22T07:39:51.090087Z","shell.execute_reply":"2022-07-22T07:39:51.100185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Reconstruction error for the pre-training of TabNet autoencoder is defined by the following formula:\n\n$$\nL_{reconstruct}=\\sum^B_{b=1}\\sum^D_{d=1}\\left|\\frac{\\left(\\hat{f}_{b,j}-f_{b,j}\\right)\\cdot S_{b,j}}{\\sqrt{\\sum^B_{b=1}\\left(f_{b,j}-1/B\\sum^{B}_{b=1}f_{b,j}\\right)^2}}\\right|\n$$\n\n$𝑓\\in\\mathbb{R}^{B\\times D}$ is input feature, and $\\hat{f}\\in\\mathbb{R}^{B\\times D}$ is decoder's prediction. $S\\in \\{0, 1\\}^{B\\times D}$ is input-mask. When $S_{b, j}=1$, corresponding input feature is masked ($f_{b, j}$ becomes 0). Conversely, the prediction error $|\\hat{f}_{b,j}-f_{b,j}|$ is added to the reconstruction loss only when $S_{b, j}=1$. Please note that input-mask $S$ is different from the mask inside the TabNet encoder $M$. The denominator of the loss is a kind of normalization term. That's the original reconstruction error, but in this TabNet model from scratch, I will use mean squared error as the reconstruction loss.\n\nFinally, we will build the autoencoder model structure with connecting our TabNet encoder and TabNet decoder. ","metadata":{}},{"cell_type":"code","source":"class TabNetEncoderDecoder(layers.Layer):\n def __init__(self,\n output_dims: int=1,\n input_dims: int=12,\n n_steps: int=3,\n feature_dims: int=16,\n gamma: float=1.3,\n epsilon: float=1e-8,\n lambda_sparsity: float=1e-4,\n mask_ratio: float=0.25):\n super().__init__()\n self.output_dims = output_dims\n self.input_dims = input_dims\n self.n_steps = n_steps\n self.feature_dims = feature_dims\n self.gamma = gamma\n self.epsilon = epsilon\n self.lambda_sparsity = lambda_sparsity\n self.mask_ratio = mask_ratio\n \n def build(self,\n input_shape: tf.TensorShape):\n self.encoder = TabNetEncoder(\n output_dims=self.output_dims,\n n_steps=self.n_steps,\n feature_dims=self.feature_dims,\n gamma=self.gamma,\n epsilon=self.epsilon,\n lambda_sparsity=self.lambda_sparsity)\n self.decoder = TabNetDecoder(\n output_dims=self.input_dims,\n n_steps=self.n_steps,\n feature_dims=self.feature_dims)\n self.mask_creator = layers.Dropout(self.mask_ratio)\n \n def call(self,\n X: Union[tf.Tensor, np.ndarray],\n training: Optional[bool]=None) -> tf.Tensor:\n ## training mask\n input_mask = self.mask_creator(tf.ones_like(X), training=training)\n M = tf.where(input_mask != 0., tf.ones_like(X), tf.zeros_like(X))\n S = tf.where(input_mask ==0., tf.ones_like(X), tf.zeros_like(X))\n \n ## encoding\n X_M = X * M\n encoder_outputs, importance, encoded = self.encoder(X_M, training=training)\n #labels = tf.where(mask == 0., X, tf.zeros_like(X))\n \n ## decoding\n decoder_outputs = self.decoder(encoded, training=training)\n \n ## reconstruction-loss\n yt = X * S\n preds = decoder_outputs * S\n mse = tf.keras.losses.MeanSquaredError()\n self.add_loss(mse(yt, preds))\n \n return encoder_outputs, importance, encoded","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:39:51.103261Z","iopub.execute_input":"2022-07-22T07:39:51.103724Z","iopub.status.idle":"2022-07-22T07:39:51.123839Z","shell.execute_reply.started":"2022-07-22T07:39:51.103689Z","shell.execute_reply":"2022-07-22T07:39:51.122660Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class TabNetAutoencoder(tf.keras.Model):\n def __init__(self,\n output_dims: int=1,\n input_dims: int=12,\n n_steps: int=3,\n feature_dims: int=16,\n gamma: float=1.3,\n epsilon: float=1e-8,\n lambda_sparsity: float=1e-4,\n mask_ratio: float=0.25):\n super().__init__()\n self.output_dims = output_dims\n self.input_dims = input_dims\n self.n_steps = n_steps\n self.feature_dims = feature_dims\n self.gamma = gamma\n self.epsilon = epsilon\n self.lambda_sparsity = lambda_sparsity\n self.mask_ratio = mask_ratio\n \n self.ae = TabNetEncoderDecoder(output_dims=output_dims,\n input_dims=input_dims,\n n_steps=n_steps,\n feature_dims=feature_dims,\n gamma=gamma,\n epsilon=epsilon,\n lambda_sparsity=lambda_sparsity,\n mask_ratio=mask_ratio)\n \n def forward(self,\n X: Union[tf.Tensor, np.ndarray],\n training: Optional[bool]=None) -> Tuple[tf.Tensor]:\n encoder_outputs, importance, encoded = self.ae(X, training=training)\n \n return encoder_outputs, importance, encoded\n \n def call(self,\n X: Union[tf.Tensor, np.ndarray],\n training: Optional[bool]=None) -> tf.Tensor:\n encoder_outputs, importance, encoded = self.forward(X)\n \n return encoder_outputs\n \n def explain(self,\n X: Union[tf.Tensor, np.ndarray],\n training: Optional[bool]=None) -> tf.Tensor:\n _, importance, _ = self.forward(X)\n return importance\n \n def transform(self,\n X: Union[tf.Tensor, np.ndarray],\n training: Optional[bool]=None) -> tf.Tensor:\n _, _, encoded = self.forward(X)\n return encoded","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:39:51.125699Z","iopub.execute_input":"2022-07-22T07:39:51.126463Z","iopub.status.idle":"2022-07-22T07:39:51.144000Z","shell.execute_reply.started":"2022-07-22T07:39:51.126424Z","shell.execute_reply":"2022-07-22T07:39:51.142836Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def create_tabnet_ae(emb_dim,\n numerical_columns,\n categorical_columns,\n num_columns,\n input_dtypes,\n lookup_layers,\n output_dims,\n n_steps,\n feature_dims,\n gamma,\n epsilon,\n lambda_sparsity,\n mask_ratio):\n \n model_inputs = create_model_inputs(numerical_columns,\n categorical_columns,\n input_dtypes)\n \n numerical_features, encoded_categorical_features = create_embedding_layers(model_inputs,\n numerical_columns,\n categorical_columns,\n lookup_layers,\n emb_dim)\n \n encoded_categorical_features = layers.Flatten(\n )(encoded_categorical_features)\n \n \n ## Numerical features\n if len(numerical_columns) > 0:\n numerical_features = layers.LayerNormalization(\n name=f'numerical_norm', \n epsilon=1e-6\n )(numerical_features)\n \n ## Concatenate categorical features with numerical features\n features = layers.Concatenate()([\n encoded_categorical_features,\n numerical_features\n ])\n else:\n features = encoded_categorical_features\n \n tabnet_ae = TabNetAutoencoder(\n output_dims=output_dims,\n input_dims=input_dims,\n n_steps=n_steps,\n feature_dims=feature_dims,\n gamma=gamma,\n epsilon=epsilon,\n lambda_sparsity=lambda_sparsity,\n mask_ratio=mask_ratio)\n \n model_outputs = tabnet_ae(features)\n \n ## Create model\n training_model = keras.Model(inputs=model_inputs,\n outputs=model_outputs)\n \n return training_model","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:39:51.145781Z","iopub.execute_input":"2022-07-22T07:39:51.147019Z","iopub.status.idle":"2022-07-22T07:39:51.165275Z","shell.execute_reply.started":"2022-07-22T07:39:51.146963Z","shell.execute_reply":"2022-07-22T07:39:51.164137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Settings for TabNet Encoder-Decoder\ninput_dims = model_config['input_dims']\noutput_dims = model_config['output_dims']\nn_steps = model_config['n_steps']\nfeature_dims = model_config['feature_dims']\nlambda_sparsity = model_config['lambda_sparsity']\ngamma = model_config['gamma']\nepsilon = model_config['epsilon']\nlambda_sparsity = model_config['lambda_sparsity']\nmask_ratio = model_config['mask_ratio']\n\n## Create TabNet Encoder-Decoder\ntabnet_ae = create_tabnet_ae(\n emb_dim,\n numerical_columns,\n categorical_columns,\n num_columns,\n input_dtypes,\n lookup_layers,\n output_dims=output_dims,\n n_steps=n_steps,\n feature_dims=feature_dims,\n gamma=gamma,\n epsilon=epsilon,\n lambda_sparsity=lambda_sparsity,\n mask_ratio=mask_ratio)\n\n## Model Compile and Build\nlr = exp_config['pretrain_lr']\noptimizer = keras.optimizers.Adam(learning_rate=lr)\n\n@tf.function\ndef dummy_loss(y, t):\n return 0.\n\ntabnet_ae.compile(optimizer=optimizer,\n loss=dummy_loss)\n\n#tabnet_ae.build(input_shape=(None, input_dims))\ntabnet_ae.summary()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:39:51.166872Z","iopub.execute_input":"2022-07-22T07:39:51.167806Z","iopub.status.idle":"2022-07-22T07:39:52.690159Z","shell.execute_reply.started":"2022-07-22T07:39:51.167761Z","shell.execute_reply":"2022-07-22T07:39:52.688876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

5.3.2 Pre-Training and Fine-Tuning

","metadata":{}},{"cell_type":"code","source":"## Model Pre-training\nepochs = exp_config['pretrain_epochs']\n\ntabnet_ae.fit(train_ds,\n epochs=epochs)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:39:52.691945Z","iopub.execute_input":"2022-07-22T07:39:52.692838Z","iopub.status.idle":"2022-07-22T07:40:22.310669Z","shell.execute_reply.started":"2022-07-22T07:39:52.692777Z","shell.execute_reply":"2022-07-22T07:40:22.309693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Creating Pre-trained Model\npretrained_model = create_tabnet(\n emb_dim,\n numerical_columns,\n categorical_columns,\n num_columns,\n input_dtypes,\n lookup_layers,\n output_dims=output_dims,\n n_steps=n_steps,\n feature_dims=feature_dims,\n gamma=gamma,\n epsilon=epsilon,\n lambda_sparsity=lambda_sparsity,\n pretrained_encoder=tabnet_ae.layers[-1].ae.encoder)\n\n\n## Pre-trained Model Compile and Build\nemb_learning_rate = exp_config['emb_finetune_lr']\nlearning_rate = exp_config['finetune_lr']\nbatch_size = exp_config['batch_size']\ntrain_epochs = exp_config['train_epochs']\nsteps_per_epoch = len(train_fold) // batch_size\n\nlearning_schedule_1 = tf.keras.optimizers.schedules.CosineDecay(\n initial_learning_rate=emb_learning_rate,\n decay_steps=train_epochs * steps_per_epoch, \n alpha=0.0)\n\nlearning_schedule_2 = tf.keras.optimizers.schedules.CosineDecay(\n initial_learning_rate=learning_rate,\n decay_steps=train_epochs * steps_per_epoch, \n alpha=0.0)\n\noptimizers = [\n tf.keras.optimizers.Adam(learning_rate=learning_schedule_1),\n tf.keras.optimizers.Adam(learning_rate=learning_schedule_2)\n]\noptimizers_and_layers = [(optimizers[0], pretrained_model.layers[:26]),\n (optimizers[1], pretrained_model.layers[26:])]\noptimizer = tfa.optimizers.MultiOptimizer(optimizers_and_layers)\n\nloss_fn = keras.losses.BinaryCrossentropy(\n from_logits=False, \n label_smoothing=model_config['label_smoothing'])\n\npretrained_model.compile(optimizer=optimizer,\n loss=loss_fn,\n metrics=['accuracy', keras.metrics.AUC()])\n\n#pretrained_model.build(input_shape=(None, input_dims))\npretrained_model.summary()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:40:22.312112Z","iopub.execute_input":"2022-07-22T07:40:22.312852Z","iopub.status.idle":"2022-07-22T07:40:22.969594Z","shell.execute_reply.started":"2022-07-22T07:40:22.312810Z","shell.execute_reply":"2022-07-22T07:40:22.967734Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Checkpoint Callback\ncheckpoint_filepath = exp_config['checkpoint_filepath']\nmodel_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n filepath=checkpoint_filepath, \n save_weights_only=True, \n monitor='val_loss', \n mode='min', \n save_best_only=True)\n\n## Model Fine-tuning\nhistory = pretrained_model.fit(\n train_ds,\n epochs=train_epochs,\n shuffle=True,\n validation_data=valid_ds,\n callbacks=[model_checkpoint_callback])\n\n## Load the Best Parameters\npretrained_model.load_weights(checkpoint_filepath)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-22T07:40:22.971160Z","iopub.execute_input":"2022-07-22T07:40:22.972620Z","iopub.status.idle":"2022-07-22T07:41:33.072572Z","shell.execute_reply.started":"2022-07-22T07:40:22.972546Z","shell.execute_reply":"2022-07-22T07:41:33.071640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Plot Losses \nhist = pd.DataFrame(history.history)\nplot_history(hist)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:41:33.073949Z","iopub.execute_input":"2022-07-22T07:41:33.074734Z","iopub.status.idle":"2022-07-22T07:41:33.282656Z","shell.execute_reply.started":"2022-07-22T07:41:33.074687Z","shell.execute_reply":"2022-07-22T07:41:33.281355Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Plot Feature Importance\nlayer_names = [l.name for l in pretrained_model.layers]\nname = layer_names[-2]\n\nimportance_model = tf.keras.Model(inputs=pretrained_model.input,\n outputs=pretrained_model.get_layer(name).output[1])\n\nimportance_list = []\nfor data, label in iter(train_ds):\n importance = importance_model(data)\n importance = importance.numpy()\n importance_list.append(importance)\n \nimportance = np.concatenate(importance_list, axis=0)\nimportance = np.mean(importance, axis=0)\nimportance_df = pd.DataFrame(importance,\n index=list(pretrained_model.input.keys()),\n columns=['importance'])\nimportance_df.sort_values(by='importance',\n ascending=False,\n inplace=True)\ntop10_importance = np.array(importance_df[:10].values.reshape([-1]))\ntop10_features = importance_df[:10].index\n\n#importance_df[:10].plot.bar()\n\nplt.figure(figsize=(7, 5))\nplt.bar(x=np.arange(10),\n height=top10_importance)\nplt.xticks(np.arange(10), top10_features, rotation='vertical')\nplt.title('Feature Importance')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:41:33.284065Z","iopub.execute_input":"2022-07-22T07:41:33.284437Z","iopub.status.idle":"2022-07-22T07:41:34.414677Z","shell.execute_reply.started":"2022-07-22T07:41:33.284403Z","shell.execute_reply":"2022-07-22T07:41:34.413344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

5.3.3 Inference

","metadata":{}},{"cell_type":"code","source":"## Inference_model = preprocessing_model + training_model\ninference_inputs = preprocessing_model.input\ninference_outputs = pretrained_model(preprocessing_model(inference_inputs))\ninference_model = tf.keras.Model(inputs=inference_inputs,\n outputs=inference_outputs)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:41:34.425656Z","iopub.execute_input":"2022-07-22T07:41:34.426067Z","iopub.status.idle":"2022-07-22T07:41:34.979294Z","shell.execute_reply.started":"2022-07-22T07:41:34.426032Z","shell.execute_reply":"2022-07-22T07:41:34.978210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Inference and Submission\nprobas = inference_model.predict(test_ds)\nprobas = np.squeeze(probas)\n\npreds = np.where(probas > 0.5, True, False)\n\nsubmission_df['Transported'] = preds\nsubmission_df.to_csv('submission_tf_pretrain.csv', index=False)\nsubmission_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T07:41:34.981020Z","iopub.execute_input":"2022-07-22T07:41:34.981607Z","iopub.status.idle":"2022-07-22T07:41:36.556874Z","shell.execute_reply.started":"2022-07-22T07:41:34.981539Z","shell.execute_reply":"2022-07-22T07:41:36.555748Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n#### Public Score: ~ 0.786\n\n---","metadata":{}}]}