{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Predicting States of Manufacturing Control Data; Using Neuronal Nets ⚙️","metadata":{}},{"cell_type":"markdown","source":"**Note: Use GPU for Training...**\n\n**Objective:** Build a powerfull NN Model that can provide a good estimation.\n\n**Strategy:** I think I will follow this strategy:\n\n**Level 1 Getting Started**\n\n* Quick EDA to identify potential opportunities.\n* Simple pre-processing step to encode categorical features.\n* A basic CV strategy using 90% for TRaining and 10% for Testing.\n* Looking at the feature importances.\n* Creating a submission file.\n* Submit the file to Kaggle.\n\n**Level 2 Feature Engineering**\n\n* Feature engineering using text information. (Massive boost in the score)\n* Cross validation loop.\n\n**Level 3 Model Optimization**\n* Work in Progress...\n\n---\n**Other Similar Implementations**\nI been working on other architechtures at the same time, to see what works more effiently\n\nXGBoost and LGBM Models\n\nhttps://www.kaggle.com/code/cv13j0/tps-may22-eda-gbdt\n\n---\n\n\n\n\n**Data Description**\n\nFor this challenge, you are given (simulated) manufacturing control data and are tasked to predict whether the machine is in state 0 or state 1. \nThe data has various feature interactions that may be important in determining the machine state.\n\nGood luck!\n\n**Files**\n* train.csv - the training data, which includes normalized continuous data and categorical data\n* test.csv - the test set; your task is to predict binary target variable which represents the state of a manufacturing process\n* sample_submission.csv - a sample submission file in the correct format\n\n---\n**Notebooks Ideas and Credits**\n\nI took ideas or inspiration from the following notebooks, if you enjoy my work, please take a look to the notebooks that inspire my work.\n\nTPSMAY22 Gradient-Boosting Quickstart: https://www.kaggle.com/code/ambrosm/tpsmay22-gradient-boosting-quickstart/notebook","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"# 1. Loading the Requiered Libraries","metadata":{}},{"cell_type":"code","source":"%%time\n# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-05-30T19:14:39.86036Z","iopub.execute_input":"2022-05-30T19:14:39.860728Z","iopub.status.idle":"2022-05-30T19:14:39.890468Z","shell.execute_reply.started":"2022-05-30T19:14:39.860639Z","shell.execute_reply":"2022-05-30T19:14:39.889615Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nimport datetime","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:14:39.892062Z","iopub.execute_input":"2022-05-30T19:14:39.892365Z","iopub.status.idle":"2022-05-30T19:14:39.898226Z","shell.execute_reply.started":"2022-05-30T19:14:39.892329Z","shell.execute_reply":"2022-05-30T19:14:39.897127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n","metadata":{}},{"cell_type":"markdown","source":"# 2. Setting the Notebook","metadata":{}},{"cell_type":"code","source":"%%time\n# I like to disable my Notebook Warnings.\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:14:39.899788Z","iopub.execute_input":"2022-05-30T19:14:39.900095Z","iopub.status.idle":"2022-05-30T19:14:39.906915Z","shell.execute_reply.started":"2022-05-30T19:14:39.900062Z","shell.execute_reply":"2022-05-30T19:14:39.906092Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Notebook Configuration...\n\n# Amount of data we want to load into the Model...\nDATA_ROWS = None\n# Dataframe, the amount of rows and cols to visualize...\nNROWS = 50\nNCOLS = 15\n# Main data location path...\nBASE_PATH = '...'","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:14:39.908723Z","iopub.execute_input":"2022-05-30T19:14:39.909902Z","iopub.status.idle":"2022-05-30T19:14:39.915191Z","shell.execute_reply.started":"2022-05-30T19:14:39.909868Z","shell.execute_reply":"2022-05-30T19:14:39.914525Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Configure notebook display settings to only use 2 decimal places, tables look nicer.\npd.options.display.float_format = '{:,.5f}'.format\npd.set_option('display.max_columns', NCOLS) \npd.set_option('display.max_rows', NROWS)","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:14:39.917451Z","iopub.execute_input":"2022-05-30T19:14:39.918201Z","iopub.status.idle":"2022-05-30T19:14:39.925418Z","shell.execute_reply.started":"2022-05-30T19:14:39.918146Z","shell.execute_reply":"2022-05-30T19:14:39.924568Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"# 3. Loading the Information (CSV) Into A Dataframe","metadata":{}},{"cell_type":"code","source":"%%time\n# Load the CSV information into a Pandas DataFrame...\ntrn_data = pd.read_csv('/kaggle/input/tabular-playground-series-may-2022/train.csv')\ntst_data = pd.read_csv('/kaggle/input/tabular-playground-series-may-2022/test.csv')\n\nsub = pd.read_csv('/kaggle/input/tabular-playground-series-may-2022/sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:14:39.927005Z","iopub.execute_input":"2022-05-30T19:14:39.927614Z","iopub.status.idle":"2022-05-30T19:14:52.764566Z","shell.execute_reply.started":"2022-05-30T19:14:39.927578Z","shell.execute_reply":"2022-05-30T19:14:52.763834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"# 4. Exploring the Information Available","metadata":{}},{"cell_type":"markdown","source":"## 4.1. Analysing the Trian Dataset","metadata":{}},{"cell_type":"code","source":"%%time\n# Explore the shape of the DataFrame...\ntrn_data.shape","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:14:52.765922Z","iopub.execute_input":"2022-05-30T19:14:52.766186Z","iopub.status.idle":"2022-05-30T19:14:52.774766Z","shell.execute_reply.started":"2022-05-30T19:14:52.766136Z","shell.execute_reply":"2022-05-30T19:14:52.774096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Display simple information of the variables in the dataset...\ntrn_data.info()","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:14:52.776027Z","iopub.execute_input":"2022-05-30T19:14:52.776509Z","iopub.status.idle":"2022-05-30T19:14:52.939726Z","shell.execute_reply.started":"2022-05-30T19:14:52.776472Z","shell.execute_reply":"2022-05-30T19:14:52.938846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Display the first few rows of the DataFrame...\ntrn_data.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:14:52.941101Z","iopub.execute_input":"2022-05-30T19:14:52.941409Z","iopub.status.idle":"2022-05-30T19:14:52.965723Z","shell.execute_reply.started":"2022-05-30T19:14:52.941373Z","shell.execute_reply":"2022-05-30T19:14:52.96492Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Generate a simple statistical summary of the DataFrame, Only Numerical...\ntrn_data.describe()","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:14:52.967022Z","iopub.execute_input":"2022-05-30T19:14:52.967462Z","iopub.status.idle":"2022-05-30T19:14:53.914216Z","shell.execute_reply.started":"2022-05-30T19:14:52.967427Z","shell.execute_reply":"2022-05-30T19:14:53.913456Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Calculates the total number of missing values...\ntrn_data.isnull().sum().sum()","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:14:53.915361Z","iopub.execute_input":"2022-05-30T19:14:53.916315Z","iopub.status.idle":"2022-05-30T19:14:54.046615Z","shell.execute_reply.started":"2022-05-30T19:14:53.916273Z","shell.execute_reply":"2022-05-30T19:14:54.045877Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Display the number of missing values by variable...\ntrn_data.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:14:54.047893Z","iopub.execute_input":"2022-05-30T19:14:54.049308Z","iopub.status.idle":"2022-05-30T19:14:54.181617Z","shell.execute_reply.started":"2022-05-30T19:14:54.049268Z","shell.execute_reply":"2022-05-30T19:14:54.180931Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Display the number of unique values for each variable...\ntrn_data.nunique()","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:14:54.182962Z","iopub.execute_input":"2022-05-30T19:14:54.183264Z","iopub.status.idle":"2022-05-30T19:14:55.204753Z","shell.execute_reply.started":"2022-05-30T19:14:54.183228Z","shell.execute_reply":"2022-05-30T19:14:55.204078Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Display the number of unique values for each variable, sorted by quantity...\ntrn_data.nunique().sort_values(ascending = True)","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:14:55.209164Z","iopub.execute_input":"2022-05-30T19:14:55.209362Z","iopub.status.idle":"2022-05-30T19:14:56.430632Z","shell.execute_reply.started":"2022-05-30T19:14:55.209338Z","shell.execute_reply":"2022-05-30T19:14:56.429942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Check some of the categorical variables\ncateg_cols = ['f_29','f_30','f_13', 'f_18','f_17','f_14','f_11','f_10','f_09','f_15','f_07','f_12','f_16','f_08','f_27']\ntrn_data[categ_cols].sample(5)","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:14:56.431867Z","iopub.execute_input":"2022-05-30T19:14:56.432249Z","iopub.status.idle":"2022-05-30T19:14:56.545912Z","shell.execute_reply.started":"2022-05-30T19:14:56.432214Z","shell.execute_reply":"2022-05-30T19:14:56.545203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Generate a quick correlation matrix to understand the dataset better\ncorrelation = trn_data.corr()","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:14:56.549289Z","iopub.execute_input":"2022-05-30T19:14:56.549973Z","iopub.status.idle":"2022-05-30T19:14:58.852714Z","shell.execute_reply.started":"2022-05-30T19:14:56.549933Z","shell.execute_reply":"2022-05-30T19:14:58.851097Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Diplay the correlation matrix\ncorrelation","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:14:58.854085Z","iopub.execute_input":"2022-05-30T19:14:58.854416Z","iopub.status.idle":"2022-05-30T19:14:58.882125Z","shell.execute_reply.started":"2022-05-30T19:14:58.854369Z","shell.execute_reply":"2022-05-30T19:14:58.881462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Check the most correlated variables to the target\ncorrelation['target'].sort_values(ascending = False)[:5]","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:14:58.883403Z","iopub.execute_input":"2022-05-30T19:14:58.883789Z","iopub.status.idle":"2022-05-30T19:14:58.89359Z","shell.execute_reply.started":"2022-05-30T19:14:58.883754Z","shell.execute_reply":"2022-05-30T19:14:58.892629Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Check the least correlated variables to the target\ncorrelation['target'].sort_values(ascending = True)[:5]","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:14:58.895249Z","iopub.execute_input":"2022-05-30T19:14:58.895923Z","iopub.status.idle":"2022-05-30T19:14:58.905471Z","shell.execute_reply.started":"2022-05-30T19:14:58.895884Z","shell.execute_reply":"2022-05-30T19:14:58.904584Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"## 4.2. Analysing the Trian Labels Dataset","metadata":{}},{"cell_type":"code","source":"%%time\n# Check how well balanced is the dataset\ntrn_data['target'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:14:58.907002Z","iopub.execute_input":"2022-05-30T19:14:58.907425Z","iopub.status.idle":"2022-05-30T19:14:58.92239Z","shell.execute_reply.started":"2022-05-30T19:14:58.907389Z","shell.execute_reply":"2022-05-30T19:14:58.921628Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Check some statistics on the target variable\ntrn_data['target'].describe()","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:14:58.92373Z","iopub.execute_input":"2022-05-30T19:14:58.9241Z","iopub.status.idle":"2022-05-30T19:14:58.955716Z","shell.execute_reply.started":"2022-05-30T19:14:58.924058Z","shell.execute_reply":"2022-05-30T19:14:58.954918Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"# 5. Feature Engineering","metadata":{}},{"cell_type":"code","source":"%%time\n# Generate a simple statistical summary of the DataFrame, Only Numerical...\ntrn_data.describe()","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:14:58.957015Z","iopub.execute_input":"2022-05-30T19:14:58.957297Z","iopub.status.idle":"2022-05-30T19:14:59.899962Z","shell.execute_reply.started":"2022-05-30T19:14:58.95726Z","shell.execute_reply":"2022-05-30T19:14:59.899226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trn_data.shape","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:14:59.9014Z","iopub.execute_input":"2022-05-30T19:14:59.901865Z","iopub.status.idle":"2022-05-30T19:14:59.908135Z","shell.execute_reply.started":"2022-05-30T19:14:59.901829Z","shell.execute_reply":"2022-05-30T19:14:59.90734Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ncontinuous_feat = ['f_00', 'f_01', 'f_02', 'f_03', 'f_04', 'f_05', 'f_06', 'f_19', 'f_20', 'f_21', 'f_22', 'f_23', 'f_24', 'f_25', 'f_26', 'f_28']\n\ndef remove_outliers(df, continuous_feat):\n    for col in df[continuous_feat].columns:\n        #df = df[(df[col] > df[col].quantile(0.01)) & (df[col] < df[col].quantile(0.99))]\n        df[col + '_outlier'] = np.where((df[col] > df[col].quantile(0.01)) & (df[col] < df[col].quantile(0.99)),0,1)\n    \n    \n    outliers = [col for col in df.columns if '_outlier' in col]\n    df['total_outliers']  = df[outliers].sum(axis=1)\n    df = df.drop(columns = outliers)\n    return df\n\ntrn_data = remove_outliers(trn_data, continuous_feat)\ntst_data = remove_outliers(tst_data, continuous_feat)","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:14:59.909507Z","iopub.execute_input":"2022-05-30T19:14:59.910497Z","iopub.status.idle":"2022-05-30T19:15:01.493934Z","shell.execute_reply.started":"2022-05-30T19:14:59.910435Z","shell.execute_reply":"2022-05-30T19:15:01.493197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trn_data.shape","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:15:01.495355Z","iopub.execute_input":"2022-05-30T19:15:01.495596Z","iopub.status.idle":"2022-05-30T19:15:01.500517Z","shell.execute_reply.started":"2022-05-30T19:15:01.495562Z","shell.execute_reply":"2022-05-30T19:15:01.4998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trn_data.sample(10)","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:15:01.501721Z","iopub.execute_input":"2022-05-30T19:15:01.502106Z","iopub.status.idle":"2022-05-30T19:15:01.545454Z","shell.execute_reply.started":"2022-05-30T19:15:01.50206Z","shell.execute_reply":"2022-05-30T19:15:01.544627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Generate a simple statistical summary of the DataFrame, Only Numerical...\ntrn_data.describe()","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:15:01.546968Z","iopub.execute_input":"2022-05-30T19:15:01.547241Z","iopub.status.idle":"2022-05-30T19:15:02.500988Z","shell.execute_reply.started":"2022-05-30T19:15:01.547206Z","shell.execute_reply":"2022-05-30T19:15:02.499409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.1 Text Base Features","metadata":{}},{"cell_type":"code","source":"%%time\n# The idea is to create a simple funtion to count the amount of letters on feature 27.\n# feature 27 seems quite important \n\ndef count_sequence(df, field):\n    '''\n    For each letter of the provided suquence it return new feature with the number of occurences.\n    '''\n    alphabet = ['A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','W','X','Y','Z']    \n    \n    for letter in alphabet:\n        df[letter + '_count'] = df[field].str.count(letter)\n    \n    df[\"unique_characters\"] = df['f_27'].apply(lambda s: len(set(s)))\n    return df","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:15:02.502326Z","iopub.execute_input":"2022-05-30T19:15:02.502583Z","iopub.status.idle":"2022-05-30T19:15:02.510866Z","shell.execute_reply.started":"2022-05-30T19:15:02.502541Z","shell.execute_reply":"2022-05-30T19:15:02.510094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Utilizes the new created funtions to generate more features.\n#trn_data = count_sequence(trn_data, 'f_27')\n#tst_data = count_sequence(tst_data, 'f_27')","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:15:02.512462Z","iopub.execute_input":"2022-05-30T19:15:02.51296Z","iopub.status.idle":"2022-05-30T19:15:02.519197Z","shell.execute_reply.started":"2022-05-30T19:15:02.512926Z","shell.execute_reply":"2022-05-30T19:15:02.518526Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ndef count_chars(df, field):\n    '''\n    Describe something...\n    '''\n    \n    for i in range(10):\n        df[f'ch_{i}'] = df[field].str.get(i).apply(ord) - ord('A')\n        \n    df[\"unique_characters\"] = df[field].apply(lambda s: len(set(s)))\n    return df","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:15:02.520558Z","iopub.execute_input":"2022-05-30T19:15:02.5211Z","iopub.status.idle":"2022-05-30T19:15:02.530174Z","shell.execute_reply.started":"2022-05-30T19:15:02.521064Z","shell.execute_reply":"2022-05-30T19:15:02.529379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Utilizes the new created funtions to generate more features.\ntrn_data = count_chars(trn_data, 'f_27')\ntst_data = count_chars(tst_data, 'f_27')","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:15:02.531796Z","iopub.execute_input":"2022-05-30T19:15:02.532069Z","iopub.status.idle":"2022-05-30T19:15:20.70456Z","shell.execute_reply.started":"2022-05-30T19:15:02.532032Z","shell.execute_reply":"2022-05-30T19:15:20.703816Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.2 Statistical Features","metadata":{}},{"cell_type":"code","source":"%%time\ncontinuous_feat = ['f_00', 'f_01', 'f_02', 'f_03', 'f_04', 'f_05', 'f_06', 'f_19', 'f_20', 'f_21', 'f_22', 'f_23', 'f_24', 'f_25', 'f_26', 'f_28']\n\ndef stat_features(df, cols = continuous_feat):\n    '''\n    Calculate aggregated features across the selected continuous columns\n    \n    '''\n    # Base statistical features.\n    df['f_sum']  = df[continuous_feat].sum(axis=1)\n    df['f_min']  = df[continuous_feat].min(axis=1)\n    df['f_max']  = df[continuous_feat].max(axis=1)\n    df['f_std']  = df[continuous_feat].std(axis=1)    \n    df['f_mad']  = df[continuous_feat].mad(axis=1)\n    df['f_mean'] = df[continuous_feat].mean(axis=1)\n    df['f_kurt'] = df[continuous_feat].kurt(axis=1)\n\n    # Extra statistical features.\n    df['f_prod'] = df[continuous_feat].prod(axis=1)\n    df['f_range'] = df[continuous_feat].max(axis=1) - df[continuous_feat].min(axis=1)\n    df['f_count_pos']  = df[df[continuous_feat].gt(0)].count(axis=1)\n    df['f_count_neg']  = df[df[continuous_feat].lt(0)].count(axis=1)\n\n    return df","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:15:20.708246Z","iopub.execute_input":"2022-05-30T19:15:20.710361Z","iopub.status.idle":"2022-05-30T19:15:20.830786Z","shell.execute_reply.started":"2022-05-30T19:15:20.710323Z","shell.execute_reply":"2022-05-30T19:15:20.82991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntrn_data = stat_features(trn_data, continuous_feat)\ntst_data = stat_features(tst_data, continuous_feat)","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:15:20.835349Z","iopub.execute_input":"2022-05-30T19:15:20.837272Z","iopub.status.idle":"2022-05-30T19:15:52.350648Z","shell.execute_reply.started":"2022-05-30T19:15:20.837233Z","shell.execute_reply":"2022-05-30T19:15:52.349866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntrn_data.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:15:52.351984Z","iopub.execute_input":"2022-05-30T19:15:52.352485Z","iopub.status.idle":"2022-05-30T19:15:52.373792Z","shell.execute_reply.started":"2022-05-30T19:15:52.35244Z","shell.execute_reply":"2022-05-30T19:15:52.373117Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"%%time\ndef calculate_feat_int(df):\n    df['i_02_21'] = (df.f_21 + df.f_02 > 5.2).astype(int) - (df.f_21 + df.f_02 < -5.3).astype(int)\n    df['i_05_22'] = (df.f_22 + df.f_05 > 5.1).astype(int) - (df.f_22 + df.f_05 < -5.4).astype(int)\n    i_00_01_26 = df.f_00 + df.f_01 + df.f_26\n    df['i_00_01_26'] = (i_00_01_26 > 5.0).astype(int) - (i_00_01_26 < -5.0).astype(int)\n    return df\n\ntrn_data = calculate_feat_int(trn_data)\ntst_data = calculate_feat_int(tst_data)","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:15:52.375177Z","iopub.execute_input":"2022-05-30T19:15:52.375491Z","iopub.status.idle":"2022-05-30T19:15:52.433237Z","shell.execute_reply.started":"2022-05-30T19:15:52.375456Z","shell.execute_reply":"2022-05-30T19:15:52.432461Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"# 7. Pre-Processing Labels","metadata":{}},{"cell_type":"markdown","source":"## 7.1 Label Encoding","metadata":{}},{"cell_type":"code","source":"%%time\n# Define a label encoding function\nfrom sklearn.preprocessing import LabelEncoder\nencoder = LabelEncoder()\ndef encode_features(df, cols = ['f_27']):\n    '''\n    Apply one-hot encode to the selected columns, return a df\n    '''\n    for col in cols:\n        df[col + '_enc'] = encoder.fit_transform(df[col])\n    return df\n\ntrn_data = encode_features(trn_data)\ntst_data = encode_features(tst_data)","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:15:52.434603Z","iopub.execute_input":"2022-05-30T19:15:52.43502Z","iopub.status.idle":"2022-05-30T19:15:58.57169Z","shell.execute_reply.started":"2022-05-30T19:15:52.434981Z","shell.execute_reply":"2022-05-30T19:15:58.570941Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Check the results of the transformation\ntrn_data.sample(10)","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:15:58.573113Z","iopub.execute_input":"2022-05-30T19:15:58.57338Z","iopub.status.idle":"2022-05-30T19:15:58.772664Z","shell.execute_reply.started":"2022-05-30T19:15:58.573346Z","shell.execute_reply":"2022-05-30T19:15:58.77184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 7.2 - One-Hot Encode","metadata":{}},{"cell_type":"code","source":"# We will process to One-Hot encode all this variables...\n# f_29           2\n# f_30           3\n# f_13          13\n# f_18          14\n# f_17          14\n# f_14          14\n# f_11          14\n# f_10          15\n# f_09          15\n# f_15          15\n# f_07          16\n# f_12          16\n# f_16          16\n# f_08          16","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:15:58.774116Z","iopub.execute_input":"2022-05-30T19:15:58.774437Z","iopub.status.idle":"2022-05-30T19:15:58.778752Z","shell.execute_reply.started":"2022-05-30T19:15:58.774395Z","shell.execute_reply":"2022-05-30T19:15:58.777791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ndef one_hot_encoder(df_trn, df_tst, var_list):\n    '''\n    Apply one-hot encode to the selected columns, and merged train and test \n    dataframes to avoid issues with missing categories.\n    \n    '''\n    df_trn['is_train'] = 1\n    df_tst['is_train'] = 0\n\n    combined = df_trn.append(df_tst)\n    combined = pd.get_dummies(combined, columns = var_list)\n    return combined[combined['is_train'] == 1], combined[combined['is_train'] == 0]","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:15:58.780113Z","iopub.execute_input":"2022-05-30T19:15:58.780414Z","iopub.status.idle":"2022-05-30T19:15:58.791705Z","shell.execute_reply.started":"2022-05-30T19:15:58.780377Z","shell.execute_reply":"2022-05-30T19:15:58.79083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n#trn_data, tst_data = one_hot_encoder(trn_data,tst_data, [\n                                                         #'f_29',\n                                                         #'f_30',\n                                                         #'f_13',\n                                                         #'f_18',\n                                                         #'f_17',\n                                                         #'f_14',\n                                                         #'f_11',\n                                                         #'f_10',\n                                                         #'f_09',\n                                                         #'f_15',\n                                                         #'f_07',\n                                                         #'f_12',\n                                                         #'f_16',\n                                                         #'f_08'\n                                                         #])","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:15:58.793444Z","iopub.execute_input":"2022-05-30T19:15:58.793668Z","iopub.status.idle":"2022-05-30T19:15:58.803349Z","shell.execute_reply.started":"2022-05-30T19:15:58.793638Z","shell.execute_reply":"2022-05-30T19:15:58.802557Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"# 8. Feature Selection for Baseline Model","metadata":{}},{"cell_type":"code","source":"%%time\n# Define what will be used in the training stage\nignore = ['id', \n          'f_27', \n          'f_27_enc', \n          'is_train', \n          'target'] # f_27 has been label encoded...\n\nfeatures = [feat for feat in trn_data.columns if feat not in ignore]\ntarget_feature = 'target'","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:15:58.808943Z","iopub.execute_input":"2022-05-30T19:15:58.809771Z","iopub.status.idle":"2022-05-30T19:15:58.817458Z","shell.execute_reply.started":"2022-05-30T19:15:58.809716Z","shell.execute_reply":"2022-05-30T19:15:58.816695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"# 9. Creating a Simple Train / Test Split Strategy","metadata":{}},{"cell_type":"code","source":"%%time\n# Creates a simple train split breakdown for baseline model\nfrom sklearn.model_selection import train_test_split\ntest_size_pct = 0.20\nX_train, X_valid, y_train, y_valid = train_test_split(trn_data[features], trn_data[target_feature], test_size = test_size_pct, random_state = 42)","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:15:58.819796Z","iopub.execute_input":"2022-05-30T19:15:58.820344Z","iopub.status.idle":"2022-05-30T19:15:59.520509Z","shell.execute_reply.started":"2022-05-30T19:15:58.820306Z","shell.execute_reply":"2022-05-30T19:15:59.519041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"# 10. Building a Baseline NN Model, Simple Split","metadata":{}},{"cell_type":"code","source":"%%time\nimport tensorflow as tf\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau, LearningRateScheduler, EarlyStopping\nfrom tensorflow.keras.layers import Dense, Input, InputLayer, Add, BatchNormalization, Dropout\n\nfrom sklearn.preprocessing import StandardScaler, RobustScaler, MinMaxScaler\nimport random","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:15:59.521913Z","iopub.execute_input":"2022-05-30T19:15:59.522221Z","iopub.status.idle":"2022-05-30T19:16:05.187991Z","shell.execute_reply.started":"2022-05-30T19:15:59.52214Z","shell.execute_reply":"2022-05-30T19:16:05.187239Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nnp.random.seed(1)\nrandom.seed(1)\ntf.random.set_seed(1)","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:16:05.190094Z","iopub.execute_input":"2022-05-30T19:16:05.190369Z","iopub.status.idle":"2022-05-30T19:16:05.197274Z","shell.execute_reply.started":"2022-05-30T19:16:05.190333Z","shell.execute_reply":"2022-05-30T19:16:05.196594Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\n\n\ndef nn_model():\n    \n    '''\n    '''\n    \n    L2 = 65e-6\n    activation_func = 'swish'\n    inputs = Input(shape = (len(features)))\n    \n    x = Dense(512, \n              #use_bias  = True, \n              kernel_regularizer = tf.keras.regularizers.l2(L2), \n              activation = activation_func)(inputs)\n    \n    x = BatchNormalization()(x)\n    \n    x = Dense(512, \n          #use_bias  = True, \n          kernel_regularizer = tf.keras.regularizers.l2(L2), \n          activation = activation_func)(x)\n    \n    x = BatchNormalization()(x)\n    \n    x = Dense(64, \n              #use_bias  = True, \n              kernel_regularizer = tf.keras.regularizers.l2(L2), \n              activation = activation_func)(x)\n    \n    x = BatchNormalization()(x)\n    \n    x = Dense(64, \n          #use_bias  = True, \n          kernel_regularizer = tf.keras.regularizers.l2(L2), \n          activation = activation_func)(x)\n    \n    x = BatchNormalization()(x)\n\n    x = Dense(16, \n              #use_bias  = True, \n              kernel_regularizer = tf.keras.regularizers.l2(L2), \n              activation = activation_func)(x)\n    \n    x = BatchNormalization()(x)\n\n    x = Dense(1 , \n              #use_bias  = True, \n              #kernel_regularizer = tf.keras.regularizers.l2(L2),\n              activation = 'sigmoid')(x)\n    \n    model = Model(inputs, x)\n    \n    return model","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:16:05.198729Z","iopub.execute_input":"2022-05-30T19:16:05.199191Z","iopub.status.idle":"2022-05-30T19:16:05.217668Z","shell.execute_reply.started":"2022-05-30T19:16:05.199143Z","shell.execute_reply":"2022-05-30T19:16:05.216835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\narchitecture = nn_model()\narchitecture.summary()","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:16:05.219492Z","iopub.execute_input":"2022-05-30T19:16:05.219792Z","iopub.status.idle":"2022-05-30T19:16:08.143433Z","shell.execute_reply.started":"2022-05-30T19:16:05.21974Z","shell.execute_reply":"2022-05-30T19:16:08.141562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Defining model parameters...\nBATCH_SIZE         = 2048\nEPOCHS             = 200 \nEPOCHS_COSINEDECAY = 150 \nDIAGRAMS           = True\nUSE_PLATEAU        = False\nINFERENCE          = False\nVERBOSE            = 0 \nTARGET             = 'target'","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:16:08.144828Z","iopub.execute_input":"2022-05-30T19:16:08.145065Z","iopub.status.idle":"2022-05-30T19:16:08.151371Z","shell.execute_reply.started":"2022-05-30T19:16:08.145023Z","shell.execute_reply":"2022-05-30T19:16:08.150636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":" %%time\n# Defining model training function...\ndef fit_model(X_train, y_train, X_val, y_val, run = 0):\n    '''\n    '''\n    lr_start = 0.01\n    start_time = datetime.datetime.now()\n    \n    scaler = StandardScaler()\n    #scaler = RobustScaler()\n    #scaler = MinMaxScaler()\n\n    X_train = scaler.fit_transform(X_train)\n\n    epochs = EPOCHS    \n    lr = ReduceLROnPlateau(monitor = 'val_loss', factor = 0.7, patience = 4, verbose = VERBOSE)\n    es = EarlyStopping(monitor = 'val_loss',patience = 12, verbose = 1, mode = 'min', restore_best_weights = True)\n    tm = tf.keras.callbacks.TerminateOnNaN()\n    callbacks = [lr, es, tm]\n    \n    # Cosine Learning Rate Decay\n    if USE_PLATEAU == False:\n        epochs = EPOCHS_COSINEDECAY\n        lr_end = 0.0002\n\n        def cosine_decay(epoch):\n            if epochs > 1:\n                w = (1 + math.cos(epoch / (epochs - 1) * math.pi)) / 2\n            else:\n                w = 1\n            return w * lr_start + (1 - w) * lr_end\n        \n        lr = LearningRateScheduler(cosine_decay, verbose = 0)\n        callbacks = [lr, tm]\n        \n    model = nn_model()\n    optimizer_func = tf.keras.optimizers.Adam(learning_rate = lr_start)\n    loss_func = tf.keras.losses.BinaryCrossentropy()\n    model.compile(optimizer = optimizer_func, loss = loss_func)\n    \n    X_val = scaler.transform(X_val)\n    validation_data = (X_val, y_val)\n    \n    history = model.fit(X_train, \n                        y_train, \n                        validation_data = validation_data, \n                        epochs          = epochs,\n                        verbose         = VERBOSE,\n                        batch_size      = BATCH_SIZE,\n                        shuffle         = True,\n                        callbacks       = callbacks\n                       )\n    \n    history_list.append(history.history)\n    print(f'Training loss:{history_list[-1][\"loss\"][-1]:.5f}')\n    callbacks, es, lr, tm, history = None, None, None, None, None\n    \n    \n    y_val_pred = model.predict(X_val, batch_size = BATCH_SIZE, verbose = VERBOSE)\n    score = roc_auc_score(y_val, y_val_pred)\n    print(f'Fold {run}.{fold} | {str(datetime.datetime.now() - start_time)[-12:-7]}'\n          f'| AUC: {score:.5f}')\n    \n    score_list.append(score)\n    \n    tst_data_scaled = scaler.transform(tst_data[features])\n    tst_pred = model.predict(tst_data_scaled)\n    predictions.append(tst_pred)\n    \n    return model","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:16:08.152728Z","iopub.execute_input":"2022-05-30T19:16:08.153954Z","iopub.status.idle":"2022-05-30T19:16:08.16907Z","shell.execute_reply.started":"2022-05-30T19:16:08.153917Z","shell.execute_reply":"2022-05-30T19:16:08.168215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import roc_auc_score, roc_curve\nimport math\n\n# Create empty lists to store NN information...\nhistory_list = []\nscore_list   = []\npredictions  = []\n\n# Define kfolds for training purposes...\nkf = KFold(n_splits = 5)\n\nfor fold, (trn_idx, val_idx) in enumerate(kf.split(trn_data)):\n    X_train, X_val = trn_data.iloc[trn_idx][features], trn_data.iloc[val_idx][features]\n    y_train, y_val = trn_data.iloc[trn_idx][TARGET], trn_data.iloc[val_idx][TARGET]\n    \n    fit_model(X_train, y_train, X_val, y_val)\n    \nprint(f'OOF AUC: {np.mean(score_list):.5f}')","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:16:08.170708Z","iopub.execute_input":"2022-05-30T19:16:08.170962Z","iopub.status.idle":"2022-05-30T19:54:32.191197Z","shell.execute_reply.started":"2022-05-30T19:16:08.170928Z","shell.execute_reply":"2022-05-30T19:54:32.190342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# OOF AUC: 0.99658... 10 Folds, Batch Normalization, Using One-Hot Features, Epochs = 150, [64,64,64,16,1]...\n# OOF AUC: 0.99653... 05 Folds, No Batch Normalization, Using Partial One-Hot Features, Epochs = 150, [64,64,64,16,1]...\n# OOF AUC: 0.99757... 05 Folds, No Batch Normalization, No One-Hot Features, Epochs = 150, [64,64,64,16,1]...\n# OOF AUC: 0.99766... 05 Folds, No Batch Normalization, No One-Hot Features, Epochs = 200, [64,64,64,16,1]...\n# OOF AUC: 0.99771... 05 Folds, No Batch Normalization, No One-Hot Features, Epochs = 300, [64,64,64,16,1]...\n# OOF AUC: 0.99759... 05 Folds, No Batch Normalization, No One-Hot Features, Epochs = 300, [96,64,64,16,1]...\n# OOF AUC: 0.99772... 05 Folds, No Batch Normalization, No One-Hot Features, Epochs = 300, [64,64,32,16,1]...\n# OOF AUC: 0.99772... 05 Folds, No Batch Normalization, No One-Hot Features, Epochs = 300, [64,64,32,16,1]...\n# OOF AUC: 0.99769... 05 Folds, No Batch Normalization, No One-Hot Features, Epochs = 300, [64,64,64,16,1], Stat Features = Yes\n# OOF AUC: 0.99769... 05 Folds, No Batch Normalization, No One-Hot Features, Epochs = 300, [256,64,64,16,1], Stat Features = Yes\n# OOF AUC: 0.99774... 05 Folds, No Batch Normalization, No One-Hot Features, Epochs = 300, [64,64,64,64,16,1], Stat Features = Yes, StandarScaler ****\n# OOF AUC: 0.99769... 05 Folds, No Batch Normalization, No One-Hot Features, Epochs = 300, [64,64,64,64,16,1], Stat Features = Yes, RobustScaler\n# OOF AUC: ???????... 05 Folds, No Batch Normalization, No One-Hot Features, Epochs = 300, [64,64,64,64,16,1], Stat Features = Yes, MinMaxScaler Terminated...\n\n# OOF AUC: 0.99771... 05 Folds, No Batch Normalization, No One-Hot Features, Epochs = 300, [64,64,64,64,16,1], Stat Features = Yes + Extra, StandarScaler ****\n\n# OOF AUC: 0.99767...\n# OOF AUC: 0.99784...\n# OOF AUC: 0.99763 >> 0.99816...\n\n# OOF AUC: 0.99794 V0, Plain features... (40e-6)\n# OOF AUC: 0.99793 V1, Added MAD...\n\n# OOF AUC: 0.99797 ...  45e-6\n# OOF AUC: 0.99798 ...  50e-6\n# OOF AUC: 0.99799 ...  60e-6\n\n# OOF AUC: 0.99803\n# OOF AUC: 0.99798 ...","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:54:32.195451Z","iopub.execute_input":"2022-05-30T19:54:32.197601Z","iopub.status.idle":"2022-05-30T19:54:32.205806Z","shell.execute_reply.started":"2022-05-30T19:54:32.197536Z","shell.execute_reply":"2022-05-30T19:54:32.205197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"# 11. Undertanding Model Behavior, Feature Importance","metadata":{}},{"cell_type":"code","source":"# Work in Progress...","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:54:32.210573Z","iopub.execute_input":"2022-05-30T19:54:32.213253Z","iopub.status.idle":"2022-05-30T19:54:32.219931Z","shell.execute_reply.started":"2022-05-30T19:54:32.213187Z","shell.execute_reply":"2022-05-30T19:54:32.219203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"# 12. Baseline Model Submission File Generation","metadata":{}},{"cell_type":"code","source":"%%time\n# Review the format of the submission file\nsub.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:54:32.224601Z","iopub.execute_input":"2022-05-30T19:54:32.226934Z","iopub.status.idle":"2022-05-30T19:54:32.24396Z","shell.execute_reply.started":"2022-05-30T19:54:32.226896Z","shell.execute_reply":"2022-05-30T19:54:32.243345Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Populated the prediction on the submission dataset and creates an output file\nsub['target'] = np.array(predictions).mean(axis = 0)\nsub.to_csv('my_submission_052222_v5.csv', index = False)","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:54:32.247648Z","iopub.execute_input":"2022-05-30T19:54:32.249551Z","iopub.status.idle":"2022-05-30T19:54:34.087894Z","shell.execute_reply.started":"2022-05-30T19:54:32.249516Z","shell.execute_reply":"2022-05-30T19:54:34.087162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n%%script false --no-raise-error\n# Create submission\nprint(f\"{len(features)} features\")\n\npred_list = []\nfor seed in range(10):\n    model = fit_model(X_tr, y_tr, run = seed)\n    model.fit(X_tr.values, y_tr)\n    pred_list.append(scipy.stats.rankdata(model.predict(tst_data[features].values, batch_size = BATCH_SIZE)))\n    print(f\"{seed:2}\", pred_list[-1])\nprint()\n\nsubmission = tst_data[['id']].copy()\nsubmission[TARGET] = np.array(pred_list).mean(axis = 0)\n\nsubmission.to_csv('submission_nn_05222022_v0.csv', index = False)","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:54:34.089044Z","iopub.execute_input":"2022-05-30T19:54:34.089707Z","iopub.status.idle":"2022-05-30T19:54:34.16554Z","shell.execute_reply.started":"2022-05-30T19:54:34.089667Z","shell.execute_reply":"2022-05-30T19:54:34.164665Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Review the submission file as a final step to upload to Kaggle.\nsub.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-30T19:54:34.167052Z","iopub.execute_input":"2022-05-30T19:54:34.167786Z","iopub.status.idle":"2022-05-30T19:54:34.180609Z","shell.execute_reply.started":"2022-05-30T19:54:34.167743Z","shell.execute_reply":"2022-05-30T19:54:34.179839Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}}]}