{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🤓 Tabular Playground Series - Aug 2022 - EDA\n\n![Coding](https://cdn.dribbble.com/users/1068771/screenshots/14247776/media/fbf5f8ae629e3a6248006e748ddd6b67.jpg)\n\n### [Read the dataset](#dataset)\n\n### [Exploratory data analysis (EDA)](#eda)\n- overall stats\n- univeriate stats\n- multivariate stats\n\n### [Train - Valid - Test Datasets](#tvt)\n\n### [Handling imbalanced Datasets](#im)\n\n### [Handing Missing Values](#missing)\n- drop columns with missing values\n- drop rows with missing values\n- impute missing values with .fillna()\n- impute missing values with sklearn's simpleimputer\n","metadata":{}},{"cell_type":"markdown","source":"<div id=\"dataset\">\n    <h3>Dataset</h3></div>\nStart by importing training data into Pandas","metadata":{}},{"cell_type":"code","source":"import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\n#import the training data into pandas\ndf = pd.read_csv('../input/tabular-playground-series-aug-2022/train.csv')","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-08-14T15:29:15.798637Z","iopub.execute_input":"2022-08-14T15:29:15.799185Z","iopub.status.idle":"2022-08-14T15:29:16.071100Z","shell.execute_reply.started":"2022-08-14T15:29:15.799060Z","shell.execute_reply":"2022-08-14T15:29:16.069843Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<div id=\"eda\">\n    <h3>EDA</h3></div>\nOverall stats - Have a look at the rows, columns and general statistics of this dataset","metadata":{}},{"cell_type":"code","source":"# view the first six rows of data\ndf.head(6)","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:16.077811Z","iopub.execute_input":"2022-08-14T15:29:16.079275Z","iopub.status.idle":"2022-08-14T15:29:16.128412Z","shell.execute_reply.started":"2022-08-14T15:29:16.079232Z","shell.execute_reply":"2022-08-14T15:29:16.127017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# let's see the data types and non-null values for each column\ndf.info()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:16.130483Z","iopub.execute_input":"2022-08-14T15:29:16.131056Z","iopub.status.idle":"2022-08-14T15:29:16.166262Z","shell.execute_reply.started":"2022-08-14T15:29:16.131017Z","shell.execute_reply":"2022-08-14T15:29:16.165003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# theis prints basic stats for numerical columns\ndf.describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:16.169618Z","iopub.execute_input":"2022-08-14T15:29:16.170430Z","iopub.status.idle":"2022-08-14T15:29:16.292524Z","shell.execute_reply.started":"2022-08-14T15:29:16.170364Z","shell.execute_reply":"2022-08-14T15:29:16.291353Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We want to choose the model features and model target. \n\nLets first print the dataset column labels via numpy. ","metadata":{}},{"cell_type":"code","source":"import numpy as np\nnp.set_printoptions(threshold=np.inf)\n\n# print the column labels of the dataframe\nprint(df.columns.values)","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:16.294340Z","iopub.execute_input":"2022-08-14T15:29:16.295019Z","iopub.status.idle":"2022-08-14T15:29:16.302969Z","shell.execute_reply.started":"2022-08-14T15:29:16.294968Z","shell.execute_reply":"2022-08-14T15:29:16.301664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_features = df.columns.drop('failure')\nmodel_target = 'failure'\n\nprint('Model Features: ', model_features)\nprint('Model target: ', model_target)","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:16.304986Z","iopub.execute_input":"2022-08-14T15:29:16.305787Z","iopub.status.idle":"2022-08-14T15:29:16.316685Z","shell.execute_reply.started":"2022-08-14T15:29:16.305713Z","shell.execute_reply":"2022-08-14T15:29:16.315601Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Basic Plots\n\n**Bar Plots**\nBar plots show counts of categorical data fields. We next examine some of the categorical variables, including the target.\n\nvalue_counts() function yields the count of each unique value.","metadata":{}},{"cell_type":"code","source":"df['product_code'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:33:27.950058Z","iopub.execute_input":"2022-08-14T15:33:27.950462Z","iopub.status.idle":"2022-08-14T15:33:27.961762Z","shell.execute_reply.started":"2022-08-14T15:33:27.950430Z","shell.execute_reply":"2022-08-14T15:33:27.960501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\ndf['product_code'].value_counts().plot.bar()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:33:59.981449Z","iopub.execute_input":"2022-08-14T15:33:59.981915Z","iopub.status.idle":"2022-08-14T15:34:00.181948Z","shell.execute_reply.started":"2022-08-14T15:33:59.981878Z","shell.execute_reply":"2022-08-14T15:34:00.180718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['loading'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:35:21.409606Z","iopub.execute_input":"2022-08-14T15:35:21.410704Z","iopub.status.idle":"2022-08-14T15:35:21.434288Z","shell.execute_reply.started":"2022-08-14T15:35:21.410649Z","shell.execute_reply":"2022-08-14T15:35:21.432339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['loading'].value_counts().plot.bar()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:35:33.414436Z","iopub.execute_input":"2022-08-14T15:35:33.414882Z","iopub.status.idle":"2022-08-14T15:37:34.650249Z","shell.execute_reply.started":"2022-08-14T15:35:33.414842Z","shell.execute_reply":"2022-08-14T15:37:34.648044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['attribute_0'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:16.318159Z","iopub.execute_input":"2022-08-14T15:29:16.319003Z","iopub.status.idle":"2022-08-14T15:29:16.332069Z","shell.execute_reply.started":"2022-08-14T15:29:16.318967Z","shell.execute_reply":"2022-08-14T15:29:16.330767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Another way to view data is via a plot.bar()","metadata":{}},{"cell_type":"code","source":"df['attribute_0'].value_counts().plot.bar()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:34:08.974961Z","iopub.execute_input":"2022-08-14T15:34:08.975524Z","iopub.status.idle":"2022-08-14T15:34:09.184767Z","shell.execute_reply.started":"2022-08-14T15:34:08.975463Z","shell.execute_reply":"2022-08-14T15:34:09.183494Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['attribute_1'].value_counts().plot.bar()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:16.606010Z","iopub.execute_input":"2022-08-14T15:29:16.606943Z","iopub.status.idle":"2022-08-14T15:29:16.825000Z","shell.execute_reply.started":"2022-08-14T15:29:16.606892Z","shell.execute_reply":"2022-08-14T15:29:16.823846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['attribute_2'].value_counts().plot.bar()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:16.828866Z","iopub.execute_input":"2022-08-14T15:29:16.829228Z","iopub.status.idle":"2022-08-14T15:29:17.025511Z","shell.execute_reply.started":"2022-08-14T15:29:16.829199Z","shell.execute_reply":"2022-08-14T15:29:17.024284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['attribute_3'].value_counts().plot.bar()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:17.027078Z","iopub.execute_input":"2022-08-14T15:29:17.028275Z","iopub.status.idle":"2022-08-14T15:29:17.221489Z","shell.execute_reply.started":"2022-08-14T15:29:17.028222Z","shell.execute_reply":"2022-08-14T15:29:17.220250Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_0'].value_counts().plot.bar()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:17.223336Z","iopub.execute_input":"2022-08-14T15:29:17.224546Z","iopub.status.idle":"2022-08-14T15:29:17.552044Z","shell.execute_reply.started":"2022-08-14T15:29:17.224504Z","shell.execute_reply":"2022-08-14T15:29:17.550607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Target distribution","metadata":{}},{"cell_type":"code","source":"df[model_target].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:17.553527Z","iopub.execute_input":"2022-08-14T15:29:17.554064Z","iopub.status.idle":"2022-08-14T15:29:17.568122Z","shell.execute_reply.started":"2022-08-14T15:29:17.554016Z","shell.execute_reply":"2022-08-14T15:29:17.566904Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[model_target].value_counts().plot.bar()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:17.570123Z","iopub.execute_input":"2022-08-14T15:29:17.571324Z","iopub.status.idle":"2022-08-14T15:29:17.889671Z","shell.execute_reply.started":"2022-08-14T15:29:17.571264Z","shell.execute_reply":"2022-08-14T15:29:17.888217Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There is an imbalance in our target dataset. The 0 is much larger (nearly 4x) than 1. There are many ways to deal with this such as upsampling to address the imbalance. ","metadata":{}},{"cell_type":"markdown","source":"Histograms, min, max and cleanup\n\nHistograms help to view the distributions of numeric data fields and divide them into bins (or buckets).","metadata":{}},{"cell_type":"code","source":"df['loading'].value_counts().hist(bins=5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:37:34.652208Z","iopub.execute_input":"2022-08-14T15:37:34.652566Z","iopub.status.idle":"2022-08-14T15:37:34.858093Z","shell.execute_reply.started":"2022-08-14T15:37:34.652532Z","shell.execute_reply":"2022-08-14T15:37:34.856993Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['loading'].min()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:37:34.859625Z","iopub.execute_input":"2022-08-14T15:37:34.859972Z","iopub.status.idle":"2022-08-14T15:37:34.866426Z","shell.execute_reply.started":"2022-08-14T15:37:34.859940Z","shell.execute_reply":"2022-08-14T15:37:34.865607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['loading'].max()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:37:34.868253Z","iopub.execute_input":"2022-08-14T15:37:34.869113Z","iopub.status.idle":"2022-08-14T15:37:34.879905Z","shell.execute_reply.started":"2022-08-14T15:37:34.869080Z","shell.execute_reply":"2022-08-14T15:37:34.878996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['loading'].value_counts(bins=10, sort=False)","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:37:34.881207Z","iopub.execute_input":"2022-08-14T15:37:34.882299Z","iopub.status.idle":"2022-08-14T15:37:34.898980Z","shell.execute_reply.started":"2022-08-14T15:37:34.882263Z","shell.execute_reply":"2022-08-14T15:37:34.897929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_0'].value_counts().hist(bins=5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:17.890852Z","iopub.execute_input":"2022-08-14T15:29:17.891826Z","iopub.status.idle":"2022-08-14T15:29:18.112428Z","shell.execute_reply.started":"2022-08-14T15:29:17.891790Z","shell.execute_reply":"2022-08-14T15:29:18.111019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This looks reasonable well distributed but we can still look at the min, max and value counts in those bins","metadata":{}},{"cell_type":"code","source":"df['measurement_2'].min()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:18.114015Z","iopub.execute_input":"2022-08-14T15:29:18.114926Z","iopub.status.idle":"2022-08-14T15:29:18.122693Z","shell.execute_reply.started":"2022-08-14T15:29:18.114887Z","shell.execute_reply":"2022-08-14T15:29:18.121584Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_2'].max()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:18.124267Z","iopub.execute_input":"2022-08-14T15:29:18.124946Z","iopub.status.idle":"2022-08-14T15:29:18.138343Z","shell.execute_reply.started":"2022-08-14T15:29:18.124909Z","shell.execute_reply":"2022-08-14T15:29:18.137009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_0'].value_counts(bins=10, sort=False)","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:18.140144Z","iopub.execute_input":"2022-08-14T15:29:18.142672Z","iopub.status.idle":"2022-08-14T15:29:18.175293Z","shell.execute_reply.started":"2022-08-14T15:29:18.142619Z","shell.execute_reply":"2022-08-14T15:29:18.174177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_1'].value_counts().hist(bins=5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:18.177242Z","iopub.execute_input":"2022-08-14T15:29:18.178120Z","iopub.status.idle":"2022-08-14T15:29:18.390972Z","shell.execute_reply.started":"2022-08-14T15:29:18.178068Z","shell.execute_reply":"2022-08-14T15:29:18.389813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_2'].value_counts().hist(bins=5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:18.392644Z","iopub.execute_input":"2022-08-14T15:29:18.393378Z","iopub.status.idle":"2022-08-14T15:29:18.603230Z","shell.execute_reply.started":"2022-08-14T15:29:18.393334Z","shell.execute_reply":"2022-08-14T15:29:18.601831Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_3'].value_counts().hist(bins=5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:18.604364Z","iopub.execute_input":"2022-08-14T15:29:18.604699Z","iopub.status.idle":"2022-08-14T15:29:18.806254Z","shell.execute_reply.started":"2022-08-14T15:29:18.604668Z","shell.execute_reply":"2022-08-14T15:29:18.805023Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_3'].min()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:18.807652Z","iopub.execute_input":"2022-08-14T15:29:18.809076Z","iopub.status.idle":"2022-08-14T15:29:18.817331Z","shell.execute_reply.started":"2022-08-14T15:29:18.809022Z","shell.execute_reply":"2022-08-14T15:29:18.815958Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_3'].max()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:18.819427Z","iopub.execute_input":"2022-08-14T15:29:18.820141Z","iopub.status.idle":"2022-08-14T15:29:18.830347Z","shell.execute_reply.started":"2022-08-14T15:29:18.820052Z","shell.execute_reply":"2022-08-14T15:29:18.828596Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_3'].value_counts(bins=10, sort=False)","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:18.832368Z","iopub.execute_input":"2022-08-14T15:29:18.833163Z","iopub.status.idle":"2022-08-14T15:29:18.853331Z","shell.execute_reply.started":"2022-08-14T15:29:18.833106Z","shell.execute_reply":"2022-08-14T15:29:18.852053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_4'].value_counts().hist(bins=5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:18.854633Z","iopub.execute_input":"2022-08-14T15:29:18.855575Z","iopub.status.idle":"2022-08-14T15:29:19.083865Z","shell.execute_reply.started":"2022-08-14T15:29:18.855534Z","shell.execute_reply":"2022-08-14T15:29:19.082713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_5'].value_counts().hist(bins=5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:19.085576Z","iopub.execute_input":"2022-08-14T15:29:19.085953Z","iopub.status.idle":"2022-08-14T15:29:19.296900Z","shell.execute_reply.started":"2022-08-14T15:29:19.085918Z","shell.execute_reply":"2022-08-14T15:29:19.295928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_6'].value_counts().hist(bins=5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:19.298400Z","iopub.execute_input":"2022-08-14T15:29:19.299028Z","iopub.status.idle":"2022-08-14T15:29:19.518672Z","shell.execute_reply.started":"2022-08-14T15:29:19.298992Z","shell.execute_reply":"2022-08-14T15:29:19.517041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_7'].value_counts().hist(bins=5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:19.527215Z","iopub.execute_input":"2022-08-14T15:29:19.528149Z","iopub.status.idle":"2022-08-14T15:29:19.756782Z","shell.execute_reply.started":"2022-08-14T15:29:19.528102Z","shell.execute_reply":"2022-08-14T15:29:19.755442Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_8'].value_counts().hist(bins=5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:19.759592Z","iopub.execute_input":"2022-08-14T15:29:19.760152Z","iopub.status.idle":"2022-08-14T15:29:19.987475Z","shell.execute_reply.started":"2022-08-14T15:29:19.760105Z","shell.execute_reply":"2022-08-14T15:29:19.986147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_9'].value_counts().hist(bins=5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:19.988902Z","iopub.execute_input":"2022-08-14T15:29:19.989566Z","iopub.status.idle":"2022-08-14T15:29:20.149917Z","shell.execute_reply.started":"2022-08-14T15:29:19.989527Z","shell.execute_reply":"2022-08-14T15:29:20.148714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_10'].value_counts().hist(bins=5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:20.151700Z","iopub.execute_input":"2022-08-14T15:29:20.152499Z","iopub.status.idle":"2022-08-14T15:29:20.315076Z","shell.execute_reply.started":"2022-08-14T15:29:20.152448Z","shell.execute_reply":"2022-08-14T15:29:20.313957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_11'].value_counts().hist(bins=5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:20.316341Z","iopub.execute_input":"2022-08-14T15:29:20.316688Z","iopub.status.idle":"2022-08-14T15:29:20.486740Z","shell.execute_reply.started":"2022-08-14T15:29:20.316657Z","shell.execute_reply":"2022-08-14T15:29:20.485808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_12'].value_counts().hist(bins=5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:20.488666Z","iopub.execute_input":"2022-08-14T15:29:20.489549Z","iopub.status.idle":"2022-08-14T15:29:20.706192Z","shell.execute_reply.started":"2022-08-14T15:29:20.489491Z","shell.execute_reply":"2022-08-14T15:29:20.705359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_13'].value_counts().hist(bins=5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:20.707944Z","iopub.execute_input":"2022-08-14T15:29:20.708645Z","iopub.status.idle":"2022-08-14T15:29:20.923605Z","shell.execute_reply.started":"2022-08-14T15:29:20.708599Z","shell.execute_reply":"2022-08-14T15:29:20.922440Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_14'].value_counts().hist(bins=5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:20.925437Z","iopub.execute_input":"2022-08-14T15:29:20.926299Z","iopub.status.idle":"2022-08-14T15:29:21.151457Z","shell.execute_reply.started":"2022-08-14T15:29:20.926249Z","shell.execute_reply":"2022-08-14T15:29:21.150655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_15'].value_counts().hist(bins=5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:21.152875Z","iopub.execute_input":"2022-08-14T15:29:21.153441Z","iopub.status.idle":"2022-08-14T15:29:21.384106Z","shell.execute_reply.started":"2022-08-14T15:29:21.153406Z","shell.execute_reply":"2022-08-14T15:29:21.382215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_16'].value_counts().hist(bins=5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:21.386221Z","iopub.execute_input":"2022-08-14T15:29:21.386663Z","iopub.status.idle":"2022-08-14T15:29:21.780917Z","shell.execute_reply.started":"2022-08-14T15:29:21.386627Z","shell.execute_reply":"2022-08-14T15:29:21.779655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_17'].value_counts().hist(bins=5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:21.782392Z","iopub.execute_input":"2022-08-14T15:29:21.783099Z","iopub.status.idle":"2022-08-14T15:29:22.003306Z","shell.execute_reply.started":"2022-08-14T15:29:21.783061Z","shell.execute_reply":"2022-08-14T15:29:22.001845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_17'].min()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:22.004867Z","iopub.execute_input":"2022-08-14T15:29:22.005210Z","iopub.status.idle":"2022-08-14T15:29:22.012476Z","shell.execute_reply.started":"2022-08-14T15:29:22.005180Z","shell.execute_reply":"2022-08-14T15:29:22.011549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_17'].max()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:22.013694Z","iopub.execute_input":"2022-08-14T15:29:22.014112Z","iopub.status.idle":"2022-08-14T15:29:22.028411Z","shell.execute_reply.started":"2022-08-14T15:29:22.014078Z","shell.execute_reply":"2022-08-14T15:29:22.027227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_17'].value_counts(bins=10, sort=False)","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:22.029424Z","iopub.execute_input":"2022-08-14T15:29:22.029800Z","iopub.status.idle":"2022-08-14T15:29:22.053234Z","shell.execute_reply.started":"2022-08-14T15:29:22.029753Z","shell.execute_reply":"2022-08-14T15:29:22.051866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It might make sense to remove the outlier ","metadata":{}},{"cell_type":"code","source":"dropIndexes = df[df['measurement_17'] > 1201.194].index\ndf.drop(dropIndexes, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:22.054896Z","iopub.execute_input":"2022-08-14T15:29:22.055420Z","iopub.status.idle":"2022-08-14T15:29:22.072099Z","shell.execute_reply.started":"2022-08-14T15:29:22.055381Z","shell.execute_reply":"2022-08-14T15:29:22.070156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['measurement_17'].value_counts(bins=10, sort=False)","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:29:22.074104Z","iopub.execute_input":"2022-08-14T15:29:22.075120Z","iopub.status.idle":"2022-08-14T15:29:22.092065Z","shell.execute_reply.started":"2022-08-14T15:29:22.075081Z","shell.execute_reply":"2022-08-14T15:29:22.090857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You may notice a trend within the data histograms. This would mean we could probably drop ones that have very similar distributions since the data is very similar. This would speed up run time. ","metadata":{}},{"cell_type":"markdown","source":"**Scatter Plot**\n\nWe can also look at the relationship between two variables. Lets try product codes vs. attributres/measurement. ","metadata":{}},{"cell_type":"code","source":"df.plot.scatter(x='loading', y='product_code')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:40:13.544197Z","iopub.execute_input":"2022-08-14T15:40:13.545501Z","iopub.status.idle":"2022-08-14T15:40:13.747946Z","shell.execute_reply.started":"2022-08-14T15:40:13.545455Z","shell.execute_reply":"2022-08-14T15:40:13.746653Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From scatter plot above you cannot identify a relationship between these variables. You can try out any number of variables to see if you might find some that have a relationship. ","metadata":{}},{"cell_type":"markdown","source":"**Coorelation Matrix Heatmap**\n\nBeing able to see a heatmap on a matrix is a great way to quickly see if there is a coorelation. ","metadata":{}},{"cell_type":"code","source":"import seaborn as sns\n\nsns.heatmap(df.corr());","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:47:18.876031Z","iopub.execute_input":"2022-08-14T15:47:18.876455Z","iopub.status.idle":"2022-08-14T15:47:21.145979Z","shell.execute_reply.started":"2022-08-14T15:47:18.876411Z","shell.execute_reply":"2022-08-14T15:47:21.144752Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Increase the size of the heatmap.\nplt.figure(figsize=(16, 6))\n# Store heatmap object in a variable to easily access it when you want to include more features (such as title).\n# Set the range of values to be displayed on the colormap from -1 to 1, and set the annotation to True to display the correlation values on the heatmap.\nheatmap = sns.heatmap(df.corr(), vmin=-1, vmax=1, annot=True)\n# Give a title to the heatmap. Pad defines the distance of the title from the top of the heatmap.\nheatmap.set_title('Correlation Heatmap', fontdict={'fontsize':12}, pad=12);","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:48:16.610982Z","iopub.execute_input":"2022-08-14T15:48:16.611384Z","iopub.status.idle":"2022-08-14T15:48:19.137065Z","shell.execute_reply.started":"2022-08-14T15:48:16.611351Z","shell.execute_reply":"2022-08-14T15:48:19.135909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Correlation of Independent Variables with the Dependent Variable","metadata":{}},{"cell_type":"code","source":"df.corr()[['failure']].sort_values(by='failure', ascending=False)","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:49:44.166625Z","iopub.execute_input":"2022-08-14T15:49:44.167018Z","iopub.status.idle":"2022-08-14T15:49:44.224486Z","shell.execute_reply.started":"2022-08-14T15:49:44.166987Z","shell.execute_reply":"2022-08-14T15:49:44.223402Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(8, 12))\nheatmap = sns.heatmap(df.corr()[['failure']].sort_values(by='failure', ascending=False), vmin=-1, vmax=1, annot=True, cmap='BrBG')\nheatmap.set_title('Features Correlating with failure', fontdict={'fontsize':18}, pad=16);","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:50:28.066205Z","iopub.execute_input":"2022-08-14T15:50:28.066582Z","iopub.status.idle":"2022-08-14T15:50:28.632684Z","shell.execute_reply.started":"2022-08-14T15:50:28.066552Z","shell.execute_reply":"2022-08-14T15:50:28.631762Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Not a lot of great coorelations but you can also see many that are really poor with negatives. ","metadata":{}},{"cell_type":"markdown","source":"<div id=\"tvt\">\n    <h3>Train - Validation - Test datasets</h3></div>\n\nWe have train and test as csv files but we can also add in validation.","metadata":{}},{"cell_type":"code","source":"# add test data\ntest_data = pd.read_csv('../input/tabular-playground-series-aug-2022/test.csv')\n\nprint('shape of dataset: ', test_data.shape)","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:55:31.727246Z","iopub.execute_input":"2022-08-14T15:55:31.727770Z","iopub.status.idle":"2022-08-14T15:55:31.793871Z","shell.execute_reply.started":"2022-08-14T15:55:31.727709Z","shell.execute_reply":"2022-08-14T15:55:31.792453Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets now split the train into train/validation set using sklearn","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\ntrain_data, val_data, = train_test_split(df, test_size=0.15, shuffle=True, random_state=2022)","metadata":{"execution":{"iopub.status.busy":"2022-08-14T15:57:17.022622Z","iopub.execute_input":"2022-08-14T15:57:17.023061Z","iopub.status.idle":"2022-08-14T15:57:17.162612Z","shell.execute_reply.started":"2022-08-14T15:57:17.023027Z","shell.execute_reply":"2022-08-14T15:57:17.161511Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print the shapes of train - validation - test\nprint('train - validation - test dataset shapes: ', train_data.shape, val_data.shape, test_data.shape)","metadata":{"execution":{"iopub.status.busy":"2022-08-14T16:00:28.344302Z","iopub.execute_input":"2022-08-14T16:00:28.344920Z","iopub.status.idle":"2022-08-14T16:00:28.352319Z","shell.execute_reply.started":"2022-08-14T16:00:28.344871Z","shell.execute_reply":"2022-08-14T16:00:28.350949Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<div id=\"im\">\n    <h3>Impute Imbalance</h3></div>\nThere was a clear imbalance in our target label - failure. So we can upsample the class with less data, which is 1 in this case. \n\nNote: we will fix the imbalance only in training set. We don't want to change it in validation or test. ","metadata":{}},{"cell_type":"code","source":"# print the distribution for failure in the train dataset\nprint('train dataset class 0 samples: ', sum(train_data[model_target] == 0))\nprint('train dataset class 1 samples: ', sum(train_data[model_target] == 1))","metadata":{"execution":{"iopub.status.busy":"2022-08-14T16:03:46.375590Z","iopub.execute_input":"2022-08-14T16:03:46.375973Z","iopub.status.idle":"2022-08-14T16:03:46.388358Z","shell.execute_reply.started":"2022-08-14T16:03:46.375941Z","shell.execute_reply":"2022-08-14T16:03:46.386834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# upsample the lesser number of 1 class for failure to level the playing field\nfrom sklearn.utils import shuffle\n\nfail_0 = train_data[train_data[model_target] == 0]\nfail_1 = train_data[train_data[model_target] == 1]\n\nupsample_failure = fail_1.sample(n=len(fail_0), replace=True, random_state=2022)\n\ntrain_data = pd.concat([fail_0, upsample_failure])\ntrain_data = shuffle(train_data)","metadata":{"execution":{"iopub.status.busy":"2022-08-14T16:07:03.143410Z","iopub.execute_input":"2022-08-14T16:07:03.143806Z","iopub.status.idle":"2022-08-14T16:07:03.181280Z","shell.execute_reply.started":"2022-08-14T16:07:03.143775Z","shell.execute_reply":"2022-08-14T16:07:03.180190Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print the new distribution for failure in the train dataset\nprint('train dataset class 0 samples: ', sum(train_data[model_target] == 0))\nprint('train dataset class 1 samples: ', sum(train_data[model_target] == 1))","metadata":{"execution":{"iopub.status.busy":"2022-08-14T16:07:29.301751Z","iopub.execute_input":"2022-08-14T16:07:29.302182Z","iopub.status.idle":"2022-08-14T16:07:29.317666Z","shell.execute_reply.started":"2022-08-14T16:07:29.302146Z","shell.execute_reply":"2022-08-14T16:07:29.316825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<div id=\"missing\">\n    <h3>Missing Values</h3></div>\nlets check how many missing values there are for each column. ","metadata":{}},{"cell_type":"code","source":"df.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T16:08:55.275122Z","iopub.execute_input":"2022-08-14T16:08:55.275581Z","iopub.status.idle":"2022-08-14T16:08:55.297057Z","shell.execute_reply.started":"2022-08-14T16:08:55.275529Z","shell.execute_reply":"2022-08-14T16:08:55.295977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**There are a lot of missing values and grow throughout the measurements. \n**\nLets handle each of these differently. As we saw above, many of the measurements were similar in their distributions so we are going to just drop the columns that have a significant amount of missing data. ","metadata":{}},{"cell_type":"code","source":"df_columns_dropped = df.drop(columns = [\"measurement_3\", \"measurement_4\", \"measurement_5\",\n                                       \"measurement_6\", \"measurement_7\", \"measurement_8\",\n                                       \"measurement_9\", \"measurement_10\", \"measurement_11\",\n                                       \"measurement_12\", \"measurement_13\", \"measurement_14\",\n                                       \"measurement_15\", \"measurement_16\", \"measurement_17\"])","metadata":{"execution":{"iopub.status.busy":"2022-08-14T16:12:49.363846Z","iopub.execute_input":"2022-08-14T16:12:49.364253Z","iopub.status.idle":"2022-08-14T16:12:49.372871Z","shell.execute_reply.started":"2022-08-14T16:12:49.364220Z","shell.execute_reply":"2022-08-14T16:12:49.371240Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_columns_dropped.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T16:13:01.768814Z","iopub.execute_input":"2022-08-14T16:13:01.769310Z","iopub.status.idle":"2022-08-14T16:13:01.789900Z","shell.execute_reply.started":"2022-08-14T16:13:01.769267Z","shell.execute_reply":"2022-08-14T16:13:01.788822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_columns_dropped.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T16:13:24.947768Z","iopub.execute_input":"2022-08-14T16:13:24.948918Z","iopub.status.idle":"2022-08-14T16:13:24.962347Z","shell.execute_reply.started":"2022-08-14T16:13:24.948877Z","shell.execute_reply":"2022-08-14T16:13:24.961198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Impute missing values with .fillna()**\n\nThis allows us to get the mean value for the missing values. We will do this for loading instead of removing it. ","metadata":{}},{"cell_type":"code","source":"# impute loading by using mean\nload_impute = [\"loading\"]\n\n# assign a new df\ndf_imputed = df_columns_dropped.copy()\nprint(df_imputed[load_impute].isna().sum())\n\ndf_imputed[load_impute] = df_imputed[load_impute].fillna(df_imputed[load_impute].mean())\nprint(df_imputed[load_impute].isna().sum())","metadata":{"execution":{"iopub.status.busy":"2022-08-14T16:20:43.535776Z","iopub.execute_input":"2022-08-14T16:20:43.536162Z","iopub.status.idle":"2022-08-14T16:20:43.554962Z","shell.execute_reply.started":"2022-08-14T16:20:43.536132Z","shell.execute_reply":"2022-08-14T16:20:43.553776Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_imputed.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-14T16:20:46.366643Z","iopub.execute_input":"2022-08-14T16:20:46.367124Z","iopub.status.idle":"2022-08-14T16:20:46.385761Z","shell.execute_reply.started":"2022-08-14T16:20:46.367091Z","shell.execute_reply":"2022-08-14T16:20:46.384149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"😄 ⬆️ If you enjoy please don't forget to upvote!","metadata":{}}]}