{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Tabular Playground Series - August 2022 - EDA tamplate","metadata":{}},{"cell_type":"markdown","source":"This is only the Exploratory Data Analysis of the TPS-Aug-22. Althogh it needs to be tuned and improved, I think it is a good start for beginners. I hope you find it useful.","metadata":{}},{"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nfrom collections import Counter\nimport scipy.stats as ss\n\nimport seaborn as sns\nfrom matplotlib import pyplot as plt\n%matplotlib inline\n\nimport missingno as msno\nfrom dython import nominal as dy","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:28.301273Z","iopub.execute_input":"2022-08-10T08:33:28.301817Z","iopub.status.idle":"2022-08-10T08:33:28.941828Z","shell.execute_reply.started":"2022-08-10T08:33:28.301690Z","shell.execute_reply":"2022-08-10T08:33:28.940767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Loading datasets  <a class=\"anchor\" id=\"1-Load\"></a>","metadata":{}},{"cell_type":"code","source":"# Set the working directory\nos.chdir(\"/kaggle/input/tabular-playground-series-aug-2022\")","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:28.944001Z","iopub.execute_input":"2022-08-10T08:33:28.944348Z","iopub.status.idle":"2022-08-10T08:33:28.949809Z","shell.execute_reply.started":"2022-08-10T08:33:28.944319Z","shell.execute_reply":"2022-08-10T08:33:28.948516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1.1. Load data and verify","metadata":{}},{"cell_type":"code","source":"# Load the train dataset\ntrain_df = pd.read_csv('train.csv', sep=',', encoding='UTF-8', index_col='id')\n\nprint(f'Train: dimensions {train_df.shape}')\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:28.950961Z","iopub.execute_input":"2022-08-10T08:33:28.951339Z","iopub.status.idle":"2022-08-10T08:33:29.106027Z","shell.execute_reply.started":"2022-08-10T08:33:28.951309Z","shell.execute_reply":"2022-08-10T08:33:29.104780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load the test dataset\ntest_df = pd.read_csv('test.csv', sep=',', encoding='UTF-8', index_col='id')\n\nprint(f'Test: dimensions {test_df.shape}')\ntest_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:29.107503Z","iopub.execute_input":"2022-08-10T08:33:29.107861Z","iopub.status.idle":"2022-08-10T08:33:29.212057Z","shell.execute_reply.started":"2022-08-10T08:33:29.107830Z","shell.execute_reply":"2022-08-10T08:33:29.210848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1.2. Concatenate datasets","metadata":{}},{"cell_type":"markdown","source":"In order to make easier the data analysis and the feature engineering, better merge both datasets.","metadata":{}},{"cell_type":"code","source":"# First, let's check the target variable:\ntrain_df['failure'].unique()","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:29.214801Z","iopub.execute_input":"2022-08-10T08:33:29.215157Z","iopub.status.idle":"2022-08-10T08:33:29.225321Z","shell.execute_reply.started":"2022-08-10T08:33:29.215102Z","shell.execute_reply":"2022-08-10T08:33:29.224131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# There is only 2 values, so I assign a '2' as a dummy value:\ntest_df['failure'] = 2","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:29.226864Z","iopub.execute_input":"2022-08-10T08:33:29.227233Z","iopub.status.idle":"2022-08-10T08:33:29.235690Z","shell.execute_reply.started":"2022-08-10T08:33:29.227201Z","shell.execute_reply":"2022-08-10T08:33:29.234848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Now, both datasets have the same columns, so I can concatenate them:\ncombi_df = pd.concat([train_df, test_df])\n\n# Verify lengths:\nprint(len(train_df))\nprint(len(test_df))\nprint(len(combi_df))","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:29.236607Z","iopub.execute_input":"2022-08-10T08:33:29.236962Z","iopub.status.idle":"2022-08-10T08:33:29.263288Z","shell.execute_reply.started":"2022-08-10T08:33:29.236928Z","shell.execute_reply":"2022-08-10T08:33:29.261556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# I create a 'rowid' column for graphical representations\ncombi_df['rowid'] = combi_df.index","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:29.265027Z","iopub.execute_input":"2022-08-10T08:33:29.265838Z","iopub.status.idle":"2022-08-10T08:33:29.274015Z","shell.execute_reply.started":"2022-08-10T08:33:29.265787Z","shell.execute_reply":"2022-08-10T08:33:29.272430Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Exploratory data analysis <a class=\"anchor\" id=\"2-analysis\"></a>","metadata":{}},{"cell_type":"markdown","source":"First checks of the data to analize tha data quality, data types, missing values and outliers, patterns, ... The purpose is to have an idea of the data in order to be able to define what kind of data transformation may be needed.","metadata":{}},{"cell_type":"markdown","source":"## 2.1. Look for duplicates","metadata":{}},{"cell_type":"code","source":"# Check number of duplicates while ignoring the index feature (train dataset)\ntrain_df.duplicated().sum()\n\n# In case of duplicates, the rows could be removed.","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:29.276328Z","iopub.execute_input":"2022-08-10T08:33:29.277002Z","iopub.status.idle":"2022-08-10T08:33:29.323363Z","shell.execute_reply.started":"2022-08-10T08:33:29.276966Z","shell.execute_reply":"2022-08-10T08:33:29.321971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.2. Cardinality and data types","metadata":{}},{"cell_type":"code","source":"# Let's validate the unique values:\ncombi_df.nunique()","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:29.325672Z","iopub.execute_input":"2022-08-10T08:33:29.326185Z","iopub.status.idle":"2022-08-10T08:33:29.369797Z","shell.execute_reply.started":"2022-08-10T08:33:29.326139Z","shell.execute_reply":"2022-08-10T08:33:29.368405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# .. and the data types\ncombi_df.info()","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:29.372997Z","iopub.execute_input":"2022-08-10T08:33:29.373797Z","iopub.status.idle":"2022-08-10T08:33:29.402865Z","shell.execute_reply.started":"2022-08-10T08:33:29.373748Z","shell.execute_reply":"2022-08-10T08:33:29.401594Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Make groups of columns, acconding to the cardinality and datatype:\ncols_categories = ['product_code', 'attribute_0', 'attribute_1', 'attribute_2', 'attribute_3']\ncombi_df[cols_categories] = combi_df[cols_categories].astype(str)\n\ncols_continuous = ['loading', 'measurement_0', 'measurement_1', 'measurement_2', 'measurement_3', 'measurement_4', 'measurement_5', \n                   'measurement_6', 'measurement_7', 'measurement_8', 'measurement_9', 'measurement_10', 'measurement_11', \n                   'measurement_12', 'measurement_13', 'measurement_14', 'measurement_15', 'measurement_16', 'measurement_17']","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:29.404902Z","iopub.execute_input":"2022-08-10T08:33:29.405579Z","iopub.status.idle":"2022-08-10T08:33:29.489029Z","shell.execute_reply.started":"2022-08-10T08:33:29.405518Z","shell.execute_reply":"2022-08-10T08:33:29.487621Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.3. Missing values","metadata":{}},{"cell_type":"code","source":"# There were some missing values. Let's see the proportion:\ntotal = len(combi_df)\ncombi_df.isna().sum()/total","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:29.490756Z","iopub.execute_input":"2022-08-10T08:33:29.491243Z","iopub.status.idle":"2022-08-10T08:33:29.516146Z","shell.execute_reply.started":"2022-08-10T08:33:29.491195Z","shell.execute_reply":"2022-08-10T08:33:29.514865Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 2/3rd of the features have missing values, but at the most, it is a 8.5%. Let's see if there is recognizable distribution:\nmsno.heatmap(combi_df, figsize=(6,4), fontsize=10)","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:29.521737Z","iopub.execute_input":"2022-08-10T08:33:29.522075Z","iopub.status.idle":"2022-08-10T08:33:30.153444Z","shell.execute_reply.started":"2022-08-10T08:33:29.522047Z","shell.execute_reply":"2022-08-10T08:33:30.151975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Conclusions:\n# There is no relationship among the missing values. I have to deal with them individualy.","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:30.154649Z","iopub.execute_input":"2022-08-10T08:33:30.155004Z","iopub.status.idle":"2022-08-10T08:33:30.160855Z","shell.execute_reply.started":"2022-08-10T08:33:30.154965Z","shell.execute_reply":"2022-08-10T08:33:30.159454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.4. Value distribution","metadata":{}},{"cell_type":"markdown","source":"#### Continuous variables","metadata":{}},{"cell_type":"code","source":"# Features with several values (continuous)\n\nfig = plt.figure(figsize=(20,15))\n          \nfor i,f in enumerate(cols_continuous):\n    plt.subplot(5, 4, i+1)\n    sns.kdeplot(data=combi_df, x=f, hue='failure', fill=False, palette='Set1')\n    plt.xlabel(f)\n\nplt.tight_layout()    \nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:30.162654Z","iopub.execute_input":"2022-08-10T08:33:30.163794Z","iopub.status.idle":"2022-08-10T08:33:39.744247Z","shell.execute_reply.started":"2022-08-10T08:33:30.163753Z","shell.execute_reply":"2022-08-10T08:33:39.742902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Conclusions:\n# The distributions are very alike, so no insights can be excracted.","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:39.745727Z","iopub.execute_input":"2022-08-10T08:33:39.746136Z","iopub.status.idle":"2022-08-10T08:33:39.751498Z","shell.execute_reply.started":"2022-08-10T08:33:39.746082Z","shell.execute_reply":"2022-08-10T08:33:39.750057Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Categorical variables ","metadata":{}},{"cell_type":"code","source":"fig = plt.figure(figsize=(12,6))\n          \nfor i,f in enumerate(cols_categories):\n    plt.subplot(2, 3, i+1)\n    sns.histplot(combi_df.sort_values(by=f), x=f, hue='failure', multiple='dodge', shrink=.8, palette='Set1', alpha=0.5)\n    plt.xlabel(f)\n\nplt.tight_layout()    \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:39.752997Z","iopub.execute_input":"2022-08-10T08:33:39.753415Z","iopub.status.idle":"2022-08-10T08:33:41.264954Z","shell.execute_reply.started":"2022-08-10T08:33:39.753382Z","shell.execute_reply":"2022-08-10T08:33:41.263682Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Conclusions:\n# The variable product_code is the 'question': what is the prediction for these new products?\n# The ratio between failure values (0, 1) is more or less the same in any case. We cannot obtain information.","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:41.266666Z","iopub.execute_input":"2022-08-10T08:33:41.267016Z","iopub.status.idle":"2022-08-10T08:33:41.271417Z","shell.execute_reply.started":"2022-08-10T08:33:41.266986Z","shell.execute_reply":"2022-08-10T08:33:41.270387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.5. Data distribution in dataset","metadata":{}},{"cell_type":"markdown","source":"Complementary value distribution analysis to know better the data and to identify unwanted entries or recording errors.","metadata":{}},{"cell_type":"markdown","source":"#### Continuous variables","metadata":{}},{"cell_type":"code","source":"# Continuous variables\ncombi_df[cols_continuous].plot(lw=0, \n                               marker=\".\", \n                               subplots=True, \n                               layout=(-1, 4), \n                               figsize=(18, 18), \n                               markersize=1);","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:41.272681Z","iopub.execute_input":"2022-08-10T08:33:41.273016Z","iopub.status.idle":"2022-08-10T08:33:47.479592Z","shell.execute_reply.started":"2022-08-10T08:33:41.272985Z","shell.execute_reply":"2022-08-10T08:33:47.478209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Conclusions\n# We can see vertical bands in several of features. I must investigate.","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:47.481068Z","iopub.execute_input":"2022-08-10T08:33:47.482221Z","iopub.status.idle":"2022-08-10T08:33:47.486386Z","shell.execute_reply.started":"2022-08-10T08:33:47.482179Z","shell.execute_reply":"2022-08-10T08:33:47.485411Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Categorical variables ","metadata":{}},{"cell_type":"code","source":"# Categorical features\nfig, axes = plt.subplots(ncols=1, nrows=5, figsize=(10, 8))\n\nfor c, ax in zip(combi_df[cols_categories].columns, axes.ravel()):\n    combi_df.plot.scatter(x='rowid', y=c, c='DarkBlue', s=1, ax=ax)\n    \nplt.tight_layout();","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:47.487859Z","iopub.execute_input":"2022-08-10T08:33:47.489208Z","iopub.status.idle":"2022-08-10T08:33:48.516701Z","shell.execute_reply.started":"2022-08-10T08:33:47.489169Z","shell.execute_reply":"2022-08-10T08:33:48.515766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Conclusions\n# From the 'product_code' distribution we can see the origin of the vertical bands in the continuous features graphics.","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:48.517928Z","iopub.execute_input":"2022-08-10T08:33:48.518967Z","iopub.status.idle":"2022-08-10T08:33:48.523761Z","shell.execute_reply.started":"2022-08-10T08:33:48.518923Z","shell.execute_reply":"2022-08-10T08:33:48.522387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.6. Outliers","metadata":{}},{"cell_type":"code","source":"fig = plt.figure(figsize=(20,15))\n          \nfor i,f in enumerate(cols_continuous):\n    plt.subplot(5, 4, i+1)\n    sns.boxplot(data=combi_df, x='failure', y=f, palette='Set1', boxprops=dict(alpha=.5))\n    plt.xlabel(f)\n\nplt.tight_layout()    \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:48.525618Z","iopub.execute_input":"2022-08-10T08:33:48.526440Z","iopub.status.idle":"2022-08-10T08:33:51.620123Z","shell.execute_reply.started":"2022-08-10T08:33:48.526374Z","shell.execute_reply":"2022-08-10T08:33:51.618867Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Conclusions:\n# Every feature has outliers, but none of them are extreme values and seem to fit in the distrubution previously seen.","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:51.621732Z","iopub.execute_input":"2022-08-10T08:33:51.622143Z","iopub.status.idle":"2022-08-10T08:33:51.627887Z","shell.execute_reply.started":"2022-08-10T08:33:51.622088Z","shell.execute_reply":"2022-08-10T08:33:51.626532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.7. Feature correlation","metadata":{}},{"cell_type":"markdown","source":"#### Continuous variables","metadata":{}},{"cell_type":"code","source":"# Let's check the correlation among the numeric features\nplt.figure(figsize=(9, 6))\nsns.heatmap(combi_df.corr(), vmin=-1, vmax=1, cmap='vlag')","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:51.629803Z","iopub.execute_input":"2022-08-10T08:33:51.630296Z","iopub.status.idle":"2022-08-10T08:33:52.336583Z","shell.execute_reply.started":"2022-08-10T08:33:51.630251Z","shell.execute_reply":"2022-08-10T08:33:52.335349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Conclusions:\n# There is a slight correlation between 'measurement_17' and 'measurement_4' to '_8', but I don't see it  significative.","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:52.337932Z","iopub.execute_input":"2022-08-10T08:33:52.338284Z","iopub.status.idle":"2022-08-10T08:33:52.343741Z","shell.execute_reply.started":"2022-08-10T08:33:52.338252Z","shell.execute_reply":"2022-08-10T08:33:52.342569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" #### Categorical variables","metadata":{}},{"cell_type":"code","source":"# I'll use the 'dython' library to get the asymmetric correlation between the variables (Theils U)\ndy.associations(combi_df[cols_categories], nom_nom_assoc='theil', figsize=(5, 5))","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:52.345652Z","iopub.execute_input":"2022-08-10T08:33:52.345997Z","iopub.status.idle":"2022-08-10T08:33:54.647515Z","shell.execute_reply.started":"2022-08-10T08:33:52.345968Z","shell.execute_reply":"2022-08-10T08:33:54.646476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Conclusions:\n# There some correlation between some of the discrete variables, mainly with 'product_code' and 'attribute_3'\n# We can see that, given a categorical value, we surely know the product code, but not otherwise. For attribute_3 it is not so clear.","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:54.649259Z","iopub.execute_input":"2022-08-10T08:33:54.649604Z","iopub.status.idle":"2022-08-10T08:33:54.655589Z","shell.execute_reply.started":"2022-08-10T08:33:54.649574Z","shell.execute_reply":"2022-08-10T08:33:54.654282Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Continuous vs. categorical variables","metadata":{}},{"cell_type":"code","source":"# Let's see if there is a significative correlation among categorical and continuous variables.\n# I'm using the 'dython' libraries.\n\nTHRESHOLD = 0.4\nfor cat in cols_categories:\n    for num in cols_continuous:\n        corr = dy.correlation_ratio(categories=combi_df[cat], measurements=combi_df[num])\n        if corr > THRESHOLD:\n            print(f'{cat} / {num}: {round(corr, 3)}')","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:54.658952Z","iopub.execute_input":"2022-08-10T08:33:54.660056Z","iopub.status.idle":"2022-08-10T08:33:58.112551Z","shell.execute_reply.started":"2022-08-10T08:33:54.660017Z","shell.execute_reply":"2022-08-10T08:33:58.111339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Visually\nfig, ax = plt.subplots(figsize=(8, 4))\n\nplt.subplot(1, 2, 1)\nsns.kdeplot(data=combi_df, x='measurement_0', hue='product_code', fill=False, palette='Set1')\nplt.xlabel('measurement_0')\n\nplt.subplot(1, 2, 2)\nsns.kdeplot(data=combi_df, x='measurement_1', hue='product_code', fill=False, palette='Set1')\nplt.xlabel('measurement_1')\n    \nplt.tight_layout()   \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:58.114012Z","iopub.execute_input":"2022-08-10T08:33:58.114359Z","iopub.status.idle":"2022-08-10T08:33:59.131423Z","shell.execute_reply.started":"2022-08-10T08:33:58.114330Z","shell.execute_reply":"2022-08-10T08:33:59.130050Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The partial distributions do not overlap completely, but don't offer definitive information.","metadata":{"execution":{"iopub.status.busy":"2022-08-10T08:33:59.133242Z","iopub.execute_input":"2022-08-10T08:33:59.133681Z","iopub.status.idle":"2022-08-10T08:33:59.140196Z","shell.execute_reply.started":"2022-08-10T08:33:59.133628Z","shell.execute_reply":"2022-08-10T08:33:59.138720Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.8. Conclusions and action plan","metadata":{}},{"cell_type":"markdown","source":"Insights:\n* The amount of missing values is very low. We can impute the missing data.\n* I have divided the columns in 2 types, categorical and numerical.\n* The value distributions doesn't tell much.\n* No significant outliers.\n* There is no significant correlation among features.","metadata":{}},{"cell_type":"markdown","source":"# 3. Feature engineering <a class=\"anchor\" id=\"3-feature\"></a>","metadata":{}},{"cell_type":"markdown","source":"...","metadata":{}}]}