{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom sklearn import ensemble\nfrom sklearn import metrics\nfrom sklearn import model_selection\nimport matplotlib.pyplot as plt\nimport seaborn as sns\npd.set_option('display.max_columns', 500)","metadata":{"execution":{"iopub.status.busy":"2022-08-01T17:31:30.863214Z","iopub.execute_input":"2022-08-01T17:31:30.863658Z","iopub.status.idle":"2022-08-01T17:31:30.870301Z","shell.execute_reply.started":"2022-08-01T17:31:30.863622Z","shell.execute_reply":"2022-08-01T17:31:30.868811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# TPS - AUG 22 Exploratory Analysis\nby: [Vinicius Duzac Cerutti](https://www.linkedin.com/in/vinicius-duzac-cerutti/)\n\n<br>\n<div style=\"display:inline-block;\">\n    <span style=\"font-size:20px\"> Summary:</span>\n<ul>\n<li><a href=\"#problem_definition\">Problem Definition</a></li>\n<li><a href=\"#data_analysis\">Data Analysis</a></li>\n<ul>\n<li><a href=\"#missing_values\">Missing Values</a></li>\n<li><a href=\"#numeric_distribution\">Numeric Data Distribution</a></li>\n<li><a href=\"#categoric_distribution\">Categoric Data Distribution</a></li>\n<li><a href=\"#correlation\">Correlation Between Features</a></li>\n</ul>\n<li><a href=\"#conclusion\">Conclusion</a></li>\n\n</div>\n<div style=\"display:inline-block;vertical-align:top; margin-left:100px\">\n<img src=\"https://i.makeagif.com/media/7-22-2014/NWEt1x.gif\" alt=\"img\" width=\"350\" height=\"350\"/>\n</div>\n\n***","metadata":{}},{"cell_type":"markdown","source":"<a id='problem_definition'></a>\n### Problem Definition:\nThis data represents the results of a large product testing study. For each product_code you are given a number of product attributes (fixed for the code) as well as a number of measurement values for each individual product, representing various lab testing methods. Each product is used in a simulated real-world environment experiment, and and absorbs a certain amount of fluid (loading) to see whether or not it fails.\n\nYour task is to use the data to predict individual product failures of new codes with their individual lab test results.\n<a id='data_analysis'></a>\n### Data Analysis:","metadata":{}},{"cell_type":"code","source":"train_data = pd.read_csv('../input/tabular-playground-series-aug-2022/train.csv')","metadata":{"execution":{"iopub.status.busy":"2022-08-01T17:31:30.882122Z","iopub.execute_input":"2022-08-01T17:31:30.883563Z","iopub.status.idle":"2022-08-01T17:31:31.014516Z","shell.execute_reply.started":"2022-08-01T17:31:30.883519Z","shell.execute_reply":"2022-08-01T17:31:31.012971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.sample(10,random_state = 10)","metadata":{"execution":{"iopub.status.busy":"2022-08-01T17:31:31.016678Z","iopub.execute_input":"2022-08-01T17:31:31.017022Z","iopub.status.idle":"2022-08-01T17:31:31.057577Z","shell.execute_reply.started":"2022-08-01T17:31:31.016990Z","shell.execute_reply":"2022-08-01T17:31:31.056311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def bar_plot(x, y, hue=None, title=None, h=True, ax=None):\n    \n    g = sns.barplot(x=x,y=y,hue=hue,ax=ax)\n   \n    sns.despine(bottom = True, left = True)\n    \n    for i in range(0, len(g.containers)):\n        g.bar_label(g.containers[i], label_type='edge', color='black',\n                fmt='%.2f%%',padding = 10, fontsize=12)\n    \n    if h:\n        g.set(yticks=[])\n    else:\n        g.set(xticks=[])\n        \n    g.axes.set_title(title,fontsize=12)\n    g.tick_params(labelsize=12)\n    if hue is not None:\n        ax.get_legend().remove()\n        \ndef groupby_pct(data, groupby, values):\n    data_groupby = data.groupby(groupby)[values].value_counts().copy()\n    return  (data_groupby / data_groupby.groupby(level=0).sum()) * 100","metadata":{"execution":{"iopub.status.busy":"2022-08-01T17:32:09.578523Z","iopub.execute_input":"2022-08-01T17:32:09.579025Z","iopub.status.idle":"2022-08-01T17:32:09.593472Z","shell.execute_reply.started":"2022-08-01T17:32:09.578983Z","shell.execute_reply":"2022-08-01T17:32:09.592269Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have an unbalanced class dataset, which is good because we have fewer faulty products.","metadata":{}},{"cell_type":"code","source":"class_train = train_data['failure'].value_counts(normalize=True) * 100\nfig = plt.figure(figsize=(2,3),tight_layout=True)\nbar_plot(class_train.index, class_train.values)\nfig.suptitle('Distribution of Fault or Not', x=0.01, y=1, horizontalalignment='left', verticalalignment='top', fontsize = 16)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-01T17:31:31.074779Z","iopub.execute_input":"2022-08-01T17:31:31.075106Z","iopub.status.idle":"2022-08-01T17:31:31.192942Z","shell.execute_reply.started":"2022-08-01T17:31:31.075077Z","shell.execute_reply":"2022-08-01T17:31:31.191467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, there isn't rows duplicates on the sample train dataset.","metadata":{}},{"cell_type":"code","source":"number_rows = train_data.shape[0]\nprint(f'Number of Rows: {number_rows}')\nprint(f'Drop Duplicates: {train_data.drop_duplicates().shape[0] - number_rows}')","metadata":{"execution":{"iopub.status.busy":"2022-08-01T17:31:31.195135Z","iopub.execute_input":"2022-08-01T17:31:31.195641Z","iopub.status.idle":"2022-08-01T17:31:31.269572Z","shell.execute_reply.started":"2022-08-01T17:31:31.195595Z","shell.execute_reply":"2022-08-01T17:31:31.268291Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_type_column_and_na(data, column):\n    print(f\"{column}, {data[column].dtype}, {data[column].isna().mean().round(2)}\")","metadata":{"execution":{"iopub.status.busy":"2022-08-01T17:31:31.271302Z","iopub.execute_input":"2022-08-01T17:31:31.272115Z","iopub.status.idle":"2022-08-01T17:31:31.280693Z","shell.execute_reply.started":"2022-08-01T17:31:31.272064Z","shell.execute_reply":"2022-08-01T17:31:31.279209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='missing_values'></a>\nWe see that we have features with null values, we need to check whether or not these values have an impact on the product failure. To do this, we will apply a high negative value to these null values and check the patterns.","metadata":{}},{"cell_type":"code","source":"print(\"column_name, dtype, na_values\")\nfor column in train_data.columns:\n    get_type_column_and_na(train_data,column)\nprint(train_data.shape)","metadata":{"execution":{"iopub.status.busy":"2022-08-01T17:31:31.283870Z","iopub.execute_input":"2022-08-01T17:31:31.284908Z","iopub.status.idle":"2022-08-01T17:31:31.312291Z","shell.execute_reply.started":"2022-08-01T17:31:31.284869Z","shell.execute_reply":"2022-08-01T17:31:31.311366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For columns with null values, we can apply a negative value or zero value. This depends on the interpretation of the columns, I imagine that zero value could be a value and negative value would be impossible.","metadata":{}},{"cell_type":"code","source":"na_columns = ['loading','measurement_3','measurement_4',\n 'measurement_5','measurement_6','measurement_7',\n 'measurement_8','measurement_9','measurement_10','measurement_11',\n 'measurement_12','measurement_13','measurement_14',\n 'measurement_15','measurement_16','measurement_17']\ntrain_data[na_columns].describe().T","metadata":{"execution":{"iopub.status.busy":"2022-08-01T17:31:31.313349Z","iopub.execute_input":"2022-08-01T17:31:31.314233Z","iopub.status.idle":"2022-08-01T17:31:31.403550Z","shell.execute_reply.started":"2022-08-01T17:31:31.314198Z","shell.execute_reply":"2022-08-01T17:31:31.402075Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"numeric_features = [column for column in train_data.columns if train_data[column].dtype == 'float64']\nnumeric_features.extend(['attribute_2','attribute_3'])","metadata":{"execution":{"iopub.status.busy":"2022-08-01T17:31:31.405171Z","iopub.execute_input":"2022-08-01T17:31:31.406248Z","iopub.status.idle":"2022-08-01T17:31:31.411787Z","shell.execute_reply.started":"2022-08-01T17:31:31.406206Z","shell.execute_reply":"2022-08-01T17:31:31.410618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='numeric_distribution'></a>\nWe can see that there is no difference between a product failing or not failing. The distribution pattern is the same, there are only changes in density. Also, we can see that the features have a normal distribution approximation, so we can apply some linear models.","metadata":{}},{"cell_type":"code","source":"fig, axes = plt.subplots(3,6, figsize=(25,10))\naxes = axes.flatten()\nfor i in range(0,len(axes)):\n    if i < len(numeric_features):\n        col = numeric_features[i]\n        sns.kdeplot(data=train_data, x=col, ax=axes[i],hue='failure')\n        fig.tight_layout()  \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-01T17:31:31.415271Z","iopub.execute_input":"2022-08-01T17:31:31.415975Z","iopub.status.idle":"2022-08-01T17:31:41.457677Z","shell.execute_reply.started":"2022-08-01T17:31:31.415923Z","shell.execute_reply":"2022-08-01T17:31:41.456346Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='categoric_distribution'></a>\nFor attribute_0 the product containing material 7 is more likely to fail than material 5, and for attribute_1 it is material_8","metadata":{}},{"cell_type":"code","source":"fig, axes = plt.subplots(1, 2,figsize=(10,4), tight_layout=True)\natt_columns = ['attribute_0', 'attribute_1']\naxes = axes.flatten()\n\nfor i in range(0, len(axes)):\n    col = att_columns[i]\n    pct_att = groupby_pct(train_data, col, 'failure').reset_index(name='percentage')\n    bar_plot(pct_att['percentage'], pct_att[col], pct_att['failure'],ax=axes[i])\nplt.legend(loc='lower right', bbox_to_anchor=(1,0),title='failure',ncol=2, bbox_transform=fig.transFigure)\nfig.suptitle('Data Distribution of Products', x=0.01, y=1, horizontalalignment='left', verticalalignment='top', fontsize = 16)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-01T17:32:14.503697Z","iopub.execute_input":"2022-08-01T17:32:14.504065Z","iopub.status.idle":"2022-08-01T17:32:14.892147Z","shell.execute_reply.started":"2022-08-01T17:32:14.504036Z","shell.execute_reply":"2022-08-01T17:32:14.890730Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='correlation'></a>\nAs we can see, we have a low positive and negative correlation between the features.\n* Feature attribute_3 has some correlation with attribute_2 -0.54;\n* measurement_0 has some correlation with attribute_2, attribute_3;\n* measurement_1 has some correlation with attribute_2, attribute_3 and measurement_0;\n* measurement_17 has some correlation with attribute measurement_4 - measurement_9.","metadata":{}},{"cell_type":"code","source":"df_correlation = train_data.drop(['id','failure'],axis=1).copy(0)\ncorrelation = df_correlation.corr().round(2)","metadata":{"execution":{"iopub.status.busy":"2022-08-01T17:32:22.680247Z","iopub.execute_input":"2022-08-01T17:32:22.680678Z","iopub.status.idle":"2022-08-01T17:32:22.731842Z","shell.execute_reply.started":"2022-08-01T17:32:22.680643Z","shell.execute_reply":"2022-08-01T17:32:22.731019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = plt.figure(figsize=(11,9),tight_layout=True)\nmask = np.triu(np.ones_like(correlation, dtype=bool))\nsns.heatmap(correlation, annot=True, vmax=1, vmin=-1, center=0, cmap='vlag',cbar=False, mask=mask)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-01T17:32:26.460693Z","iopub.execute_input":"2022-08-01T17:32:26.461098Z","iopub.status.idle":"2022-08-01T17:32:27.971287Z","shell.execute_reply.started":"2022-08-01T17:32:26.461063Z","shell.execute_reply":"2022-08-01T17:32:27.970105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='conclusion'></a>\n### Conclusion:\n* We have an unbalanced class dataset 21.26% of the products failed the tests;\n* There are no duplicate rows in the training dataset;\n* Some features have null values 0.01% ~ 0.09%. We can apply a number arbiter for this (because 0 might be a good result).\n* All numerical features have a normal distribution approximation, we can apply some linear models.\n* For attribute_0 the product containing material 7 is more likely to fail than material 5, and for attribute_1 it is material_8.\n* We have some low positive and negative correlations between features but no relevance.\n\n<div class=\"alert alert-success\">\n  And that’s it! It has been a pleasure to make this kernel, I have learned a lot! Thank you for reading and if you like it, please upvote it!\n</div>\n\n***\n\n<span style=\"font-size:12px\"><b>Last kernels that I worked:</b></span>\n\n<a style=\"font-size:12px\" href=\"https://www.kaggle.com/code/ceruttivini/recommender-systems-for-article-comments\">Recommender Systems for Article Comments</a><br>\n<a style=\"font-size:12px\" href=\"https://www.kaggle.com/code/ceruttivini/cientista-ou-analista-de-dados-qual-a-diferen-a\">Data Scientist or Data Analyst? What is the difference? (Brazil)</a><br>\n<a style=\"font-size:12px\" href=\"https://www.kaggle.com/code/ceruttivini/association-rules-mining-market-basket-analysis\">Association Rules Mining/Market Basket Analysis</a>","metadata":{}}]}