{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-07-21T02:22:46.802809Z","iopub.execute_input":"2022-07-21T02:22:46.803949Z","iopub.status.idle":"2022-07-21T02:22:46.834141Z","shell.execute_reply.started":"2022-07-21T02:22:46.803808Z","shell.execute_reply":"2022-07-21T02:22:46.833155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Imputation**\n\nDatasets may have missing values, and this can cause problems for many machine learning algorithms.\n\nAs such, it is good practice to identify and replace missing values for each column in your input data prior to modeling your prediction task. This is called missing data imputation, or imputing for short.\n\n**Imputation is a technique used for replacing the missing data with some substitute value to retain most of the data/information of the dataset. These techniques are used because removing the data from the dataset every time is not feasible and can lead to a reduction in the size of the dataset to a large extend, which not only raises concerns for biasing the dataset but also leads to incorrect analysis.**\n\n![imp1](https://miro.medium.com/max/1400/1*PovaJ2Ka7PdlJinqjHAwDQ.png)\n\n#### **Note**\n\n* *Missing values must be marked with NaN values and can be replaced with statistical measures to calculate the column of values.*\n* *How to load a CSV value with missing values and mark the missing values with NaN values and report the number and percentage of missing values for each column.*\n* *How to impute missing values with statistics as a data preparation method when evaluating models and when fitting a final model to make predictions on new data.*\n\n## **Why Its Needed?**\n\nWe use imputation because Missing data can cause the below issues: –\n\n1. **Incompatible with most of the Python libraries used in Machine Learning:-** Yes, you read it right. While using the libraries for ML(the most common is skLearn), they don’t have a provision to automatically handle these missing data and can lead to errors.\n2. **Distortion in Dataset:-** A huge amount of missing data can cause distortions in the variable distribution i.e it can increase or decrease the value of a particular category in the dataset.\n3. **Affects the Final Model:-** the missing data can cause a bias in the dataset and can lead to a faulty analysis by the model.\n\nImportantly “We want to restore the complete dataset”. This is mostly in the case when we do not want to lose any data from our dataset as all of it is important, If dataset size is not very big, and removing some part of it can have a significant impact on the final model.\n\n## **Types**\n\n![imp2](https://1.bp.blogspot.com/-gF6uDeQi8ck/X0KeUfSFrgI/AAAAAAAAKUQ/PwsXhRPAjDsiWQlwnQn7085QAkU6IJJ_wCLcBGAsYHQ/s1357/data%2Bvariables.PNG)","metadata":{}},{"cell_type":"markdown","source":"# **Top 6 Techniques with Code**","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline","metadata":{"execution":{"iopub.status.busy":"2022-07-21T02:22:46.836449Z","iopub.execute_input":"2022-07-21T02:22:46.837102Z","iopub.status.idle":"2022-07-21T02:22:47.374174Z","shell.execute_reply.started":"2022-07-21T02:22:46.837064Z","shell.execute_reply":"2022-07-21T02:22:47.373234Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data=pd.read_csv('/kaggle/input/tabular-playground-series-jun-2022/data.csv',index_col='row_id')\ndata","metadata":{"execution":{"iopub.status.busy":"2022-07-21T02:22:47.375497Z","iopub.execute_input":"2022-07-21T02:22:47.376894Z","iopub.status.idle":"2022-07-21T02:23:04.067116Z","shell.execute_reply.started":"2022-07-21T02:22:47.376858Z","shell.execute_reply":"2022-07-21T02:23:04.066127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Null Analysis**","metadata":{}},{"cell_type":"code","source":"data.info()","metadata":{"execution":{"iopub.status.busy":"2022-07-21T02:23:04.068580Z","iopub.execute_input":"2022-07-21T02:23:04.068939Z","iopub.status.idle":"2022-07-21T02:23:04.223372Z","shell.execute_reply.started":"2022-07-21T02:23:04.068892Z","shell.execute_reply":"2022-07-21T02:23:04.222417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Number of Missing Values in Each of the Columns**","metadata":{}},{"cell_type":"code","source":"n_null=data.isnull().sum()\nn_null","metadata":{"execution":{"iopub.status.busy":"2022-07-21T02:23:04.226618Z","iopub.execute_input":"2022-07-21T02:23:04.226945Z","iopub.status.idle":"2022-07-21T02:23:04.355583Z","shell.execute_reply.started":"2022-07-21T02:23:04.226917Z","shell.execute_reply":"2022-07-21T02:23:04.354589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Visualize**","metadata":{}},{"cell_type":"code","source":"import missingno as msno\nmsno.matrix(data.sample(500),color=(1, 0.38, 0.27))","metadata":{"execution":{"iopub.status.busy":"2022-07-21T02:55:58.856196Z","iopub.execute_input":"2022-07-21T02:55:58.856554Z","iopub.status.idle":"2022-07-21T02:55:59.272175Z","shell.execute_reply.started":"2022-07-21T02:55:58.856522Z","shell.execute_reply":"2022-07-21T02:55:59.271286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**The white bars represent Null Values**","metadata":{}},{"cell_type":"markdown","source":"## **Columns with NULL Values**","metadata":{}},{"cell_type":"code","source":"null=data.columns[data.isnull().any()]\nnull","metadata":{"execution":{"iopub.status.busy":"2022-07-21T02:23:04.803557Z","iopub.execute_input":"2022-07-21T02:23:04.804000Z","iopub.status.idle":"2022-07-21T02:23:04.860098Z","shell.execute_reply.started":"2022-07-21T02:23:04.803961Z","shell.execute_reply":"2022-07-21T02:23:04.859021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Distribution of Features**","metadata":{}},{"cell_type":"code","source":"plt.rcParams[\"figure.figsize\"] = (25,20)\n\nfig, ax = plt.subplots(9,9)\n\n# adds title to figure            \nfig.text(0.35,1,'Distribution of Features',{'size': 24})\n\ni = 0\nj = 0\nfor col in data.columns: #iterate thru all dataset columns\n    if col not in ['row_id']: \n        ax[j, i].hist(data[col], bins=100) #plots histogram on subplot [j, i]\n        ax[j, i].set_title(col, #adds a title to the subplot\n                           {'size': '14', 'weight': 'bold'}) \n        if i == 8: #if we reach the last column of the row, drop down a row and reset\n            i = 0\n            j += 1\n        else: #if not at the end of the row, move over a column\n            i += 1\n\nplt.rcParams.update({'axes.facecolor':'lightgreen'})\nplt.figure(facecolor='red') \nplt.show() ","metadata":{"execution":{"iopub.status.busy":"2022-07-21T02:23:04.861510Z","iopub.execute_input":"2022-07-21T02:23:04.861942Z","iopub.status.idle":"2022-07-21T02:23:25.166615Z","shell.execute_reply.started":"2022-07-21T02:23:04.861902Z","shell.execute_reply":"2022-07-21T02:23:25.165658Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Correlation-Heatmap**","metadata":{}},{"cell_type":"code","source":"import missingno as msno\nmsno.heatmap(data,cmap=\"RdYlGn\")","metadata":{"execution":{"iopub.status.busy":"2022-07-21T02:57:18.292342Z","iopub.execute_input":"2022-07-21T02:57:18.292714Z","iopub.status.idle":"2022-07-21T02:57:28.911134Z","shell.execute_reply.started":"2022-07-21T02:57:18.292682Z","shell.execute_reply":"2022-07-21T02:57:28.910237Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Feature-Wise NAN Value Distribution**","metadata":{}},{"cell_type":"code","source":"import seaborn as sns\nplt.rcParams.update({'axes.facecolor':'black'})\nplot = sns.histplot(data=n_null, bins=10, stat=\"percent\")\nplot.set_xlabel('The number of NaN values')","metadata":{"execution":{"iopub.status.busy":"2022-07-21T02:23:36.073633Z","iopub.execute_input":"2022-07-21T02:23:36.074098Z","iopub.status.idle":"2022-07-21T02:23:36.335299Z","shell.execute_reply.started":"2022-07-21T02:23:36.074056Z","shell.execute_reply":"2022-07-21T02:23:36.334359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Dendogram**","metadata":{}},{"cell_type":"code","source":"import missingno as msno\nmsno.dendrogram(data, figsize=(20,15), fontsize=12);","metadata":{"execution":{"iopub.status.busy":"2022-07-21T03:00:56.189771Z","iopub.execute_input":"2022-07-21T03:00:56.190144Z","iopub.status.idle":"2022-07-21T03:01:00.367983Z","shell.execute_reply.started":"2022-07-21T03:00:56.190114Z","shell.execute_reply":"2022-07-21T03:01:00.366998Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Top 6 Popular Imputers**\n\n***These are some popular techniques. There are many other third-party imputers (and many are comming up)***","metadata":{}},{"cell_type":"markdown","source":"## **1. Constant Imputer:**\n\nYou may replace the missing value with a contant. The constant can be a numeric constant or even a string constant.\n\n![fillna](https://www.w3resource.com/w3r_images/pandas-series-fillna-image-2.svg)\n\nThat’s an easy one. You just let the algorithm handle the missing data. Some algorithms can factor in the missing values and learn the best imputation values for the missing data based on the training loss reduction (ie. XGBoost). Some others have the option to just ignore them (ie. LightGBM — use_missing=false). However, other algorithms will panic and throw an error complaining about the missing values (ie. Scikit learn — LinearRegression). In that case, you will need to handle the missing data and clean it before feeding it to the algorithm.","metadata":{}},{"cell_type":"code","source":"# Fill with Numeric\ndata_num=data.fillna(0)\ndata_num.isnull().sum().sum()","metadata":{"execution":{"iopub.status.busy":"2022-07-21T02:23:36.336741Z","iopub.execute_input":"2022-07-21T02:23:36.337636Z","iopub.status.idle":"2022-07-21T02:23:36.798235Z","shell.execute_reply.started":"2022-07-21T02:23:36.337596Z","shell.execute_reply":"2022-07-21T02:23:36.797349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Fill with String\ndata_str=data.fillna('kaggle')\ndata_str.isnull().sum().sum()","metadata":{"execution":{"iopub.status.busy":"2022-07-21T02:23:36.799888Z","iopub.execute_input":"2022-07-21T02:23:36.800265Z","iopub.status.idle":"2022-07-21T02:23:43.955613Z","shell.execute_reply.started":"2022-07-21T02:23:36.800228Z","shell.execute_reply":"2022-07-21T02:23:43.954545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **2. Numerical : Mean/Median Imputer:**\n\nThis works by calculating the mean/median of the non-missing values in a column and then replacing the missing values within each column separately and independently from the others. It can only be used with numeric data.\n\n![mimp](https://miro.medium.com/max/1400/1*MiJ_HpTbZECYjjF1qepNNQ.png)\n\n#### **Pros:**\n* Easy and fast.\n* Works well with small numerical datasets.\n\n#### **Cons:**\n* Doesn’t factor the correlations between features. It only works on the column level.\n* Will give poor results on encoded categorical features (do NOT use it on categorical features).\n* Not very accurate.\n* Doesn’t account for the uncertainty in the imputations.","metadata":{}},{"cell_type":"code","source":"# Impute with Mean\nfrom sklearn.impute import SimpleImputer\n\nimp=SimpleImputer(strategy='mean')\n\ndata_mean=imp.fit_transform(data)\ndata_mean","metadata":{"execution":{"iopub.status.busy":"2022-07-21T02:23:43.957473Z","iopub.execute_input":"2022-07-21T02:23:43.957845Z","iopub.status.idle":"2022-07-21T02:23:46.166351Z","shell.execute_reply.started":"2022-07-21T02:23:43.957809Z","shell.execute_reply":"2022-07-21T02:23:46.165334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **3. Numerical & Categorical : Most Frequent Imputer:**\n\nMost Frequent is another statistical strategy to impute missing values and YES!! It works with categorical features (strings or numerical representations) by replacing missing data with the most frequent values within each column.\n\n![mimp](https://miro.medium.com/max/1208/1*bgzrL1JLxno2igi4M20tgA.png)\n\n#### **Pros:**\n* Works well with categorical features.\n\n#### **Cons:**\n* It also doesn’t factor the correlations between features.\n* It can introduce bias in the data.\n","metadata":{}},{"cell_type":"code","source":"# Impute with Most Frequent (also for Categorical Features)\nfrom sklearn.impute import SimpleImputer\n\nimp=SimpleImputer(strategy='most_frequent')\n\ndata_mode=imp.fit_transform(data)\ndata_mode","metadata":{"execution":{"iopub.status.busy":"2022-07-21T02:23:46.170615Z","iopub.execute_input":"2022-07-21T02:23:46.170914Z","iopub.status.idle":"2022-07-21T02:23:53.987585Z","shell.execute_reply.started":"2022-07-21T02:23:46.170880Z","shell.execute_reply":"2022-07-21T02:23:53.986463Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **4-k-NN Imputer:**\n\nThe k nearest neighbours is an algorithm that is used for simple classification. The algorithm uses ‘feature similarity’ to predict the values of any new data points. This means that the new point is assigned a value based on how closely it resembles the points in the training set. This can be very useful in making predictions about the missing values by finding the k’s closest neighbours to the observation with missing data and then imputing them based on the non-missing values in the neighbourhood\n\n![kimp](https://miro.medium.com/max/1280/1*b9BXv0uAkbSAn8MJIa4-_Q.gif)\n\n#### **Pros:**\n* Can be much more accurate than the mean, median or most frequent imputation methods (It depends on the dataset).\n\n#### **Cons:**\n* Computationally expensive. KNN works by storing the whole training dataset in memory.\n* K-NN is quite sensitive to outliers in the data (unlike SVM)","metadata":{}},{"cell_type":"code","source":"# from sklearn.impute import KNNImputer\n\n# knn = KNNImputer(n_neighbors=5)\n\n# data_knn=knn.fit_transform(data)\n\n# data_knn    ","metadata":{"execution":{"iopub.status.busy":"2022-07-21T02:23:53.989318Z","iopub.execute_input":"2022-07-21T02:23:53.990168Z","iopub.status.idle":"2022-07-21T02:23:53.994741Z","shell.execute_reply.started":"2022-07-21T02:23:53.990121Z","shell.execute_reply":"2022-07-21T02:23:53.993744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**kNN Imputers are conputationally very  Resource-Consuming. Here it was causing this notebook to crash & restart. So commented.Howeveeer you can reference the code**","metadata":{}},{"cell_type":"markdown","source":"## **5-Indicator Imputer**\n\nImputation is the standard approach, and it usually works well. However, imputed values may be systematically above or below their actual values (which weren't collected in the dataset). Or rows with missing values may be unique in some other way. In that case, your model would make better predictions by considering which values were originally missing.\n\n![indi](https://www.researchgate.net/profile/Antonio-Pereira-Barata-2/publication/338597970/figure/fig1/AS:849726220029952@1579601926055/Imputation-through-missing-indicator-The-table-to-the-left-represents-a-4-rows-slice-of.png)","metadata":{}},{"cell_type":"code","source":"from sklearn.impute import SimpleImputer\n\nimp=SimpleImputer(strategy='median',add_indicator=True)\n\ndata_ind=imp.fit_transform(data)\ndata_ind=pd.DataFrame(data_ind)\ndata_ind","metadata":{"execution":{"iopub.status.busy":"2022-07-21T02:23:53.996580Z","iopub.execute_input":"2022-07-21T02:23:53.997583Z","iopub.status.idle":"2022-07-21T02:24:10.768273Z","shell.execute_reply.started":"2022-07-21T02:23:53.997535Z","shell.execute_reply":"2022-07-21T02:24:10.766967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **6-Iterative Imputer**\n\nUseful only when working with multivariate data, the IterativeImputer in scikit-learn utilizes the data available in other features in order to estimate the missing values being imputed. \n\n***It does so through an iterated round-robin fashion: at each step, a feature column is designated as output y and the other feature columns are treated as inputs X. A regressor is fit on (X, y) for known y. Then, the regressor is used to predict the missing values of y. This is done for each feature in an iterative fashion, and then is repeated for max_iter imputation rounds. The results of the final imputation round are returned***\n\n![ii](https://amueller.github.io/COMS4995-s18/slides/aml-08-021218-imputation-feature-selection/images/img_4.png)","metadata":{}},{"cell_type":"code","source":"# # This estimator is still experimental for now: the predictions and the API might change without any deprecation cycle\n\n# from sklearn.experimental import enable_iterative_imputer\n# from sklearn.impute import IterativeImputer\n# from sklearn.linear_model import LinearRegression\n\n# imp=IterativeImputer(estimator=LinearRegression(),missing_values=np.nan)\n\n# data_ii=imp.fit_transform(data)\n# data_ii","metadata":{"execution":{"iopub.status.busy":"2022-07-21T02:36:41.276049Z","iopub.execute_input":"2022-07-21T02:36:41.276906Z","iopub.status.idle":"2022-07-21T02:36:41.281554Z","shell.execute_reply.started":"2022-07-21T02:36:41.276862Z","shell.execute_reply":"2022-07-21T02:36:41.280346Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Taking too long, Commented!","metadata":{}},{"cell_type":"markdown","source":"# Suggestions:-\n* Kaggle - https://www.kaggle.com/pythonkumar\n* GitHub - https://github.com/KumarPython​\n* Twitter - https://twitter.com/KumarPython\n* LinkedIn - https://www.linkedin.com/in/kumarpython/","metadata":{}},{"cell_type":"markdown","source":"#  **Submission**","metadata":{}},{"cell_type":"code","source":"sub=pd.read_csv('../input/tabular-playground-series-jun-2022/sample_submission.csv')\n\nsplit1=sub['row-col'].str.split(pat=\"-\",expand=True)\nsplit2=split1.iloc[:,1].str.split(pat=\"_\",expand=True)\n\nrow=split1.iloc[:,0].astype('int64')\ncol=split2.iloc[:,1].astype('int64')\n\nval=[data_ind.iloc[row[i],col[i]] for i in range(len(row))]\nsub['value']=val   \n\nsub.to_csv('submission.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-21T02:30:51.893010Z","iopub.execute_input":"2022-07-21T02:30:51.893370Z","iopub.status.idle":"2022-07-21T02:31:35.765532Z","shell.execute_reply.started":"2022-07-21T02:30:51.893339Z","shell.execute_reply":"2022-07-21T02:31:35.764455Z"},"trusted":true},"execution_count":null,"outputs":[]}]}