{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":81933,"databundleVersionId":9643020,"sourceType":"competition"},{"sourceId":9548329,"sourceType":"datasetVersion","datasetId":5817528}],"dockerImageVersionId":30786,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# REPLACING OUTLIER for Physical-Weight feature  \nIn this notebook we will replace outliers for the feature Physical-Weight ","metadata":{}},{"cell_type":"markdown","source":"## Data Importation","metadata":{}},{"cell_type":"code","source":"import pandas as pd \ndf= pd.read_csv('/kaggle/input/child-mind-institute-problematic-internet-use/train.csv')","metadata":{"execution":{"iopub.status.busy":"2024-10-04T20:02:41.839948Z","iopub.execute_input":"2024-10-04T20:02:41.840890Z","iopub.status.idle":"2024-10-04T20:02:41.894585Z","shell.execute_reply.started":"2024-10-04T20:02:41.840838Z","shell.execute_reply":"2024-10-04T20:02:41.893714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data discovery ","metadata":{}},{"cell_type":"code","source":"df.describe()","metadata":{"execution":{"iopub.status.busy":"2024-10-04T20:02:41.896273Z","iopub.execute_input":"2024-10-04T20:02:41.896660Z","iopub.status.idle":"2024-10-04T20:02:42.062027Z","shell.execute_reply.started":"2024-10-04T20:02:41.896613Z","shell.execute_reply":"2024-10-04T20:02:42.061007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n# Assuming df is your DataFrame\nplt.figure(figsize=(8,6))\nplt.boxplot(df['Physical-Weight'].dropna())  \nplt.title('Box Plot of BIA-BIA_LDM')\nplt.ylabel('BIA-BIA_LDM')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-10-04T20:02:42.214489Z","iopub.execute_input":"2024-10-04T20:02:42.215479Z","iopub.status.idle":"2024-10-04T20:02:42.458114Z","shell.execute_reply.started":"2024-10-04T20:02:42.215437Z","shell.execute_reply":"2024-10-04T20:02:42.457097Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Feature selection ","metadata":{}},{"cell_type":"markdown","source":"we have selected these features cuz they are the most related features for a weight :'Basic_Demos-Age', 'Basic_Demos-Sex','Physical-Height' ","metadata":{}},{"cell_type":"code","source":"Train= df[['Basic_Demos-Age', 'Basic_Demos-Sex','Physical-Height']]\nTrain_columns=Train.columns","metadata":{"execution":{"iopub.status.busy":"2024-10-04T20:02:42.538406Z","iopub.execute_input":"2024-10-04T20:02:42.539377Z","iopub.status.idle":"2024-10-04T20:02:42.545040Z","shell.execute_reply.started":"2024-10-04T20:02:42.539321Z","shell.execute_reply":"2024-10-04T20:02:42.543998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Droping missing values ","metadata":{}},{"cell_type":"markdown","source":"deleting the features that doesn't exist in the test data ","metadata":{}},{"cell_type":"code","source":"df_test=  pd.read_csv(\"/kaggle/input/child-mind-institute-problematic-internet-use/test.csv\")\ntrain_cols = df.columns.tolist()  \ntest_cols = df_test.columns.tolist()\ndrop_cols = list(set(train_cols) - set(test_cols))  \ndrop_cols.remove('sii')\ndf1=df.drop(drop_cols ,axis=1 ,inplace=False)","metadata":{"execution":{"iopub.status.busy":"2024-10-04T20:02:42.618278Z","iopub.execute_input":"2024-10-04T20:02:42.619266Z","iopub.status.idle":"2024-10-04T20:02:42.632725Z","shell.execute_reply.started":"2024-10-04T20:02:42.619218Z","shell.execute_reply":"2024-10-04T20:02:42.631645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"deleting nans in the target sii ","metadata":{}},{"cell_type":"code","source":"df=df1.dropna(subset=['sii'])","metadata":{"execution":{"iopub.status.busy":"2024-10-04T20:02:42.855703Z","iopub.execute_input":"2024-10-04T20:02:42.856136Z","iopub.status.idle":"2024-10-04T20:02:42.864757Z","shell.execute_reply.started":"2024-10-04T20:02:42.856097Z","shell.execute_reply":"2024-10-04T20:02:42.863600Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"deleting the columns where the nan is upper than 20%","metadata":{}},{"cell_type":"code","source":"half_missing = [val for val in df.columns[df.isnull().sum()>len(df)/5]]\ndf2=[i for i in df if i not in half_missing]\ndf=df[df2]\ndf=df.dropna()","metadata":{"execution":{"iopub.status.busy":"2024-10-04T20:02:42.869435Z","iopub.execute_input":"2024-10-04T20:02:42.869778Z","iopub.status.idle":"2024-10-04T20:02:42.885204Z","shell.execute_reply.started":"2024-10-04T20:02:42.869744Z","shell.execute_reply":"2024-10-04T20:02:42.884141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Outlier Detection ","metadata":{}},{"cell_type":"markdown","source":"The box plot shows that aprroxiamtely that  the values grater 150 are outliers like the previous plot shows","metadata":{}},{"cell_type":"code","source":"outlier_threshold = 150\n# Mask for values greater than 150 (outliers)\noutlier_mask = df['Physical-Weight'] > outlier_threshold","metadata":{"execution":{"iopub.status.busy":"2024-10-04T20:02:42.887143Z","iopub.execute_input":"2024-10-04T20:02:42.887519Z","iopub.status.idle":"2024-10-04T20:02:42.892706Z","shell.execute_reply.started":"2024-10-04T20:02:42.887483Z","shell.execute_reply":"2024-10-04T20:02:42.891679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Selecting the Data Frame  when the 'Physical-Weight' has outliers ","metadata":{}},{"cell_type":"code","source":"from sklearn.linear_model import LinearRegression\nnon_outliers = df[~outlier_mask]","metadata":{"execution":{"iopub.status.busy":"2024-10-04T20:02:42.961817Z","iopub.execute_input":"2024-10-04T20:02:42.962566Z","iopub.status.idle":"2024-10-04T20:02:42.969502Z","shell.execute_reply.started":"2024-10-04T20:02:42.962511Z","shell.execute_reply":"2024-10-04T20:02:42.968360Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model Creation\n* initialising the features and the target\n* training the model on the lines where there no outliers \n* predict the new values to replace the outliers ","metadata":{}},{"cell_type":"code","source":"X_train = non_outliers[Train_columns]\ny_train = non_outliers['Physical-Weight']\n# Train the Linear Regression model\nregressor = LinearRegression()\nregressor.fit(X_train, y_train)\nX_outliers = df[outlier_mask][Train_columns]\npredicted_outliers = regressor.predict(X_outliers)","metadata":{"execution":{"iopub.status.busy":"2024-10-04T20:02:43.131182Z","iopub.execute_input":"2024-10-04T20:02:43.131963Z","iopub.status.idle":"2024-10-04T20:02:43.145929Z","shell.execute_reply.started":"2024-10-04T20:02:43.131919Z","shell.execute_reply":"2024-10-04T20:02:43.144742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Outlier Replacement","metadata":{}},{"cell_type":"code","source":"df.loc[outlier_mask, 'Physical-Weight'] = predicted_outliers","metadata":{"execution":{"iopub.status.busy":"2024-10-04T20:02:43.164513Z","iopub.execute_input":"2024-10-04T20:02:43.164848Z","iopub.status.idle":"2024-10-04T20:02:43.170551Z","shell.execute_reply.started":"2024-10-04T20:02:43.164814Z","shell.execute_reply":"2024-10-04T20:02:43.169353Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Result Visaulization","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(8,6))\nplt.boxplot(df['Physical-Weight'])  \nplt.title('Box Plot of BIA-BIA_LDM')\nplt.ylabel('BIA-BIA_LDM')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-10-04T20:02:43.172776Z","iopub.execute_input":"2024-10-04T20:02:43.173726Z","iopub.status.idle":"2024-10-04T20:02:43.433024Z","shell.execute_reply.started":"2024-10-04T20:02:43.173689Z","shell.execute_reply":"2024-10-04T20:02:43.431945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Conclusion\nThe primary goal of this notebook was to address and replace outliers in the Physical-Weight feature, so the focus was not on extensive data cleaning or pre-processing. I hope this notebook proves helpful to you all. I look forward to your feedback and comments.\n\nThank you for taking the time to read through it!","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}