{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":81933,"databundleVersionId":9643020,"sourceType":"competition"}],"dockerImageVersionId":30786,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"* [Section 1](#Section-one)\n","metadata":{}},{"cell_type":"markdown","source":"**Aim:** predict the Severity Impairment Index (sii) - measures the level of problematic internet use among children and adolescents, based on physical activity data and other features.\n\nTarget Variable (sii) is defined as:\n\n- 0: None (PCIAT-PCIAT_Total from 0 to 30)\n- 1: Mild (PCIAT-PCIAT_Total from 31 to 49)\n- 2: Moderate (PCIAT-PCIAT_Total from 50 to 79)\n- 3: Severe (PCIAT-PCIAT_Total 80 and more\n\nNote:\n- sii is derived from PCIAT-PCIAT_Total, the sum of scores from the Parent-Child Internet Addiction Test (PCIAT: 20 questions, scored 0-5).\n- The test dataset doesn't have any PCIAT columns (otherwise predictions would be trivial).\n\n**Insight:**\n\n- We should focus on predicting the target from all other features except the PCIAT results.\n\n- We know the target only for two thirds of the samples. The samples without target can perhaps be used for semi-supervised learning.\n\n- We can directly predict sii (this is the value we have to submit), or we can predict PCIAT-PCIAT_Total and then transform this prediction to a sii prediction for submission. As PCIAT-PCIAT_Total is more granular and informative than sii, training to predict PCIAT-PCIAT_Total has the potential to produce a better model.\n","metadata":{}},{"cell_type":"markdown","source":"# Data Preview","metadata":{}},{"cell_type":"code","source":"import pandas as pd\npath = '/kaggle/input/child-mind-institute-problematic-internet-use/train.csv'\ntrain = pd.read_csv(path)\npath = '/kaggle/input/child-mind-institute-problematic-internet-use/test.csv'\ntest = pd.read_csv(path)\n\npath= '/kaggle/input/child-mind-institute-problematic-internet-use/data_dictionary.csv'\ndata_dict = pd.read_csv(path)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T07:54:30.781268Z","iopub.execute_input":"2024-12-02T07:54:30.781801Z","iopub.status.idle":"2024-12-02T07:54:32.528505Z","shell.execute_reply.started":"2024-12-02T07:54:30.78173Z","shell.execute_reply":"2024-12-02T07:54:32.527062Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T07:54:32.530793Z","iopub.execute_input":"2024-12-02T07:54:32.531273Z","iopub.status.idle":"2024-12-02T07:54:32.589334Z","shell.execute_reply.started":"2024-12-02T07:54:32.531222Z","shell.execute_reply":"2024-12-02T07:54:32.588059Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train.info()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T07:54:32.590668Z","iopub.execute_input":"2024-12-02T07:54:32.591083Z","iopub.status.idle":"2024-12-02T07:54:32.628238Z","shell.execute_reply.started":"2024-12-02T07:54:32.591033Z","shell.execute_reply":"2024-12-02T07:54:32.627002Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T07:54:32.630711Z","iopub.execute_input":"2024-12-02T07:54:32.631474Z","iopub.status.idle":"2024-12-02T07:54:32.65911Z","shell.execute_reply.started":"2024-12-02T07:54:32.631424Z","shell.execute_reply":"2024-12-02T07:54:32.657995Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data_dict.head(80)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T07:54:32.660374Z","iopub.execute_input":"2024-12-02T07:54:32.660676Z","iopub.status.idle":"2024-12-02T07:54:32.685695Z","shell.execute_reply.started":"2024-12-02T07:54:32.660648Z","shell.execute_reply":"2024-12-02T07:54:32.684223Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"groups = data_dict.groupby('Instrument')['Field'].apply(list).to_dict()\n\nfor instrument, features in groups.items():\n    print(f\"{instrument}: {features}\\n\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T07:54:32.687207Z","iopub.execute_input":"2024-12-02T07:54:32.687633Z","iopub.status.idle":"2024-12-02T07:54:32.704102Z","shell.execute_reply.started":"2024-12-02T07:54:32.687584Z","shell.execute_reply":"2024-12-02T07:54:32.7025Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Missing value","metadata":{}},{"cell_type":"markdown","source":"All columns have a substantial proportion of missing values, except id (not surprisingly) and the three basic demographic columns for sex, age and season of enrollment. Even the target sii has missing values:","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\nfrom matplotlib.ticker import PercentFormatter\n\nmissing_count = train.isnull().sum().sort_values(ascending=False)\nmissing_ratio = missing_count / len(train)\n\nplt.figure(figsize=(6, 15))\nplt.title('Missing values over the whole training dataset')\nplt.barh(np.arange(len(missing_count)), missing_ratio, color='coral', label='missing')\nplt.barh(np.arange(len(missing_count)), \n         1 - missing_ratio,\n         left=missing_ratio,\n         color='darkseagreen', label='available')\nplt.yticks(np.arange(len(missing_count)), missing_count.index)\nplt.gca().xaxis.set_major_formatter(PercentFormatter(xmax=1, decimals=0))\nplt.xlim(0, 1)\nplt.legend()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T07:54:32.705948Z","iopub.execute_input":"2024-12-02T07:54:32.707274Z","iopub.status.idle":"2024-12-02T07:54:33.974436Z","shell.execute_reply.started":"2024-12-02T07:54:32.707216Z","shell.execute_reply":"2024-12-02T07:54:33.973221Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Lọc các mẫu có giá trị 'sii' không rỗng\nsupervised_usable = train[train['sii'].notnull()]\n\n# Đếm số lượng giá trị bị thiếu\nmissing_count = supervised_usable.isnull().sum().sort_values(ascending=False)\nmissing_ratio = missing_count / len(supervised_usable)\n\nplt.figure(figsize=(6, 15))\nplt.title(f'Missing values over the {len(supervised_usable)} samples which have a target')\nplt.barh(np.arange(len(missing_count)), \n         missing_ratio, \n         color='coral', \n         label='missing')\nplt.barh(np.arange(len(missing_count)), \n         1 - missing_ratio,\n         left=missing_ratio,\n         color='darkseagreen', \n         label='available')\nplt.yticks(np.arange(len(missing_count)), missing_count.index)\nplt.gca().xaxis.set_major_formatter(PercentFormatter(xmax=1, decimals=0))\nplt.xlim(0, 1)\nplt.legend()\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T07:54:33.975916Z","iopub.execute_input":"2024-12-02T07:54:33.976365Z","iopub.status.idle":"2024-12-02T07:54:35.336013Z","shell.execute_reply.started":"2024-12-02T07:54:33.976319Z","shell.execute_reply":"2024-12-02T07:54:35.334778Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Features EAD by Groups","metadata":{}},{"cell_type":"markdown","source":"## Demographics","metadata":{}},{"cell_type":"code","source":"import seaborn as sns\n\nfig, axes = plt.subplots(1, 2, figsize=(12, 5))\n\n# Season of Enrollment\nseason_counts = train['Basic_Demos-Enroll_Season'].value_counts(dropna=False)\n\naxes[0].pie(\n    season_counts, labels=season_counts.index,\n    autopct='%1.1f%%', startangle=90,\n    colors=sns.color_palette(\"Set3\")\n)\naxes[0].set_title('Season of Enrollment')\naxes[0].axis('equal')\n\n# Age Distribution by Sex\nsns.histplot(\n    data=train, x='Basic_Demos-Age',\n    hue='Basic_Demos-Sex', multiple='dodge',\n    palette=\"Set2\", bins=20, ax=axes[1]\n)\naxes[1].set_title('Age Distribution by Sex')\naxes[1].set_xlabel('Age')\naxes[1].set_ylabel('Count')\n\nplt.tight_layout()\nplt.show()\nprint('0=Male, 1=Female')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T07:54:35.337641Z","iopub.execute_input":"2024-12-02T07:54:35.338017Z","iopub.status.idle":"2024-12-02T07:54:36.97766Z","shell.execute_reply.started":"2024-12-02T07:54:35.337982Z","shell.execute_reply":"2024-12-02T07:54:36.976566Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"calculate_stats(train, 'Basic_Demos-Age')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T07:54:36.980614Z","iopub.execute_input":"2024-12-02T07:54:36.981124Z","iopub.status.idle":"2024-12-02T07:54:37.335148Z","shell.execute_reply.started":"2024-12-02T07:54:36.981088Z","shell.execute_reply":"2024-12-02T07:54:37.333541Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Physical Measures","metadata":{}},{"cell_type":"code","source":"train_cols = set(train.columns)\ntest_cols = set(test.columns)\ncolumns_not_in_test = sorted(list(train_cols - test_cols))\nprint('Columns missing in test:')\nprint(columns_not_in_test)\ndata_dict[data_dict['Field'].isin(columns_not_in_test)]\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T07:54:37.336108Z","iopub.status.idle":"2024-12-02T07:54:37.33647Z","shell.execute_reply.started":"2024-12-02T07:54:37.336301Z","shell.execute_reply":"2024-12-02T07:54:37.336319Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"# 2.Processing data","metadata":{}},{"cell_type":"code","source":"!pip install --upgrade category_encoders","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T07:54:37.33842Z","iopub.status.idle":"2024-12-02T07:54:37.338817Z","shell.execute_reply.started":"2024-12-02T07:54:37.338636Z","shell.execute_reply":"2024-12-02T07:54:37.338655Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"numerical_cols = train.select_dtypes(exclude='object').columns\ncategorical_cols = train.select_dtypes(include='object').columns\ntrain = train.dropna(subset=['sii'])\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T07:54:37.340136Z","iopub.status.idle":"2024-12-02T07:54:37.340551Z","shell.execute_reply.started":"2024-12-02T07:54:37.340349Z","shell.execute_reply":"2024-12-02T07:54:37.340369Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train[categorical_cols].head(10)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T07:54:37.341541Z","iopub.status.idle":"2024-12-02T07:54:37.34193Z","shell.execute_reply.started":"2024-12-02T07:54:37.341725Z","shell.execute_reply":"2024-12-02T07:54:37.341744Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import category_encoders as ce\n\nencoder = ce.TargetEncoder(cols=categorical_cols, smoothing=1.0)\ntrain[categorical_cols] = encoder.fit_transform(train[categorical_cols], train['sii'])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T07:54:37.34352Z","iopub.status.idle":"2024-12-02T07:54:37.34407Z","shell.execute_reply.started":"2024-12-02T07:54:37.343779Z","shell.execute_reply":"2024-12-02T07:54:37.343807Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train[categorical_cols].head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T07:54:37.345674Z","iopub.status.idle":"2024-12-02T07:54:37.346252Z","shell.execute_reply.started":"2024-12-02T07:54:37.345983Z","shell.execute_reply":"2024-12-02T07:54:37.346011Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 3.Univariate analysis","metadata":{}},{"cell_type":"code","source":"train.describe()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T07:54:37.347805Z","iopub.status.idle":"2024-12-02T07:54:37.348363Z","shell.execute_reply.started":"2024-12-02T07:54:37.348092Z","shell.execute_reply":"2024-12-02T07:54:37.34812Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train.hist(figsize=30,40), xrot=40);","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-12-02T07:54:37.350211Z","iopub.status.idle":"2024-12-02T07:54:37.350569Z","shell.execute_reply.started":"2024-12-02T07:54:37.350394Z","shell.execute_reply":"2024-12-02T07:54:37.350411Z"}},"outputs":[],"execution_count":null}]}