\n\n* We now have 16 selected features based on a) correlation with the target and b) relatively few missing values.\n* The idea is to create a robust model that focuses on key signals in the data and reduces some of the excess noise from large numbers of features.\n* Some of the min and max values appear to be impossible (such as a minimum weight of zero) or very unlikely.","metadata":{"execution":{"iopub.status.busy":"2024-10-05T13:59:54.552101Z","iopub.execute_input":"2024-10-05T13:59:54.552525Z","iopub.status.idle":"2024-10-05T13:59:54.560829Z","shell.execute_reply.started":"2024-10-05T13:59:54.552484Z","shell.execute_reply":"2024-10-05T13:59:54.559313Z"}}},{"cell_type":"code","source":"describe = train[selection].describe().T\ndescribe = describe[['min','max']].sort_index()\ndescribe.style.background_gradient(cmap='YlOrRd')","metadata":{"execution":{"iopub.status.busy":"2025-01-07T10:45:27.114089Z","iopub.execute_input":"2025-01-07T10:45:27.114451Z","iopub.status.idle":"2025-01-07T10:45:27.175094Z","shell.execute_reply.started":"2025-01-07T10:45:27.114417Z","shell.execute_reply":"2025-01-07T10:45:27.173791Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train[selection].hist(figsize=(10,10), grid = True, color = 'chocolate')\nplt.tight_layout()","metadata":{"execution":{"iopub.status.busy":"2025-01-07T10:45:27.176207Z","iopub.execute_input":"2025-01-07T10:45:27.176563Z","iopub.status.idle":"2025-01-07T10:45:31.718285Z","shell.execute_reply.started":"2025-01-07T10:45:27.176528Z","shell.execute_reply":"2025-01-07T10:45:31.716988Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#
\n\n* In this section I train an XGBoost classification model as a benchmark and use sii as the target.\n* The model was tuned offline using Optuna to get the hyperparameters.\n* This gives a CV score of **.375**. ","metadata":{"execution":{"iopub.status.busy":"2024-10-01T11:23:08.558733Z","iopub.execute_input":"2024-10-01T11:23:08.559238Z","iopub.status.idle":"2024-10-01T11:23:08.568315Z","shell.execute_reply.started":"2024-10-01T11:23:08.559196Z","shell.execute_reply":"2024-10-01T11:23:08.566315Z"}}},{"cell_type":"code","source":"X = train[selection]\ntest = test[selection]\ny = train.sii","metadata":{"execution":{"iopub.status.busy":"2025-01-07T10:45:31.719854Z","iopub.execute_input":"2025-01-07T10:45:31.720461Z","iopub.status.idle":"2025-01-07T10:45:31.732093Z","shell.execute_reply.started":"2025-01-07T10:45:31.720403Z","shell.execute_reply":"2025-01-07T10:45:31.730605Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def quadratic_kappa(y_true, y_pred):\n return cohen_kappa_score(y_true, y_pred, weights='quadratic')\nkappa_scorer = make_scorer(quadratic_kappa)","metadata":{"execution":{"iopub.status.busy":"2025-01-07T10:45:31.733494Z","iopub.execute_input":"2025-01-07T10:45:31.733921Z","iopub.status.idle":"2025-01-07T10:45:31.749989Z","shell.execute_reply.started":"2025-01-07T10:45:31.733872Z","shell.execute_reply":"2025-01-07T10:45:31.748445Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"params = {'max_depth': 3, \n 'n_estimators': 202, \n 'learning_rate': 0.07956777025142073, \n 'subsample': 0.8197358255094112, \n 'colsample_bytree': 0.645036755035947}\nskf = StratifiedKFold(n_splits=10)\nclf = xgb.XGBClassifier(**params)","metadata":{"execution":{"iopub.status.busy":"2025-01-07T10:45:31.751652Z","iopub.execute_input":"2025-01-07T10:45:31.752313Z","iopub.status.idle":"2025-01-07T10:45:31.769389Z","shell.execute_reply.started":"2025-01-07T10:45:31.752247Z","shell.execute_reply":"2025-01-07T10:45:31.768090Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"scores = cross_val_score(clf, X, y, cv=skf, scoring=kappa_scorer)\nprint(\"QWK Scores:\", scores)\nprint(\"Mean QWK Score:\", np.mean(scores))","metadata":{"execution":{"iopub.status.busy":"2025-01-07T10:45:31.771247Z","iopub.execute_input":"2025-01-07T10:45:31.771655Z","iopub.status.idle":"2025-01-07T10:45:36.800216Z","shell.execute_reply.started":"2025-01-07T10:45:31.771606Z","shell.execute_reply":"2025-01-07T10:45:36.799305Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"clf.fit(X,y)\nfeature_imp = pd.Series(clf.feature_importances_,index=X.columns).sort_values(ascending=False)\nfeature_imp","metadata":{"execution":{"iopub.status.busy":"2025-01-07T10:45:36.801239Z","iopub.execute_input":"2025-01-07T10:45:36.801594Z","iopub.status.idle":"2025-01-07T10:45:37.327179Z","shell.execute_reply.started":"2025-01-07T10:45:36.801560Z","shell.execute_reply":"2025-01-07T10:45:37.325837Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.barplot(x=feature_imp, y=feature_imp.index)\nplt.xlabel('Feature Importance Score')\nplt.title(\"Feature Importances\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2025-01-07T10:45:37.332534Z","iopub.execute_input":"2025-01-07T10:45:37.332966Z","iopub.status.idle":"2025-01-07T10:45:37.793885Z","shell.execute_reply.started":"2025-01-07T10:45:37.332925Z","shell.execute_reply":"2025-01-07T10:45:37.792429Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"perm = PermutationImportance(clf, random_state=1).fit(X,y)\neli5.show_weights(perm, feature_names = X.columns.tolist())","metadata":{"execution":{"iopub.status.busy":"2025-01-07T10:45:37.795935Z","iopub.execute_input":"2025-01-07T10:45:37.796438Z","iopub.status.idle":"2025-01-07T10:45:39.537259Z","shell.execute_reply.started":"2025-01-07T10:45:37.796388Z","shell.execute_reply":"2025-01-07T10:45:39.536057Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#
11/ Regression Model
","metadata":{}},{"cell_type":"markdown","source":"
\n\n* In this section I train an XGBoost regression model as a benchmark and use PCIAT-PCIAT_Total as the target.\n* We tweak the quadratic kappa function to convert PCIAT total scores to sii categories, which gives a better cross-validation result.\n* The model was tuned offline using Optuna to get the hyperparameters.\n* I found that adding a scaling factor of 1.25 to the scores gives a better overall QWK score.\n* The effect of this is to reduce the score thresholds from 30, 50, 80 to 24, 40, 64. It's quite possible there are even better values to be found.","metadata":{"execution":{"iopub.status.busy":"2024-10-01T12:35:10.605389Z","iopub.execute_input":"2024-10-01T12:35:10.605904Z","iopub.status.idle":"2024-10-01T12:35:10.613456Z","shell.execute_reply.started":"2024-10-01T12:35:10.605858Z","shell.execute_reply":"2024-10-01T12:35:10.612056Z"}}},{"cell_type":"code","source":"X = train[selection]\ntest = test[selection]\ny = train['PCIAT-PCIAT_Total']","metadata":{"execution":{"iopub.status.busy":"2025-01-07T10:45:39.538461Z","iopub.execute_input":"2025-01-07T10:45:39.538927Z","iopub.status.idle":"2025-01-07T10:45:39.547996Z","shell.execute_reply.started":"2025-01-07T10:45:39.538876Z","shell.execute_reply":"2025-01-07T10:45:39.546767Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def convert(scores):\n scores = np.array(scores)*1.27\n bins = np.zeros_like(scores)\n bins[scores <= 30] = 0\n bins[(scores > 30) & (scores < 50)] = 1\n bins[(scores >= 50) & (scores < 80)] = 2\n bins[scores >= 80] = 3\n return bins","metadata":{"execution":{"iopub.status.busy":"2025-01-07T10:45:39.549659Z","iopub.execute_input":"2025-01-07T10:45:39.550096Z","iopub.status.idle":"2025-01-07T10:45:39.559452Z","shell.execute_reply.started":"2025-01-07T10:45:39.550059Z","shell.execute_reply":"2025-01-07T10:45:39.558408Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def quadratic_kappa(y_true, y_pred):\n y_true_cat = convert(y_true)\n y_pred_cat = convert(y_pred)\n return cohen_kappa_score(y_true_cat, y_pred_cat, weights='quadratic')\n\nkappa_scorer = make_scorer(quadratic_kappa, greater_is_better=True)","metadata":{"execution":{"iopub.status.busy":"2025-01-07T10:45:39.561277Z","iopub.execute_input":"2025-01-07T10:45:39.561669Z","iopub.status.idle":"2025-01-07T10:45:39.577131Z","shell.execute_reply.started":"2025-01-07T10:45:39.561635Z","shell.execute_reply":"2025-01-07T10:45:39.575628Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"params = {'max_depth': 3, \n 'n_estimators': 59, \n 'learning_rate': 0.07327652118259573, \n 'subsample': 0.5968194045365575, \n 'colsample_bytree': 0.9123669348125403}","metadata":{"execution":{"iopub.status.busy":"2025-01-07T10:45:39.578881Z","iopub.execute_input":"2025-01-07T10:45:39.579697Z","iopub.status.idle":"2025-01-07T10:45:39.592892Z","shell.execute_reply.started":"2025-01-07T10:45:39.579640Z","shell.execute_reply":"2025-01-07T10:45:39.591659Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model = xgb.XGBRegressor(**params)","metadata":{"execution":{"iopub.status.busy":"2025-01-07T10:45:39.594372Z","iopub.execute_input":"2025-01-07T10:45:39.594859Z","iopub.status.idle":"2025-01-07T10:45:39.607461Z","shell.execute_reply.started":"2025-01-07T10:45:39.594783Z","shell.execute_reply":"2025-01-07T10:45:39.606393Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"scores = cross_val_score(model, X, y, cv=skf, scoring=kappa_scorer)\nprint(\"QWK Scores:\", scores)\nprint(\"Mean QWK Score:\", np.mean(scores))","metadata":{"execution":{"iopub.status.busy":"2025-01-07T10:45:39.608705Z","iopub.execute_input":"2025-01-07T10:45:39.609052Z","iopub.status.idle":"2025-01-07T10:45:40.367864Z","shell.execute_reply.started":"2025-01-07T10:45:39.609021Z","shell.execute_reply":"2025-01-07T10:45:40.367015Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model.fit(X,y)\nfeature_imp = pd.Series(model.feature_importances_,index=X.columns).sort_values(ascending=False)\nfeature_imp","metadata":{"execution":{"iopub.status.busy":"2025-01-07T10:45:40.368829Z","iopub.execute_input":"2025-01-07T10:45:40.369484Z","iopub.status.idle":"2025-01-07T10:45:40.438301Z","shell.execute_reply.started":"2025-01-07T10:45:40.369451Z","shell.execute_reply":"2025-01-07T10:45:40.437102Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sns.barplot(x=feature_imp, y=feature_imp.index)\nplt.xlabel('Feature Importance Score')\nplt.title(\"Feature Importances\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2025-01-07T10:45:40.439581Z","iopub.execute_input":"2025-01-07T10:45:40.439974Z","iopub.status.idle":"2025-01-07T10:45:40.803485Z","shell.execute_reply.started":"2025-01-07T10:45:40.439936Z","shell.execute_reply":"2025-01-07T10:45:40.802190Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"perm = PermutationImportance(model, random_state=1).fit(X,y)\neli5.show_weights(perm, feature_names = X.columns.tolist())","metadata":{"execution":{"iopub.status.busy":"2025-01-07T10:45:40.805190Z","iopub.execute_input":"2025-01-07T10:45:40.805672Z","iopub.status.idle":"2025-01-07T10:45:41.311104Z","shell.execute_reply.started":"2025-01-07T10:45:40.805621Z","shell.execute_reply":"2025-01-07T10:45:41.309602Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#
12/ Submission
","metadata":{}},{"cell_type":"code","source":"model.fit(X,y)\npreds = model.predict(test)\npreds = convert(preds) # convert raw scores to sii categories if using regressor\npreds = pd.Series(preds)\npreds.index = test.index\npreds.to_csv('submission.csv')","metadata":{"execution":{"iopub.status.busy":"2025-01-07T10:45:41.312941Z","iopub.execute_input":"2025-01-07T10:45:41.313608Z","iopub.status.idle":"2025-01-07T10:45:41.388243Z","shell.execute_reply.started":"2025-01-07T10:45:41.313552Z","shell.execute_reply":"2025-01-07T10:45:41.387336Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#
13/ Conclusion
","metadata":{}},{"cell_type":"markdown","source":"
\n \n* This notebook has provided a general introduction to the competition with some EDA and feature selection.\n* We have created two models, a classifier model and a regression model, to demonstrate how the modelling process works.\n* The regression models gets a benchmark LB score of **.463**.\n* At the time of writing the very best LB scores demonstrate only **moderate** levels of agreement between predictive machine learning models and the test data.\n* PIU may well be linked to reduced levels of physical activity but these indicators do not necessarily demonstrate PIU.\n* Some key risk factors appear to be the age of the child, the level of sleep disturbance experienced and - of course - hours per week of internet usage.","metadata":{}}]}