![\"psp_solution_overview\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5563440%2F010fd90d184db7dc2858562025f3867a%2Fpsp_solution_overview.drawio%20(1).png?generation=1688311769280377&alt=media\")
I respect all of you for your tough and long term fight and I am glad that we were able to fight together. I also want to thank my teammates ( @tereka, @deepkun1995, @ryotak12, @yurimaeda) for their hard work.
\nI'm happy because this is the first time I got a gold medal.
\nWe're not doing anything special in our solution. We used 1 NN, 1 LightGBM and 4 XGBoost with various features for the Stage 1, and MLP and Logistic Regression stacking for the Stage 2. We used average and threshold optimization for the Stage 3.
\n
I created XGBoost for each level_group. The base features are not so different from those in the public code. It is an aggregate feature of elapsed_time_diff and hover_duration, and other numerical features. However, in addition to these, I used previous level_group features and predicted probability as current level_group features.
\nI used numpy and numba to create them. Initially, I had used polars, but I switched to numba which is my teammate @yurimaeda 's approach. The submission time was significantly reduced from 2 hours with polars to just 13 minutes with numpy and numba. I used 5-StratifiedGroupKFold as cross-validation strategy.
![\"psp_solution_xgboost\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5563440%2Faabaecb685f9740c8b542c8282791901%2Fpsp_solution_xgb.drawio%20(1).png?generation=1688311791487937&alt=media\")
We created MLP and Logistic Regression for each question. Thus, there are 18 models each, and the output dimension of each model is (n_samples, 1).
\nSince stacking was very easy to overfit, we kept the model architecture simple.
\nHere is the code for MLP.
(nn.Module):\n ():\n ().__init__()\n self.fc1 = nn.Linear(input_size, hidden_size)\n self.head = nn.Linear(hidden_size, output_size)\n self.dropout = nn.Dropout()\n self.relu = nn.ReLU()\n\n ():\n x = self.relu(self.fc1(x))\n x = self.dropout(x)\n x = self.head(x)\n x\nWe take the average of the predictions of the 2 models in the Stage 2 and optimize the threshold for each question.
\n numpy np\n sklearn.metrics f1_score\n scipy.optimize minimize\n\n\n ():\n y_pred_binary = (y_pred_prob > thresholds).astype()\n score = f1_score(y_true.flatten(), y_pred_binary.flatten(), average=)\n score\n\n\n ():\n n_labels = y_pred_prob.shape[]\n init_thresholds = np.full(n_labels, )\n\n objective = thresholds: -f1_score_macro_for_thresholds(\n y_true, y_pred_prob, thresholds\n )\n result = minimize(\n objective, init_thresholds, bounds=[(, )] * n_labels, method=method\n )\n\n result.x\nWe tried some optimization methods, but Powell worked best.
\nThis method improved CV by 0.008.
I respect all of you for your tough and long term fight and I am glad that we were able to fight together. I also want to thank my teammates ( @tereka, @deepkun1995, @ryotak12, @yurimaeda) for their hard work.
\nI'm happy because this is the first time I got a gold medal.
\nWe're not doing anything special in our solution. We used 1 NN, 1 LightGBM and 4 XGBoost with various features for the Stage 1, and MLP and Logistic Regression stacking for the Stage 2. We used average and threshold optimization for the Stage 3.
\n
I created XGBoost for each level_group. The base features are not so different from those in the public code. It is an aggregate feature of elapsed_time_diff and hover_duration, and other numerical features. However, in addition to these, I used previous level_group features and predicted probability as current level_group features.
\nI used numpy and numba to create them. Initially, I had used polars, but I switched to numba which is my teammate @yurimaeda 's approach. The submission time was significantly reduced from 2 hours with polars to just 13 minutes with numpy and numba. I used 5-StratifiedGroupKFold as cross-validation strategy.
We created MLP and Logistic Regression for each question. Thus, there are 18 models each, and the output dimension of each model is (n_samples, 1).
\nSince stacking was very easy to overfit, we kept the model architecture simple.
\nHere is the code for MLP.
(nn.Module):\n ():\n ().__init__()\n self.fc1 = nn.Linear(input_size, hidden_size)\n self.head = nn.Linear(hidden_size, output_size)\n self.dropout = nn.Dropout()\n self.relu = nn.ReLU()\n\n ():\n x = self.relu(self.fc1(x))\n x = self.dropout(x)\n x = self.head(x)\n x\nWe take the average of the predictions of the 2 models in the Stage 2 and optimize the threshold for each question.
\n numpy np\n sklearn.metrics f1_score\n scipy.optimize minimize\n\n\n ():\n y_pred_binary = (y_pred_prob > thresholds).astype()\n score = f1_score(y_true.flatten(), y_pred_binary.flatten(), average=)\n score\n\n\n ():\n n_labels = y_pred_prob.shape[]\n init_thresholds = np.full(n_labels, )\n\n objective = thresholds: -f1_score_macro_for_thresholds(\n y_true, y_pred_prob, thresholds\n )\n result = minimize(\n objective, init_thresholds, bounds=[(, )] * n_labels, method=method\n )\n\n result.x\nWe tried some optimization methods, but Powell worked best.
\nThis method improved CV by 0.008.
Congrats on coming 10th place @shu421 🎉🎉🎉
", "rawMarkdown": "Congrats on coming 10th place @shu421 🎉🎉🎉", "votes": null }, { "id": "2323633", "postDate": "06/30/2023 05:01:18", "content": "@shu421, @tereka, @deepkun1995, @ryotak12, @yurimaeda congrats with 10 place! You are winning fight and get medals!
", "rawMarkdown": "shu421, @tereka, @deepkun1995, @ryotak12, @yurimaeda congrats with 10 place! You are winning fight and get medals!", "votes": null } ], "comments": [ { "id": 2322635, "author_name": "swapnilchowdhury", "author_url": "", "post_date": "06/29/2023 11:48:12", "content": "Congrats on coming 10th place @shu421 🎉🎉🎉
", "votes": null, "replies": [] }, { "id": 2323633, "author_name": "serangu", "author_url": "", "post_date": "06/30/2023 05:01:18", "content": "@shu421, @tereka, @deepkun1995, @ryotak12, @yurimaeda congrats with 10 place! You are winning fight and get medals!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2322563": "I respect all of you for your tough and long term fight and I am glad that we were able to fight together. I also want to thank my teammates ( @tereka, @deepkun1995, @ryotak12, @yurimaeda) for their hard work.\n\nI'm happy because this is the first time I got a gold medal.\n\n# Overview\n We're not doing anything special in our solution. We used 1 NN, 1 LightGBM and 4 XGBoost with various features for the Stage 1, and MLP and Logistic Regression stacking for the Stage 2. We used average and threshold optimization for the Stage 3.\n - CV: 0.70573\n - Public LB: 0.706\n - Private LB: 0.702\n\n.png?generation=1688311769280377&alt=media)\n\n## Code\n- Inference: https://www.kaggle.com/code/shu421/psp-10thsolution-public0706-private0702/notebook\n- shu421 XGBoost and Stacking Training: https://github.com/shu421/Kaggle_PSP_10thSolution\n\n# Models\n## Stage 1: XGBoost (shu421 part)\nI created XGBoost for each level_group. The base features are not so different from those in the public code. It is an aggregate feature of elapsed_time_diff and hover_duration, and other numerical features. However, in addition to these, I used previous level_group features and predicted probability as current level_group features. \nI used numpy and numba to create them. Initially, I had used polars, but I switched to numba which is my teammate @yurimaeda 's approach. The submission time was significantly reduced from 2 hours with polars to just 13 minutes with numpy and numba. I used 5-StratifiedGroupKFold as cross-validation strategy.\n- CV: 0.70111\n- Public LB: 0.702\n- Private LB: 0.699\n\n.png?generation=1688311791487937&alt=media)\n\n\n## Stage 2: Stacking\nWe created MLP and Logistic Regression for each question. Thus, there are 18 models each, and the output dimension of each model is (n_samples, 1).\nSince stacking was very easy to overfit, we kept the model architecture simple.\nHere is the code for MLP.\n\n```python\nclass MLP(nn.Module):\n def __init__(self, input_size, hidden_size, output_size):\n super().__init__()\n self.fc1 = nn.Linear(input_size, hidden_size)\n self.head = nn.Linear(hidden_size, output_size)\n self.dropout = nn.Dropout(0.2)\n self.relu = nn.ReLU()\n\n def forward(self, x):\n x = self.relu(self.fc1(x))\n x = self.dropout(x)\n x = self.head(x)\n return x\n```\n\n\n## Stage 3: Threshold Optimization\nWe take the average of the predictions of the 2 models in the Stage 2 and optimize the threshold for each question.\n\n```python\nimport numpy as np\nfrom sklearn.metrics import f1_score\nfrom scipy.optimize import minimize\n\n\ndef f1_score_macro_for_thresholds(y_true, y_pred_prob, thresholds):\n y_pred_binary = (y_pred_prob > thresholds).astype(int)\n score = f1_score(y_true.flatten(), y_pred_binary.flatten(), average=\"macro\")\n return score\n\n\ndef optimize_thresholds(y_true, y_pred_prob, method=\"Powell\"):\n n_labels = y_pred_prob.shape[1]\n init_thresholds = np.full(n_labels, 0.6)\n\n objective = lambda thresholds: -f1_score_macro_for_thresholds(\n y_true, y_pred_prob, thresholds\n )\n result = minimize(\n objective, init_thresholds, bounds=[(0, 1)] * n_labels, method=method\n )\n\n return result.x\n```\n\nWe tried some optimization methods, but Powell worked best.\nThis method improved CV by 0.008.\n\n# What worked\n- feature engineering\n - elapsed_time_diff and hover_duration agg features was important\n- threshold optimization\n- ensemble\n- lstm + transformer(ryota part)\n- [sort_frame](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/416963)\n\n# What didn't work\n- 1D CNN\n- stacking(below methods seemed to be overfitting)\n - CNN(1D/2D)\n - RNN\n - level_group preds\n- datetime agg features\n- use NN embedding as gbdt features\n- TimeSeriesClustering (elapsed_time_diff)\n- additional data", "2322635": "Congrats on coming 10th place @shu421 🎉🎉🎉", "2323633": "shu421, @tereka, @deepkun1995, @ryotak12, @yurimaeda congrats with 10 place! You are winning fight and get medals!" }, "source": "meta" }