First of all, I would like to thank the Kaggle community for sharing great ideas and engaging discussions. I would also like to thank the hosts for organizing this interesting task competition.
\nFirst of all, I would like to thank the Kaggle community for sharing great ideas and engaging discussions. I would also like to thank the hosts for organizing this interesting task competition.
\nCongratulations on your gold medal!
\nMay I ask why you split the training and validation data this way?
\n\n", "rawMarkdown": "Congratulations on your gold medal!\n\nMay I ask why you split the training and validation data this way?\n\n>Cross validation: Nested cross validation\n>Training data: Data for which the first four digits of the session_id are 2200 or less.\n>Validation data: Data for which the first four digits of the session_id are 2201 or more.", "votes": null }, { "id": "2322154", "postDate": "06/29/2023 05:46:22", "content": "Cross validation: Nested cross validation
\n
\n Training data: Data for which the first four digits of the session_id are 2200 or less.
\n Validation data: Data for which the first four digits of the session_id are 2201 or more.
Congratulations on your solo gold! Can you share your NN architecture? For me, NN alone doesn't work that well, but extracting embedding from NN and add them into Tabular features works pretty help. So I am curious how NN looks like in your case. And it would be greater that you could share the training details, some remarks, etc.
", "rawMarkdown": "Congratulations on your solo gold! Can you share your NN architecture? For me, NN alone doesn't work that well, but extracting embedding from NN and add them into Tabular features works pretty help. So I am curious how NN looks like in your case. And it would be greater that you could share the training details, some remarks, etc.", "votes": null }, { "id": "2322176", "postDate": "06/29/2023 06:05:14", "content": "Congratulations on gold medal.
\n\n\nI trained one model for level group 0-4 and another for level group 5-12. For these, I included features representing the target and trained a single model.For level group 13-22, I trained a distinct model for each target.
\n
May I ask the reason for this strategy? I find training model for each level_group/level has a very different cv/lb correlation.
", "rawMarkdown": "Congratulations on gold medal.\n\n>I trained one model for level group 0-4 and another for level group 5-12. For these, I included features representing the target and trained a single model.For level group 13-22, I trained a distinct model for each target.\n\nMay I ask the reason for this strategy? I find training model for each level_group/level has a very different cv/lb correlation.", "votes": null }, { "id": "2322194", "postDate": "06/29/2023 06:22:21", "content": "Thank you for your comment! I used nested cross-validation to evaluate for the following two reasons:
\nThe evaluation metric for this time fluctuates significantly based on the threshold, and just changing the threshold also significantly changed the Public Score. I wanted to create the same situation locally as the Public Score (including the ensemble of 5-fold) and check the change in CV due to changing the threshold.
For instance, in situations like the one described in the link below, I believed that nested cross-validation would be more reliable than standard cross-validation.
\nhttps://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/388682
Thank you for your comment! Also, congratulations on your gold medal!
\nFor instance, I created the following model for level group 5-12.
\nnum_cols_level2 = [,,\n ,]\n\n\ncat_cols = [,,,,]\n\n\nclass MeanPooling(nn.Module):\n def __init__(self):\n super(MeanPooling, self).__init__()\n\n def forward(self, last_hidden_state, attention_mask):\n input_mask_expanded = attention_mask.unsqueeze(-1).expand(last_hidden_state.size()).float()\n sum_embeddings = torch.sum(last_hidden_state * input_mask_expanded, 1)\n sum_mask = input_mask_expanded.sum(1)\n sum_mask = torch.clamp(sum_mask, =1e-9)\n mean_embeddings = sum_embeddings / sum_mask\n return mean_embeddings\n\nclass PspTransformerLevel2Model(nn.Module):\n def __init__(\n self, =0.2,\n =4,\n =7, name_embedding_size = 12,\n =12, event_embedding_size = 12,\n =130, fqid_embedding_size = 24,\n =20, room_fqid_embedding_size = 12,\n =24,level_embedding_size=12,\n categorical_linear_size = 120,\n numeraical_linear_size = 24,\n model_size = 160,\n nhead = 16,\n =480,\n =10):\n super(PspTransformerLevel2Model, self).__init__()\n self.name_embedding = nn.Embedding(=input_name_nunique, \n =name_embedding_size)\n self.event_embedding = nn.Embedding(=input_event_nunique, \n =event_embedding_size)\n self.fqid_embedding = nn.Embedding(=input_fqid_nunique, \n =fqid_embedding_size)\n self.room_fqid_embedding = nn.Embedding(=input_room_fqid_nunique, \n =room_fqid_embedding_size)\n self.level_embedding = nn.Embedding(=input_level_nunique, \n =level_embedding_size)\n self.categorical_linear = nn.Sequential(\n nn.Linear(name_embedding_size + event_embedding_size + \n fqid_embedding_size + room_fqid_embedding_size + \n level_embedding_size, categorical_linear_size),\n nn.LayerNorm(categorical_linear_size)\n )\n self.numerical_linear = nn.Sequential(\n nn.Linear(input_numerical_size, numeraical_linear_size),\n nn.LayerNorm(numeraical_linear_size)\n )\n\n self.linear1 = nn.Sequential(\n nn.Linear(categorical_linear_size + numeraical_linear_size, \n model_size),\n nn.LayerNorm(model_size),\n )\n self.transformer_encoder = TransformerEncoder(\n encoder_layer = nn.TransformerEncoderLayer(=model_size, \n =nhead,\n =dim_feedforward ,\n =dropout),\n =1)\n self.gru = nn.GRU(model_size, model_size,\n num_layers = 1, \n =,\n =)\n self.linear_out = nn.Sequential(\n nn.Linear(model_size, \n out_size)\n )\n self.pool = MeanPooling()\n\n def forward(self, numerical_array, name_array,event_array,\n fqid_array, room_fqid_array, level_array, \n mask,mask_for_pooling):\n\n name_embedding = self.name_embedding(name_array)\n event_embedding = self.event_embedding(event_array)\n fqid_embedding = self.fqid_embedding(fqid_array)\n room_fqid_embedding = self.room_fqid_embedding(room_fqid_array)\n level_embedding = self.level_embedding(level_array)\n categorical_emedding = torch.cat([name_embedding,\n event_embedding,\n fqid_embedding, \n room_fqid_embedding,\n level_embedding\n ], =2)\n categorical_emedding = self.categorical_linear(categorical_emedding)\n numerical_embedding = self.numerical_linear(numerical_array)\n concat_embedding = torch.cat([categorical_emedding,\n numerical_embedding],=2)\n concat_embedding = self.linear1(concat_embedding)\n concat_embedding = concat_embedding.permute(1,0,2).contiguous()\n output = self.transformer_encoder(concat_embedding, \n =mask)\n output = output.permute(1,0,2).contiguous()\n output,_ = self.gru(output)\n output = self.pool(output,mask_for_pooling)\n output = self.linear_out(output)\n return output\nThe main improvements I made are listed below. Through a series of experiments, I found that the following practices were advantageous:
\nThank you for your comment!
\nRegarding level groups 0-4 and 5-12, I adopted the above method because creating a single model for them all, rather than creating a model for each target, resulted in better CV and public score.
Hi @takoihiraokazu,
\nCongratulations on your solo gold medal! I have some questions about your solutions described as follows:
\nlevel_group model?Thanks a lot for the sharing!
", "rawMarkdown": "Hi @takoihiraokazu,\n\nCongratulations on your solo gold medal! I have some questions about your solutions described as follows:\n1. For LightGBM, how many features are considered for each `level_group` model?\n2. For NN, how did you choose the sequence length of the event records?\n\nThanks a lot for the sharing!", "votes": null }, { "id": "2322499", "postDate": "06/29/2023 10:11:40", "content": "@takoihiraokazu congrats on solo gold medal!
", "rawMarkdown": "takoihiraokazu congrats on solo gold medal!", "votes": null }, { "id": "2322729", "postDate": "06/29/2023 13:08:32", "content": "Thanks!
\nFor the LightGBM model, the number of features considered for each level_group model is as follows: For level_group 0-4, I consider 1570 features. For level_group 5-12, I consider 3992 features. And for level_group 13-22, I consider 5290 features.
As for the Neural Network, the sequence length of the event records was determined based on cross-validation results. Specifically, for level_group 0-4, I chose a sequence length of 250. For level_group 5-12, the sequence length is 500. And for level_group 13-22, the sequence length is 800.
@takoihiraokazu Congratulations on your 13th place finish! Your strategy of training separate models for different level groups and incorporating features like categorical and numerical data counts demonstrates a thoughtful approach.
", "rawMarkdown": "takoihiraokazu Congratulations on your 13th place finish! Your strategy of training separate models for different level groups and incorporating features like categorical and numerical data counts demonstrates a thoughtful approach.", "votes": null }, { "id": "2322853", "postDate": "06/29/2023 14:28:12", "content": "Thanks for the quick reply, what a robust DL-based model! One more question if you don't mind sharing, did you use early stopping to choose the best checkpoints or just let the training process converge to some satisfactory position?
", "rawMarkdown": "Thanks for the quick reply, what a robust DL-based model! One more question if you don't mind sharing, did you use early stopping to choose the best checkpoints or just let the training process converge to some satisfactory position?", "votes": null }, { "id": "2323866", "postDate": "06/30/2023 07:54:35", "content": "Congratulations on your 13th place solution! It's impressive that you combined LightGBM and neural networks (NN) in an ensemble approach. Your use of nested cross-validation and careful splitting of training and validation data demonstrates a thoughtful approach. Well done!
", "rawMarkdown": "Congratulations on your 13th place solution! It's impressive that you combined LightGBM and neural networks (NN) in an ensemble approach. Your use of nested cross-validation and careful splitting of training and validation data demonstrates a thoughtful approach. Well done!", "votes": null }, { "id": "2324140", "postDate": "06/30/2023 12:10:16", "content": "+1 congrats on gold
", "rawMarkdown": "1 congrats on gold", "votes": null }, { "id": "2324257", "postDate": "06/30/2023 13:51:48", "content": "I didn't exactly use early stopping. Instead, I set the number of epochs to 20 and saved the checkpoint that produced the best validation score. The choice to use 20 epochs was based on several experiments, where this setting resulted in the best cross-validation performance.\"
", "rawMarkdown": "I didn't exactly use early stopping. Instead, I set the number of epochs to 20 and saved the checkpoint that produced the best validation score. The choice to use 20 epochs was based on several experiments, where this setting resulted in the best cross-validation performance.\"", "votes": null }, { "id": "2324389", "postDate": "06/30/2023 15:10:00", "content": "As mentioned in the NN section, you train separate model for each level_group. If I'm not mistaken, the model checkpoints used to obtain the best CV performance could be different for different level_groups. For example, epoch 15 for level_group \"0-4\", 17 for \"5-12\" and 11 for \"13-22\" can lead to the best overall Macro-F1 on local CV. Then, what's the better way to find out this combination. During the competition, I found it hard to figure out an efficient way to determine which checkpoints to choose. Thanks a lot for your clarification and patience 😓
Thank you! Your practices are interesting. My NN was not super good, maybe it's because I kept the architecture and the input features complicated.
", "rawMarkdown": "Thank you! Your practices are interesting. My NN was not super good, maybe it's because I kept the architecture and the input features complicated.", "votes": null }, { "id": "2324819", "postDate": "06/30/2023 22:40:26", "content": "Yes, as you said, the optimal epoch does differ for each level group. As for the checkpoints, I decided on the one where the AUC was highest. Even then, if the AUC improved for each level group, the final macro-f1 score also improved.
", "rawMarkdown": "Yes, as you said, the optimal epoch does differ for each level group. As for the checkpoints, I decided on the one where the AUC was highest. Even then, if the AUC improved for each level group, the final macro-f1 score also improved.", "votes": null }, { "id": "2325133", "postDate": "07/01/2023 06:21:08", "content": "I see! What I did was to choose those with the highest Macro-F1 @ 0.63 (but, I knew optimizing locally on level_group models might not trasfer to global optimization…). I forgot to try to select checkpoints with other metrics. I'll go back to give them a try and do some late submissions.
Thanks for the reply and kindness. Good luck with your next competition!
", "rawMarkdown": "I see! What I did was to choose those with the highest Macro-F1 @ 0.63 (but, I knew optimizing locally on `level_group` models might not trasfer to global optimization...). I forgot to try to select checkpoints with other metrics. I'll go back to give them a try and do some late submissions. \n\nThanks for the reply and kindness. Good luck with your next competition!", "votes": null } ], "comments": [ { "id": 2322148, "author_name": "hoangnguyen719", "author_url": "", "post_date": "06/29/2023 05:38:59", "content": "Congratulations on your gold medal!
\nMay I ask why you split the training and validation data this way?
\n\n", "votes": null, "replies": [ { "id": 2322194, "author_name": "takoihiraokazu", "author_url": "", "post_date": "06/29/2023 06:22:21", "content": "Cross validation: Nested cross validation
\n
\n Training data: Data for which the first four digits of the session_id are 2200 or less.
\n Validation data: Data for which the first four digits of the session_id are 2201 or more.
Thank you for your comment! I used nested cross-validation to evaluate for the following two reasons:
\nThe evaluation metric for this time fluctuates significantly based on the threshold, and just changing the threshold also significantly changed the Public Score. I wanted to create the same situation locally as the Public Score (including the ensemble of 5-fold) and check the change in CV due to changing the threshold.
For instance, in situations like the one described in the link below, I believed that nested cross-validation would be more reliable than standard cross-validation.
\nhttps://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/388682
Congratulations on your solo gold! Can you share your NN architecture? For me, NN alone doesn't work that well, but extracting embedding from NN and add them into Tabular features works pretty help. So I am curious how NN looks like in your case. And it would be greater that you could share the training details, some remarks, etc.
", "votes": null, "replies": [ { "id": 2322229, "author_name": "takoihiraokazu", "author_url": "", "post_date": "06/29/2023 06:49:04", "content": "Thank you for your comment! Also, congratulations on your gold medal!
\nFor instance, I created the following model for level group 5-12.
\nnum_cols_level2 = [,,\n ,]\n\n\ncat_cols = [,,,,]\n\n\nclass MeanPooling(nn.Module):\n def __init__(self):\n super(MeanPooling, self).__init__()\n\n def forward(self, last_hidden_state, attention_mask):\n input_mask_expanded = attention_mask.unsqueeze(-1).expand(last_hidden_state.size()).float()\n sum_embeddings = torch.sum(last_hidden_state * input_mask_expanded, 1)\n sum_mask = input_mask_expanded.sum(1)\n sum_mask = torch.clamp(sum_mask, =1e-9)\n mean_embeddings = sum_embeddings / sum_mask\n return mean_embeddings\n\nclass PspTransformerLevel2Model(nn.Module):\n def __init__(\n self, =0.2,\n =4,\n =7, name_embedding_size = 12,\n =12, event_embedding_size = 12,\n =130, fqid_embedding_size = 24,\n =20, room_fqid_embedding_size = 12,\n =24,level_embedding_size=12,\n categorical_linear_size = 120,\n numeraical_linear_size = 24,\n model_size = 160,\n nhead = 16,\n =480,\n =10):\n super(PspTransformerLevel2Model, self).__init__()\n self.name_embedding = nn.Embedding(=input_name_nunique, \n =name_embedding_size)\n self.event_embedding = nn.Embedding(=input_event_nunique, \n =event_embedding_size)\n self.fqid_embedding = nn.Embedding(=input_fqid_nunique, \n =fqid_embedding_size)\n self.room_fqid_embedding = nn.Embedding(=input_room_fqid_nunique, \n =room_fqid_embedding_size)\n self.level_embedding = nn.Embedding(=input_level_nunique, \n =level_embedding_size)\n self.categorical_linear = nn.Sequential(\n nn.Linear(name_embedding_size + event_embedding_size + \n fqid_embedding_size + room_fqid_embedding_size + \n level_embedding_size, categorical_linear_size),\n nn.LayerNorm(categorical_linear_size)\n )\n self.numerical_linear = nn.Sequential(\n nn.Linear(input_numerical_size, numeraical_linear_size),\n nn.LayerNorm(numeraical_linear_size)\n )\n\n self.linear1 = nn.Sequential(\n nn.Linear(categorical_linear_size + numeraical_linear_size, \n model_size),\n nn.LayerNorm(model_size),\n )\n self.transformer_encoder = TransformerEncoder(\n encoder_layer = nn.TransformerEncoderLayer(=model_size, \n =nhead,\n =dim_feedforward ,\n =dropout),\n =1)\n self.gru = nn.GRU(model_size, model_size,\n num_layers = 1, \n =,\n =)\n self.linear_out = nn.Sequential(\n nn.Linear(model_size, \n out_size)\n )\n self.pool = MeanPooling()\n\n def forward(self, numerical_array, name_array,event_array,\n fqid_array, room_fqid_array, level_array, \n mask,mask_for_pooling):\n\n name_embedding = self.name_embedding(name_array)\n event_embedding = self.event_embedding(event_array)\n fqid_embedding = self.fqid_embedding(fqid_array)\n room_fqid_embedding = self.room_fqid_embedding(room_fqid_array)\n level_embedding = self.level_embedding(level_array)\n categorical_emedding = torch.cat([name_embedding,\n event_embedding,\n fqid_embedding, \n room_fqid_embedding,\n level_embedding\n ], =2)\n categorical_emedding = self.categorical_linear(categorical_emedding)\n numerical_embedding = self.numerical_linear(numerical_array)\n concat_embedding = torch.cat([categorical_emedding,\n numerical_embedding],=2)\n concat_embedding = self.linear1(concat_embedding)\n concat_embedding = concat_embedding.permute(1,0,2).contiguous()\n output = self.transformer_encoder(concat_embedding, \n =mask)\n output = output.permute(1,0,2).contiguous()\n output,_ = self.gru(output)\n output = self.pool(output,mask_for_pooling)\n output = self.linear_out(output)\n return output\nThe main improvements I made are listed below. Through a series of experiments, I found that the following practices were advantageous:
\nThank you! Your practices are interesting. My NN was not super good, maybe it's because I kept the architecture and the input features complicated.
", "votes": null, "replies": [] } ] } ] }, { "id": 2322176, "author_name": "ataraxian", "author_url": "", "post_date": "06/29/2023 06:05:14", "content": "Congratulations on gold medal.
\n\n\nI trained one model for level group 0-4 and another for level group 5-12. For these, I included features representing the target and trained a single model.For level group 13-22, I trained a distinct model for each target.
\n
May I ask the reason for this strategy? I find training model for each level_group/level has a very different cv/lb correlation.
", "votes": null, "replies": [ { "id": 2322243, "author_name": "takoihiraokazu", "author_url": "", "post_date": "06/29/2023 06:59:53", "content": "Thank you for your comment!
\nRegarding level groups 0-4 and 5-12, I adopted the above method because creating a single model for them all, rather than creating a model for each target, resulted in better CV and public score.
Hi @takoihiraokazu,
\nCongratulations on your solo gold medal! I have some questions about your solutions described as follows:
\nlevel_group model?Thanks a lot for the sharing!
", "votes": null, "replies": [ { "id": 2322729, "author_name": "takoihiraokazu", "author_url": "", "post_date": "06/29/2023 13:08:32", "content": "Thanks!
\nFor the LightGBM model, the number of features considered for each level_group model is as follows: For level_group 0-4, I consider 1570 features. For level_group 5-12, I consider 3992 features. And for level_group 13-22, I consider 5290 features.
As for the Neural Network, the sequence length of the event records was determined based on cross-validation results. Specifically, for level_group 0-4, I chose a sequence length of 250. For level_group 5-12, the sequence length is 500. And for level_group 13-22, the sequence length is 800.
Thanks for the quick reply, what a robust DL-based model! One more question if you don't mind sharing, did you use early stopping to choose the best checkpoints or just let the training process converge to some satisfactory position?
", "votes": null, "replies": [ { "id": 2324257, "author_name": "takoihiraokazu", "author_url": "", "post_date": "06/30/2023 13:51:48", "content": "I didn't exactly use early stopping. Instead, I set the number of epochs to 20 and saved the checkpoint that produced the best validation score. The choice to use 20 epochs was based on several experiments, where this setting resulted in the best cross-validation performance.\"
", "votes": null, "replies": [ { "id": 2324389, "author_name": "abaojiang", "author_url": "", "post_date": "06/30/2023 15:10:00", "content": "As mentioned in the NN section, you train separate model for each level_group. If I'm not mistaken, the model checkpoints used to obtain the best CV performance could be different for different level_groups. For example, epoch 15 for level_group \"0-4\", 17 for \"5-12\" and 11 for \"13-22\" can lead to the best overall Macro-F1 on local CV. Then, what's the better way to find out this combination. During the competition, I found it hard to figure out an efficient way to determine which checkpoints to choose. Thanks a lot for your clarification and patience 😓
Yes, as you said, the optimal epoch does differ for each level group. As for the checkpoints, I decided on the one where the AUC was highest. Even then, if the AUC improved for each level group, the final macro-f1 score also improved.
", "votes": null, "replies": [ { "id": 2325133, "author_name": "abaojiang", "author_url": "", "post_date": "07/01/2023 06:21:08", "content": "I see! What I did was to choose those with the highest Macro-F1 @ 0.63 (but, I knew optimizing locally on level_group models might not trasfer to global optimization…). I forgot to try to select checkpoints with other metrics. I'll go back to give them a try and do some late submissions.
Thanks for the reply and kindness. Good luck with your next competition!
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] }, { "id": 2322499, "author_name": "serangu", "author_url": "", "post_date": "06/29/2023 10:11:40", "content": "@takoihiraokazu congrats on solo gold medal!
", "votes": null, "replies": [ { "id": 2324140, "author_name": "joek47", "author_url": "", "post_date": "06/30/2023 12:10:16", "content": "+1 congrats on gold
", "votes": null, "replies": [] } ] }, { "id": 2322764, "author_name": "akshayvyas02", "author_url": "", "post_date": "06/29/2023 13:37:04", "content": "@takoihiraokazu Congratulations on your 13th place finish! Your strategy of training separate models for different level groups and incorporating features like categorical and numerical data counts demonstrates a thoughtful approach.
", "votes": null, "replies": [] }, { "id": 2323866, "author_name": "poojach7611", "author_url": "", "post_date": "06/30/2023 07:54:35", "content": "Congratulations on your 13th place solution! It's impressive that you combined LightGBM and neural networks (NN) in an ensemble approach. Your use of nested cross-validation and careful splitting of training and validation data demonstrates a thoughtful approach. Well done!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2322141": "# 13th place solution\nFirst of all, I would like to thank the Kaggle community for sharing great ideas and engaging discussions. I would also like to thank the hosts for organizing this interesting task competition.\n\n## Summary\n- Ensemble of LightGBM and NN\n- Cross validation: Nested cross validation\n - Training data: Data for which the first four digits of the session_id are 2200 or less.\n - Validation data: Data for which the first four digits of the session_id are 2201 or more.\n - I trained the model on the training data using a 5-fold cross validation strategy, and evaluated it on the validation data using predictions from all 5 trained models.\n - For the final submission, I trained the model on the entire dataset using a 5-fold cross validation approach.\n\n## LightGBM\n- I trained one model for level group 0-4 and another for level group 5-12. For these, I included features representing the target and trained a single model.\n- For level group 13-22, I trained a distinct model for each target.\n- Main features:\n - The count of categorical data for each session_id\n - The statistical measures of numerical data for each session_id\n - An aggregate of the next action taken \n- Scores\n - CV : 0.7032\n - Public Score : 0.704\n - Private Score : 0.701\n\n\n## NN\n- Model: Transformer + GRU\n - The standalone Transformer didn't perform very well.\n - The addition of GRU improved the score.\n- Trained with fewer features\n- I trained a separate model for each level group.\n- Scores\n - CV : 0.7010\n - Public Score: 0.700\n - Private Score: 0.700\n\n## Ensemble\n - LightGBM * 0.66 + NN * 0.34\n - CV : 0.7053\n - Public Score : 0.706\n - Private Score : 0.702", "2322148": "Congratulations on your gold medal!\n\nMay I ask why you split the training and validation data this way?\n\n>Cross validation: Nested cross validation\n>Training data: Data for which the first four digits of the session_id are 2200 or less.\n>Validation data: Data for which the first four digits of the session_id are 2201 or more.", "2322154": "Congratulations on your solo gold! Can you share your NN architecture? For me, NN alone doesn't work that well, but extracting embedding from NN and add them into Tabular features works pretty help. So I am curious how NN looks like in your case. And it would be greater that you could share the training details, some remarks, etc.", "2322176": "Congratulations on gold medal.\n\n>I trained one model for level group 0-4 and another for level group 5-12. For these, I included features representing the target and trained a single model.For level group 13-22, I trained a distinct model for each target.\n\nMay I ask the reason for this strategy? I find training model for each level_group/level has a very different cv/lb correlation.", "2322194": "Thank you for your comment! I used nested cross-validation to evaluate for the following two reasons:\n\n- The evaluation metric for this time fluctuates significantly based on the threshold, and just changing the threshold also significantly changed the Public Score. I wanted to create the same situation locally as the Public Score (including the ensemble of 5-fold) and check the change in CV due to changing the threshold.\n\n- For instance, in situations like the one described in the link below, I believed that nested cross-validation would be more reliable than standard cross-validation.\nhttps://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/388682", "2322229": "Thank you for your comment! Also, congratulations on your gold medal! \nFor instance, I created the following model for level group 5-12.\n\n```\n# Numerical features used for NN\nnum_cols_level2 = [\"elapsed_time_log1p\",\"elapsed_time_diff_log1p\",\n \"room_coor_x\",\"room_coor_y\"]\n\n# Categorical features used for NN\ncat_cols = [\"event_name\",\"name\",\"fqid\",\"room_fqid\",\"level\"]\n\n\nclass MeanPooling(nn.Module):\n def __init__(self):\n super(MeanPooling, self).__init__()\n \n def forward(self, last_hidden_state, attention_mask):\n input_mask_expanded = attention_mask.unsqueeze(-1).expand(last_hidden_state.size()).float()\n sum_embeddings = torch.sum(last_hidden_state * input_mask_expanded, 1)\n sum_mask = input_mask_expanded.sum(1)\n sum_mask = torch.clamp(sum_mask, min=1e-9)\n mean_embeddings = sum_embeddings / sum_mask\n return mean_embeddings\n\nclass PspTransformerLevel2Model(nn.Module):\n def __init__(\n self, dropout=0.2,\n input_numerical_size=4,\n input_name_nunique=7, name_embedding_size = 12,\n input_event_nunique=12, event_embedding_size = 12,\n input_fqid_nunique=130, fqid_embedding_size = 24,\n input_room_fqid_nunique=20, room_fqid_embedding_size = 12,\n input_level_nunique=24,level_embedding_size=12,\n categorical_linear_size = 120,\n numeraical_linear_size = 24,\n model_size = 160,\n nhead = 16,\n dim_feedforward=480,\n out_size=10):\n super(PspTransformerLevel2Model, self).__init__()\n self.name_embedding = nn.Embedding(num_embeddings=input_name_nunique, \n embedding_dim=name_embedding_size)\n self.event_embedding = nn.Embedding(num_embeddings=input_event_nunique, \n embedding_dim=event_embedding_size)\n self.fqid_embedding = nn.Embedding(num_embeddings=input_fqid_nunique, \n embedding_dim=fqid_embedding_size)\n self.room_fqid_embedding = nn.Embedding(num_embeddings=input_room_fqid_nunique, \n embedding_dim=room_fqid_embedding_size)\n self.level_embedding = nn.Embedding(num_embeddings=input_level_nunique, \n embedding_dim=level_embedding_size)\n self.categorical_linear = nn.Sequential(\n nn.Linear(name_embedding_size + event_embedding_size + \n fqid_embedding_size + room_fqid_embedding_size + \n level_embedding_size, categorical_linear_size),\n nn.LayerNorm(categorical_linear_size)\n )\n self.numerical_linear = nn.Sequential(\n nn.Linear(input_numerical_size, numeraical_linear_size),\n nn.LayerNorm(numeraical_linear_size)\n )\n \n self.linear1 = nn.Sequential(\n nn.Linear(categorical_linear_size + numeraical_linear_size, \n model_size),\n nn.LayerNorm(model_size),\n )\n self.transformer_encoder = TransformerEncoder(\n encoder_layer = nn.TransformerEncoderLayer(d_model=model_size, \n nhead=nhead,\n dim_feedforward=dim_feedforward ,\n dropout=dropout),\n num_layers=1)\n self.gru = nn.GRU(model_size, model_size,\n num_layers = 1, \n batch_first=True,\n bidirectional=True)\n self.linear_out = nn.Sequential(\n nn.Linear(model_size*2, \n out_size)\n )\n self.pool = MeanPooling()\n \n def forward(self, numerical_array, name_array,event_array,\n fqid_array, room_fqid_array, level_array, \n mask,mask_for_pooling):\n \n name_embedding = self.name_embedding(name_array)\n event_embedding = self.event_embedding(event_array)\n fqid_embedding = self.fqid_embedding(fqid_array)\n room_fqid_embedding = self.room_fqid_embedding(room_fqid_array)\n level_embedding = self.level_embedding(level_array)\n categorical_emedding = torch.cat([name_embedding,\n event_embedding,\n fqid_embedding, \n room_fqid_embedding,\n level_embedding\n ], axis=2)\n categorical_emedding = self.categorical_linear(categorical_emedding)\n numerical_embedding = self.numerical_linear(numerical_array)\n concat_embedding = torch.cat([categorical_emedding,\n numerical_embedding],axis=2)\n concat_embedding = self.linear1(concat_embedding)\n concat_embedding = concat_embedding.permute(1,0,2).contiguous()\n output = self.transformer_encoder(concat_embedding, \n src_key_padding_mask=mask)\n output = output.permute(1,0,2).contiguous()\n output,_ = self.gru(output)\n output = self.pool(output,mask_for_pooling)\n output = self.linear_out(output)\n return output\n\n```\n\n\nThe main improvements I made are listed below. Through a series of experiments, I found that the following practices were advantageous:\n\n- Keeping the model size small:\n - For instance, I kept the number of layers for both the Transformer and GRU at 1.\n - I didn't enlarge the encoder layer or hidden size.\n - I also maintained a relatively small embedding size for categorical features.\n- Using fewer features:\n - For instance, I didn't use features such as screen_coor_x, screen_coor_y, hover_duration, and text.", "2322243": "Thank you for your comment! \nRegarding level groups 0-4 and 5-12, I adopted the above method because creating a single model for them all, rather than creating a model for each target, resulted in better CV and public score.", "2322245": "Hi @takoihiraokazu,\n\nCongratulations on your solo gold medal! I have some questions about your solutions described as follows:\n1. For LightGBM, how many features are considered for each `level_group` model?\n2. For NN, how did you choose the sequence length of the event records?\n\nThanks a lot for the sharing!", "2322499": "takoihiraokazu congrats on solo gold medal!", "2322729": "Thanks!\n\n1. For the LightGBM model, the number of features considered for each level_group model is as follows: For level_group 0-4, I consider 1570 features. For level_group 5-12, I consider 3992 features. And for level_group 13-22, I consider 5290 features.\n\n2. As for the Neural Network, the sequence length of the event records was determined based on cross-validation results. Specifically, for level_group 0-4, I chose a sequence length of 250. For level_group 5-12, the sequence length is 500. And for level_group 13-22, the sequence length is 800.", "2322764": "takoihiraokazu Congratulations on your 13th place finish! Your strategy of training separate models for different level groups and incorporating features like categorical and numerical data counts demonstrates a thoughtful approach.", "2322853": "Thanks for the quick reply, what a robust DL-based model! One more question if you don't mind sharing, did you use early stopping to choose the best checkpoints or just let the training process converge to some satisfactory position?", "2323866": "Congratulations on your 13th place solution! It's impressive that you combined LightGBM and neural networks (NN) in an ensemble approach. Your use of nested cross-validation and careful splitting of training and validation data demonstrates a thoughtful approach. Well done!", "2324140": "1 congrats on gold", "2324257": "I didn't exactly use early stopping. Instead, I set the number of epochs to 20 and saved the checkpoint that produced the best validation score. The choice to use 20 epochs was based on several experiments, where this setting resulted in the best cross-validation performance.\"", "2324389": "As mentioned in the NN section, you train separate model for each `level_group`. If I'm not mistaken, the model checkpoints used to obtain the best CV performance could be different for different `level_group`s. For example, epoch 15 for `level_group` \"0-4\", 17 for \"5-12\" and 11 for \"13-22\" can lead to the best overall Macro-F1 on local CV. Then, what's the better way to find out this combination. During the competition, I found it hard to figure out an efficient way to determine which checkpoints to choose. Thanks a lot for your clarification and patience 😓", "2324447": "Thank you! Your practices are interesting. My NN was not super good, maybe it's because I kept the architecture and the input features complicated.", "2324819": "Yes, as you said, the optimal epoch does differ for each level group. As for the checkpoints, I decided on the one where the AUC was highest. Even then, if the AUC improved for each level group, the final macro-f1 score also improved.", "2325133": "I see! What I did was to choose those with the highest Macro-F1 @ 0.63 (but, I knew optimizing locally on `level_group` models might not trasfer to global optimization...). I forgot to try to select checkpoints with other metrics. I'll go back to give them a try and do some late submissions. \n\nThanks for the reply and kindness. Good luck with your next competition!" }, "source": "meta" }