![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1146523%2F3f2943b27feb7d884472c201cd032d5b%2F1.jpg?generation=1610075778291231&alt=media\")
First of all, thanks to Hoon Pyo (Tim) Jeon and Kaggle team for such an interesting competition.
\nAnd congratulates to all the winning teams!
The following is the team takoi + kurupical solution.
\n
\n
@tito's validation strategy. https://www.kaggle.com/its7171/cv-strategy
\n
\n
I would like to have three kaggler to thank.
\n@takoi for inviting me to form a team. If it weren't for you, I couldn't reach this rank!
\n@limerobot for sharing DSB 3rd solution. I'm beginner in transformer for time-series data, so I learned a lot from your solution!
\n@wangsg for sharing notebook https://www.kaggle.com/wangsg/a-self-attentive-model-for-knowledge-tracing.!
\nI used this notebook as a baseline and finally get 0.809 CV for single transformer.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1146523%2F411c8ee779411bb443284b572980ef21%2F2.jpg?generation=1610075796425059&alt=media\")
I did over 300 experiments, and only about 20 of them were successful.
\n
\n
I made 1 LightGBM and 8 NN models. The model that combined Transformer and LSTM had the best CV. Here is architecture and brief description.
\n
\n
\n
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1146523%2Fcab7462101930cf01ccbdd867e3b8a68%2F2%20(1).jpeg?generation=1610097396888437&alt=media\")
I used 17 features. 15 features were computed per user_id. 2 features were computed per content_id.
\nI used 97 features. The following are the main features.
\nFirst of all, thanks to Hoon Pyo (Tim) Jeon and Kaggle team for such an interesting competition.
\nAnd congratulates to all the winning teams!
The following is the team takoi + kurupical solution.
\n
\n
@tito's validation strategy. https://www.kaggle.com/its7171/cv-strategy
\n
\n
I would like to have three kaggler to thank.
\n@takoi for inviting me to form a team. If it weren't for you, I couldn't reach this rank!
\n@limerobot for sharing DSB 3rd solution. I'm beginner in transformer for time-series data, so I learned a lot from your solution!
\n@wangsg for sharing notebook https://www.kaggle.com/wangsg/a-self-attentive-model-for-knowledge-tracing.!
\nI used this notebook as a baseline and finally get 0.809 CV for single transformer.
I did over 300 experiments, and only about 20 of them were successful.
\n
\n
I made 1 LightGBM and 8 NN models. The model that combined Transformer and LSTM had the best CV. Here is architecture and brief description.
\n
\n
\n
\n
I used 17 features. 15 features were computed per user_id. 2 features were computed per content_id.
\nI used 97 features. The following are the main features.
\nCongrats on 13th place @kurupical and @takoihiraokazu and thanks for sharing your solution
", "rawMarkdown": "Congrats on 13th place @kurupical and @takoihiraokazu and thanks for sharing your solution", "votes": null }, { "id": "1144389", "postDate": "01/08/2021 12:39:23", "content": "Very nice write-up, thanks for sharing ! And congratz on the gold finish!
", "rawMarkdown": "Very nice write-up, thanks for sharing ! And congratz on the gold finish!", "votes": null }, { "id": "1145474", "postDate": "01/09/2021 07:04:38", "content": "congratulations！
", "rawMarkdown": "congratulations！", "votes": null }, { "id": "1145485", "postDate": "01/09/2021 07:13:36", "content": "Congratulations on a cool finish!
\nalpha*new_value + (1-alpha)*old_value?Thanks a lot!
", "rawMarkdown": "Congratulations on a cool finish!\n\n- What's same_question_index and target?\n- distance between the same content_id -> so this is the difference b/w the sequence no when a particular content_id was last seen Vs their current sequence number assigned, right?\n- decayed features (average of answered correctly) -> How did you decay them? Did you do something like `alpha*new_value + (1-alpha)*old_value`?\n- LIT structure in EncoderLayer. What's LIT?\n\nThanks a lot!", "votes": null }, { "id": "1145614", "postDate": "01/09/2021 08:57:14", "content": "Thank you!
\nI answer question about my side.
distance between the same content_id -> so this is the difference b/w the sequence no when a particular content_id was last seen Vs their current sequence number assigned, right?\nYes.
\ndecayed features (average of answered correctly) -> How did you decay them? Did you do something like alpha*new_value + (1-alpha)*old_value?\nI decayed the value as 0.3 * new_vlaue + (1 - 0.3) * old_value .
Hi @adityaecdrid !
\n\n\nWhat's same_question_index and target?
\n
-> same as takoi's distance between the same content_id
\n\n\nLIT structure in EncoderLayer. What's LIT?
\n
-> Sorry, I forget to explain detail. https://arxiv.org/pdf/2012.14164.pdf. My implements is below. TransformerEncoder is copy from torch.nn.TransformerEncoder, I just replaced the linear part of src2 with a LITLayer.
\nclass LITLayer(nn.Module):\n \"\"\"\n https://arxiv.org/pdf/2012.14164.pdf\n \"\"\"\n def __init__(self, input_dim, embed_dim, dropout, activation):\n super(LITLayer, self).__init__()\n self.input_dim = input_dim\n self.embed_dim = embed_dim\n self.activation = activation\n\n self.linear1 = nn.Linear(input_dim, embed_dim)\n self.lstm = nn.LSTM(embed_dim, embed_dim)\n self.linear2 = nn.Linear(embed_dim, input_dim)\n\n self.norm_lstm = nn.LayerNorm(embed_dim)\n\n self.dropout = nn.Dropout(dropout)\n\n def forward(self, x):\n x = self.linear1(x)\n x = self.activation(x)\n x = self.dropout(x)\n x, _ = self.lstm(x)\n x = self.norm_lstm(x)\n x = self.dropout(x)\n x = self.linear2(x)\n\n return x\n\nclass TransformerEncoderLayer(nn.Module):\n r\"\"\"TransformerEncoderLayer is made up of self-attn and feedforward network.\n This standard encoder layer is based on the paper \"Attention Is All You Need\".\n Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,\n Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in\n Neural Information Processing Systems, pages 6000-6010. Users may modify or implement\n in a different way during application.\n\n Args:\n d_model: the number of expected features in the input (required).\n nhead: the number of heads in the multiheadattention models (required).\n dim_feedforward: the dimension of the feedforward network model (default=2048).\n dropout: the dropout value (default=0.1).\n activation: the activation function of intermediate layer, relu or gelu (default=relu).\n\n Examples::\n >>> encoder_layer = nn.TransformerEncoderLayer(d_model=512, nhead=8)\n >>> src = torch.rand(10, 32, 512)\n >>> out = encoder_layer(src)\n \"\"\"\n\n def __init__(self, d_model, nhead, dim_feedforward=256, dropout=0.1, activation=\"relu\"):\n super(TransformerEncoderLayer, self).__init__()\n self.self_attn = nn.MultiheadAttention(d_model, nhead, dropout=dropout)\n # Implementation of Feedforward model\n\n self.norm1 = nn.LayerNorm(d_model)\n self.norm2 = nn.LayerNorm(d_model)\n self.dropout1 = nn.Dropout(dropout)\n self.dropout2 = nn.Dropout(dropout)\n\n self.activation = _get_activation_fn(activation)\n\n self.lit_layer = LITLayer(input_dim=d_model, embed_dim=dim_feedforward, dropout=dropout, activation=self.activation)\n\n def __setstate__(self, state):\n if 'activation' not in state:\n state['activation'] = nn.ReLU\n super(TransformerEncoderLayer, self).__setstate__(state)\n\n def forward(self, src: torch.Tensor, src_mask: Optional[torch.Tensor] = None, src_key_padding_mask: Optional[torch.Tensor] = None) -> torch.Tensor:\n r\"\"\"Pass the input through the encoder layer.\n\n Args:\n src: the sequence to the encoder layer (required).\n src_mask: the mask for the src sequence (optional).\n src_key_padding_mask: the mask for the src keys per batch (optional).\n\n Shape:\n see the docs in Transformer class.\n \"\"\"\n src2 = self.self_attn(src, src, src, attn_mask=src_mask,\n key_padding_mask=src_key_padding_mask)[0]\n src = src + self.dropout1(src2)\n src = self.norm1(src)\n # src2 = self.linear2(self.dropout(self.activation(self.linear1(src))))\n src2 = self.lit_layer(src)\n src = src + self.dropout2(src2)\n src = self.norm2(src)\n return src\nThanks guys for the quick response!
", "rawMarkdown": "Thanks guys for the quick response!", "votes": null }, { "id": "1146884", "postDate": "01/10/2021 06:23:05", "content": "Thank you for sharing a great solution and congratulations on gold medal!
\nIt might be a dumb question but what makes you stack a LSTM layer on a Transformer layer?
\nI am studying NN and it would be a great help if you guide me to understand how LSTM is superior to Transformer.
And, how did you decide which features put in transformer or dense layer?
", "rawMarkdown": "Thank you for sharing a great solution and congratulations on gold medal!\n\nIt might be a dumb question but what makes you stack a LSTM layer on a Transformer layer?\nI am studying NN and it would be a great help if you guide me to understand how LSTM is superior to Transformer.\n\nAnd, how did you decide which features put in transformer or dense layer?", "votes": null }, { "id": "1146914", "postDate": "01/10/2021 06:52:20", "content": "Thanks!
\nIt might be a dumb question but what makes you stack a LSTM layer on a Transformer layer?\nI am studying NN and it would be a great help if you guide me to understand how LSTM is superior to Transformer.\nI think LSTM is better at capturing nearby information than Transformer, but Transformer is better at capturing distant information. That's why I included both.
\nAnd, how did you decide which features put in transformer or dense layer?\nI included cumulative and average features for all previous problems of the user, and features that greatly increased the score of LightGBM.
", "rawMarkdown": "Thanks!\n```\nIt might be a dumb question but what makes you stack a LSTM layer on a Transformer layer?\nI am studying NN and it would be a great help if you guide me to understand how LSTM is superior to Transformer.\n```\nI think LSTM is better at capturing nearby information than Transformer, but Transformer is better at capturing distant information. That's why I included both.\n\n```\nAnd, how did you decide which features put in transformer or dense layer?\n```\nI included cumulative and average features for all previous problems of the user, and features that greatly increased the score of LightGBM.", "votes": null }, { "id": "1146929", "postDate": "01/10/2021 07:15:34", "content": "\n\nI think LSTM is better at capturing nearby information than Transformer, but Transformer is better at capturing distant information. That's why I included both.
\n
Thank you for the explanation. I understand you try to have each layers capture different information.
\nI will try by myself, thanks!
\n\nI included cumulative and average features for all previous problems of the user, and features that greatly increased the score of LightGBM.
\n
Sorry for my unclear asking. I meant your NN seems to have two branches one gets features like content id, lag_time..(left side), the another one gets features like maybe cumulative and average predictions you mentioned (right side). How did you split features into two group?
", "rawMarkdown": "> I think LSTM is better at capturing nearby information than Transformer, but Transformer is better at capturing distant information. That's why I included both.\n\nThank you for the explanation. I understand you try to have each layers capture different information.\nI will try by myself, thanks!\n\n> I included cumulative and average features for all previous problems of the user, and features that greatly increased the score of LightGBM.\n\nSorry for my unclear asking. I meant your NN seems to have two branches one gets features like content id, lag_time..(left side), the another one gets features like maybe cumulative and average predictions you mentioned (right side). How did you split features into two group?", "votes": null }, { "id": "1146941", "postDate": "01/10/2021 07:35:38", "content": "I'm sorry, I misunderstood.
\nThe sequence (content_id, answered correctly etc…) are on the Transformer side, and the other features that are not sequences as explained earlier are on the Dence side. Is that the answer?
Thank you for answering. I thought cumulative features seem kind of sequence features but that is wrong?
", "rawMarkdown": "Thank you for answering. I thought cumulative features seem kind of sequence features but that is wrong?", "votes": null }, { "id": "1147012", "postDate": "01/10/2021 08:43:19", "content": "The cumulative features can be represented to some extent by sequences, but as shown in the figure, the length of my sequences are 100, so it do not represent the entire history of the user. That's why I added the cumulative sum and average features to the dense layer.
", "rawMarkdown": "The cumulative features can be represented to some extent by sequences, but as shown in the figure, the length of my sequences are 100, so it do not represent the entire history of the user. That's why I added the cumulative sum and average features to the dense layer.", "votes": null }, { "id": "1147049", "postDate": "01/10/2021 08:54:33", "content": "I understand well. I will try.
\nThank you very much for kindly explaining and taking your precious time!
Congrats on 13th place @kurupical and @takoihiraokazu and thanks for sharing your solution
", "votes": null, "replies": [] }, { "id": 1144389, "author_name": "theoviel", "author_url": "", "post_date": "01/08/2021 12:39:23", "content": "Very nice write-up, thanks for sharing ! And congratz on the gold finish!
", "votes": null, "replies": [] }, { "id": 1145474, "author_name": "zjjszj2", "author_url": "", "post_date": "01/09/2021 07:04:38", "content": "congratulations！
", "votes": null, "replies": [] }, { "id": 1145485, "author_name": "adityaecdrid", "author_url": "", "post_date": "01/09/2021 07:13:36", "content": "Congratulations on a cool finish!
\nalpha*new_value + (1-alpha)*old_value?Thanks a lot!
", "votes": null, "replies": [ { "id": 1145614, "author_name": "takoihiraokazu", "author_url": "", "post_date": "01/09/2021 08:57:14", "content": "Thank you!
\nI answer question about my side.
distance between the same content_id -> so this is the difference b/w the sequence no when a particular content_id was last seen Vs their current sequence number assigned, right?\nYes.
\ndecayed features (average of answered correctly) -> How did you decay them? Did you do something like alpha*new_value + (1-alpha)*old_value?\nI decayed the value as 0.3 * new_vlaue + (1 - 0.3) * old_value .
Hi @adityaecdrid !
\n\n\nWhat's same_question_index and target?
\n
-> same as takoi's distance between the same content_id
\n\n\nLIT structure in EncoderLayer. What's LIT?
\n
-> Sorry, I forget to explain detail. https://arxiv.org/pdf/2012.14164.pdf. My implements is below. TransformerEncoder is copy from torch.nn.TransformerEncoder, I just replaced the linear part of src2 with a LITLayer.
\nclass LITLayer(nn.Module):\n \"\"\"\n https://arxiv.org/pdf/2012.14164.pdf\n \"\"\"\n def __init__(self, input_dim, embed_dim, dropout, activation):\n super(LITLayer, self).__init__()\n self.input_dim = input_dim\n self.embed_dim = embed_dim\n self.activation = activation\n\n self.linear1 = nn.Linear(input_dim, embed_dim)\n self.lstm = nn.LSTM(embed_dim, embed_dim)\n self.linear2 = nn.Linear(embed_dim, input_dim)\n\n self.norm_lstm = nn.LayerNorm(embed_dim)\n\n self.dropout = nn.Dropout(dropout)\n\n def forward(self, x):\n x = self.linear1(x)\n x = self.activation(x)\n x = self.dropout(x)\n x, _ = self.lstm(x)\n x = self.norm_lstm(x)\n x = self.dropout(x)\n x = self.linear2(x)\n\n return x\n\nclass TransformerEncoderLayer(nn.Module):\n r\"\"\"TransformerEncoderLayer is made up of self-attn and feedforward network.\n This standard encoder layer is based on the paper \"Attention Is All You Need\".\n Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,\n Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in\n Neural Information Processing Systems, pages 6000-6010. Users may modify or implement\n in a different way during application.\n\n Args:\n d_model: the number of expected features in the input (required).\n nhead: the number of heads in the multiheadattention models (required).\n dim_feedforward: the dimension of the feedforward network model (default=2048).\n dropout: the dropout value (default=0.1).\n activation: the activation function of intermediate layer, relu or gelu (default=relu).\n\n Examples::\n >>> encoder_layer = nn.TransformerEncoderLayer(d_model=512, nhead=8)\n >>> src = torch.rand(10, 32, 512)\n >>> out = encoder_layer(src)\n \"\"\"\n\n def __init__(self, d_model, nhead, dim_feedforward=256, dropout=0.1, activation=\"relu\"):\n super(TransformerEncoderLayer, self).__init__()\n self.self_attn = nn.MultiheadAttention(d_model, nhead, dropout=dropout)\n # Implementation of Feedforward model\n\n self.norm1 = nn.LayerNorm(d_model)\n self.norm2 = nn.LayerNorm(d_model)\n self.dropout1 = nn.Dropout(dropout)\n self.dropout2 = nn.Dropout(dropout)\n\n self.activation = _get_activation_fn(activation)\n\n self.lit_layer = LITLayer(input_dim=d_model, embed_dim=dim_feedforward, dropout=dropout, activation=self.activation)\n\n def __setstate__(self, state):\n if 'activation' not in state:\n state['activation'] = nn.ReLU\n super(TransformerEncoderLayer, self).__setstate__(state)\n\n def forward(self, src: torch.Tensor, src_mask: Optional[torch.Tensor] = None, src_key_padding_mask: Optional[torch.Tensor] = None) -> torch.Tensor:\n r\"\"\"Pass the input through the encoder layer.\n\n Args:\n src: the sequence to the encoder layer (required).\n src_mask: the mask for the src sequence (optional).\n src_key_padding_mask: the mask for the src keys per batch (optional).\n\n Shape:\n see the docs in Transformer class.\n \"\"\"\n src2 = self.self_attn(src, src, src, attn_mask=src_mask,\n key_padding_mask=src_key_padding_mask)[0]\n src = src + self.dropout1(src2)\n src = self.norm1(src)\n # src2 = self.linear2(self.dropout(self.activation(self.linear1(src))))\n src2 = self.lit_layer(src)\n src = src + self.dropout2(src2)\n src = self.norm2(src)\n return src\nThanks guys for the quick response!
", "votes": null, "replies": [] } ] }, { "id": 1146884, "author_name": "yaaku123", "author_url": "", "post_date": "01/10/2021 06:23:05", "content": "Thank you for sharing a great solution and congratulations on gold medal!
\nIt might be a dumb question but what makes you stack a LSTM layer on a Transformer layer?
\nI am studying NN and it would be a great help if you guide me to understand how LSTM is superior to Transformer.
And, how did you decide which features put in transformer or dense layer?
", "votes": null, "replies": [ { "id": 1146914, "author_name": "takoihiraokazu", "author_url": "", "post_date": "01/10/2021 06:52:20", "content": "Thanks!
\nIt might be a dumb question but what makes you stack a LSTM layer on a Transformer layer?\nI am studying NN and it would be a great help if you guide me to understand how LSTM is superior to Transformer.\nI think LSTM is better at capturing nearby information than Transformer, but Transformer is better at capturing distant information. That's why I included both.
\nAnd, how did you decide which features put in transformer or dense layer?\nI included cumulative and average features for all previous problems of the user, and features that greatly increased the score of LightGBM.
", "votes": null, "replies": [] }, { "id": 1146929, "author_name": "yaaku123", "author_url": "", "post_date": "01/10/2021 07:15:34", "content": "\n\nI think LSTM is better at capturing nearby information than Transformer, but Transformer is better at capturing distant information. That's why I included both.
\n
Thank you for the explanation. I understand you try to have each layers capture different information.
\nI will try by myself, thanks!
\n\nI included cumulative and average features for all previous problems of the user, and features that greatly increased the score of LightGBM.
\n
Sorry for my unclear asking. I meant your NN seems to have two branches one gets features like content id, lag_time..(left side), the another one gets features like maybe cumulative and average predictions you mentioned (right side). How did you split features into two group?
", "votes": null, "replies": [] }, { "id": 1146941, "author_name": "takoihiraokazu", "author_url": "", "post_date": "01/10/2021 07:35:38", "content": "I'm sorry, I misunderstood.
\nThe sequence (content_id, answered correctly etc…) are on the Transformer side, and the other features that are not sequences as explained earlier are on the Dence side. Is that the answer?
Thank you for answering. I thought cumulative features seem kind of sequence features but that is wrong?
", "votes": null, "replies": [] }, { "id": 1147012, "author_name": "takoihiraokazu", "author_url": "", "post_date": "01/10/2021 08:43:19", "content": "The cumulative features can be represented to some extent by sequences, but as shown in the figure, the length of my sequences are 100, so it do not represent the entire history of the user. That's why I added the cumulative sum and average features to the dense layer.
", "votes": null, "replies": [] }, { "id": 1147049, "author_name": "yaaku123", "author_url": "", "post_date": "01/10/2021 08:54:33", "content": "I understand well. I will try.
\nThank you very much for kindly explaining and taking your precious time!