Hi Kagglers, here is the paper link.
\nPaper Link : https://arxiv.org/abs/2102.05038
\nPlease refer the paper for details.
I wanted to use Transformer but I could not input long history to the model because QK matrix multiplication in Transformer has O(L^2) time complexity when input length is L.
\nMy approach is to use only last input as Query, because I only predict last question's answer correctness per history inputs. It means I will only compare between last question(query) and other questions(key), and not between other questions. It makes QK matrix multiplication in Transformer to have O(L) time complexity(because len(Q)=1, len(K)=L), which allows me to input much longer history.
\nIn final submission, I ensemble 5 models with 1728 length history inputs.
\nI didn't do feature engineering much, since I can use extremely long history, I wanted the model to learn it by itself. 5 input features I used are question id, question part, answer correctness, current question elapsed time, and timestamp difference.
Thank you @limerobot, to share amazing transformer model to approach table data problem in the 3rd place solution of 2019 Data Science Bowl. It was a good motivation to work on transformer encoder. https://www.kaggle.com/c/data-science-bowl-2019/discussion/127891
\nThanks to competition sponsers and organizers for hosting this fun competition.
\nThank you for reading.
", "messages": [ { "id": 1194242, "postDate": "2021-02-10T05:40:56.623Z", "content": "Hi Kagglers, here is the paper link.
\nPaper Link : https://arxiv.org/abs/2102.05038
\nPlease refer the paper for details.
I wanted to use Transformer but I could not input long history to the model because QK matrix multiplication in Transformer has O(L^2) time complexity when input length is L.
\nMy approach is to use only last input as Query, because I only predict last question's answer correctness per history inputs. It means I will only compare between last question(query) and other questions(key), and not between other questions. It makes QK matrix multiplication in Transformer to have O(L) time complexity(because len(Q)=1, len(K)=L), which allows me to input much longer history.
\nIn final submission, I ensemble 5 models with 1728 length history inputs.
\nI didn't do feature engineering much, since I can use extremely long history, I wanted the model to learn it by itself. 5 input features I used are question id, question part, answer correctness, current question elapsed time, and timestamp difference.
Thank you @limerobot, to share amazing transformer model to approach table data problem in the 3rd place solution of 2019 Data Science Bowl. It was a good motivation to work on transformer encoder. https://www.kaggle.com/c/data-science-bowl-2019/discussion/127891
\nThanks to competition sponsers and organizers for hosting this fun competition.
\nThank you for reading.
", "rawMarkdown": " Hi Kagglers, here is the paper link.\nPaper Link : https://arxiv.org/abs/2102.05038\nPlease refer the paper for details.\n# Summary\nI wanted to use Transformer but I could not input long history to the model because QK matrix multiplication in Transformer has O(L^2) time complexity when input length is L. \n# \nMy approach is to use only last input as Query, because I only predict last question's answer correctness per history inputs. It means I will only compare between last question(query) and other questions(key), and not between other questions. It makes QK matrix multiplication in Transformer to have O(L) time complexity(because len(Q)=1, len(K)=L), which allows me to input much longer history. \n# \nIn final submission, I ensemble 5 models with 1728 length history inputs.\nI didn't do feature engineering much, since I can use extremely long history, I wanted the model to learn it by itself. 5 input features I used are question id, question part, answer correctness, current question elapsed time, and timestamp difference.\n# Acknowledgement\nThank you @limerobot, to share amazing transformer model to approach table data problem in the 3rd place solution of 2019 Data Science Bowl. It was a good motivation to work on transformer encoder. https://www.kaggle.com/c/data-science-bowl-2019/discussion/127891 \n# \nThanks to competition sponsers and organizers for hosting this fun competition.\n\n# \nThank you for reading.", "votes": 170 }, { "id": 1195053, "postDate": "2021-02-10T14:28:02.963Z", "content": "Thank you @keetar for sharing this approach. Very very nice, and great to see such long histories used. If you have published code somewhere, it would be great to see the encoder layer set up - if it is not public, no worries. Well done again, well deserved.
", "rawMarkdown": "Thank you @keetar for sharing this approach. Very very nice, and great to see such long histories used. If you have published code somewhere, it would be great to see the encoder layer set up - if it is not public, no worries. Well done again, well deserved. ", "votes": 3, "replies": [ { "id": 1195096, "postDate": "2021-02-10T14:56:00.353Z", "content": "Hi @darraghdog, thank you.
\nActually, it's really simple to implement the model.
\nI only changed the one line from normal transformer encoder.
\nIn encoder layer,
\nI changed
\n attn_output, _ = self.mha(x, x, x[:,:,:], mask) #mha : multi head attention
\nto
\n attn_output, _ = self.mha(x, x, x[:,-1:,:], mask)
\nHope it helps!
Hey @keetar Congrats.
\nI had a couple of doubts, can you please answer them-
Assume that I represent Query, Key and Value as Q, K, and V respectively, and let's assume that I have an input sequence with-
\nD- Dimension of model
\nS - Sequence Length of input
According to the paper first we need to multiply the Q and K transpose so -
\nresult1 = Q * K ---> (S , D) * ( D ,S) ---> (S,S) dimension
Now, we divide by sq root of a scaling factor and then apply softmax and get result2 with dim (S,S).
\nresult2 = softmax( result1 / sqRoot(ScalingFactor) )
Now we multiply the result2 with our Values V-->
\nFinalResult = result2 * V ---> i.e. (S,S) * (S,D) ---> (S,D) dimension
My doubts-
\n1. If you keep seq len = 1 in the Query Q, then the final output will be having dim (1, D). \nThis is just a single feature with dim D, then how are you passing it through the LSTM? \n( Or you are assuming the single feature with dim D as a sequence and passing it through the LSTM ? If so then can you tell the intuition behind this please )
2. In one of the comments you mentioned the following -
\nattn_output, _ = self.mha(x, x, x[:,-1:,:], mask)
\nCan you please elaborate more on the code, What is x and what is x[:,-1:,:] ?\nWhich one of those 3 is Query Q ?\n( If the code above is used from torch.nn.MultiheadAttention then in that case you are passing the values V as x[:,-1:,:], which will raise an error. )
Thanks
", "rawMarkdown": "Hey @keetar Congrats. \nI had a couple of doubts, can you please answer them-\n\nAssume that I represent Query, Key and Value as Q, K, and V respectively, and let's assume that I have an input sequence with- \n**D**- Dimension of model \n**S** - Sequence Length of input \n\nAccording to the [paper](https://arxiv.org/abs/1706.03762) first we need to multiply the Q and K transpose so - \n**result1 = Q * K ---> (S , D) * ( D ,S) ---> (S,S)** dimension \n\nNow, we divide by sq root of a scaling factor and then apply softmax and get result2 with dim (S,S). \n**result2 = softmax( result1 / sqRoot(ScalingFactor) )** \n\nNow we multiply the result2 with our Values V--> \n**FinalResult = result2 * V ---> i.e. (S,S) * (S,D) ---> (S,D)** dimension\n\nMy doubts- \n**1. If you keep seq len = 1 in the Query Q, then the final output will be having dim (1, D). \nThis is just a single feature with dim D, then how are you passing it through the LSTM? \n( Or you are assuming the single feature with dim D as a sequence and passing it through the LSTM ? If so then can you tell the intuition behind this please )** \n\n**2. In one of the comments you mentioned the following -**\n`attn_output, _ = self.mha(x, x, x[:,-1:,:], mask)` \n**Can you please elaborate more on the code, What is x and what is x[:,-1:,:] ?\nWhich one of those 3 is Query Q ?\n( If the code above is used from [torch.nn.MultiheadAttention](https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html) then in that case you are passing the values V as x[:,-1:,:], which will raise an error. )** \n\nThanks\n\n \n \n\n", "votes": 4, "replies": [ { "id": 1201657, "postDate": "2021-02-15T14:51:42.267Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1201901, "postDate": "2021-02-15T18:19:58.613Z", "content": "Based on my understanding of transformers and what @keetar said, it looks like x[:,-1:,:] is the query. I don't think he uses torch's MHA. I've seen other MHAs where the prototype is KVQ instead of QKV.
In one of the reference implementations for Transformers, x represents Q and has a shape of (batch_size, sequence_len, embedding_dim_size). By tweaking x to x[:,-1,:] you change the sequence_length to 1.
Same doubts, especial the first doubt
", "rawMarkdown": "Same doubts, especial the first doubt" }, { "id": 1202115, "postDate": "2021-02-15T21:10:03.073Z", "content": "Hi,
\n1. If you keep seq len = 1 in the Query Q, then the final output will be having dim (1, D). This is just a single feature with dim D, then how are you passing it through the LSTM? ( Or you are assuming the single feature with dim D as a sequence and passing it through the LSTM ? If so then can you tell the intuition behind this please )\nIn transformer encoder, there's a residual addition, which adds 'multi head attention output' to original input. In multi head attention, I use Query with len=1 to get some speed advantages in QK matrix multiplication. In result, the output of multi head attention becomes (batch size, 1, d_model) instead of (batch size, sequence length, d_model). So, for residual addition in transformer encoder, operands are original input with shape (batch size, sequence length, d_model) and multi head attention output with shape (batch size, 1, d_model). Because two shapes are different, I add multi head attention output to every point of original input.(like broadcasting) Therefore, transformer encoder output is (batch size, sequence length, d_model).
\n2. In one of the comments you mentioned the following -\nattn_output, _ = self.mha(x, x, x[:,-1:,:], mask)\nCan you please elaborate more on the code, What is x and what is x[:,-1:,:] ? Which one of those 3 is Query Q ? ( If the code above is used from torch.nn.MultiheadAttention then in that case you are passing the values V as x[:,-1:,:], which will raise an error. )\nx is an original input to encoder. x's shape is (batch size, seq_len, d_model). The order of mha(multihead attention) parameters is v,k,q,mask.
", "rawMarkdown": "Hi,\n```\n1. If you keep seq len = 1 in the Query Q, then the final output will be having dim (1, D). This is just a single feature with dim D, then how are you passing it through the LSTM? ( Or you are assuming the single feature with dim D as a sequence and passing it through the LSTM ? If so then can you tell the intuition behind this please )\n```\n# \nIn transformer encoder, there's a residual addition, which adds 'multi head attention output' to original input. In multi head attention, I use Query with len=1 to get some speed advantages in QK matrix multiplication. In result, the output of multi head attention becomes (batch size, 1, d_model) instead of (batch size, sequence length, d_model). So, for residual addition in transformer encoder, operands are original input with shape (batch size, sequence length, d_model) and multi head attention output with shape (batch size, 1, d_model). Because two shapes are different, I add multi head attention output to every point of original input.(like broadcasting) Therefore, transformer encoder output is (batch size, sequence length, d_model).\n\n```\n2. In one of the comments you mentioned the following -\nattn_output, _ = self.mha(x, x, x[:,-1:,:], mask)\nCan you please elaborate more on the code, What is x and what is x[:,-1:,:] ? Which one of those 3 is Query Q ? ( If the code above is used from torch.nn.MultiheadAttention then in that case you are passing the values V as x[:,-1:,:], which will raise an error. )\n```\nx is an original input to encoder. x's shape is (batch size, seq_len, d_model). The order of mha(multihead attention) parameters is v,k,q,mask. ", "votes": 12 }, { "id": 1204329, "postDate": "2021-02-16T04:16:51.840Z", "content": "Thanks @keetar for the reply. Once again congrats, its a great solution.
", "rawMarkdown": "Thanks @keetar for the reply. Once again congrats, its a great solution.", "votes": 1 }, { "id": 1204407, "postDate": "2021-02-16T05:53:12.453Z", "content": "Thank you :)
", "rawMarkdown": "Thank you :)", "votes": 1 }, { "id": 1332599, "postDate": "2021-06-02T07:52:51.703Z", "content": "Thanks for sharing this amazing solution! One question I wanna ask is you add 'last query embedding' to the original input(which contains the last query), which seems like adding a 'scaler' to all sequences, does this make sense?
", "rawMarkdown": "Thanks for sharing this amazing solution! One question I wanna ask is you add 'last query embedding' to the original input(which contains the last query), which seems like adding a 'scaler' to all sequences, does this make sense?" } ] }, { "id": 1485709, "postDate": "2021-08-22T10:56:56.997Z", "content": "Pytorch Implementation of 1st place solution - https://github.com/arshadshk/Last_Query_Transformer_RNN-PyTorch
\n( Let me know if any changes are needed or any bugs present )
Thank you keetar for your great solution, it was really fun to compete with you. Finally I lost, though.
\nIt is not surprising to use only the last input as query, because I heard @its7171, @takoihiraokazu and @nadare did the same thing and I suppose many people did this, as it's easy to implement.
\nHowever, they said using the last input as query is very time-consuming (I remember @its7171 said training takes 1 week). How long is your training time, why is it so fast and what is your trick? Did you use TPU?
My another question is about the sequence length and the input features. I was very surprised that your sequence length is 1728 and you used only 5 features. How did you notice sequence length is so important? Did you throw away other useful features to make the sequence length longer? I would like to know your journey to L = 1728 :)
", "rawMarkdown": "Thank you keetar for your great solution, it was really fun to compete with you. Finally I lost, though.\nIt is not surprising to use only the last input as query, because I heard @its7171, @takoihiraokazu and @nadare did the same thing and I suppose many people did this, as it's easy to implement.\nHowever, they said using the last input as query is very time-consuming (I remember @its7171 said training takes 1 week). How long is your training time, why is it so fast and what is your trick? Did you use TPU? \n\nMy another question is about the sequence length and the input features. I was very surprised that your sequence length is 1728 and you used only 5 features. How did you notice sequence length is so important? Did you throw away other useful features to make the sequence length longer? I would like to know your journey to L = 1728 :)", "votes": 2, "replies": [ { "id": 1195672, "postDate": "2021-02-11T01:51:18.893Z", "content": "Hi @mamasinkgs, thank you.
\nI'm surprised to hear many people tried this, as I saw none. Could you provide some links?
Yes, it's not clearly written in many people's solution, but I believe this was done by many people. I noticed my japanese friends did this in a meeting after the competition ends. For example, please look at the takoi's figure of their solution. it uses only the last query for training. tito also said only trained and predicted for answered_correctly of last question of the sequence. in their solution, which is based on the public kernel. Moreover, I didn't use the last query technique in training phase (instead I used masking), but I used last query technique in the inference phase, as can be seen in my kernel. The corresponding part of my code is here.
#key and value process 2\n query_clone = query.clone()\n query_clone[token_idx, -2] = 0\n memory_cat = torch.cat([query_clone, ohe_explanation, ohe_correctness, \n normed_elapsed.unsqueeze(2), ohe_user_answer], dim = 2)\n memory = self.norm4(self.linear8(self.dropout(F.relu(self.linear7(memory_cat)))))\n memory = memory[:, :-1]\n\n #transpose\n query = query.transpose(0, 1)\n memory = memory.transpose(0, 1)\n spc_query = query[-1:]\n\n #MyEncoder\n out = self.MyEncoderLayer1(spc_query, memory, memory, None, padding_mask)\n out = self.MyEncoderLayer2(out, memory, memory, None, padding_mask)\n out = self.MyEncoderLayer3(out, memory, memory, None, padding_mask)\n out = self.MyEncoderLayer4(out, memory, memory, None, padding_mask).squeeze(0)\nI think they're totally different.
\nNormally, even if you train and predict for only last input, you don't change query vector in transformer to have length 1. In usual case, all query and value and key are set to have length L.(when input length is L). Because that's how transformer works. I believe the cases you mentioned have the same configurations.
\nChanging query vector in transformer to have length 1 changes the inner workings of transformer significantly. When multi head attention output is added to each inputs, last query's attention output is added to each inputs like constant, instead of each query's attention output. I found that it does not hurt the performance much, while making QK matrix multiplication in transformer to have O(L) instead of O(L^2).
\nTo answer why my model is faster than others who also used last query for training, it's simply because my QK matrix multiplication is O(L) and others is O(L^2).
Yes, as you say, in usual all query and key have length L and thus the matrix multiplication is O(L^2). But I think my inference code above is O(L), because the size of the query is 1 as can be seen from spc_query = query[-1:] for the input of my custom transformer, so I just guessed other competitors who used the last input for training did the same thing like my inference phase (i.e. O(L) matrix multiplication). But it may be wrong, as I don't perfectly understand others' solution. Anyway, you are the kaggler I respect the most and your solution is really great without any doubt :) Thank you again!
Thank you :) And congratulations on your successful finish.
", "rawMarkdown": "Thank you :) And congratulations on your successful finish.", "votes": 1 } ] }, { "id": 1233189, "postDate": "2021-03-10T08:19:17.753Z", "content": "Hi, I had some doubts about your inputs, can you please anwer it.
\nCategorical embedding is used for first three features, and continuous embedding is used for last two continuous features.
\nbut how you make continuous embedding?
'Timestamp difference' feature indicates the difference from the past question timestamp to the current question timestamp, and it is clipped by maximum value, 3 day.
\nhow you fill the first 'Timestamp difference' in a sequence?
Thanks!
", "rawMarkdown": "Hi, I had some doubts about your inputs, can you please anwer it.\n\n1. **Categorical embedding is used for first three features, and continuous embedding is used for last two continuous features.**\nbut how you make continuous embedding?\n\n2. **'Timestamp difference' feature indicates the difference from the past question timestamp to the current question timestamp, and it is clipped by maximum value, 3 day.**\nhow you fill the first 'Timestamp difference' in a sequence?\n\nThanks!\n\n", "votes": 1 }, { "id": 1195385, "postDate": "2021-02-10T18:51:53.747Z", "content": "Cong! What a simple but sweet solution:)
", "rawMarkdown": "Cong! What a simple but sweet solution:)", "votes": 1 }, { "id": 1194577, "postDate": "2021-02-10T08:53:02.880Z", "content": "Congrats for this really elegant and original solution !
", "rawMarkdown": "Congrats for this really elegant and original solution !", "votes": 1, "replies": [ { "id": 1195111, "postDate": "2021-02-10T15:04:36.273Z", "content": "Thank you :)
", "rawMarkdown": "Thank you :)", "votes": 1 } ] }, { "id": 1349555, "postDate": "2021-06-14T22:25:27.257Z", "content": "Hi,
\ndoes the LSTM use a mask as well? It looks like Tensorflow's LSTM accepts mask.
\nthe residual addition in the transformer includes the padding tokens which we may want to ignore in the lstm.
great point
", "rawMarkdown": "great point" }, { "id": 1219606, "postDate": "2021-02-27T01:55:10.553Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1216721, "postDate": "2021-02-24T12:53:15.533Z", "content": "Good work! Thanks for sharing.
", "rawMarkdown": "Good work! Thanks for sharing." } ], "comments": [ { "id": 1195053, "author_name": "Darragh", "author_url": "", "post_date": "2021-02-10T14:28:02.963000", "content": "Thank you @keetar for sharing this approach. Very very nice, and great to see such long histories used. If you have published code somewhere, it would be great to see the encoder layer set up - if it is not public, no worries. Well done again, well deserved.
", "votes": 3, "replies": [ { "id": 1195096, "author_name": "keetar", "author_url": "", "post_date": "2021-02-10T14:56:00.353000", "content": "Hi @darraghdog, thank you.
\nActually, it's really simple to implement the model.
\nI only changed the one line from normal transformer encoder.
\nIn encoder layer,
\nI changed
\n attn_output, _ = self.mha(x, x, x[:,:,:], mask) #mha : multi head attention
\nto
\n attn_output, _ = self.mha(x, x, x[:,-1:,:], mask)
\nHope it helps!
Hey @keetar Congrats.
\nI had a couple of doubts, can you please answer them-
Assume that I represent Query, Key and Value as Q, K, and V respectively, and let's assume that I have an input sequence with-
\nD- Dimension of model
\nS - Sequence Length of input
According to the paper first we need to multiply the Q and K transpose so -
\nresult1 = Q * K ---> (S , D) * ( D ,S) ---> (S,S) dimension
Now, we divide by sq root of a scaling factor and then apply softmax and get result2 with dim (S,S).
\nresult2 = softmax( result1 / sqRoot(ScalingFactor) )
Now we multiply the result2 with our Values V-->
\nFinalResult = result2 * V ---> i.e. (S,S) * (S,D) ---> (S,D) dimension
My doubts-
\n1. If you keep seq len = 1 in the Query Q, then the final output will be having dim (1, D). \nThis is just a single feature with dim D, then how are you passing it through the LSTM? \n( Or you are assuming the single feature with dim D as a sequence and passing it through the LSTM ? If so then can you tell the intuition behind this please )
2. In one of the comments you mentioned the following -
\nattn_output, _ = self.mha(x, x, x[:,-1:,:], mask)
\nCan you please elaborate more on the code, What is x and what is x[:,-1:,:] ?\nWhich one of those 3 is Query Q ?\n( If the code above is used from torch.nn.MultiheadAttention then in that case you are passing the values V as x[:,-1:,:], which will raise an error. )
Thanks
", "votes": 4, "replies": [ { "id": 1201657, "author_name": "", "author_url": "", "post_date": "2021-02-15T14:51:42.267000", "content": "", "votes": 0, "replies": [] }, { "id": 1201901, "author_name": "Philip Dhingra", "author_url": "", "post_date": "2021-02-15T18:19:58.613000", "content": "Based on my understanding of transformers and what @keetar said, it looks like x[:,-1:,:] is the query. I don't think he uses torch's MHA. I've seen other MHAs where the prototype is KVQ instead of QKV.
In one of the reference implementations for Transformers, x represents Q and has a shape of (batch_size, sequence_len, embedding_dim_size). By tweaking x to x[:,-1,:] you change the sequence_length to 1.
Same doubts, especial the first doubt
", "votes": 0, "replies": [] }, { "id": 1202115, "author_name": "keetar", "author_url": "", "post_date": "2021-02-15T21:10:03.073000", "content": "Hi,
\n1. If you keep seq len = 1 in the Query Q, then the final output will be having dim (1, D). This is just a single feature with dim D, then how are you passing it through the LSTM? ( Or you are assuming the single feature with dim D as a sequence and passing it through the LSTM ? If so then can you tell the intuition behind this please )\nIn transformer encoder, there's a residual addition, which adds 'multi head attention output' to original input. In multi head attention, I use Query with len=1 to get some speed advantages in QK matrix multiplication. In result, the output of multi head attention becomes (batch size, 1, d_model) instead of (batch size, sequence length, d_model). So, for residual addition in transformer encoder, operands are original input with shape (batch size, sequence length, d_model) and multi head attention output with shape (batch size, 1, d_model). Because two shapes are different, I add multi head attention output to every point of original input.(like broadcasting) Therefore, transformer encoder output is (batch size, sequence length, d_model).
\n2. In one of the comments you mentioned the following -\nattn_output, _ = self.mha(x, x, x[:,-1:,:], mask)\nCan you please elaborate more on the code, What is x and what is x[:,-1:,:] ? Which one of those 3 is Query Q ? ( If the code above is used from torch.nn.MultiheadAttention then in that case you are passing the values V as x[:,-1:,:], which will raise an error. )\nx is an original input to encoder. x's shape is (batch size, seq_len, d_model). The order of mha(multihead attention) parameters is v,k,q,mask.
", "votes": 12, "replies": [] }, { "id": 1204329, "author_name": "Arshad Shaikh", "author_url": "", "post_date": "2021-02-16T04:16:51.840000", "content": "Thanks @keetar for the reply. Once again congrats, its a great solution.
", "votes": 1, "replies": [] }, { "id": 1204407, "author_name": "keetar", "author_url": "", "post_date": "2021-02-16T05:53:12.453000", "content": "Thank you :)
", "votes": 1, "replies": [] }, { "id": 1332599, "author_name": "Gerryl", "author_url": "", "post_date": "2021-06-02T07:52:51.703000", "content": "Thanks for sharing this amazing solution! One question I wanna ask is you add 'last query embedding' to the original input(which contains the last query), which seems like adding a 'scaler' to all sequences, does this make sense?
", "votes": 0, "replies": [] } ] }, { "id": 1485709, "author_name": "Arshad Shaikh", "author_url": "", "post_date": "2021-08-22T10:56:56.997000", "content": "Pytorch Implementation of 1st place solution - https://github.com/arshadshk/Last_Query_Transformer_RNN-PyTorch
\n( Let me know if any changes are needed or any bugs present )
Thank you keetar for your great solution, it was really fun to compete with you. Finally I lost, though.
\nIt is not surprising to use only the last input as query, because I heard @its7171, @takoihiraokazu and @nadare did the same thing and I suppose many people did this, as it's easy to implement.
\nHowever, they said using the last input as query is very time-consuming (I remember @its7171 said training takes 1 week). How long is your training time, why is it so fast and what is your trick? Did you use TPU?
My another question is about the sequence length and the input features. I was very surprised that your sequence length is 1728 and you used only 5 features. How did you notice sequence length is so important? Did you throw away other useful features to make the sequence length longer? I would like to know your journey to L = 1728 :)
", "votes": 2, "replies": [ { "id": 1195672, "author_name": "keetar", "author_url": "", "post_date": "2021-02-11T01:51:18.893000", "content": "Hi @mamasinkgs, thank you.
\nI'm surprised to hear many people tried this, as I saw none. Could you provide some links?
Yes, it's not clearly written in many people's solution, but I believe this was done by many people. I noticed my japanese friends did this in a meeting after the competition ends. For example, please look at the takoi's figure of their solution. it uses only the last query for training. tito also said only trained and predicted for answered_correctly of last question of the sequence. in their solution, which is based on the public kernel. Moreover, I didn't use the last query technique in training phase (instead I used masking), but I used last query technique in the inference phase, as can be seen in my kernel. The corresponding part of my code is here.
#key and value process 2\n query_clone = query.clone()\n query_clone[token_idx, -2] = 0\n memory_cat = torch.cat([query_clone, ohe_explanation, ohe_correctness, \n normed_elapsed.unsqueeze(2), ohe_user_answer], dim = 2)\n memory = self.norm4(self.linear8(self.dropout(F.relu(self.linear7(memory_cat)))))\n memory = memory[:, :-1]\n\n #transpose\n query = query.transpose(0, 1)\n memory = memory.transpose(0, 1)\n spc_query = query[-1:]\n\n #MyEncoder\n out = self.MyEncoderLayer1(spc_query, memory, memory, None, padding_mask)\n out = self.MyEncoderLayer2(out, memory, memory, None, padding_mask)\n out = self.MyEncoderLayer3(out, memory, memory, None, padding_mask)\n out = self.MyEncoderLayer4(out, memory, memory, None, padding_mask).squeeze(0)\nI think they're totally different.
\nNormally, even if you train and predict for only last input, you don't change query vector in transformer to have length 1. In usual case, all query and value and key are set to have length L.(when input length is L). Because that's how transformer works. I believe the cases you mentioned have the same configurations.
\nChanging query vector in transformer to have length 1 changes the inner workings of transformer significantly. When multi head attention output is added to each inputs, last query's attention output is added to each inputs like constant, instead of each query's attention output. I found that it does not hurt the performance much, while making QK matrix multiplication in transformer to have O(L) instead of O(L^2).
\nTo answer why my model is faster than others who also used last query for training, it's simply because my QK matrix multiplication is O(L) and others is O(L^2).
Yes, as you say, in usual all query and key have length L and thus the matrix multiplication is O(L^2). But I think my inference code above is O(L), because the size of the query is 1 as can be seen from spc_query = query[-1:] for the input of my custom transformer, so I just guessed other competitors who used the last input for training did the same thing like my inference phase (i.e. O(L) matrix multiplication). But it may be wrong, as I don't perfectly understand others' solution. Anyway, you are the kaggler I respect the most and your solution is really great without any doubt :) Thank you again!
Thank you :) And congratulations on your successful finish.
", "votes": 1, "replies": [] } ] }, { "id": 1233189, "author_name": "hlf", "author_url": "", "post_date": "2021-03-10T08:19:17.753000", "content": "Hi, I had some doubts about your inputs, can you please anwer it.
\nCategorical embedding is used for first three features, and continuous embedding is used for last two continuous features.
\nbut how you make continuous embedding?
'Timestamp difference' feature indicates the difference from the past question timestamp to the current question timestamp, and it is clipped by maximum value, 3 day.
\nhow you fill the first 'Timestamp difference' in a sequence?
Thanks!
", "votes": 1, "replies": [] }, { "id": 1195385, "author_name": "HenryHZY", "author_url": "", "post_date": "2021-02-10T18:51:53.747000", "content": "Cong! What a simple but sweet solution:)
", "votes": 1, "replies": [] }, { "id": 1194577, "author_name": "TitiTest", "author_url": "", "post_date": "2021-02-10T08:53:02.880000", "content": "Congrats for this really elegant and original solution !
", "votes": 1, "replies": [ { "id": 1195111, "author_name": "keetar", "author_url": "", "post_date": "2021-02-10T15:04:36.273000", "content": "Thank you :)
", "votes": 1, "replies": [] } ] }, { "id": 1349555, "author_name": "Pierre", "author_url": "", "post_date": "2021-06-14T22:25:27.257000", "content": "Hi,
\ndoes the LSTM use a mask as well? It looks like Tensorflow's LSTM accepts mask.
\nthe residual addition in the transformer includes the padding tokens which we may want to ignore in the lstm.
great point
", "votes": 0, "replies": [] }, { "id": 1219606, "author_name": "", "author_url": "", "post_date": "2021-02-27T01:55:10.553000", "content": "", "votes": 0, "replies": [] }, { "id": 1216721, "author_name": "ZavodRobotov", "author_url": "", "post_date": "2021-02-24T12:53:15.533000", "content": "Good work! Thanks for sharing.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1194242": " Hi Kagglers, here is the paper link.\nPaper Link : https://arxiv.org/abs/2102.05038\nPlease refer the paper for details.\n# Summary\nI wanted to use Transformer but I could not input long history to the model because QK matrix multiplication in Transformer has O(L^2) time complexity when input length is L. \n# \nMy approach is to use only last input as Query, because I only predict last question's answer correctness per history inputs. It means I will only compare between last question(query) and other questions(key), and not between other questions. It makes QK matrix multiplication in Transformer to have O(L) time complexity(because len(Q)=1, len(K)=L), which allows me to input much longer history. \n# \nIn final submission, I ensemble 5 models with 1728 length history inputs.\nI didn't do feature engineering much, since I can use extremely long history, I wanted the model to learn it by itself. 5 input features I used are question id, question part, answer correctness, current question elapsed time, and timestamp difference.\n# Acknowledgement\nThank you @limerobot, to share amazing transformer model to approach table data problem in the 3rd place solution of 2019 Data Science Bowl. It was a good motivation to work on transformer encoder. https://www.kaggle.com/c/data-science-bowl-2019/discussion/127891 \n# \nThanks to competition sponsers and organizers for hosting this fun competition.\n\n# \nThank you for reading.", "1195053": "Thank you @keetar for sharing this approach. Very very nice, and great to see such long histories used. If you have published code somewhere, it would be great to see the encoder layer set up - if it is not public, no worries. Well done again, well deserved. ", "1200641": "Hey @keetar Congrats. \nI had a couple of doubts, can you please answer them-\n\nAssume that I represent Query, Key and Value as Q, K, and V respectively, and let's assume that I have an input sequence with- \n**D**- Dimension of model \n**S** - Sequence Length of input \n\nAccording to the [paper](https://arxiv.org/abs/1706.03762) first we need to multiply the Q and K transpose so - \n**result1 = Q * K ---> (S , D) * ( D ,S) ---> (S,S)** dimension \n\nNow, we divide by sq root of a scaling factor and then apply softmax and get result2 with dim (S,S). \n**result2 = softmax( result1 / sqRoot(ScalingFactor) )** \n\nNow we multiply the result2 with our Values V--> \n**FinalResult = result2 * V ---> i.e. (S,S) * (S,D) ---> (S,D)** dimension\n\nMy doubts- \n**1. If you keep seq len = 1 in the Query Q, then the final output will be having dim (1, D). \nThis is just a single feature with dim D, then how are you passing it through the LSTM? \n( Or you are assuming the single feature with dim D as a sequence and passing it through the LSTM ? If so then can you tell the intuition behind this please )** \n\n**2. In one of the comments you mentioned the following -**\n`attn_output, _ = self.mha(x, x, x[:,-1:,:], mask)` \n**Can you please elaborate more on the code, What is x and what is x[:,-1:,:] ?\nWhich one of those 3 is Query Q ?\n( If the code above is used from [torch.nn.MultiheadAttention](https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html) then in that case you are passing the values V as x[:,-1:,:], which will raise an error. )** \n\nThanks\n\n \n \n\n", "1485709": "Pytorch Implementation of 1st place solution - https://github.com/arshadshk/Last_Query_Transformer_RNN-PyTorch \n( Let me know if any changes are needed or any bugs present )", "1195536": "Thank you keetar for your great solution, it was really fun to compete with you. Finally I lost, though.\nIt is not surprising to use only the last input as query, because I heard @its7171, @takoihiraokazu and @nadare did the same thing and I suppose many people did this, as it's easy to implement.\nHowever, they said using the last input as query is very time-consuming (I remember @its7171 said training takes 1 week). How long is your training time, why is it so fast and what is your trick? Did you use TPU? \n\nMy another question is about the sequence length and the input features. I was very surprised that your sequence length is 1728 and you used only 5 features. How did you notice sequence length is so important? Did you throw away other useful features to make the sequence length longer? I would like to know your journey to L = 1728 :)", "1233189": "Hi, I had some doubts about your inputs, can you please anwer it.\n\n1. **Categorical embedding is used for first three features, and continuous embedding is used for last two continuous features.**\nbut how you make continuous embedding?\n\n2. **'Timestamp difference' feature indicates the difference from the past question timestamp to the current question timestamp, and it is clipped by maximum value, 3 day.**\nhow you fill the first 'Timestamp difference' in a sequence?\n\nThanks!\n\n", "1195385": "Cong! What a simple but sweet solution:)", "1194577": "Congrats for this really elegant and original solution !", "1349555": "Hi,\ndoes the LSTM use a mask as well? It looks like Tensorflow's LSTM accepts mask.\nthe residual addition in the transformer includes the padding tokens which we may want to ignore in the lstm.", "1230528": "great point", "1219606": "", "1216721": "Good work! Thanks for sharing." } }