![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F762938%2F39ec58b701e6d37701cc794763d3a473%2F2nd_place.png?generation=1610213712807228&alt=media\")
Thank you all teams who competed with me, all the people who participated in this competition, and the organizer that hosts such a great competition with the well-designed API!
\nCongrats @keetar, who defeats me and becomes the winner in this competition.
I'm happy because it's my first time I get solo prize!
\nIt's my 4th kaggle competition and it was fun to compete with my past teammates ( @nyanpn, @pocketsuteado) and people who I competed with past competitions (e.g. @aerdem4, @its7171).
\nHere, I will explain the summary of my model and my features.
I uploaded 2 kaggle notebooks for the explanation. As the ensemble is not so important in my solution, I will only explain my single model.
\n
my model is similar to SAKT, with 400 sequence length, 512 dimension, 4 nheads.
\nI don't use lecture information for the input of transformer model, which I guess is why I lost in this competition. For the query and key/value of SAKT-like model, I used LSTM-encoded features, whose input is as follows.
\"Query\" features
\ncontent_id
\npart
\ntags
\nnormalized timedelta
\nnormalized log timestamp
\ncorrect answer
\ntask_container_id delta
\ncontent_type_id delta
\nnormalized absolute position
\"Memory\" features
\nexplanation
\ncorrectness
\nnormalized elapsed time
\nuser_answer
I tried a very precise indexing/masking technique to avoid data leakage in training process, I guess which is partly why I became 2nd place. OK, suppose a very simple model of the sequence length = 5, and a task_container_id history of a specific user (without lecture) is like this.
\n[0, 0, 0, 1, 1, 2, 2, 3, 3, 4, 5, 8, 7, 7, 6]
\nAs there is 15 measurements in this user, I made 15/5 = 3 training samples and loss mask.
(1) input task_container_id: [pad, pad, pad, pad, pad, 0, 0, 0, 1, 1]
\n(1) loss_mask: [False, False, False, False, False, True, True, True, True, True]
\n(2) input task_container_id: [0, 0, 0, 1, 1, 2, 2, 3, 3, 4]
\n(2) loss_mask: [False, False, False, False, False, True, True, True, True, True]
\n(3) input task_container_id: [2, 2, 3, 3, 4, 5, 8, 7, 7, 6]
\n(3) loss_mask: [False, False, False, False, False, True, True, True, True, True]
In this competition, the handling of the task_container_id is very important, as it is not allowed to use \"memory\" features of the same task container id to avoid leakage.
\nSo, after applying LSTM to the features, such fancy indexing is required to avoid the leakage for 3 training samples, where -1 means this position can't attend any position.
(1) indices: [-1, -1, -1, -1, -1, 4, 4, 4, 7, 7]
\n(2) indices: [-1, -1, -1, 2, 2, 4, 4, 6, 6, 8]
\n(3) indices: [-1, -1, 1, 1, 3, 4, 5, 6, 6, 8]
To get this indices very fast in the training/inference phase, I wrote such cython (#1) function (vectorized implementation):
\n%%cython\nimport numpy as np\ncimport numpy as np\ncpdef np.ndarray[int] cget_memory_indices(np.ndarray task):\n cdef Py_ssize_t n = task.shape[1]\n cdef np.ndarray[int, ndim = 2] res = np.zeros_like(task, dtype = np.int32)\n cdef np.ndarray[int] tmp_counter = np.full(task.shape[0], -1, dtype = np.int32)\n cdef np.ndarray[int] u_counter = np.full(task.shape[0], task.shape[1] - 1, dtype = np.int32)\n for i in range(n):\n res[:, i] = u_counter\n tmp_counter += 1\n if i != n - 1:\n mask = (task[:, i] != task[:, i + 1])\n u_counter[mask] = tmp_counter[mask]\n return res\nAfter applying such fancy indexing to the output of LSTM features, I concatenated them with \"Query\" features and apply MLP, then we can get query for SAKT model. Then, I obtained key/value for SAKT model by applying MLP to query concatenated with \"Memory\" features.
\nTo train SAKT-like model, precise memory masking is also required to avoid leakage. I used 3D attention mask of torch.nn.MultiheadAttention. It should be noted torch.repeat_interleave must be leveraged to make 3D mask (batchsize * nhead, sequence length, sequence length).
\nThe memory mask for these 3 samples are like this. Here, 1 means True and 0 means False. Please remember the documentation says
attn_mask ensure that position i is allowed to attend the unmasked positions. If a BoolTensor is provided, positions with True is not allowed to attend while False values will be unchanged.\n(1) memory_mask:
\narray([[1, 1, 1, 1, 1, 1, 1, 1, 1, 0],\n [1, 1, 1, 1, 1, 1, 1, 1, 1, 0],\n [1, 1, 1, 1, 1, 1, 1, 1, 1, 0],\n [1, 1, 1, 1, 1, 1, 1, 1, 1, 0],\n [1, 1, 1, 1, 1, 1, 1, 1, 1, 0],\n [0, 0, 0, 0, 0, 1, 1, 1, 1, 1],\n [0, 0, 0, 0, 0, 1, 1, 1, 1, 1],\n [0, 0, 0, 0, 0, 1, 1, 1, 1, 1],\n [1, 1, 1, 0, 0, 0, 0, 0, 1, 1],\n [1, 1, 1, 0, 0, 0, 0, 0, 1, 1]])\n(2) memory_mask:
\narray([[1, 1, 1, 1, 1, 1, 1, 1, 1, 0],\n [1, 1, 1, 1, 1, 1, 1, 1, 1, 0],\n [1, 1, 1, 1, 1, 1, 1, 1, 1, 0],\n [0, 0, 0, 1, 1, 1, 1, 1, 1, 1],\n [0, 0, 0, 1, 1, 1, 1, 1, 1, 1],\n [0, 0, 0, 0, 0, 1, 1, 1, 1, 1],\n [0, 0, 0, 0, 0, 1, 1, 1, 1, 1],\n [1, 1, 0, 0, 0, 0, 0, 1, 1, 1],\n [1, 1, 0, 0, 0, 0, 0, 1, 1, 1],\n [1, 1, 1, 1, 0, 0, 0, 0, 0, 1]])\n(3) memory_mask:
\narray([[1, 1, 1, 1, 1, 1, 1, 1, 1, 0],\n [1, 1, 1, 1, 1, 1, 1, 1, 1, 0],\n [0, 0, 1, 1, 1, 1, 1, 1, 1, 1],\n [0, 0, 1, 1, 1, 1, 1, 1, 1, 1],\n [0, 0, 0, 0, 1, 1, 1, 1, 1, 1],\n [0, 0, 0, 0, 0, 1, 1, 1, 1, 1],\n [1, 0, 0, 0, 0, 0, 1, 1, 1, 1],\n [1, 1, 0, 0, 0, 0, 0, 1, 1, 1],\n [1, 1, 0, 0, 0, 0, 0, 1, 1, 1],\n [1, 1, 1, 1, 0, 0, 0, 0, 0, 1]])\nTo make this memory mask very fast in the training phase, I wrote such cython (#2) function (not vectorized implementation):
\n%%cython\nimport numpy as np\ncimport numpy as np\ncpdef np.ndarray[int] cget_memory_mask(np.ndarray task, int n_length):\n cdef Py_ssize_t n = task.shape[0]\n cdef np.ndarray[int, ndim = 2] res = np.full((n, n), 1, dtype = np.int32)\n cdef int tmp_counter = 0\n cdef int u_counter = 0\n for i in range(n):\n tmp_counter += 1\n if i == n - 1 or task[i] != task[i + 1]:\n res[i - tmp_counter + 1 : i + 1, :u_counter] = 0\n if u_counter == 0:\n res[i - tmp_counter + 1 : i + 1, n - 1] = 0\n if u_counter > n_length:\n res[i - tmp_counter + 1: i + 1, :(u_counter - n_length)] = 1\n u_counter += tmp_counter\n tmp_counter = 0\n return res\nIn the inference phase, memory mask is not required. For more details, please look at my single model notebook.
\nTransformer is great, but it suffers from a problem that the sequence length cannot be infinite. I mean, it cannot consider the information of very old samples. To tackle this problem and to leverage the lecture information, I made simple features and concatenated them with the output features of SAKT-like model, and applied MLP and sigmoid.
\nI uploaded the names of 90 features as attachments. For the implementation of these features, I didn't use any pd.merge or df.join and most of the implementation are done using numpy. To make user-content features, I used scipy.sparse.lil_matrix to spare memory usage. I think my feature is not so good as other competitors (about 0.795 when using GBDT), but still improved score 0.001 ~ 0.002.
\nI made some tricky features (e.g. obtained by SVD), but it did not improve the score of NN model (improved GBDT model, though.), probably because the information of NN features includes that of such tricky features.
My CV strategy is completely different from tito( @its7171 )'s one. First, I probed the number of new users in test set and I found there are about 7000 new users. as we know there is 2.5M rows in test set and we know the average length of the history of all users, we can estimate the number of rows of the new users and the number of rows of the existing users. After the calculation, I found
\nthe number of rows of new users (i.e. user split): the number of rows of existing users (i.e. timeseries split) = 2 : 1.\nSo, For validation, I decided to use 1M rows for timeseries split and 2M rows for user split.
\nWhen making validation set of timeseries split, I was so careful that leakage can't happen. I mean, my training dataset and validation dataset never shares same task_container_id for a given user.
I was very sad to see the private score bug problem of kaggle. Of course this competition is very great, but I think this competition could have been one of the most successful competition in kaggle without this problem.
\nAnyway, I really enjoyed my 4th kaggle competition. I'll continue kaggle and will surely become the winner in the next competition!
P.S. The word memory_mask may be confusing because it is different from pytorch Transformer's memory_mask.
\nThe word memory_mask is more like mask in pytorch's TransformerEncoder.
\nThe reason my word is confusing is, I used Encoder-Decoder model at first, then gave up it and started using Encoder-only model, but I didn't change the function names. :(
Thank you all teams who competed with me, all the people who participated in this competition, and the organizer that hosts such a great competition with the well-designed API!
\nCongrats @keetar, who defeats me and becomes the winner in this competition.
I'm happy because it's my first time I get solo prize!
\nIt's my 4th kaggle competition and it was fun to compete with my past teammates ( @nyanpn, @pocketsuteado) and people who I competed with past competitions (e.g. @aerdem4, @its7171).
\nHere, I will explain the summary of my model and my features.
I uploaded 2 kaggle notebooks for the explanation. As the ensemble is not so important in my solution, I will only explain my single model.
\n
my model is similar to SAKT, with 400 sequence length, 512 dimension, 4 nheads.
\nI don't use lecture information for the input of transformer model, which I guess is why I lost in this competition. For the query and key/value of SAKT-like model, I used LSTM-encoded features, whose input is as follows.
\"Query\" features
\ncontent_id
\npart
\ntags
\nnormalized timedelta
\nnormalized log timestamp
\ncorrect answer
\ntask_container_id delta
\ncontent_type_id delta
\nnormalized absolute position
\"Memory\" features
\nexplanation
\ncorrectness
\nnormalized elapsed time
\nuser_answer
I tried a very precise indexing/masking technique to avoid data leakage in training process, I guess which is partly why I became 2nd place. OK, suppose a very simple model of the sequence length = 5, and a task_container_id history of a specific user (without lecture) is like this.
\n[0, 0, 0, 1, 1, 2, 2, 3, 3, 4, 5, 8, 7, 7, 6]
\nAs there is 15 measurements in this user, I made 15/5 = 3 training samples and loss mask.
(1) input task_container_id: [pad, pad, pad, pad, pad, 0, 0, 0, 1, 1]
\n(1) loss_mask: [False, False, False, False, False, True, True, True, True, True]
\n(2) input task_container_id: [0, 0, 0, 1, 1, 2, 2, 3, 3, 4]
\n(2) loss_mask: [False, False, False, False, False, True, True, True, True, True]
\n(3) input task_container_id: [2, 2, 3, 3, 4, 5, 8, 7, 7, 6]
\n(3) loss_mask: [False, False, False, False, False, True, True, True, True, True]
In this competition, the handling of the task_container_id is very important, as it is not allowed to use \"memory\" features of the same task container id to avoid leakage.
\nSo, after applying LSTM to the features, such fancy indexing is required to avoid the leakage for 3 training samples, where -1 means this position can't attend any position.
(1) indices: [-1, -1, -1, -1, -1, 4, 4, 4, 7, 7]
\n(2) indices: [-1, -1, -1, 2, 2, 4, 4, 6, 6, 8]
\n(3) indices: [-1, -1, 1, 1, 3, 4, 5, 6, 6, 8]
To get this indices very fast in the training/inference phase, I wrote such cython (#1) function (vectorized implementation):
\n%%cython\nimport numpy as np\ncimport numpy as np\ncpdef np.ndarray[int] cget_memory_indices(np.ndarray task):\n cdef Py_ssize_t n = task.shape[1]\n cdef np.ndarray[int, ndim = 2] res = np.zeros_like(task, dtype = np.int32)\n cdef np.ndarray[int] tmp_counter = np.full(task.shape[0], -1, dtype = np.int32)\n cdef np.ndarray[int] u_counter = np.full(task.shape[0], task.shape[1] - 1, dtype = np.int32)\n for i in range(n):\n res[:, i] = u_counter\n tmp_counter += 1\n if i != n - 1:\n mask = (task[:, i] != task[:, i + 1])\n u_counter[mask] = tmp_counter[mask]\n return res\nAfter applying such fancy indexing to the output of LSTM features, I concatenated them with \"Query\" features and apply MLP, then we can get query for SAKT model. Then, I obtained key/value for SAKT model by applying MLP to query concatenated with \"Memory\" features.
\nTo train SAKT-like model, precise memory masking is also required to avoid leakage. I used 3D attention mask of torch.nn.MultiheadAttention. It should be noted torch.repeat_interleave must be leveraged to make 3D mask (batchsize * nhead, sequence length, sequence length).
\nThe memory mask for these 3 samples are like this. Here, 1 means True and 0 means False. Please remember the documentation says
attn_mask ensure that position i is allowed to attend the unmasked positions. If a BoolTensor is provided, positions with True is not allowed to attend while False values will be unchanged.\n(1) memory_mask:
\narray([[1, 1, 1, 1, 1, 1, 1, 1, 1, 0],\n [1, 1, 1, 1, 1, 1, 1, 1, 1, 0],\n [1, 1, 1, 1, 1, 1, 1, 1, 1, 0],\n [1, 1, 1, 1, 1, 1, 1, 1, 1, 0],\n [1, 1, 1, 1, 1, 1, 1, 1, 1, 0],\n [0, 0, 0, 0, 0, 1, 1, 1, 1, 1],\n [0, 0, 0, 0, 0, 1, 1, 1, 1, 1],\n [0, 0, 0, 0, 0, 1, 1, 1, 1, 1],\n [1, 1, 1, 0, 0, 0, 0, 0, 1, 1],\n [1, 1, 1, 0, 0, 0, 0, 0, 1, 1]])\n(2) memory_mask:
\narray([[1, 1, 1, 1, 1, 1, 1, 1, 1, 0],\n [1, 1, 1, 1, 1, 1, 1, 1, 1, 0],\n [1, 1, 1, 1, 1, 1, 1, 1, 1, 0],\n [0, 0, 0, 1, 1, 1, 1, 1, 1, 1],\n [0, 0, 0, 1, 1, 1, 1, 1, 1, 1],\n [0, 0, 0, 0, 0, 1, 1, 1, 1, 1],\n [0, 0, 0, 0, 0, 1, 1, 1, 1, 1],\n [1, 1, 0, 0, 0, 0, 0, 1, 1, 1],\n [1, 1, 0, 0, 0, 0, 0, 1, 1, 1],\n [1, 1, 1, 1, 0, 0, 0, 0, 0, 1]])\n(3) memory_mask:
\narray([[1, 1, 1, 1, 1, 1, 1, 1, 1, 0],\n [1, 1, 1, 1, 1, 1, 1, 1, 1, 0],\n [0, 0, 1, 1, 1, 1, 1, 1, 1, 1],\n [0, 0, 1, 1, 1, 1, 1, 1, 1, 1],\n [0, 0, 0, 0, 1, 1, 1, 1, 1, 1],\n [0, 0, 0, 0, 0, 1, 1, 1, 1, 1],\n [1, 0, 0, 0, 0, 0, 1, 1, 1, 1],\n [1, 1, 0, 0, 0, 0, 0, 1, 1, 1],\n [1, 1, 0, 0, 0, 0, 0, 1, 1, 1],\n [1, 1, 1, 1, 0, 0, 0, 0, 0, 1]])\nTo make this memory mask very fast in the training phase, I wrote such cython (#2) function (not vectorized implementation):
\n%%cython\nimport numpy as np\ncimport numpy as np\ncpdef np.ndarray[int] cget_memory_mask(np.ndarray task, int n_length):\n cdef Py_ssize_t n = task.shape[0]\n cdef np.ndarray[int, ndim = 2] res = np.full((n, n), 1, dtype = np.int32)\n cdef int tmp_counter = 0\n cdef int u_counter = 0\n for i in range(n):\n tmp_counter += 1\n if i == n - 1 or task[i] != task[i + 1]:\n res[i - tmp_counter + 1 : i + 1, :u_counter] = 0\n if u_counter == 0:\n res[i - tmp_counter + 1 : i + 1, n - 1] = 0\n if u_counter > n_length:\n res[i - tmp_counter + 1: i + 1, :(u_counter - n_length)] = 1\n u_counter += tmp_counter\n tmp_counter = 0\n return res\nIn the inference phase, memory mask is not required. For more details, please look at my single model notebook.
\nTransformer is great, but it suffers from a problem that the sequence length cannot be infinite. I mean, it cannot consider the information of very old samples. To tackle this problem and to leverage the lecture information, I made simple features and concatenated them with the output features of SAKT-like model, and applied MLP and sigmoid.
\nI uploaded the names of 90 features as attachments. For the implementation of these features, I didn't use any pd.merge or df.join and most of the implementation are done using numpy. To make user-content features, I used scipy.sparse.lil_matrix to spare memory usage. I think my feature is not so good as other competitors (about 0.795 when using GBDT), but still improved score 0.001 ~ 0.002.
\nI made some tricky features (e.g. obtained by SVD), but it did not improve the score of NN model (improved GBDT model, though.), probably because the information of NN features includes that of such tricky features.
My CV strategy is completely different from tito( @its7171 )'s one. First, I probed the number of new users in test set and I found there are about 7000 new users. as we know there is 2.5M rows in test set and we know the average length of the history of all users, we can estimate the number of rows of the new users and the number of rows of the existing users. After the calculation, I found
\nthe number of rows of new users (i.e. user split): the number of rows of existing users (i.e. timeseries split) = 2 : 1.\nSo, For validation, I decided to use 1M rows for timeseries split and 2M rows for user split.
\nWhen making validation set of timeseries split, I was so careful that leakage can't happen. I mean, my training dataset and validation dataset never shares same task_container_id for a given user.
I was very sad to see the private score bug problem of kaggle. Of course this competition is very great, but I think this competition could have been one of the most successful competition in kaggle without this problem.
\nAnyway, I really enjoyed my 4th kaggle competition. I'll continue kaggle and will surely become the winner in the next competition!
P.S. The word memory_mask may be confusing because it is different from pytorch Transformer's memory_mask.
\nThe word memory_mask is more like mask in pytorch's TransformerEncoder.
\nThe reason my word is confusing is, I used Encoder-Decoder model at first, then gave up it and started using Encoder-only model, but I didn't change the function names. :(
Good luck WINNING your next competition.
", "rawMarkdown": "Good luck WINNING your next competition.", "votes": 4 }, { "id": 1147926, "postDate": "2021-01-10T20:03:30.223Z", "content": "Thanks for the nicely detailed write-up and congratz for the impressive finish!
", "rawMarkdown": "Thanks for the nicely detailed write-up and congratz for the impressive finish!", "votes": 1 }, { "id": 1146546, "postDate": "2021-01-09T21:03:17.123Z", "content": "\n\nIn my implementation, SAINT didn't work well. In my opinion, it's logically very hard to implement SAINT without data leakage in training due to task_container_id.
\n
Couldn't you control that with tgt_mask from PyTorch's implementation ?
Yes we can control tgt_mask in decoder. but if we use the same input of the decoder as SAINT+ paper in this image, I think the problem cannot be fixed just by controlling tgt_mask, because the query of decoder produces leakage (i.e. uses the information of the sample in the same task_container_id). In my understanding, the data leakage caused by query is unavoidable by controlling masking, while the data leakage caused by key/value is avoidable by masking.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F762938%2Fb29c0c24c02cb5b71707c2d378bc5a13%2Finput_of_saint.jpg?generation=1610226842294243&alt=media\")
If I understand what you are saying, you mean for questions in the same bundle that are under prediction (but during training), the corresponding input in the decoder will see the information (i.e. user correctness) in the same bundle (but prior the current position). However, while during inference, it can only see this information at the places before the current bundle.
\nI am using the encoder-decoder approach, and during the inference time, I replace the values by the value obtained at the first question in the bundle under prediction. Basically, it treats the question in the bundle as separate questions, and use the history before them.
\nAnyway, the best I can get is 0.802, so potentially there are some problems as you mentioned.
", "rawMarkdown": "If I understand what you are saying, you mean for questions in the same bundle that are under prediction (but during training), the corresponding input in the decoder will see the information (i.e. user correctness) in the same bundle (but prior the current position). However, while during inference, it can only see this information at the places before the current bundle.\n\nI am using the encoder-decoder approach, and during the inference time, I replace the values by the value obtained at the first question in the bundle under prediction. Basically, it treats the question in the bundle as separate questions, and use the history before them.\n\nAnyway, the best I can get is 0.802, so potentially there are some problems as you mentioned." }, { "id": 1146587, "postDate": "2021-01-09T21:44:02.640Z", "content": "Yes of course the leakage doesn't happen during inference, but I think it's problematic that leakage happens in training phase, because the model can learn meaningless patterns during training.
\nI replace the values by the value obtained at the first question in the bundle under prediction.
\nHmm, I didn't notice this idea but this looks great. Why don't you try this idea in the training phase? I think it is possible by fancy indexing.
I'm still not seeing which way the leak could go :?
", "rawMarkdown": "I'm still not seeing which way the leak could go :?" }, { "id": 1146610, "postDate": "2021-01-09T22:02:29.533Z", "content": "I didn't think of the training will be problematic. I am aware of some kind of inconsistency between decoder training / inference input by this approach, but it is a comprising approach. Treating each question in the bundle under the current inference time step is not difficult, although still need to be careful.
\nTo do so in training, I feel it becomes more difficult. Because we are not just dealing the last bundle in the truncated sequence, we extract exact one question from each budle in that sequence. Maybe not so difficult, but I kind feel we also loss information that given by bundle. Anyway, without trying, I can't say if we avoid leakage or lossing info if we apply this approach to training
", "rawMarkdown": "I didn't think of the training will be problematic. I am aware of some kind of inconsistency between decoder training / inference input by this approach, but it is a comprising approach. Treating each question in the bundle under the current inference time step is not difficult, although still need to be careful.\n\nTo do so in training, I feel it becomes more difficult. Because we are not just dealing the last bundle in the truncated sequence, we extract exact one question from each budle in that sequence. Maybe not so difficult, but I kind feel we also loss information that given by bundle. Anyway, without trying, I can't say if we avoid leakage or lossing info if we apply this approach to training", "votes": 1 }, { "id": 1146611, "postDate": "2021-01-09T22:05:18.623Z", "content": "@bacicnikola
\nFor example, suppose task_container_id of a specific user is [0, 0, 0, 1, 1, 2, 2, 3, 3, 4, 5, 8, 7, 7, 6]. Then,
\nposition [0, 1, 2] cannot use any information.
\nposition [3, 4] can use the information of [0, 1, 2].
\nposition [5, 6] can use the information of [0, 1, 2, 3, 4].
\nposition [7, 8] can use the information of [0, 1, 2, 3, 4, 5, 6].
However, when simply using the input of the SAINT paper, position 1 can use the information (e.g. correctness) of position 0 and position 2 can use the information of position [0, 1]. I think this is problematic as the model uses the information that should not be utilized, during training.
", "rawMarkdown": "@bacicnikola\nFor example, suppose task_container_id of a specific user is [0, 0, 0, 1, 1, 2, 2, 3, 3, 4, 5, 8, 7, 7, 6]. Then, \nposition [0, 1, 2] cannot use any information.\nposition [3, 4] can use the information of [0, 1, 2].\nposition [5, 6] can use the information of [0, 1, 2, 3, 4].\nposition [7, 8] can use the information of [0, 1, 2, 3, 4, 5, 6].\n\nHowever, when simply using the input of the SAINT paper, position 1 can use the information (e.g. correctness) of position 0 and position 2 can use the information of position [0, 1]. I think this is problematic as the model uses the information that should not be utilized, during training.", "votes": 3 }, { "id": 1146612, "postDate": "2021-01-09T22:06:17.623Z", "content": "Not clear to me about what kind of leakage neither, but at least some form of inconsistency exist
", "rawMarkdown": "Not clear to me about what kind of leakage neither, but at least some form of inconsistency exist" }, { "id": 1146617, "postDate": "2021-01-09T22:12:44.273Z", "content": "Yes, it can be called \"incosistency\", as the problem doesn't happen at least in inference phase.
", "rawMarkdown": "Yes, it can be called \"incosistency\", as the problem doesn't happen at least in inference phase. ", "votes": 1 }, { "id": 1146618, "postDate": "2021-01-09T22:14:48.053Z", "content": "To be clear, my approach mentioned above , for decoder, only applies to the last bundle in the truncated sequence. The bundles before it, even they appear in inference time, but already being predicted, the special processing is not applied to them anymore
", "rawMarkdown": "To be clear, my approach mentioned above , for decoder, only applies to the last bundle in the truncated sequence. The bundles before it, even they appear in inference time, but already being predicted, the special processing is not applied to them anymore", "votes": 1 }, { "id": 1146621, "postDate": "2021-01-09T22:19:53.660Z", "content": "To do so in training, I feel it becomes more difficult. Because we are not just dealing the last bundle in the truncated sequence, we extract exact one question from each budle in that sequence. Maybe not so difficult, but I kind feel we also loss information that given by bundle.\nNow I understand what you mean. it seems the problem cannot be solved by simply using fancy indexing. Hmm…
", "rawMarkdown": "```\nTo do so in training, I feel it becomes more difficult. Because we are not just dealing the last bundle in the truncated sequence, we extract exact one question from each budle in that sequence. Maybe not so difficult, but I kind feel we also loss information that given by bundle.\n```\nNow I understand what you mean. it seems the problem cannot be solved by simply using fancy indexing. Hmm...", "votes": 1 }, { "id": 1146623, "postDate": "2021-01-09T22:20:17.523Z", "content": "@mamasinkgs
\nYes, I agree. But what confusses me is why tgt_mask can't control that. In example you mentioned, it should look like this:
[0, 0, 0, 1, 1, 2, 2, 3, 3, 4, 5, 8, 7, 7, 6]
\n(1 mask, 0 no mask)
\n1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 (for 0 position)
\n1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 (for 1 position)
\n1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 (for 2 position)
\n0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 (for 3 position)
\n0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 (for 4 position)
\n0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 (for 5 position)
\n0 0 0 0 0 1 1 1 1 1 1 1 1 1 (for 6 position)
\netc.
Although in my solution, I'm only using encoder part of the transformer, but that shouldn't make any difference.
", "rawMarkdown": "@mamasinkgs \nYes, I agree. But what confusses me is why tgt_mask can't control that. In example you mentioned, it should look like this:\n\n[0, 0, 0, 1, 1, 2, 2, 3, 3, 4, 5, 8, 7, 7, 6]\n\n(1 mask, 0 no mask)\n\n1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 (for 0 position)\n1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 (for 1 position)\n1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 (for 2 position)\n0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 (for 3 position)\n0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 (for 4 position)\n0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 (for 5 position)\n0 0 0 0 0 1 1 1 1 1 1 1 1 1 (for 6 position)\netc.\n\nAlthough in my solution, I'm only using encoder part of the transformer, but that shouldn't make any difference.\n" }, { "id": 1146630, "postDate": "2021-01-09T22:28:16.987Z", "content": "@bacicnikola
\nWhen using encoder-only model, there is no problem. However, when using decoder of SAINT+ model, the sequence is shifted like the image above. So, by simply using tgt_mask you showed (actually the mask you showed is inherently same as the mask I showed in my solution), the problem cannot be solved.
@mamasinkgs
\nYes, I see now. My implementation of this was only partly good because even though I did mask positions from the same task_container, responses were shifted to the right and question could always see previous response, which could be from the same task_container (since I used encoder only, position could always see itself)
\nI think I should've put responses into a encoder, and the rest of the features into decoder and control task_container with memory_mask.
Thank you for your time!
", "rawMarkdown": "@mamasinkgs\nYes, I see now. My implementation of this was only partly good because even though I did mask positions from the same task_container, responses were shifted to the right and question could always see previous response, which could be from the same task_container (since I used encoder only, position could always see itself)\nI think I should've put responses into a encoder, and the rest of the features into decoder and control task_container with memory_mask.\n\nThank you for your time!", "votes": 2 }, { "id": 1146647, "postDate": "2021-01-09T22:57:28.953Z", "content": "@bacicnikola ， just a remark, the mask you shows for decoder also has an issue. For example , at a place, it can't look at the current position. Therefore, position embedding of that place becomes no meaning while predicting that place, if we use the usual way of position embedding
", "rawMarkdown": "@bacicnikola ， just a remark, the mask you shows for decoder also has an issue. For example , at a place, it can't look at the current position. Therefore, position embedding of that place becomes no meaning while predicting that place, if we use the usual way of position embedding" }, { "id": 1146651, "postDate": "2021-01-09T23:06:13.917Z", "content": "@yihdarshieh Yes, you're right. But I couldn't get anything out of positions anyway.
", "rawMarkdown": "@yihdarshieh Yes, you're right. But I couldn't get anything out of positions anyway.", "votes": 1 } ] }, { "id": 1146767, "postDate": "2021-01-10T03:13:26.980Z", "content": "Big congrats to you on 2nd solo prize. Thanks for the detail writeup, very intelligence solution @mamasinkgs
", "rawMarkdown": "Big congrats to you on 2nd solo prize. Thanks for the detail writeup, very intelligence solution @mamasinkgs ", "votes": 2 }, { "id": 1146439, "postDate": "2021-01-09T19:19:19.120Z", "content": "@mamasinkgs , thank you for sharing. See you in the next one!
\nAbout your CPython implementation for getting the masks, it is amazing, you must be an advanced Python programmer!
\nFor me, I use TensorFlow, and it is quite straightforward and fast to get these mask by using tf operations, like
def get_causal_attention_mask(nd, ns, dtype, only_before):\n \"\"\"\n 1's in the lower triangle, counting from the lower right corner. Same as tf.matrix_band_part(tf.ones([nd, ns]),\n -1, ns-nd), but doesn't produce garbage on TPUs.\n \"\"\"\n\n # Remark: Think `nd` as the number of queries and `ns` as the number of keys.\n # In encoder-decoder case, the queries are the decoder features and the keys are the encoder features.\n\n i = tf.range(nd)[:, tf.newaxis] # repeat along dim 1\n j = tf.range(ns) # repeat along dim 0 \n m = i >= (j - ns + nd) + tf.cast(only_before, dtype=tf.int32)\n\n return tf.cast(m, dtype)\n\ndef get_attention_mask_from_timestamp_batch(timestamp_tensors, dtype, only_before):\n \"\"\"\n Args:\n timestamp_tensors: 2-D tf.int32 tensor, representing a batch of sequences of non-decreasing timestamps.\n\n Returns:\n attention_mask: 3-D tf.int32 tensor of shape = [batch_size, query_len, key_len], consisting of 0 and 1.\n Here `query_len` and `key_len` are actually `seq_len`. It should be reshpaed, when used to calculate \n attention scores, to [batch_size, nb_attn_head, query_len, key_len].\n \"\"\"\n\n t = timestamp_tensors\n\n batch_size = tf.math.reduce_sum(tf.ones_like(t[:, :1], dtype=tf.int32))\n seq_len = tf.math.reduce_sum(tf.ones_like(t[:1, :], dtype=tf.int32))\n\n x = tf.broadcast_to(t[:, :, tf.newaxis], shape=[batch_size, seq_len, seq_len]) # repeat along dim 2\n y = tf.broadcast_to(t[:, tf.newaxis, :], shape=[batch_size, seq_len, seq_len]) + tf.cast(only_before, dtype=tf.int64) # repeat along dim 1\n\n m = x >= y\n\n return tf.cast(m, dtype)\nThanks! I'm not familiar with tensorflow, but I'll try it in future competitions.
", "rawMarkdown": "Thanks! I'm not familiar with tensorflow, but I'll try it in future competitions.", "votes": 1 } ] }, { "id": 1590856, "postDate": "2021-11-21T17:37:10.020Z", "content": "Hi, I have few questions
\nHi Nimra,
\n@mamasinkgs thanks for your reply.
\nActually I have run your notebook. Everything seems good, no errors. But from where can I see results? Do submission.csv file is getting updated every time I run the notebook?
\nAlso one more question, I am a newbie in machine learning and python. Can you suggest me what courses should I take or any thing else that can help me understand code?
@mamasinkgs , Very Nice Approach of Using Transformer . Few Questions , I wanted to ask , if you don't mind sharing
\n1) How did you estimate New Users in Private Test set by probing ?
\n2) Is memory masking neccessary while training Model , what happens if We train Model without masking ?
1) I uploaded https://www.kaggle.com/mamasinkgs/submission-with-7000-limit?scriptVersionId=45368594, so please check it. I tried binary search by changing the limit.
\n2) I thought models will learn some useless patterns. But, judging from others' solutions, precise masking is not necessarily needed. I think it depends on architectures, though.
Okay Thanks .
", "rawMarkdown": "Okay Thanks ." } ] }, { "id": 1152889, "postDate": "2021-01-14T14:46:23.917Z", "content": "Very nice approach, I was looking forward to this. A side question though, in this comment it was inferred that you were not using a transformer. When did you shift to this? or was it a transformer all along?
", "rawMarkdown": "Very nice approach, I was looking forward to this. A side question though, in this [comment](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/195527#1071365) it was inferred that you were not using a transformer. When did you shift to this? or was it a transformer all along?", "replies": [ { "id": 1153011, "postDate": "2021-01-14T15:24:59.587Z", "content": "Nice question! In the beginning of the competition, I used LightGBM as usual and got to 0.792 (1st place) soon. Then, I started using Transformer by reading this blog, soon I got to 0.801 using Transformer and 0.803 by ensembling GBDT and Transformer. I remember I wrote this comment then. (so, The fact is I was using both GBDT and Transformer). After that, my Transformer improved a lot and I decided to throw away GBDT and use my hand-crafted features for the additional input of Transformer. \"Is Attention All You Need?\" means I was wondering whether Transformer is better than GBDT when I wrote this comment.
", "rawMarkdown": "Nice question! In the beginning of the competition, I used LightGBM as usual and got to 0.792 (1st place) soon. Then, I started using Transformer by reading this [blog](http://jalammar.github.io/illustrated-transformer/), soon I got to 0.801 using Transformer and 0.803 by ensembling GBDT and Transformer. I remember I wrote this comment then. (so, The fact is I was using both GBDT and Transformer). After that, my Transformer improved a lot and I decided to throw away GBDT and use my hand-crafted features for the additional input of Transformer. \"Is Attention All You Need?\" means I was wondering whether Transformer is better than GBDT when I wrote this comment.", "votes": 3 } ] }, { "id": 1150057, "postDate": "2021-01-12T10:28:19.493Z", "content": "a dump question from a newbie. Which software do you use to draw that model structure graph? thank you !
", "rawMarkdown": "a dump question from a newbie. Which software do you use to draw that model structure graph? thank you !", "replies": [ { "id": 1150308, "postDate": "2021-01-12T14:02:33.230Z", "content": "I simply used powerpoint.
", "rawMarkdown": "I simply used powerpoint.", "votes": 1 } ] }, { "id": 1148892, "postDate": "2021-01-11T13:07:43.350Z", "content": "\n\nI uploaded the names of 90 features as attachments.
\n
Where did you upload the attachments?
", "rawMarkdown": "> I uploaded the names of 90 features as attachments.\n\nWhere did you upload the attachments?", "replies": [ { "id": 1148939, "postDate": "2021-01-11T13:44:16.993Z", "content": "Thank you, I uploaded it now!
", "rawMarkdown": "Thank you, I uploaded it now!" } ] }, { "id": 1148231, "postDate": "2021-01-11T02:46:41.373Z", "content": "Congratulations and thank for detailed explanation!
\nMay I ask a question about how to deal with the tags features?
\nThank you.
Thank you! I used one-hot encoding (like multilabel problem) for tags and concatenate them with other features, then applied MLP.
", "rawMarkdown": "Thank you! I used one-hot encoding (like multilabel problem) for tags and concatenate them with other features, then applied MLP.", "votes": 2 } ] }, { "id": 1146837, "postDate": "2021-01-10T05:09:53.260Z", "content": "Congrats and thank you for your detailed explanation.
\nI have two questions about your solution.
I think if we did not carefully split train/valid datasets and evaluate the model, CV score would be decreased when using task_container-aware masking, as the model cannot utilize the intra-container leakage you mentioned.
\nOne of the major CV strategies in this competition by @its7171 did not care task_container_id, though it split train/valid datasets by timestamp + random int. [update] I confirmed that there are no such data at all.
", "rawMarkdown": "Congrats and thank you for your detailed explanation.\nI have two questions about your solution.\n- What is your cross-validation and model evaluating strategy?\n- Did your task_container-aware masking improved CV score (and how much)?\n\nI think if we did not carefully split train/valid datasets and evaluate the model, CV score would be decreased when using task_container-aware masking, as the model cannot utilize the intra-container leakage you mentioned. \n\n[One of the major CV strategies in this competition](https://www.kaggle.com/its7171/cv-strategy) by @its7171 did not care task_container_id, though it split train/valid datasets by timestamp + random int. ~~This CV strategy might split some task_containers into train/valid datasets, and might cause some intra-container leakage. (I should note that I used these datasets with good CV-LB relationship)~~ [update] I confirmed that there are no such data at all.", "replies": [ { "id": 1146874, "postDate": "2021-01-10T06:03:54.940Z", "content": "What is your cross-validation and model evaluating strategy?
\nMy CV strategy is completely different from tito's one. First, I probed the number of new users in test set and I found there is about 7000 new users. as we know there is 2.5M rows in test set and we know the average length of the history of all users, we can estimate the number of rows of the new users and the number of rows of the existing users. After the calculation, I found
\n# rows for user split (i.e. new users): # rows for timeseries split (i.e. existing users) = 2 : 1.
\nSo, For validation, I decided to use 1M rows for timeseries split and 2M rows for user split.
\nWhen making validation set of timeseries split, I was so careful that inter-container leakage problem can't happen. I mean, my training dataset and validation dataset never shares same task_container_id for a specific user.
Did your task_container-aware masking improved CV score (and how much)?
\nI don't know how much it improves CV/LB score, because I was really careful about inter-container leakage problem from the beginning and have never tried task_container-unaware masking and indexing (I hate leakage). So, there is no evidence that shows task_container-aware masking is important. That's just my guess from LB score.
Thank you for your clarification.
\nIt is so sophisticated CV strategy and very logical consequence.
Hi @tomooinubushi,
\n\n\nThis CV strategy might split some task_containers into train/valid datasets
\n
Please check whether this kind of data is exists or not by your self if you think so.
\nI believe there is no such data. :)
\nSince:
\nI am sorry. I should have checked it before making a sloppy speculation.
\nI found there are no data which share the same timestamp or task_container_id between train/valid datasets. So, there are no intra-container leakage at all.
\nI found some of task_container_ids in valid dataset are lower than those in train dataset, which might be caused by known inconsistency between timestamp and task_container_id. I am not sure how to deal with these data when creating attention masks.
\nAnyway, thank you for pointing out my mistake @its7171 and congrats to @mamasinkgs.
Thank you for your confirmation!
", "rawMarkdown": "Thank you for your confirmation!", "votes": 1 } ] }, { "id": 1146747, "postDate": "2021-01-10T02:35:49.287Z", "content": "thanks for sharing! a great job! this is a big project!
", "rawMarkdown": "thanks for sharing! a great job! this is a big project!" }, { "id": 1146571, "postDate": "2021-01-09T21:34:07.693Z", "content": "@mamasisking , congratulations! Can you please also share info about the resources you put into the competition? hardware, amount of effort, time etc.
", "rawMarkdown": "@mamasisking , congratulations! Can you please also share info about the resources you put into the competition? hardware, amount of effort, time etc. ", "replies": [ { "id": 1146590, "postDate": "2021-01-09T21:46:33.083Z", "content": "I use V100 * 4 (VRAM 64GB) and training takes about 30h. I think I spent more than 500 hours for this competition. I did nothing other than this competition since 10/16.
", "rawMarkdown": "I use V100 * 4 (VRAM 64GB) and training takes about 30h. I think I spent more than 500 hours for this competition. I did nothing other than this competition since 10/16.", "votes": 9 }, { "id": 1147123, "postDate": "2021-01-10T09:55:45.060Z", "content": "@mamasinkgs You don't have a full-time job currently …? I can only work on this competition after my daily job …😭
", "rawMarkdown": "@mamasinkgs You don't have a full-time job currently ...? I can only work on this competition after my daily job ...😭", "votes": 1 }, { "id": 1147357, "postDate": "2021-01-10T13:05:35.330Z", "content": "I'm a student 😄
", "rawMarkdown": "I'm a student 😄", "votes": 3 }, { "id": 1167076, "postDate": "2021-01-24T02:52:43.677Z", "content": "Tesla V100 seems not cheap. That was a lot investment money and time wise!
", "rawMarkdown": "Tesla V100 seems not cheap. That was a lot investment money and time wise!" } ] }, { "id": 1146503, "postDate": "2021-01-09T20:03:23.923Z", "content": "@mamasinkgs , sorry to bother, but could you share your insight about this statement
\nIn my implementation, SAINT didn't work well. In my opinion, it's logically very hard to implement SAINT without data leakage in training due to task_container_id.\nWhy you think task_container_id cause data leakage during training, and what kind of leakage?
I mean, in my opinion, the input of the decoder of SAINT+ architecture cannot be implemented without any information loss or any data leakage, due to task_container_id. ![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F762938%2F8467453b525282d6754cbac4f2c9473e%2Finput_of_saint.jpg?generation=1610223284475779&alt=media\")
Good luck WINNING your next competition.
", "votes": 4, "replies": [] }, { "id": 1147926, "author_name": "Theo Viel", "author_url": "", "post_date": "2021-01-10T20:03:30.223000", "content": "Thanks for the nicely detailed write-up and congratz for the impressive finish!
", "votes": 1, "replies": [] }, { "id": 1146546, "author_name": "Nikola Bacic", "author_url": "", "post_date": "2021-01-09T21:03:17.123000", "content": "\n\nIn my implementation, SAINT didn't work well. In my opinion, it's logically very hard to implement SAINT without data leakage in training due to task_container_id.
\n
Couldn't you control that with tgt_mask from PyTorch's implementation ?
Yes we can control tgt_mask in decoder. but if we use the same input of the decoder as SAINT+ paper in this image, I think the problem cannot be fixed just by controlling tgt_mask, because the query of decoder produces leakage (i.e. uses the information of the sample in the same task_container_id). In my understanding, the data leakage caused by query is unavoidable by controlling masking, while the data leakage caused by key/value is avoidable by masking.
If I understand what you are saying, you mean for questions in the same bundle that are under prediction (but during training), the corresponding input in the decoder will see the information (i.e. user correctness) in the same bundle (but prior the current position). However, while during inference, it can only see this information at the places before the current bundle.
\nI am using the encoder-decoder approach, and during the inference time, I replace the values by the value obtained at the first question in the bundle under prediction. Basically, it treats the question in the bundle as separate questions, and use the history before them.
\nAnyway, the best I can get is 0.802, so potentially there are some problems as you mentioned.
", "votes": 0, "replies": [] }, { "id": 1146587, "author_name": "mamas", "author_url": "", "post_date": "2021-01-09T21:44:02.640000", "content": "Yes of course the leakage doesn't happen during inference, but I think it's problematic that leakage happens in training phase, because the model can learn meaningless patterns during training.
\nI replace the values by the value obtained at the first question in the bundle under prediction.
\nHmm, I didn't notice this idea but this looks great. Why don't you try this idea in the training phase? I think it is possible by fancy indexing.
I'm still not seeing which way the leak could go :?
", "votes": 0, "replies": [] }, { "id": 1146610, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2021-01-09T22:02:29.533000", "content": "I didn't think of the training will be problematic. I am aware of some kind of inconsistency between decoder training / inference input by this approach, but it is a comprising approach. Treating each question in the bundle under the current inference time step is not difficult, although still need to be careful.
\nTo do so in training, I feel it becomes more difficult. Because we are not just dealing the last bundle in the truncated sequence, we extract exact one question from each budle in that sequence. Maybe not so difficult, but I kind feel we also loss information that given by bundle. Anyway, without trying, I can't say if we avoid leakage or lossing info if we apply this approach to training
", "votes": 1, "replies": [] }, { "id": 1146611, "author_name": "mamas", "author_url": "", "post_date": "2021-01-09T22:05:18.623000", "content": "@bacicnikola
\nFor example, suppose task_container_id of a specific user is [0, 0, 0, 1, 1, 2, 2, 3, 3, 4, 5, 8, 7, 7, 6]. Then,
\nposition [0, 1, 2] cannot use any information.
\nposition [3, 4] can use the information of [0, 1, 2].
\nposition [5, 6] can use the information of [0, 1, 2, 3, 4].
\nposition [7, 8] can use the information of [0, 1, 2, 3, 4, 5, 6].
However, when simply using the input of the SAINT paper, position 1 can use the information (e.g. correctness) of position 0 and position 2 can use the information of position [0, 1]. I think this is problematic as the model uses the information that should not be utilized, during training.
", "votes": 3, "replies": [] }, { "id": 1146612, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2021-01-09T22:06:17.623000", "content": "Not clear to me about what kind of leakage neither, but at least some form of inconsistency exist
", "votes": 0, "replies": [] }, { "id": 1146617, "author_name": "mamas", "author_url": "", "post_date": "2021-01-09T22:12:44.273000", "content": "Yes, it can be called \"incosistency\", as the problem doesn't happen at least in inference phase.
", "votes": 1, "replies": [] }, { "id": 1146618, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2021-01-09T22:14:48.053000", "content": "To be clear, my approach mentioned above , for decoder, only applies to the last bundle in the truncated sequence. The bundles before it, even they appear in inference time, but already being predicted, the special processing is not applied to them anymore
", "votes": 1, "replies": [] }, { "id": 1146621, "author_name": "mamas", "author_url": "", "post_date": "2021-01-09T22:19:53.660000", "content": "To do so in training, I feel it becomes more difficult. Because we are not just dealing the last bundle in the truncated sequence, we extract exact one question from each budle in that sequence. Maybe not so difficult, but I kind feel we also loss information that given by bundle.\nNow I understand what you mean. it seems the problem cannot be solved by simply using fancy indexing. Hmm…
", "votes": 1, "replies": [] }, { "id": 1146623, "author_name": "Nikola Bacic", "author_url": "", "post_date": "2021-01-09T22:20:17.523000", "content": "@mamasinkgs
\nYes, I agree. But what confusses me is why tgt_mask can't control that. In example you mentioned, it should look like this:
[0, 0, 0, 1, 1, 2, 2, 3, 3, 4, 5, 8, 7, 7, 6]
\n(1 mask, 0 no mask)
\n1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 (for 0 position)
\n1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 (for 1 position)
\n1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 (for 2 position)
\n0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 (for 3 position)
\n0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 (for 4 position)
\n0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 (for 5 position)
\n0 0 0 0 0 1 1 1 1 1 1 1 1 1 (for 6 position)
\netc.
Although in my solution, I'm only using encoder part of the transformer, but that shouldn't make any difference.
", "votes": 0, "replies": [] }, { "id": 1146630, "author_name": "mamas", "author_url": "", "post_date": "2021-01-09T22:28:16.987000", "content": "@bacicnikola
\nWhen using encoder-only model, there is no problem. However, when using decoder of SAINT+ model, the sequence is shifted like the image above. So, by simply using tgt_mask you showed (actually the mask you showed is inherently same as the mask I showed in my solution), the problem cannot be solved.
@mamasinkgs
\nYes, I see now. My implementation of this was only partly good because even though I did mask positions from the same task_container, responses were shifted to the right and question could always see previous response, which could be from the same task_container (since I used encoder only, position could always see itself)
\nI think I should've put responses into a encoder, and the rest of the features into decoder and control task_container with memory_mask.
Thank you for your time!
", "votes": 2, "replies": [] }, { "id": 1146647, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2021-01-09T22:57:28.953000", "content": "@bacicnikola ， just a remark, the mask you shows for decoder also has an issue. For example , at a place, it can't look at the current position. Therefore, position embedding of that place becomes no meaning while predicting that place, if we use the usual way of position embedding
", "votes": 0, "replies": [] }, { "id": 1146651, "author_name": "Nikola Bacic", "author_url": "", "post_date": "2021-01-09T23:06:13.917000", "content": "@yihdarshieh Yes, you're right. But I couldn't get anything out of positions anyway.
", "votes": 1, "replies": [] } ] }, { "id": 1146767, "author_name": "KhanhVD", "author_url": "", "post_date": "2021-01-10T03:13:26.980000", "content": "Big congrats to you on 2nd solo prize. Thanks for the detail writeup, very intelligence solution @mamasinkgs
", "votes": 2, "replies": [] }, { "id": 1146439, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2021-01-09T19:19:19.120000", "content": "@mamasinkgs , thank you for sharing. See you in the next one!
\nAbout your CPython implementation for getting the masks, it is amazing, you must be an advanced Python programmer!
\nFor me, I use TensorFlow, and it is quite straightforward and fast to get these mask by using tf operations, like
def get_causal_attention_mask(nd, ns, dtype, only_before):\n \"\"\"\n 1's in the lower triangle, counting from the lower right corner. Same as tf.matrix_band_part(tf.ones([nd, ns]),\n -1, ns-nd), but doesn't produce garbage on TPUs.\n \"\"\"\n\n # Remark: Think `nd` as the number of queries and `ns` as the number of keys.\n # In encoder-decoder case, the queries are the decoder features and the keys are the encoder features.\n\n i = tf.range(nd)[:, tf.newaxis] # repeat along dim 1\n j = tf.range(ns) # repeat along dim 0 \n m = i >= (j - ns + nd) + tf.cast(only_before, dtype=tf.int32)\n\n return tf.cast(m, dtype)\n\ndef get_attention_mask_from_timestamp_batch(timestamp_tensors, dtype, only_before):\n \"\"\"\n Args:\n timestamp_tensors: 2-D tf.int32 tensor, representing a batch of sequences of non-decreasing timestamps.\n\n Returns:\n attention_mask: 3-D tf.int32 tensor of shape = [batch_size, query_len, key_len], consisting of 0 and 1.\n Here `query_len` and `key_len` are actually `seq_len`. It should be reshpaed, when used to calculate \n attention scores, to [batch_size, nb_attn_head, query_len, key_len].\n \"\"\"\n\n t = timestamp_tensors\n\n batch_size = tf.math.reduce_sum(tf.ones_like(t[:, :1], dtype=tf.int32))\n seq_len = tf.math.reduce_sum(tf.ones_like(t[:1, :], dtype=tf.int32))\n\n x = tf.broadcast_to(t[:, :, tf.newaxis], shape=[batch_size, seq_len, seq_len]) # repeat along dim 2\n y = tf.broadcast_to(t[:, tf.newaxis, :], shape=[batch_size, seq_len, seq_len]) + tf.cast(only_before, dtype=tf.int64) # repeat along dim 1\n\n m = x >= y\n\n return tf.cast(m, dtype)\nThanks! I'm not familiar with tensorflow, but I'll try it in future competitions.
", "votes": 1, "replies": [] } ] }, { "id": 1590856, "author_name": "Nimra Tassawar", "author_url": "", "post_date": "2021-11-21T17:37:10.020000", "content": "Hi, I have few questions
\nHi Nimra,
\n@mamasinkgs thanks for your reply.
\nActually I have run your notebook. Everything seems good, no errors. But from where can I see results? Do submission.csv file is getting updated every time I run the notebook?
\nAlso one more question, I am a newbie in machine learning and python. Can you suggest me what courses should I take or any thing else that can help me understand code?
@mamasinkgs , Very Nice Approach of Using Transformer . Few Questions , I wanted to ask , if you don't mind sharing
\n1) How did you estimate New Users in Private Test set by probing ?
\n2) Is memory masking neccessary while training Model , what happens if We train Model without masking ?
1) I uploaded https://www.kaggle.com/mamasinkgs/submission-with-7000-limit?scriptVersionId=45368594, so please check it. I tried binary search by changing the limit.
\n2) I thought models will learn some useless patterns. But, judging from others' solutions, precise masking is not necessarily needed. I think it depends on architectures, though.
Okay Thanks .
", "votes": 0, "replies": [] } ] }, { "id": 1152889, "author_name": "Wisso", "author_url": "", "post_date": "2021-01-14T14:46:23.917000", "content": "Very nice approach, I was looking forward to this. A side question though, in this comment it was inferred that you were not using a transformer. When did you shift to this? or was it a transformer all along?
", "votes": 0, "replies": [ { "id": 1153011, "author_name": "mamas", "author_url": "", "post_date": "2021-01-14T15:24:59.587000", "content": "Nice question! In the beginning of the competition, I used LightGBM as usual and got to 0.792 (1st place) soon. Then, I started using Transformer by reading this blog, soon I got to 0.801 using Transformer and 0.803 by ensembling GBDT and Transformer. I remember I wrote this comment then. (so, The fact is I was using both GBDT and Transformer). After that, my Transformer improved a lot and I decided to throw away GBDT and use my hand-crafted features for the additional input of Transformer. \"Is Attention All You Need?\" means I was wondering whether Transformer is better than GBDT when I wrote this comment.
", "votes": 3, "replies": [] } ] }, { "id": 1150057, "author_name": "yoyoyo", "author_url": "", "post_date": "2021-01-12T10:28:19.493000", "content": "a dump question from a newbie. Which software do you use to draw that model structure graph? thank you !
", "votes": 0, "replies": [ { "id": 1150308, "author_name": "mamas", "author_url": "", "post_date": "2021-01-12T14:02:33.230000", "content": "I simply used powerpoint.
", "votes": 1, "replies": [] } ] }, { "id": 1148892, "author_name": "Nya 🚀", "author_url": "", "post_date": "2021-01-11T13:07:43.350000", "content": "\n\nI uploaded the names of 90 features as attachments.
\n
Where did you upload the attachments?
", "votes": 0, "replies": [ { "id": 1148939, "author_name": "mamas", "author_url": "", "post_date": "2021-01-11T13:44:16.993000", "content": "Thank you, I uploaded it now!
", "votes": 0, "replies": [] } ] }, { "id": 1148231, "author_name": "Dean", "author_url": "", "post_date": "2021-01-11T02:46:41.373000", "content": "Congratulations and thank for detailed explanation!
\nMay I ask a question about how to deal with the tags features?
\nThank you.
Thank you! I used one-hot encoding (like multilabel problem) for tags and concatenate them with other features, then applied MLP.
", "votes": 2, "replies": [] } ] }, { "id": 1146837, "author_name": "tomoo inubushi", "author_url": "", "post_date": "2021-01-10T05:09:53.260000", "content": "Congrats and thank you for your detailed explanation.
\nI have two questions about your solution.
I think if we did not carefully split train/valid datasets and evaluate the model, CV score would be decreased when using task_container-aware masking, as the model cannot utilize the intra-container leakage you mentioned.
\nOne of the major CV strategies in this competition by @its7171 did not care task_container_id, though it split train/valid datasets by timestamp + random int. [update] I confirmed that there are no such data at all.
", "votes": 0, "replies": [ { "id": 1146874, "author_name": "mamas", "author_url": "", "post_date": "2021-01-10T06:03:54.940000", "content": "What is your cross-validation and model evaluating strategy?
\nMy CV strategy is completely different from tito's one. First, I probed the number of new users in test set and I found there is about 7000 new users. as we know there is 2.5M rows in test set and we know the average length of the history of all users, we can estimate the number of rows of the new users and the number of rows of the existing users. After the calculation, I found
\n# rows for user split (i.e. new users): # rows for timeseries split (i.e. existing users) = 2 : 1.
\nSo, For validation, I decided to use 1M rows for timeseries split and 2M rows for user split.
\nWhen making validation set of timeseries split, I was so careful that inter-container leakage problem can't happen. I mean, my training dataset and validation dataset never shares same task_container_id for a specific user.
Did your task_container-aware masking improved CV score (and how much)?
\nI don't know how much it improves CV/LB score, because I was really careful about inter-container leakage problem from the beginning and have never tried task_container-unaware masking and indexing (I hate leakage). So, there is no evidence that shows task_container-aware masking is important. That's just my guess from LB score.
Thank you for your clarification.
\nIt is so sophisticated CV strategy and very logical consequence.
Hi @tomooinubushi,
\n\n\nThis CV strategy might split some task_containers into train/valid datasets
\n
Please check whether this kind of data is exists or not by your self if you think so.
\nI believe there is no such data. :)
\nSince:
\nI am sorry. I should have checked it before making a sloppy speculation.
\nI found there are no data which share the same timestamp or task_container_id between train/valid datasets. So, there are no intra-container leakage at all.
\nI found some of task_container_ids in valid dataset are lower than those in train dataset, which might be caused by known inconsistency between timestamp and task_container_id. I am not sure how to deal with these data when creating attention masks.
\nAnyway, thank you for pointing out my mistake @its7171 and congrats to @mamasinkgs.
Thank you for your confirmation!
", "votes": 1, "replies": [] } ] }, { "id": 1146747, "author_name": "2981", "author_url": "", "post_date": "2021-01-10T02:35:49.287000", "content": "thanks for sharing! a great job! this is a big project!
", "votes": 0, "replies": [] }, { "id": 1146571, "author_name": "kobi2000", "author_url": "", "post_date": "2021-01-09T21:34:07.693000", "content": "@mamasisking , congratulations! Can you please also share info about the resources you put into the competition? hardware, amount of effort, time etc.
", "votes": 0, "replies": [ { "id": 1146590, "author_name": "mamas", "author_url": "", "post_date": "2021-01-09T21:46:33.083000", "content": "I use V100 * 4 (VRAM 64GB) and training takes about 30h. I think I spent more than 500 hours for this competition. I did nothing other than this competition since 10/16.
", "votes": 9, "replies": [] }, { "id": 1147123, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2021-01-10T09:55:45.060000", "content": "@mamasinkgs You don't have a full-time job currently …? I can only work on this competition after my daily job …😭
", "votes": 1, "replies": [] }, { "id": 1147357, "author_name": "mamas", "author_url": "", "post_date": "2021-01-10T13:05:35.330000", "content": "I'm a student 😄
", "votes": 3, "replies": [] }, { "id": 1167076, "author_name": "Toddgm", "author_url": "", "post_date": "2021-01-24T02:52:43.677000", "content": "Tesla V100 seems not cheap. That was a lot investment money and time wise!
", "votes": 0, "replies": [] } ] }, { "id": 1146503, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2021-01-09T20:03:23.923000", "content": "@mamasinkgs , sorry to bother, but could you share your insight about this statement
\nIn my implementation, SAINT didn't work well. In my opinion, it's logically very hard to implement SAINT without data leakage in training due to task_container_id.\nWhy you think task_container_id cause data leakage during training, and what kind of leakage?
I mean, in my opinion, the input of the decoder of SAINT+ architecture cannot be implemented without any information loss or any data leakage, due to task_container_id.