![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F583548%2Fa19776338ca1ba57bd1260bf8e0ec678%2Friiid_model.png?generation=1610188697948445&alt=media\")
Hi all!
\nAfter seeing everyone share their solution we have decided to also share ours in the same spirit.
We ran a modified SAINT+ that included lectures, tags and a couple of aggregate features.
\nAll our code is available on github ( https://github.com/gautierdag/riiid ). We used Pytorch/Pytorch Lightning and Hydra.
\nOur single model public LB was 0.808, and we were able to push that to LB 0.810 (Final LB 0.813) through ensembling.
\n
Lectures: lectures were included and had a special trainable embedding vector instead of the answer embedding. The loss was not masked for lecture steps. We tested both with and without, and the difference wasn't crazy. The big benefit they offered was that they made everything much simpler to handle since we did not need to filter anything.
Tags: Since questions had a maximum of six tags, we passed sequences of length 6 to a tag embedding layer and summed the output. Therefore our tag embeddings were learned for each tag but then sum across to obtain an embedding for all the tags in a question/lecture. The padding simply returned a 0 vector.
Agg Feats: This is what gave us a healthy boost of 0.002 on public LB. By including 12 aggregate features in the decoder we were able to use some of our learnings in previous LGBM implementations.
We used the following aggregate features:
\nNote: session is just a way of dividing up the questions if a difference in time between questions is greater than two hours.
\nWe switched to use a float representation of answers in the last few days. This seemed to have little effect on our LB auc unfortunately, but had an effect locally. The idea was that since we auto-regress on answers we would propagate the uncertainty that the model displayed into future predictions.
\nAll aggs and time features are floats that are ran through a linear layer (without bias). They were all normalized to either [-1,1] or [0, 1].
\nAll other embeddings are categorical.
\nWhat made a big difference early on was our sampling methodology. Unlike most approaches in public kernels, we did not take a user-centric approach to sample our training examples. Instead, we simply sampled based on row_id and would load in the previous window_size history for every row.
\nSo for instance if we sampled row 55. Then we would find the user id that row 55 corresponds to, and load in their history up until that row. The size of the window would then be the min(window_size, len(history_until_row_55)).
\nWe used Adam with 0.001 learning rate, early stopping and lr decay based on val_auc during training.
\nFor validation, we kept a holdout set of 2.5mil (like the test set) and generated randomly with a certain proportion guaranteed of new users. We used only 250,000 rows to actually validate on during training.
\nFor every row in validation, we would pick a random number of inference steps between 1 and 10. We would then infer over these steps, not allowing the model to see the true answers and having to use its own.
\nDuring inference, when predicting multiple questions for a single user, we fed back the previous prediction of our model as answers in the model. This auto-regression was helpful in propagating uncertainty and helped the auc.
\nI saw a writeup that said that this was not possible to do because it constrains you to batch size = 1. That is wrong, you can actually do this for a whole batch in parallel and it's a little tricky with indexes but it is doable. You simply have to keep track of each sequence lengths and how many steps you are predicting for each.
\nUnfortunately since some of our aggs are also based on user answers, these do not get updated until the next batch because they are calculated on CPU and not updated in that inference loop.
\nWe ensemble three models.
\nFirst model:
\nSecond model:
\nThird model:
\nWe rented three machines but we could have probably gotten farther with using a larger single machine:
\nFinally, thanks to @eeeedev for the adventure!
", "messages": [ { "id": "1145876", "postDate": "01/09/2021 11:58:53", "content": "Hi all!
\nAfter seeing everyone share their solution we have decided to also share ours in the same spirit.
We ran a modified SAINT+ that included lectures, tags and a couple of aggregate features.
\nAll our code is available on github ( https://github.com/gautierdag/riiid ). We used Pytorch/Pytorch Lightning and Hydra.
\nOur single model public LB was 0.808, and we were able to push that to LB 0.810 (Final LB 0.813) through ensembling.
\n
Lectures: lectures were included and had a special trainable embedding vector instead of the answer embedding. The loss was not masked for lecture steps. We tested both with and without, and the difference wasn't crazy. The big benefit they offered was that they made everything much simpler to handle since we did not need to filter anything.
Tags: Since questions had a maximum of six tags, we passed sequences of length 6 to a tag embedding layer and summed the output. Therefore our tag embeddings were learned for each tag but then sum across to obtain an embedding for all the tags in a question/lecture. The padding simply returned a 0 vector.
Agg Feats: This is what gave us a healthy boost of 0.002 on public LB. By including 12 aggregate features in the decoder we were able to use some of our learnings in previous LGBM implementations.
We used the following aggregate features:
\nNote: session is just a way of dividing up the questions if a difference in time between questions is greater than two hours.
\nWe switched to use a float representation of answers in the last few days. This seemed to have little effect on our LB auc unfortunately, but had an effect locally. The idea was that since we auto-regress on answers we would propagate the uncertainty that the model displayed into future predictions.
\nAll aggs and time features are floats that are ran through a linear layer (without bias). They were all normalized to either [-1,1] or [0, 1].
\nAll other embeddings are categorical.
\nWhat made a big difference early on was our sampling methodology. Unlike most approaches in public kernels, we did not take a user-centric approach to sample our training examples. Instead, we simply sampled based on row_id and would load in the previous window_size history for every row.
\nSo for instance if we sampled row 55. Then we would find the user id that row 55 corresponds to, and load in their history up until that row. The size of the window would then be the min(window_size, len(history_until_row_55)).
\nWe used Adam with 0.001 learning rate, early stopping and lr decay based on val_auc during training.
\nFor validation, we kept a holdout set of 2.5mil (like the test set) and generated randomly with a certain proportion guaranteed of new users. We used only 250,000 rows to actually validate on during training.
\nFor every row in validation, we would pick a random number of inference steps between 1 and 10. We would then infer over these steps, not allowing the model to see the true answers and having to use its own.
\nDuring inference, when predicting multiple questions for a single user, we fed back the previous prediction of our model as answers in the model. This auto-regression was helpful in propagating uncertainty and helped the auc.
\nI saw a writeup that said that this was not possible to do because it constrains you to batch size = 1. That is wrong, you can actually do this for a whole batch in parallel and it's a little tricky with indexes but it is doable. You simply have to keep track of each sequence lengths and how many steps you are predicting for each.
\nUnfortunately since some of our aggs are also based on user answers, these do not get updated until the next batch because they are calculated on CPU and not updated in that inference loop.
\nWe ensemble three models.
\nFirst model:
\nSecond model:
\nThird model:
\nWe rented three machines but we could have probably gotten farther with using a larger single machine:
\nFinally, thanks to @eeeedev for the adventure!
", "rawMarkdown": "Hi all!\nAfter seeing everyone share their solution we have decided to also share ours in the same spirit.\n\nWe ran a modified SAINT+ that **included lectures, tags and a couple of aggregate features.**\n\nAll our code is available on github ( https://github.com/gautierdag/riiid ). We used Pytorch/Pytorch Lightning and Hydra.\n\nOur single model public LB was 0.808, and we were able to push that to LB 0.810 (Final LB 0.813) through ensembling.\n\n\n\n## Modifications to original SAINT:\n\n- **Lectures**: lectures were included and had a special trainable embedding vector instead of the answer embedding. The loss was not masked for lecture steps. We tested both with and without, and the difference wasn't crazy. The big benefit they offered was that they made everything much simpler to handle since we did not need to filter anything.\n\n- **Tags**: Since questions had a maximum of six tags, we passed sequences of length 6 to a tag embedding layer and summed the output. Therefore our tag embeddings were learned for each tag but then sum across to obtain an embedding for all the tags in a question/lecture. The padding simply returned a 0 vector.\n\n- **Agg Feats**: This is what gave us a healthy boost of 0.002 on public LB. By including 12 aggregate features in the decoder we were able to use some of our learnings in previous LGBM implementations. \n\n\n#### Agg feats\n\nWe used the following aggregate features:\n- attempts: number of previous attempts on current question (clamped to 5 and normalized to 1)\n- mean_content_mean: average of all the average question accuracy (population wise) seen until this step by the user\n- mean_user_mean: average of user's accuracy until this step \n- parts_mean: seven dimensional (the average of the user's accuracy on each part)\n- m_user_session_mean: the average accuracy in the current user's session \n- session_number: the current session number (how many sessions up till now)\n\nNote: session is just a way of dividing up the questions if a difference in time between questions is greater than two hours.\n\n#### Float vs Categorical\n\nWe switched to use a float representation of answers in the last few days. This seemed to have little effect on our LB auc unfortunately, but had an effect locally. The idea was that since we auto-regress on answers we would propagate the uncertainty that the model displayed into future predictions.\n\nAll aggs and time features are floats that are ran through a linear layer (without bias). They were all normalized to either [-1,1] or [0, 1].\n\nAll other embeddings are categorical.\n\n## Training:\n\nWhat made a **big** difference early on was our sampling methodology. Unlike most approaches in public kernels, we did not take a user-centric approach to sample our training examples. Instead, we simply sampled based on row_id and would load in the previous window_size history for every row.\n\nSo for instance if we sampled row 55. Then we would find the user id that row 55 corresponds to, and load in their history up until that row. The size of the window would then be the min(window_size, len(history_until_row_55)).\n\nWe used Adam with 0.001 learning rate, early stopping and lr decay based on val_auc during training.\n\n## Validation\n\nFor validation, we kept a holdout set of 2.5mil (like the test set) and generated randomly with a certain proportion guaranteed of new users. We used only 250,000 rows to actually validate on during training.\n\nFor every row in validation, we would pick a random number of inference steps between 1 and 10. We would then infer over these steps, not allowing the model to see the true answers and having to use its own.\n\n## Inference\n\nDuring inference, when predicting multiple questions for a single user, we fed back the previous prediction of our model as answers in the model. This auto-regression was helpful in propagating uncertainty and helped the auc.\n\nI saw a writeup that said that this was not possible to do because it constrains you to batch size = 1. That is wrong, you can actually do this for a whole batch in parallel and it's a little tricky with indexes but it is doable. You simply have to keep track of each sequence lengths and how many steps you are predicting for each.\n \nUnfortunately since some of our aggs are also based on user answers, these do not get updated until the next batch because they are calculated on CPU and not updated in that inference loop.\n\n## Parameters\n\nWe ensemble three models.\n\nFirst model:\n- 64 emb_dim\n- 4/6 encoder/decoder layers\n- 256 feed foward in Transformer\n- 4 Heads\n- 100 window size\n\nSecond model:\n- 64 emb_dim\n- 4/6 encoder/decoder layers\n- 256 feed foward in Transformer\n- 4 Heads\n- 200 window size (expanded by finetuning on 200)\n\nThird model:\n- 256 emb_dim\n- 2/2 encoder/decoder layers\n- 512 feed foward in Transformer\n- 4 Heads\n- 256 window size\n\n\n## Hardware\nWe rented three machines but we could have probably gotten farther with using a larger single machine:\n\n- 3 X 1 Tesla V100(16gb VRAM)\n\n## Other things we tried\n- Noam lr schedule (super slow to converge)\n- Linear attention / Reformer / Linformer (spent way to much time on this)\n- Local Attention\n- Additional Aggs in output layer and a myriad of other aggs\n- Concatenating Embeds instead of summing\n- A lot of different configurations of num_layers / dims / .. etc\n- Custom attention based on bundle \n- Ensembling with LGBM\n- Predicting time taken to answer question as well as answer in Loss\n- K Beam decoding (predicting in parallel K possible paths of answers and taking the one that maximized joint probability of sequence)\n- Running average of agg features (different windows or averaging over time)\n- Causal 1D convolutions before the Transformer\n- Increasing window size during training\n- ...\n\nFinally, thanks to @eeeedev for the adventure!", "votes": null }, { "id": "1146399", "postDate": "01/09/2021 18:40:40", "content": "Congratulations and thanks for sharing, interesting sampling method. Was your ensemble just an average of the 3 models? When you compare Adam to Noam was the difference like half the epochs required to converge?
", "rawMarkdown": "Congratulations and thanks for sharing, interesting sampling method. Was your ensemble just an average of the 3 models? When you compare Adam to Noam was the difference like half the epochs required to converge?", "votes": null }, { "id": "1146499", "postDate": "01/09/2021 20:01:44", "content": "@brotye github link is not working.
", "rawMarkdown": "brotye github link is not working.", "votes": null }, { "id": "1146567", "postDate": "01/09/2021 21:29:28", "content": "Thanks for noticing! The parenthesis was wrongly included in the url - fixed it!
", "rawMarkdown": "Thanks for noticing! The parenthesis was wrongly included in the url - fixed it!", "votes": null }, { "id": "1146582", "postDate": "01/09/2021 21:40:33", "content": "Thank you!
\nYes the ensemble was a simple average of all models, all models were all too close individually to weight one over another.
\nThe problem with Noam was that because it's a linear increase that needs to be parametrised (you have to decide number of warmup steps) - the network really learns nothing at the beginning. Then the exponential decay hits immediately after reaching your target lr. While this might be good to warmup very deep Transformers, we found that using a simple lr and starting directly at the target lr was much faster.
\nI don't remember the exact difference, but probably something like 1.5x faster to not use Noam and no loss/auc difference. It would obviously depend on how you parametrize the Noam scheme. The default and advertised Noam scheme is 4k warm up steps but that should also depend on your batch size as well.
", "rawMarkdown": "Thank you!\n\nYes the ensemble was a simple average of all models, all models were all too close individually to weight one over another.\n\nThe problem with Noam was that because it's a linear increase that needs to be parametrised (you have to decide number of warmup steps) - the network really learns nothing at the beginning. Then the exponential decay hits immediately after reaching your target lr. While this might be good to warmup very deep Transformers, we found that using a simple lr and starting directly at the target lr was much faster. \n\nI don't remember the exact difference, but probably something like 1.5x faster to **not** use Noam and no loss/auc difference. It would obviously depend on how you parametrize the Noam scheme. The default and advertised Noam scheme is 4k warm up steps but that should also depend on your batch size as well.", "votes": null }, { "id": "1146619", "postDate": "01/09/2021 22:16:01", "content": "@brotye I was not able to find the code for preparation of .npy files. Can you point out the file which does that or is it not present in repo ?
Also, can you please explain a bit about your model fine tuning ?
", "rawMarkdown": "brotye I was not able to find the code for preparation of `.npy files`. Can you point out the file which does that or is it not present in repo ?\n\nAlso, can you please explain a bit about your model fine tuning ?", "votes": null }, { "id": "1146672", "postDate": "01/09/2021 23:53:22", "content": "Congrats and thank you for sharing. I felt that all of the efforts are excellent, considering the characteristic of this competition!
", "rawMarkdown": "Congrats and thank you for sharing. I felt that all of the efforts are excellent, considering the characteristic of this competition!", "votes": null }, { "id": "1146893", "postDate": "01/10/2021 06:27:47", "content": "Nice solution, I also had linear transformers in mind although due to time constraint did not try it. Can you give more details (lb/cv) on the linear transformers? Also, did you happen to try performers?
", "rawMarkdown": "Nice solution, I also had linear transformers in mind although due to time constraint did not try it. Can you give more details (lb/cv) on the linear transformers? Also, did you happen to try performers?", "votes": null }, { "id": "1147091", "postDate": "01/10/2021 09:29:29", "content": "Thanks!
\nWell the problem is that I never got any of the alternative attentions to even converge. I used public implementations of them so I'm quite certain it wasn't from a bug. I spent about a week on testing a lot of them (I didn't try the Performer architecture). I think they are much harder to train for some reason on this dataset, this could be due to me not being patient enough in training or having some wrong hyperparameter/initialization scheme.
", "rawMarkdown": "Thanks! \n\nWell the problem is that I never got any of the alternative attentions to even converge. I used public implementations of them so I'm quite certain it wasn't from a bug. I spent about a week on testing a lot of them (I didn't try the Performer architecture). I think they are much harder to train for some reason on this dataset, this could be due to me not being patient enough in training or having some wrong hyperparameter/initialization scheme.", "votes": null }, { "id": "1147098", "postDate": "01/10/2021 09:37:12", "content": "@abdurrehman245 you can find the generation of some of the .npy files in the preprocessing.py file. Some are also included in the repo itself in the data folder.
Apologies if some are missing, if you let me know in particular which one you are referring to I can rectify it.
\nThere is one I know is missing: questions_lectures_pct.npy, which is a content based feature one, but it is not actually needed to reproduce our results since we did not use it in the end.
Finetuning was done by simply resuming our best checkpoint after training on (90%) to a window size double of what it had seen (100 -> 200) and the remainder of the training set (10%). This made training faster (since we could train on smaller windows) and models with bigger windows tended to be a little bit more robust, but it did not greatly improve our AUC.
", "rawMarkdown": "abdurrehman245 you can find the generation of some of the `.npy` files in the `preprocessing.py` file. Some are also included in the repo itself in the `data` folder. \n\nApologies if some are missing, if you let me know in particular which one you are referring to I can rectify it. \n\nThere is one I know is missing: `questions_lectures_pct.npy`, which is a content based feature one, but it is not actually needed to reproduce our results since we did not use it in the end.\n\n\nFinetuning was done by simply resuming our best checkpoint after training on (90%) to a window size double of what it had seen (100 -> 200) and the remainder of the training set (10%). This made training faster (since we could train on smaller windows) and models with bigger windows tended to be a little bit more robust, but it did not greatly improve our AUC.", "votes": null }, { "id": "1147251", "postDate": "01/10/2021 11:39:34", "content": "Congrats @brotye 👍
\nNice solution
", "rawMarkdown": "Congrats @brotye 👍\n\nNice solution", "votes": null }, { "id": "1147265", "postDate": "01/10/2021 11:52:01", "content": "Thanks for the detailed response much appreciated
", "rawMarkdown": "Thanks for the detailed response much appreciated", "votes": null }, { "id": "1147452", "postDate": "01/10/2021 14:15:05", "content": "@brotye I think these files preparation code is not present in repo. Can you explain a bit about how you generate these files?
\nval_2500000_lec_True.npy\nval_2500000_lec_False.npy\nuser_weights.npy\nAlso, can you confirm by using neural.ipynb notebook in repo, I can reproduce one of your best single model by specifying the one of the configurations mentioned in description?
self.embed_content_id = nn.Embedding(n_content_id, emb_dim, padding_idx=13942)\nself.embed_parts = nn.Embedding(n_part, emb_dim, padding_idx=0)\nself.embed_tags = nn.Embedding(n_tags, emb_dim, padding_idx=188)\nI am newbie in transformers so asking some silly questions as well. You used padding_idxin embeddings because we can have some content_id or parts in test data that we don't have in train data ?
val_2500000_lec_True.npy and val_2500000_lec_False.npy are just numpy arrays that describe the generated val set - its just an array of length 2.5mil of row ids that belong to val.
You don't actually need them since it will get generated randomly when you run the code.
\nuser_weights.npy is not used anymore.
If you want to reproduce, it will be a little trickier than just running the notebook. You'll have to use the python main.py script. The notebook is unfortunately out of date.
padding_idx is used because we pass batches of variable lengths sequences to the network, so for instance, if we have two sequences (one length 6 and one length 3):
[[1, 2, 3, 4, 5, 6],
\n[1, 2, 3, 0, 0, 0]]
Then when we pass them through embedding we don't care about the embedding of the padding token so we can just set the output vector to 0 directly.
", "rawMarkdown": "`val_2500000_lec_True.npy` and `val_2500000_lec_False.npy` are just numpy arrays that describe the generated val set - its just an array of length 2.5mil of row ids that belong to val. \n\nYou don't actually need them since it will get generated randomly when you run the code.\n\n`user_weights.npy` is not used anymore.\n\nIf you want to reproduce, it will be a little trickier than just running the notebook. You'll have to use the python `main.py` script. The notebook is unfortunately out of date.\n\n`padding_idx` is used because we pass batches of variable lengths sequences to the network, so for instance, if we have two sequences (one length 6 and one length 3):\n\n[[1, 2, 3, 4, 5, 6],\n[1, 2, 3, 0, 0, 0]]\n\nThen when we pass them through embedding we don't care about the embedding of the padding token so we can just set the output vector to 0 directly.", "votes": null }, { "id": "1147724", "postDate": "01/10/2021 16:50:01", "content": "@brotye thanks for all the replies.
\nSo I have to go through the repo files to reproduce them in kaggle notebook.
\nAs you mentioned in above, you append 0's to sequence to make them of equal length and then we get embedding vector of 0's corresponding to padding_idx so this means in case of content_id sequnence whose length is less than MAX_SEQ_LEN you pad them with 13942 to get vector of 0's for those positions. Why you did not use the same padding_idx in each embedding instead of 0, 13942,188. Is there any specific reason of it ?
self.embed_content_id = nn.Embedding(n_content_id, emb_dim, padding_idx=13942)
Did you try to feed the aggregated features to encoder as well? I am just trying to understand the intuition of feeding them to decoder layer.
How did you come up with the hyperparameters( by some hyperparameter turning or some other method) ? I am asking this because most of the solution used 8 heads as mentioned in Attention paper but you used 4 heads in your solution.
", "rawMarkdown": "brotye thanks for all the replies.\n\nSo I have to go through the repo files to reproduce them in kaggle notebook.\n\nAs you mentioned in above, you append 0's to sequence to make them of equal length and then we get embedding vector of 0's corresponding to `padding_idx ` so this means in case of `content_id ` sequnence whose length is less than `MAX_SEQ_LEN` you pad them with `13942` to get vector of 0's for those positions. Why you did not use the same `padding_idx ` in each embedding instead of `0, 13942,188`. Is there any specific reason of it ?\n\n`self.embed_content_id = nn.Embedding(n_content_id, emb_dim, padding_idx=13942)`\n\nDid you try to feed the `aggregated features` to encoder as well? I am just trying to understand the intuition of feeding them to decoder layer.\n\nHow did you come up with the hyperparameters( by some hyperparameter turning or some other method) ? I am asking this because most of the solution used 8 heads as mentioned in Attention paper but you used 4 heads in your solution.", "votes": null } ], "comments": [ { "id": 1146399, "author_name": "darren1515", "author_url": "", "post_date": "01/09/2021 18:40:40", "content": "Congratulations and thanks for sharing, interesting sampling method. Was your ensemble just an average of the 3 models? When you compare Adam to Noam was the difference like half the epochs required to converge?
", "votes": null, "replies": [ { "id": 1146582, "author_name": "brotye", "author_url": "", "post_date": "01/09/2021 21:40:33", "content": "Thank you!
\nYes the ensemble was a simple average of all models, all models were all too close individually to weight one over another.
\nThe problem with Noam was that because it's a linear increase that needs to be parametrised (you have to decide number of warmup steps) - the network really learns nothing at the beginning. Then the exponential decay hits immediately after reaching your target lr. While this might be good to warmup very deep Transformers, we found that using a simple lr and starting directly at the target lr was much faster.
\nI don't remember the exact difference, but probably something like 1.5x faster to not use Noam and no loss/auc difference. It would obviously depend on how you parametrize the Noam scheme. The default and advertised Noam scheme is 4k warm up steps but that should also depend on your batch size as well.
", "votes": null, "replies": [] }, { "id": 1147265, "author_name": "darren1515", "author_url": "", "post_date": "01/10/2021 11:52:01", "content": "Thanks for the detailed response much appreciated
", "votes": null, "replies": [] } ] }, { "id": 1146499, "author_name": "abdurrehman245", "author_url": "", "post_date": "01/09/2021 20:01:44", "content": "@brotye github link is not working.
", "votes": null, "replies": [ { "id": 1146567, "author_name": "brotye", "author_url": "", "post_date": "01/09/2021 21:29:28", "content": "Thanks for noticing! The parenthesis was wrongly included in the url - fixed it!
", "votes": null, "replies": [] }, { "id": 1146619, "author_name": "abdurrehman245", "author_url": "", "post_date": "01/09/2021 22:16:01", "content": "@brotye I was not able to find the code for preparation of .npy files. Can you point out the file which does that or is it not present in repo ?
Also, can you please explain a bit about your model fine tuning ?
", "votes": null, "replies": [] }, { "id": 1147098, "author_name": "brotye", "author_url": "", "post_date": "01/10/2021 09:37:12", "content": "@abdurrehman245 you can find the generation of some of the .npy files in the preprocessing.py file. Some are also included in the repo itself in the data folder.
Apologies if some are missing, if you let me know in particular which one you are referring to I can rectify it.
\nThere is one I know is missing: questions_lectures_pct.npy, which is a content based feature one, but it is not actually needed to reproduce our results since we did not use it in the end.
Finetuning was done by simply resuming our best checkpoint after training on (90%) to a window size double of what it had seen (100 -> 200) and the remainder of the training set (10%). This made training faster (since we could train on smaller windows) and models with bigger windows tended to be a little bit more robust, but it did not greatly improve our AUC.
", "votes": null, "replies": [] }, { "id": 1147452, "author_name": "abdurrehman245", "author_url": "", "post_date": "01/10/2021 14:15:05", "content": "@brotye I think these files preparation code is not present in repo. Can you explain a bit about how you generate these files?
\nval_2500000_lec_True.npy\nval_2500000_lec_False.npy\nuser_weights.npy\nAlso, can you confirm by using neural.ipynb notebook in repo, I can reproduce one of your best single model by specifying the one of the configurations mentioned in description?
self.embed_content_id = nn.Embedding(n_content_id, emb_dim, padding_idx=13942)\nself.embed_parts = nn.Embedding(n_part, emb_dim, padding_idx=0)\nself.embed_tags = nn.Embedding(n_tags, emb_dim, padding_idx=188)\nI am newbie in transformers so asking some silly questions as well. You used padding_idxin embeddings because we can have some content_id or parts in test data that we don't have in train data ?
val_2500000_lec_True.npy and val_2500000_lec_False.npy are just numpy arrays that describe the generated val set - its just an array of length 2.5mil of row ids that belong to val.
You don't actually need them since it will get generated randomly when you run the code.
\nuser_weights.npy is not used anymore.
If you want to reproduce, it will be a little trickier than just running the notebook. You'll have to use the python main.py script. The notebook is unfortunately out of date.
padding_idx is used because we pass batches of variable lengths sequences to the network, so for instance, if we have two sequences (one length 6 and one length 3):
[[1, 2, 3, 4, 5, 6],
\n[1, 2, 3, 0, 0, 0]]
Then when we pass them through embedding we don't care about the embedding of the padding token so we can just set the output vector to 0 directly.
", "votes": null, "replies": [] }, { "id": 1147724, "author_name": "abdurrehman245", "author_url": "", "post_date": "01/10/2021 16:50:01", "content": "@brotye thanks for all the replies.
\nSo I have to go through the repo files to reproduce them in kaggle notebook.
\nAs you mentioned in above, you append 0's to sequence to make them of equal length and then we get embedding vector of 0's corresponding to padding_idx so this means in case of content_id sequnence whose length is less than MAX_SEQ_LEN you pad them with 13942 to get vector of 0's for those positions. Why you did not use the same padding_idx in each embedding instead of 0, 13942,188. Is there any specific reason of it ?
self.embed_content_id = nn.Embedding(n_content_id, emb_dim, padding_idx=13942)
Did you try to feed the aggregated features to encoder as well? I am just trying to understand the intuition of feeding them to decoder layer.
How did you come up with the hyperparameters( by some hyperparameter turning or some other method) ? I am asking this because most of the solution used 8 heads as mentioned in Attention paper but you used 4 heads in your solution.
", "votes": null, "replies": [] } ] }, { "id": 1146672, "author_name": "sishihara", "author_url": "", "post_date": "01/09/2021 23:53:22", "content": "Congrats and thank you for sharing. I felt that all of the efforts are excellent, considering the characteristic of this competition!
", "votes": null, "replies": [] }, { "id": 1146893, "author_name": "shujun717", "author_url": "", "post_date": "01/10/2021 06:27:47", "content": "Nice solution, I also had linear transformers in mind although due to time constraint did not try it. Can you give more details (lb/cv) on the linear transformers? Also, did you happen to try performers?
", "votes": null, "replies": [ { "id": 1147091, "author_name": "brotye", "author_url": "", "post_date": "01/10/2021 09:29:29", "content": "Thanks!
\nWell the problem is that I never got any of the alternative attentions to even converge. I used public implementations of them so I'm quite certain it wasn't from a bug. I spent about a week on testing a lot of them (I didn't try the Performer architecture). I think they are much harder to train for some reason on this dataset, this could be due to me not being patient enough in training or having some wrong hyperparameter/initialization scheme.
", "votes": null, "replies": [] } ] }, { "id": 1147251, "author_name": "jagadish13", "author_url": "", "post_date": "01/10/2021 11:39:34", "content": "Congrats @brotye 👍
\nNice solution
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1145876": "Hi all!\nAfter seeing everyone share their solution we have decided to also share ours in the same spirit.\n\nWe ran a modified SAINT+ that **included lectures, tags and a couple of aggregate features.**\n\nAll our code is available on github ( https://github.com/gautierdag/riiid ). We used Pytorch/Pytorch Lightning and Hydra.\n\nOur single model public LB was 0.808, and we were able to push that to LB 0.810 (Final LB 0.813) through ensembling.\n\n\n\n## Modifications to original SAINT:\n\n- **Lectures**: lectures were included and had a special trainable embedding vector instead of the answer embedding. The loss was not masked for lecture steps. We tested both with and without, and the difference wasn't crazy. The big benefit they offered was that they made everything much simpler to handle since we did not need to filter anything.\n\n- **Tags**: Since questions had a maximum of six tags, we passed sequences of length 6 to a tag embedding layer and summed the output. Therefore our tag embeddings were learned for each tag but then sum across to obtain an embedding for all the tags in a question/lecture. The padding simply returned a 0 vector.\n\n- **Agg Feats**: This is what gave us a healthy boost of 0.002 on public LB. By including 12 aggregate features in the decoder we were able to use some of our learnings in previous LGBM implementations. \n\n\n#### Agg feats\n\nWe used the following aggregate features:\n- attempts: number of previous attempts on current question (clamped to 5 and normalized to 1)\n- mean_content_mean: average of all the average question accuracy (population wise) seen until this step by the user\n- mean_user_mean: average of user's accuracy until this step \n- parts_mean: seven dimensional (the average of the user's accuracy on each part)\n- m_user_session_mean: the average accuracy in the current user's session \n- session_number: the current session number (how many sessions up till now)\n\nNote: session is just a way of dividing up the questions if a difference in time between questions is greater than two hours.\n\n#### Float vs Categorical\n\nWe switched to use a float representation of answers in the last few days. This seemed to have little effect on our LB auc unfortunately, but had an effect locally. The idea was that since we auto-regress on answers we would propagate the uncertainty that the model displayed into future predictions.\n\nAll aggs and time features are floats that are ran through a linear layer (without bias). They were all normalized to either [-1,1] or [0, 1].\n\nAll other embeddings are categorical.\n\n## Training:\n\nWhat made a **big** difference early on was our sampling methodology. Unlike most approaches in public kernels, we did not take a user-centric approach to sample our training examples. Instead, we simply sampled based on row_id and would load in the previous window_size history for every row.\n\nSo for instance if we sampled row 55. Then we would find the user id that row 55 corresponds to, and load in their history up until that row. The size of the window would then be the min(window_size, len(history_until_row_55)).\n\nWe used Adam with 0.001 learning rate, early stopping and lr decay based on val_auc during training.\n\n## Validation\n\nFor validation, we kept a holdout set of 2.5mil (like the test set) and generated randomly with a certain proportion guaranteed of new users. We used only 250,000 rows to actually validate on during training.\n\nFor every row in validation, we would pick a random number of inference steps between 1 and 10. We would then infer over these steps, not allowing the model to see the true answers and having to use its own.\n\n## Inference\n\nDuring inference, when predicting multiple questions for a single user, we fed back the previous prediction of our model as answers in the model. This auto-regression was helpful in propagating uncertainty and helped the auc.\n\nI saw a writeup that said that this was not possible to do because it constrains you to batch size = 1. That is wrong, you can actually do this for a whole batch in parallel and it's a little tricky with indexes but it is doable. You simply have to keep track of each sequence lengths and how many steps you are predicting for each.\n \nUnfortunately since some of our aggs are also based on user answers, these do not get updated until the next batch because they are calculated on CPU and not updated in that inference loop.\n\n## Parameters\n\nWe ensemble three models.\n\nFirst model:\n- 64 emb_dim\n- 4/6 encoder/decoder layers\n- 256 feed foward in Transformer\n- 4 Heads\n- 100 window size\n\nSecond model:\n- 64 emb_dim\n- 4/6 encoder/decoder layers\n- 256 feed foward in Transformer\n- 4 Heads\n- 200 window size (expanded by finetuning on 200)\n\nThird model:\n- 256 emb_dim\n- 2/2 encoder/decoder layers\n- 512 feed foward in Transformer\n- 4 Heads\n- 256 window size\n\n\n## Hardware\nWe rented three machines but we could have probably gotten farther with using a larger single machine:\n\n- 3 X 1 Tesla V100(16gb VRAM)\n\n## Other things we tried\n- Noam lr schedule (super slow to converge)\n- Linear attention / Reformer / Linformer (spent way to much time on this)\n- Local Attention\n- Additional Aggs in output layer and a myriad of other aggs\n- Concatenating Embeds instead of summing\n- A lot of different configurations of num_layers / dims / .. etc\n- Custom attention based on bundle \n- Ensembling with LGBM\n- Predicting time taken to answer question as well as answer in Loss\n- K Beam decoding (predicting in parallel K possible paths of answers and taking the one that maximized joint probability of sequence)\n- Running average of agg features (different windows or averaging over time)\n- Causal 1D convolutions before the Transformer\n- Increasing window size during training\n- ...\n\nFinally, thanks to @eeeedev for the adventure!", "1146399": "Congratulations and thanks for sharing, interesting sampling method. Was your ensemble just an average of the 3 models? When you compare Adam to Noam was the difference like half the epochs required to converge?", "1146499": "brotye github link is not working.", "1146567": "Thanks for noticing! The parenthesis was wrongly included in the url - fixed it!", "1146582": "Thank you!\n\nYes the ensemble was a simple average of all models, all models were all too close individually to weight one over another.\n\nThe problem with Noam was that because it's a linear increase that needs to be parametrised (you have to decide number of warmup steps) - the network really learns nothing at the beginning. Then the exponential decay hits immediately after reaching your target lr. While this might be good to warmup very deep Transformers, we found that using a simple lr and starting directly at the target lr was much faster. \n\nI don't remember the exact difference, but probably something like 1.5x faster to **not** use Noam and no loss/auc difference. It would obviously depend on how you parametrize the Noam scheme. The default and advertised Noam scheme is 4k warm up steps but that should also depend on your batch size as well.", "1146619": "brotye I was not able to find the code for preparation of `.npy files`. Can you point out the file which does that or is it not present in repo ?\n\nAlso, can you please explain a bit about your model fine tuning ?", "1146672": "Congrats and thank you for sharing. I felt that all of the efforts are excellent, considering the characteristic of this competition!", "1146893": "Nice solution, I also had linear transformers in mind although due to time constraint did not try it. Can you give more details (lb/cv) on the linear transformers? Also, did you happen to try performers?", "1147091": "Thanks! \n\nWell the problem is that I never got any of the alternative attentions to even converge. I used public implementations of them so I'm quite certain it wasn't from a bug. I spent about a week on testing a lot of them (I didn't try the Performer architecture). I think they are much harder to train for some reason on this dataset, this could be due to me not being patient enough in training or having some wrong hyperparameter/initialization scheme.", "1147098": "abdurrehman245 you can find the generation of some of the `.npy` files in the `preprocessing.py` file. Some are also included in the repo itself in the `data` folder. \n\nApologies if some are missing, if you let me know in particular which one you are referring to I can rectify it. \n\nThere is one I know is missing: `questions_lectures_pct.npy`, which is a content based feature one, but it is not actually needed to reproduce our results since we did not use it in the end.\n\n\nFinetuning was done by simply resuming our best checkpoint after training on (90%) to a window size double of what it had seen (100 -> 200) and the remainder of the training set (10%). This made training faster (since we could train on smaller windows) and models with bigger windows tended to be a little bit more robust, but it did not greatly improve our AUC.", "1147251": "Congrats @brotye 👍\n\nNice solution", "1147265": "Thanks for the detailed response much appreciated", "1147452": "brotye I think these files preparation code is not present in repo. Can you explain a bit about how you generate these files? \n\n```\nval_2500000_lec_True.npy\nval_2500000_lec_False.npy\nuser_weights.npy\n```\nAlso, can you confirm by using `neural.ipynb` notebook in repo, I can reproduce one of your best single model by specifying the one of the configurations mentioned in description?\n\n```\nself.embed_content_id = nn.Embedding(n_content_id, emb_dim, padding_idx=13942)\nself.embed_parts = nn.Embedding(n_part, emb_dim, padding_idx=0)\nself.embed_tags = nn.Embedding(n_tags, emb_dim, padding_idx=188)\n```\nI am newbie in transformers so asking some silly questions as well. You used `padding_idx `in embeddings because we can have some content_id or parts in test data that we don't have in train data ?", "1147665": "`val_2500000_lec_True.npy` and `val_2500000_lec_False.npy` are just numpy arrays that describe the generated val set - its just an array of length 2.5mil of row ids that belong to val. \n\nYou don't actually need them since it will get generated randomly when you run the code.\n\n`user_weights.npy` is not used anymore.\n\nIf you want to reproduce, it will be a little trickier than just running the notebook. You'll have to use the python `main.py` script. The notebook is unfortunately out of date.\n\n`padding_idx` is used because we pass batches of variable lengths sequences to the network, so for instance, if we have two sequences (one length 6 and one length 3):\n\n[[1, 2, 3, 4, 5, 6],\n[1, 2, 3, 0, 0, 0]]\n\nThen when we pass them through embedding we don't care about the embedding of the padding token so we can just set the output vector to 0 directly.", "1147724": "brotye thanks for all the replies.\n\nSo I have to go through the repo files to reproduce them in kaggle notebook.\n\nAs you mentioned in above, you append 0's to sequence to make them of equal length and then we get embedding vector of 0's corresponding to `padding_idx ` so this means in case of `content_id ` sequnence whose length is less than `MAX_SEQ_LEN` you pad them with `13942` to get vector of 0's for those positions. Why you did not use the same `padding_idx ` in each embedding instead of `0, 13942,188`. Is there any specific reason of it ?\n\n`self.embed_content_id = nn.Embedding(n_content_id, emb_dim, padding_idx=13942)`\n\nDid you try to feed the `aggregated features` to encoder as well? I am just trying to understand the intuition of feeding them to decoder layer.\n\nHow did you come up with the hyperparameters( by some hyperparameter turning or some other method) ? I am asking this because most of the solution used 8 heads as mentioned in Attention paper but you used 4 heads in your solution." }, "source": "meta" }