![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2Fa47293277b9e989ab5c81c269e6187a8%2Fdoc_saint.png?generation=1610121546239700&alt=media\")
Hi all,
\nHere is an insight of our 2 solutions that both scored public 0.812/private 0.815 and that reached 16th gold place.
\nThis competition was both ML and engineering optimization to make everything work in 9h with 13GB RAM/16GB GPU. We spent almost 30% of time on optimization to keep the last 512 interactions per users + per content attempts in memory + required for our features.
\nWe would like to thank Kaggle and RIIID organizers for this great competition! Congratulations to the top teams and all competitors for their motivation all along the challenge.
\nI would like to thank my teammates @rafiko1, @cdeotte @titericz and @matthiasanderer. You've been amazing, I've learnt a lot from you. I really enjoyed this competition.
The SAINT+ model is described here https://arxiv.org/pdf/2010.12042.pdf
\nThe code for our SAINT+ adaptation is available here https://github.com/rafiko1/Riiid-sharing.
Our single model SAINT++ achieved, CV: 0.812 Public LB: 0.812, Private LB: 0.815
\nWe trained with 95% of users first then fine tuned with all data using smart window technique (see diagram below for SAKT). The model is simple in terms of features. It only contains the four features of SAINT+ (pictured above), with one additional feature - the number of attempts of a user for specific content (hence SAINT++):
\nThe greatest improvement in features compared to SAINT+ came from grouping lag time into seconds, unlike minutes as done in the paper.
\nThen, we went bigger and bigger on the architecture and burned some GPU power 🔥. We increased on parameters of the model, most importantly the sequence length and number of layers. Final parameters of the model are as follows:
We used the Noam learning rate scheduler: with initial warmup and exponential decrease down to 2e-5.
\nFinal improvement came from our recursive trick during inference. Here, we rounded predictions that came from the same bundle to 0 or 1 - as their true response is unknown in time yet. The rounded predictions are then fed back to the model to predict the next response within the same bundle. This trick boosts CV LB +0.0025, but requires a batch size of 1, so we couldn’t ensemble multiple transformer models.
\nLightGBM model scored CV=0.793, public LB=0.792 with 44 features.
\nOur main features:
\nQuestion correctness per content and per user, tags 1 and 2, part, elapsed time, had explanation, number of attempts, multiple lags, running average (answer), multiple rolling means/median (answers, lags, elapsed time) + weighted mean, mean after/before 30 interactions, multiple momentums (lag, answers), per part correctness, per session (8 hours split) running average. Only 3 categories: part, tags1, tags2. Train/valid split from Tito.
Pytorch SAKT modified scored CV=0.786, LB=0.789 with additional features. Training procedure with a smart window.
When a user’s sequence length is larger than model input, i.e. N>W, then using random crops gives +0.002 CV versus tiled crops. And using smart random crops gives +0.003 CV versus tiled crops. Basic random crops have a low probability of selecting the early or late questions from a user’s sequence whereas smart random crops have an equally likely probability of selecting all questions from a user.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2Fa1239fda69769669432879b2b1a4ae39%2Fdoc_window.png?generation=1610121604322940&alt=media\")
TensorFlow Transformer model alone scored CV=0.811, LB=0.811
\nSame as solution#1 but with sequence length = 256
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F698363%2Fe4b8abca67c11258a438304b8fc67db5%2Fdoc_pp.png?generation=1610121577542081&alt=media\")
But was not applicable for us within the 9h runtime limit:
\nOne additional word to Kaggle @sohier I loved your API and the way it hides private data, it’s more realistic as in real world usage/production and it avoided chaotic blending. Congratulations for that, however, even if I guess you want to prevent probing, you should find a solution to provide better error feedback. If it is not possible (the more error codes the more probing) then you need to provide a simulator and guidelines allowing competitors to troubleshoot locally.
", "messages": [ { "id": 1144696, "postDate": "2021-01-08T16:05:25.673Z", "content": "Hi all,
\nHere is an insight of our 2 solutions that both scored public 0.812/private 0.815 and that reached 16th gold place.
\nThis competition was both ML and engineering optimization to make everything work in 9h with 13GB RAM/16GB GPU. We spent almost 30% of time on optimization to keep the last 512 interactions per users + per content attempts in memory + required for our features.
\nWe would like to thank Kaggle and RIIID organizers for this great competition! Congratulations to the top teams and all competitors for their motivation all along the challenge.
\nI would like to thank my teammates @rafiko1, @cdeotte @titericz and @matthiasanderer. You've been amazing, I've learnt a lot from you. I really enjoyed this competition.
The SAINT+ model is described here https://arxiv.org/pdf/2010.12042.pdf
\nThe code for our SAINT+ adaptation is available here https://github.com/rafiko1/Riiid-sharing.
Our single model SAINT++ achieved, CV: 0.812 Public LB: 0.812, Private LB: 0.815
\nWe trained with 95% of users first then fine tuned with all data using smart window technique (see diagram below for SAKT). The model is simple in terms of features. It only contains the four features of SAINT+ (pictured above), with one additional feature - the number of attempts of a user for specific content (hence SAINT++):
\nThe greatest improvement in features compared to SAINT+ came from grouping lag time into seconds, unlike minutes as done in the paper.
\nThen, we went bigger and bigger on the architecture and burned some GPU power 🔥. We increased on parameters of the model, most importantly the sequence length and number of layers. Final parameters of the model are as follows:
We used the Noam learning rate scheduler: with initial warmup and exponential decrease down to 2e-5.
\nFinal improvement came from our recursive trick during inference. Here, we rounded predictions that came from the same bundle to 0 or 1 - as their true response is unknown in time yet. The rounded predictions are then fed back to the model to predict the next response within the same bundle. This trick boosts CV LB +0.0025, but requires a batch size of 1, so we couldn’t ensemble multiple transformer models.
\nLightGBM model scored CV=0.793, public LB=0.792 with 44 features.
\nOur main features:
\nQuestion correctness per content and per user, tags 1 and 2, part, elapsed time, had explanation, number of attempts, multiple lags, running average (answer), multiple rolling means/median (answers, lags, elapsed time) + weighted mean, mean after/before 30 interactions, multiple momentums (lag, answers), per part correctness, per session (8 hours split) running average. Only 3 categories: part, tags1, tags2. Train/valid split from Tito.
Pytorch SAKT modified scored CV=0.786, LB=0.789 with additional features. Training procedure with a smart window.
When a user’s sequence length is larger than model input, i.e. N>W, then using random crops gives +0.002 CV versus tiled crops. And using smart random crops gives +0.003 CV versus tiled crops. Basic random crops have a low probability of selecting the early or late questions from a user’s sequence whereas smart random crops have an equally likely probability of selecting all questions from a user.
\n
TensorFlow Transformer model alone scored CV=0.811, LB=0.811
\nSame as solution#1 but with sequence length = 256
But was not applicable for us within the 9h runtime limit:
\nOne additional word to Kaggle @sohier I loved your API and the way it hides private data, it’s more realistic as in real world usage/production and it avoided chaotic blending. Congratulations for that, however, even if I guess you want to prevent probing, you should find a solution to provide better error feedback. If it is not possible (the more error codes the more probing) then you need to provide a simulator and guidelines allowing competitors to troubleshoot locally.
", "rawMarkdown": "Hi all,\n\n\nHere is an insight of our 2 solutions that both scored public 0.812/private 0.815 and that reached 16th gold place.\n\n\nThis competition was both ML and engineering optimization to make everything work in 9h with 13GB RAM/16GB GPU. We spent almost 30% of time on optimization to keep the last 512 interactions per users + per content attempts in memory + required for our features.\n\nWe would like to thank Kaggle and RIIID organizers for this great competition! Congratulations to the top teams and all competitors for their motivation all along the challenge.\nI would like to thank my teammates @rafiko1, @cdeotte @titericz and @matthiasanderer. You've been amazing, I've learnt a lot from you. I really enjoyed this competition.\n\n\n## Solution 1: Single transformer model\n\nThe SAINT+ model is described here https://arxiv.org/pdf/2010.12042.pdf\nThe code for our SAINT+ adaptation is available here https://github.com/rafiko1/Riiid-sharing.\n\n\n\n\n\n\nOur single model SAINT++ achieved, CV: 0.812 Public LB: 0.812, Private LB: 0.815\n\nWe trained with 95% of users first then fine tuned with all data using smart window technique (see diagram below for SAKT). The model is simple in terms of features. It only contains the four features of SAINT+ (pictured above), with one additional feature - the number of attempts of a user for specific content (hence SAINT++):\n\n\n\n* Content id\n* Lag time\n* Prior question elapsed time\n* Previous responses\n* Number of attempts \n\nThe greatest improvement in features compared to SAINT+ came from grouping lag time into seconds, unlike minutes as done in the paper. \nThen, we went bigger and bigger on the architecture and burned some GPU power 🔥. We increased on parameters of the model, most importantly the sequence length and number of layers. Final parameters of the model are as follows: \n\n\n\n\n1. Input Sequence length: 512\n2. Encoding layers: 4\n3. Decoding Layers: 4\n4. Embedding size: 288\n5. Dense Layer: 768\n6. heads: 8\n7. Dropout: 0.20\n\nWe used the Noam learning rate scheduler: with initial warmup and exponential decrease down to 2e-5. \n \nFinal improvement came from our **_recursive trick_** during inference. Here, we rounded predictions that came from the same bundle to **_0 or 1 _**- as their true response is unknown in time yet. The rounded predictions are then fed back to the model to predict the next response within the same bundle. This trick boosts CV LB +0.0025, but requires a batch size of 1, so we couldn’t ensemble multiple transformer models.\n\n\n### Solution 2: Ensemble of transformer, modified SAKT and LGB\n\n\n\n* LightGBM model scored CV=0.793, public LB=0.792 with 44 features. \nOur main features: \nQuestion correctness per content and per user, tags 1 and 2, part, elapsed time, had explanation, number of attempts, multiple lags, running average (answer), multiple rolling means/median (answers, lags, elapsed time) + weighted mean, mean after/before 30 interactions, multiple momentums (lag, answers), per part correctness, per session (8 hours split) running average. Only 3 categories: part, tags1, tags2. Train/valid split from [Tito](https://www.kaggle.com/its7171/cv-strategy). \n\n\n* Pytorch SAKT modified scored CV=0.786, LB=0.789 with additional features. Training procedure with a smart window. \n* When a user’s sequence length is larger than model input, i.e. N>W, then using random crops gives +0.002 CV versus tiled crops. And using smart random crops gives +0.003 CV versus tiled crops. Basic random crops have a low probability of selecting the early or late questions from a user’s sequence whereas smart random crops have an equally likely probability of selecting all questions from a user.\n\n \n\n\n \n\n\n* TensorFlow Transformer model alone scored CV=0.811, LB=0.811\n\n Same as solution#1 but with sequence length = 256\n\n\n\n### What did not work:\n\n\n\n* TabNet\n* Features with lectures for LGB. It worked on CV but not on LB (might be an issue in inference).\n* Post processing using absolute position of question aka. question sequence number. Plotting mean(answered_correctly) vs question number looked like the image below. We can see that the 30 first questions have a different distribution compared with the rest. Also looks like there are subsequently batches of 30 questions (becomes visible if we zoom in the plot below). PP using that information worked in CV improving by around 0.0009, but didn’t work on LB. \n\n \n\n\n\n\n\n### What worked partially:\n\n \nBut was not applicable for us within the 9h runtime limit:\n\n\n\n* More than 3 models ensemble\n* Level 2 model (XGB) could boost by +0.001\n\n\n### Lessons learnt:\n\n\n\n* Start inference Kernel as soon as possible when you need to deal with an API.\n* Try to simulate API locally to understand how data will be handled. [Tito](https://www.kaggle.com/its7171/time-series-api-iter-test-emulator)’s simulator was perfect for that purpose.\n* Push your inference (with the simulator) to the limits to debug it, it will avoid the frustrating “submission scoring error”. \n* Team-up at some point, your teammates always have good ideas.\n\nOne additional word to Kaggle @sohier I loved your API and the way it hides private data, it’s more realistic as in real world usage/production and it avoided chaotic blending. Congratulations for that, however, even if I guess you want to prevent probing, you should find a solution to provide better error feedback. If it is not possible (the more error codes the more probing) then you need to provide a simulator and guidelines allowing competitors to troubleshoot locally.\n", "votes": 68 }, { "id": 1146920, "postDate": "2021-01-10T07:05:16.133Z", "content": "@MPWare Thanks for sharing the \"Simple\" Pytorch code, but why does it score so much less than Tensorflow?
\nAre there any other differences? Would the Pytorch model be as good if you changed it to use all these features? Ty, I will really cherish that \"simple\" pytorch demo notebook you provided.
", "rawMarkdown": "@MPWare Thanks for sharing the \"Simple\" Pytorch code, but why does it score so much less than Tensorflow?\n\n1. The Pytorch model uses 2 layers instead of Tensorflow uses 4 layers\n2. The Pytorch model uses 100 sequence length instead of Tensorflow uses 512 sequence length\n3. The Pytorch model uses 256 embedding size instead of Tensorflow uses 288\n4. The Pytorch model uses dropout = 0.1 but Tensorflow uses dropout = 0.2\n5. The Pytorch model only uses exercise_id in encoder, but Tensorflow uses and response, but Tensorflow uses exercise_id, Lag time, prior question elapsed time, previous responses, number of attempts. Both models just use response_correct embedding in the decoder\n\nAre there any other differences? Would the Pytorch model be as good if you changed it to use all these features? Ty, I will really cherish that \"simple\" pytorch demo notebook you provided.", "votes": 3, "replies": [ { "id": 1147039, "postDate": "2021-01-10T08:50:14.530Z", "content": "@returnofsputnik As you've noticed, the main difference is that I'm not using exactly the same features. I'm not using attempts at all. The configuration to get LB=0.795 with Pytorch model is below, it requires seq_len=256. We've noticed that seq_len is a quite important to get a better model. The training procedure is also important, I'm not using the smart window described by @cdeotte. I've spent some time to fix the NaN loss issue with Pytorch MultiHeadAttention + padding masks. Root cause was right padding instead of left padding. I've stopped to try improving it once we got better results with @rafiko1 TensorFlow model which uses seq_len=512. Other difference is sigmoid vs softmax but I don't think it's really important.
class raw_conf:\n\n pad_mode = \"token\"\n pad_right = False\n flatten = True\n sampler = None # \"prob\" # Option to give long user's sequence higher probability\n\n seq_len = 256 # 100\n embedding_dim = 256\n exercices_id_size = 13523\n exercices_part_size = 7\n response_size = 2 \n elapsed_time_cat = True\n elapsed_time_size = 73 # Categories after binning\n lag_time_cat = True\n lag_time_size = 366 # Categories after binning\n explanation_size = 2\n\n # Model\n nhead = 8 \n num_encoder_layers = 4\n num_decoder_layers = 4\n dim_feedforward = 2048\n dropout = 0.1\n activation = None\n num_classes = 1\n loss = MaskedBCEWithLogitsLoss(num_classes)\n post_activation = \"sigmoid\"\n\n optimizer = \"Noam\" # \"Adam\"\n scheduler = \"Cosine\" if optimizer == \"Adam\" else None\n lr = 0.0001\n min_lr = 0.00005\n beta1 = 0.9\n\n BATCH_SIZE = 128\nWe're going to upload the full pytorch model on GitHub with the weights.
", "rawMarkdown": "We're going to upload the full pytorch model on GitHub with the weights.", "votes": 3 }, { "id": 1147512, "postDate": "2021-01-10T14:59:40.747Z", "content": "Thank you. Since I know PT better than TF, I want to use your PT Transformer model for many future competitions since to me it is the simplest since it invokes nn.Transformer.
", "rawMarkdown": "Thank you. Since I know PT better than TF, I want to use your PT Transformer model for many future competitions since to me it is the simplest since it invokes nn.Transformer.", "votes": 1 }, { "id": 1149848, "postDate": "2021-01-12T07:50:08.993Z", "content": "@returnofsputnik We've uploaded Pytorch full code here:
\nhttps://github.com/rafiko1/Riiid-sharing/tree/main/pytorch
\nIt's MODEL-PT-v14
@mpware thanks. I will go through it and will try to reproduce your solution.
", "rawMarkdown": "@mpware thanks. I will go through it and will try to reproduce your solution." }, { "id": 1149996, "postDate": "2021-01-12T09:33:39.397Z", "content": "The best model is with TensorFlow (seq_len=512), Public LB=0.812, Pytorch (seq_len=256) one is only Public LB=0.795.
Thanks for the complete src!
", "rawMarkdown": "Thanks for the complete src!", "votes": 1 } ] }, { "id": 1146671, "postDate": "2021-01-09T23:46:57.540Z", "content": "Congrats & thank you for sharing your approach and codes.
", "rawMarkdown": "Congrats & thank you for sharing your approach and codes.", "votes": 3 }, { "id": 1146361, "postDate": "2021-01-09T18:15:33.017Z", "content": "Congratulations and thanks for sharing your solution, I and many others will be able to learn a lot from it. I also used Tensorflow and implemented Saint+ however my code to sample sequences/users was not robust so I intend to see how the model would have performed using the sampling code you have shared.
\nI have read through the code in demo_riiid_train.ipynb and think I understand how you have set up training and validation sets but was hoping you could confirm my understanding?
\nYou sample using N_SELECT_PER_EPOCH = 100000, was this just a hyperparameter you tuned?
\nThank you
", "rawMarkdown": "Congratulations and thanks for sharing your solution, I and many others will be able to learn a lot from it. I also used Tensorflow and implemented Saint+ however my code to sample sequences/users was not robust so I intend to see how the model would have performed using the sampling code you have shared.\n\nI have read through the code in demo_riiid_train.ipynb and think I understand how you have set up training and validation sets but was hoping you could confirm my understanding?\n\n1. You allocate distinct users to both the training and validation set\n2. You then assign a probability to each user based on sequence length. For say user 115 in the training set their probability is their sequence length divided by the sum of all sequence lengths in training set.\n3. You then generate a one off validation set where you randomly sample users with replacement and take **random training crops** for each row. Based on the 'select_window_size' function\n4. For the training dataset you do the same but repeat step 3 each epoch\n\nYou sample using N_SELECT_PER_EPOCH = 100000, was this just a hyperparameter you tuned?\n\n\nThank you ", "votes": 3, "replies": [ { "id": 1146978, "postDate": "2021-01-10T08:15:12.450Z", "content": "select_window_size as the equivalent of WeightedRandomSampler in PytorchN_SELECT_PER_EPOCH is defining how many samples to pass within each epoch. It's not really a hyperparameter to tune. Just how long you'd like each epoch to be by specifying number of samples.
Thank you, will do a late submission and see how my score would have changed.
", "rawMarkdown": "Thank you, will do a late submission and see how my score would have changed.", "votes": 1 } ] }, { "id": 1146392, "postDate": "2021-01-09T18:35:27.793Z", "content": "Congrats, single 0.812/0.815 with SAINT is really great!
", "rawMarkdown": "Congrats, single 0.812/0.815 with SAINT is really great!", "votes": 4, "replies": [ { "id": 1146980, "postDate": "2021-01-10T08:17:43.870Z", "content": "Congrats on your amazing 2nd place @mamas, well-deserved!
", "rawMarkdown": "Congrats on your amazing 2nd place @mamas, well-deserved!", "votes": 1 } ] }, { "id": 1144929, "postDate": "2021-01-08T18:49:27.457Z", "content": "@mpware congrats on gold medal and thanks for sharing your approach. Would you like to share your code ?
", "rawMarkdown": "@mpware congrats on gold medal and thanks for sharing your approach. Would you like to share your code ?", "votes": 1, "replies": [ { "id": 1144950, "postDate": "2021-01-08T19:12:24.127Z", "content": "Thanks! Our best single model + features is with TensorFlow. @rafiko1 if you get a chance to share it …
\nIn the other thread, it was a Pytorch model (with simple features) that only reached public LB 0.795. Here it is if it can help.
\n# Model\nclass RIIIDModel(nn.Module):\n def __init__(self, cfg, verbose=False):\n super().__init__()\n self.response_size = cfg.response_size\n self.lag_time_size = cfg.lag_time_size\n self.elapsed_time_size = cfg.elapsed_time_size\n self.explanation_size = cfg.explanation_size\n self.attempt_size = cfg.attempt_size\n self.seq_len = cfg.seq_len\n self.embedding_dim = cfg.embedding_dim\n self.elapsed_time_cat = cfg.elapsed_time_cat\n self.lag_time_cat = cfg.lag_time_cat\n self.num_classes = cfg.num_classes\n self.verbose = verbose\n self.pad_mode = cfg.pad_mode\n\n self.pos_encoder1 = None\n self.pos_encoder2 = None\n\n additional_token_dim = 1 if self.pad_mode == \"token\" else 0\n\n # Exercices embeddings\n self.exercices_id_embedding = nn.Embedding(cfg.exercices_id_size + additional_token_dim, self.embedding_dim)\n self.exercices_part_embedding = nn.Embedding(cfg.exercices_part_size + additional_token_dim, self.embedding_dim) if cfg.exercices_part_size is not None else None\n\n # Response embeddings\n self.response_embedding = nn.Embedding(cfg.response_size + 1 + additional_token_dim, self.embedding_dim) # +1 to include start token\n\n if self.elapsed_time_cat is True:\n self.elapsed_time_embedding = nn.Embedding(cfg.elapsed_time_size + 1 + additional_token_dim, self.embedding_dim) if cfg.elapsed_time_size is not None else None # +1 to include start token\n else:\n self.elapsed_time_embedding = nn.Linear(1, self.embedding_dim, bias=False) if cfg.elapsed_time_size is not None else None # Continuous embedding\n\n if self.lag_time_cat is True:\n self.lag_time_embedding = nn.Embedding(cfg.lag_time_size + 1 + additional_token_dim, self.embedding_dim) if cfg.lag_time_size is not None else None # +1 to include start token\n else:\n self.lag_time_embedding = nn.Linear(1, self.embedding_dim, bias=False) if cfg.lag_time_size is not None else None # Continuous embedding\n\n self.explanation_embedding = nn.Embedding(cfg.explanation_size + 1 + additional_token_dim, self.embedding_dim) if cfg.explanation_size is not None else None # +1 to include start token\n\n input_features_dim = self.embedding_dim\n\n # Position encoder (relative or absolute position of the tokens in the sequence)\n if cfg.position_encoding_enabled is True:\n self.pos_encoder1 = PositionalEncoding(input_features_dim, cfg.dropout)\n self.pos_encoder2 = self.pos_encoder1\n\n # Transformer with default encoder/decoder \n self.transformer = nn.Transformer(d_model=input_features_dim, \n nhead=cfg.nhead, \n num_encoder_layers=cfg.num_encoder_layers,\n num_decoder_layers=cfg.num_decoder_layers, \n dim_feedforward=cfg.dim_feedforward, \n dropout=cfg.dropout, \n activation='relu', \n custom_encoder = None, \n custom_decoder = None)\n\n # Decoder\n self.fc = nn.Linear(input_features_dim, self.num_classes)\n\n\n # If a BoolTensor is provided, the positions with the value of True will be ignored while the position with the value of False will be unchanged.\n # tensor([[False, True, True, True],\n # [False, False, True, True],\n # [False, False, False, True],\n # [False, False, False, False]]) \n def generate_mask(self, size, diagonal=1): \n return torch.triu(torch.ones(size, size)==1, diagonal=diagonal)\n\n def forward(self, data, src_mask=None, tgt_mask=None, mem_mask=None, src_key_padding_mask=None, tgt_key_padding_mask=None, memory_key_padding_mask=None):\n\n # Each input is (BS, seq_len)\n # Content\n data_content_id = data[CONTENT_ID].long()\n # Answers\n data_response = data[TARGET].long() \n\n # Optional features\n data_part = data[PART].long() if self.exercices_part_embedding is not None else None\n if self.elapsed_time_cat is True:\n data_elapsed_time = data[PRIOR_QUESTION_ELAPSED_TIME].long() if self.elapsed_time_embedding is not None else None\n else:\n data_elapsed_time = data[PRIOR_QUESTION_ELAPSED_TIME].float().unsqueeze(2) if self.elapsed_time_embedding is not None else None\n if self.lag_time_cat is True:\n data_lag_time = data[LAG].long() if self.lag_time_embedding is not None else None\n else:\n data_lag_time = data[LAG].float().unsqueeze(2) if self.lag_time_embedding is not None else None\n\n data_explanation = data[PRIOR_QUESTION_HAD_EXPLANATION].long() if self.explanation_embedding is not None else None\n\n # Start token(s)\n\n # Add start token to correctness\n data_response = torch.roll(data_response, shifts=(0, 1), dims=(0, 1)) # Shift right the sequence\n data_response[:,0] = self.response_size # Start token (2)\n\n # Add start token to lag time\n if data_lag_time is not None:\n data_lag_time = torch.roll(data_lag_time, shifts=(0, 1), dims=(0, 1)) # Shift right the sequence\n data_lag_time[:,0] = self.lag_time_size # Start token\n\n # Add start token to elapsed time\n if data_elapsed_time is not None:\n data_elapsed_time = torch.roll(data_elapsed_time, shifts=(0, 1), dims=(0, 1)) # Shift right the sequence\n if self.elapsed_time_cat is True:\n data_elapsed_time[:,0] = self.elapsed_time_size # Start token\n else:\n data_elapsed_time[:,0] = 0.0\n\n # Add start token to explanation\n if data_explanation is not None:\n data_explanation = torch.roll(data_explanation, shifts=(0, 1), dims=(0, 1)) # Shift right the sequence\n data_explanation[:,0] = self.explanation_size # Start token\n\n # Questions, Part, Elapsed time, Lag embeddings\n x_content_id = self.exercices_id_embedding(data_content_id) # (BS, seq_len, embedding_dim)\n\n x_exercices_part = self.exercices_part_embedding(data_part) if self.exercices_part_embedding is not None else None # (BS, seq_len, embedding_dim)\n x_elapsed_time = self.elapsed_time_embedding(data_elapsed_time) if self.elapsed_time_embedding is not None else None # (BS, seq_len, embedding_dim)\n x_lag_time = self.lag_time_embedding(data_lag_time) if self.lag_time_embedding is not None else None # (BS, seq_len, embedding_dim)\n x_explanation = self.explanation_embedding(data_explanation) if self.explanation_embedding is not None else None # (BS, seq_len, embedding_dim)\n\n # Response embeddings\n x_correctness = self.response_embedding(data_response) # (BS, seq_len, embedding_dim)\n\n x_position = None\n\n # Ei (sum of embeddings)\n x_exercices = x_content_id\n\n if x_exercices_part is not None:\n x_exercices = x_exercices + x_exercices_part # (BS, seq_len, embedding_dim)\n\n x_position_exercices = self.pos_encoder1(x_exercices) if self.pos_encoder1 is not None else x_exercices # (BS, seq_len, embedding_dim)\n\n # Ri (sum of embeddings) [S, R1, Rk-1], S is start token\n\n x_responses = x_correctness \n\n if x_lag_time is not None:\n x_responses = x_responses + x_lag_time\n\n if x_elapsed_time is not None:\n x_responses = x_responses + x_elapsed_time # (BS, seq_len, embedding_dim)\n\n if x_explanation is not None:\n x_responses = x_responses + x_explanation # (BS, seq_len, embedding_dim)\n\n if x_attempt is not None:\n x_responses = x_responses + x_attempt # (BS, seq_len, embedding_dim) \n\n if x_exercices_task is not None:\n x_responses = x_responses + x_exercices_task # (BS, seq_len, embedding_dim)\n\n\n x_position_responses = self.pos_encoder2(x_responses) if self.pos_encoder2 is not None else x_responses # (BS, seq_len, embedding_dim)\n\n # Transformer src: (S,N,E), tgt:(T,N,E), src_mask:(S,S), tgt_mask:(T,T)\n # where S is the source sequence length, T is the target sequence length, N is the batch size, E is the feature number\n # output: (T,N,E)\n # src_key_padding_mask: (N,S), tgt_key_padding_mask: (N,T), memory_key_padding_mask: (N,S) \n x_position_exercices = x_position_exercices.transpose(1,0) # (seq_len, BS, embedding_dim)\n x_position_responses = x_position_responses.transpose(1,0) # (seq_len, BS, embedding_dim)\n\n x_transformer = self.transformer(src=x_position_exercices, tgt=x_position_responses, src_mask=src_mask, tgt_mask=tgt_mask, memory_mask=mem_mask, \n src_key_padding_mask=src_key_padding_mask, tgt_key_padding_mask=tgt_key_padding_mask, memory_key_padding_mask=memory_key_padding_mask) # (seq_len, BS, embedding_dim)\n x_transformer = x_transformer.transpose(1,0) # (BS, seq_len, embedding_dim)\n\n output = self.fc(x_transformer)\n output = output.squeeze(dim=2)\n\n return output\n@abdurrehman245 You can find the code for our Tensorflow model here
", "rawMarkdown": "@abdurrehman245 You can find the code for our Tensorflow model [here](https://github.com/rafiko1/Riiid-sharing)", "votes": 2 }, { "id": 1145005, "postDate": "2021-01-08T19:57:33.353Z", "content": "@mpware Thanks for sharing.
\nDid your best single model in tensorflow have same architecture as of pytorch model above apart from the features which achieved Private LB 0.815?
Also, what type of embedding (continuous or categorical)did you used for lag_time and Prior question elapsed time ?
Did you guys used any other feature in your best single model (tensorflow model) other than which you mentioned above as ?
\nContent id\nLag time\nPrior question elapsed time\nPrevious responses\nNumber of attempts \nIt would be great if you can share some training details like lr_scheduler, epochs for model convergence and anything special if you guys used.
", "rawMarkdown": "@mpware Thanks for sharing. \n\nDid your best single model in tensorflow have same architecture as of pytorch model above apart from the features which achieved `Private LB 0.815`?\n\nAlso, what type of embedding `(continuous or categorical) `did you used for `lag_time` and `Prior question elapsed time` ?\n\nDid you guys used any other feature in your best single model (tensorflow model) other than which you mentioned above as ?\n\n```\nContent id\nLag time\nPrior question elapsed time\nPrevious responses\nNumber of attempts \n```\n\nIt would be great if you can share some training details like lr_scheduler, epochs for model convergence and anything special if you guys used.", "votes": 1 }, { "id": 1145009, "postDate": "2021-01-08T20:00:11.827Z", "content": "@rafiko1 thanks for sharing the solution.
", "rawMarkdown": "@rafiko1 thanks for sharing the solution.", "votes": 2 }, { "id": 1145017, "postDate": "2021-01-08T20:11:19.707Z", "content": "For the Tensorflow model that achieved the private LB 0.815:
\nWe tried other features as well, but those were by far the most important for our model and others didn't really improve our CV much.
\nEmbedding were all categorical, as continuous didn't work out well for us.
\nWe transformed lag_time to 173 categories, which gave a nice boost to the model. We used the help of pandas qcut, like so:
N_ltg = 400 # number of groups 400 -> 173\ntrain[\"ltg\"] = train.groupby(\"user_id\")[\"timestamp\"].shift()\n\n# Lag in seconds\ntrain[\"ltg\"] = ((train[\"timestamp\"] - train[\"ltg\"])/(1000.0))\ntrain[\"ltg\"] = round(train[\"ltg\"])\n\ntrain[\"ltg\"], bins = pd.qcut(train[\"ltg\"], N_ltg, duplicates=\"drop\", retbins=True) # duplicated -> reduce about 1/2 of N_ltg\ntrain[\"ltg\"] = train[\"ltg\"].cat.codes # codes\n\n# Replace values\nN_ltg = train[\"ltg\"].nunique() # 173\ntrain[\"ltg\"] = train[\"ltg\"].replace(-1, N_ltg) # Replace -1 = NaN id \nFor the lr scheduler, we used Noam like in SAINT papers, and trained until convergence of the model, can be around 40-50 epochs of full training data per epoch.
", "rawMarkdown": "For the Tensorflow model that achieved the `private LB 0.815`:\nWe tried other features as well, but those were by far the most important for our model and others didn't really improve our CV much.\nEmbedding were all categorical, as continuous didn't work out well for us.\nWe transformed `lag_time` to 173 categories, which gave a nice boost to the model. We used the help of pandas `qcut`, like so:\n```\nN_ltg = 400 # number of groups 400 -> 173\ntrain[\"ltg\"] = train.groupby(\"user_id\")[\"timestamp\"].shift()\n \n# Lag in seconds\ntrain[\"ltg\"] = ((train[\"timestamp\"] - train[\"ltg\"])/(1000.0))\ntrain[\"ltg\"] = round(train[\"ltg\"])\n\ntrain[\"ltg\"], bins = pd.qcut(train[\"ltg\"], N_ltg, duplicates=\"drop\", retbins=True) # duplicated -> reduce about 1/2 of N_ltg\ntrain[\"ltg\"] = train[\"ltg\"].cat.codes # codes\n \n# Replace values\nN_ltg = train[\"ltg\"].nunique() # 173\ntrain[\"ltg\"] = train[\"ltg\"].replace(-1, N_ltg) # Replace -1 = NaN id \n \n```\n\nFor the lr scheduler, we used Noam like in SAINT papers, and trained until convergence of the model, can be around 40-50 epochs of full training data per epoch.", "votes": 3 }, { "id": 1145072, "postDate": "2021-01-08T21:22:49.230Z", "content": "@rafiko1 thanks.
\nThe model in the train notebook have achieved AUC 0.8001 even after 68 epochs. What's the reason of that even the model is using all features ?
", "rawMarkdown": "@rafiko1 thanks.\n\nThe model in the train notebook have achieved AUC 0.8001 even after 68 epochs. What's the reason of that even the model is using all features ? ", "votes": 1 }, { "id": 1145099, "postDate": "2021-01-08T22:10:22.567Z", "content": "Mainly because this is the \"demo\" version. You'll need to pass in the parameters of the transformers written in the description in order to get the higher score.
", "rawMarkdown": "Mainly because this is the \"demo\" version. You'll need to pass in the parameters of the transformers written in the description in order to get the higher score.", "votes": 2 }, { "id": 1145115, "postDate": "2021-01-08T22:49:33.730Z", "content": "yeah you are right. I did not notice the hyperparams.
", "rawMarkdown": "yeah you are right. I did not notice the hyperparams.", "votes": 1 }, { "id": 1145264, "postDate": "2021-01-09T03:04:37.637Z", "content": "@mpware @rafiko1 thanks for sharing, your team did a great job.
\nDid you try to calculate lagged time by task_container_id? I did something like below to calculate lagged time. I wonder what is the good way for calculating lagged time? If you have tried it, could you tell me what is better and how much does it boost your model?
\nclass UserLaggedTimeEncoder:\n def __init__(self):\n self.last_timestamp = defaultdict(float)\n self.last_task_ts = defaultdict(float)\n self.last_task_id = defaultdict(int)\n\n def update(self, user_id, timestamp, task_id):\n if task_id > self.last_task_id[user_id]:\n self.last_task_id[user_id] = task_id\n self.last_task_ts[user_id] = self.last_timestamp[user_id]\n self.last_timestamp[user_id] = timestamp\n\n def encode(self, df):\n lagged_time = np.zeros((df.shape[0],), dtype=np.float32)\n for i, row in enumerate(tqdm(zip(df['user_id'].to_numpy(),\n df['timestamp'].to_numpy(),\n df['task_container_id'].to_numpy()),\n total=df.shape[0])):\n self.update(row[0], row[1], row[2])\n lagged_time[i] = row[1] - self.last_task_ts[row[0]]\n return lagged_time\nThank you @nvhbk16k53. I noticed it's better to have a specific category for questions within a bundle. The lag time for questions within a bundle becomes 0 when you subtract.
\nYou can see how lag time was calculated in my code above.
Thank you for your reply @rafiko1 . I saw the way you calculate lag time, it's be all zeros for all questions that have the same task_container_id except the first one. Mine is questions have the same task_container_id will have the same lag time value. What is the advantage of your compare to mine?
I also experiment with LGBM that when calculate lag time like I did will give higher score.
I see. I haven't tried the same for lag time like you did. So I can't tell what has more advantage, your method might work better also for transformers.
@rafiko1 thanks for sharing all the code and thoughts.
\nI am asking a bit irrelevant question but it would be great if you can share your thoughts. Since you have done awesome with transformers so I just want to know that can we apply transformers on any time-series data or it depends on some characteristics of data to apply transformers on them.
\nFor example, if we want to apply the transformers to predict the fraud transaction of a user given its history of transactions(similarly we have history of user interactions in riid dataset), do you think transformers will be a good approach to apply on that dataset. I will appreciate your suggestions.
", "rawMarkdown": "@rafiko1 thanks for sharing all the code and thoughts.\n\nI am asking a bit irrelevant question but it would be great if you can share your thoughts. Since you have done awesome with transformers so I just want to know that can we apply transformers on any time-series data or it depends on some characteristics of data to apply transformers on them.\n\nFor example, if we want to apply the transformers to predict the fraud transaction of a user given its history of transactions(similarly we have history of user interactions in riid dataset), do you think transformers will be a good approach to apply on that dataset. I will appreciate your suggestions.", "votes": 2 }, { "id": 1146960, "postDate": "2021-01-10T08:00:26.043Z", "content": "@abdurrehman245
\nI have a feeling transformers can be applied to more use-cases of time series, including fraud detection.
\nCan't tell with uncertainty until one experiments with it.
About
\nAnd @mpware has a brilliant way of storing each user's attempt history using only 3 bits per time step to save memory. (3 bits can represent 0 thru 7 in binary).\nWould it be possible to share how you perform this? At the very end of this competition, I needed to deal with the time/memory issue once I introduced the performance features. And it took time, and I didn't have much time to fix the other inference bugs due to the new features.
\nIt would be great to learn from your great management of the memory, hope I could hear from you, at least a bit.
", "rawMarkdown": "@mpware @cdeotte \n\nAbout\n\n```\nAnd @mpware has a brilliant way of storing each user's attempt history using only 3 bits per time step to save memory. (3 bits can represent 0 thru 7 in binary).\n```\n\nWould it be possible to share how you perform this? At the very end of this competition, I needed to deal with the time/memory issue once I introduced the performance features. And it took time, and I didn't have much time to fix the other inference bugs due to the new features.\n\nIt would be great to learn from your great management of the memory, hope I could hear from you, at least a bit.", "votes": 2, "replies": [ { "id": 1148710, "postDate": "2021-01-11T10:33:09.883Z", "content": "@yihdarshieh I'm maintaining the following stucture in memory. I've been inspired from different threads in forum and benchmarked multiple solutions before keeping this one. Per user and per content attempts dictionary costs too much memory if stored as integers (even np.int8) because it could be up to 393k x 13k. To lower memory usage I'm using 3xbitarray (index is content_id) to address 2x2x2=8 attempts. As we're using attempts are categories we have 0-7 values and all attempts beyond 7 fall into last category which is indeed 7 or higher.
We also need to store per user history for:
\nAttempts history per user is computed on-fly based on the cseen values containing only the total (that's the trick).
Our different models (LGBM, Transfomer) can share this dictionary.
\nper_user_dict = defaultdict(CustomDictNP)
def bit_array():\n b = bitarray(13530, endian='little') # Higher than totals question if any new question in test set\n b.setall(False)\n return b\n\nclass CustomDictNP:\n def __init__(self):\n # Count/sum for further average\n self.qc = 0 # answers count\n self.qs = 0 # answers sum (correct)\n self.s = 0 # session id\n self.sc = 1 # session count\n self.ss = 0 # total sessions\n self.qes = 0.0 # total elapsed time\n\n # History\n self.ha = np.array([], dtype=np.bool) # answers\n self.hp = np.array([], dtype=np.int8) # parts\n self.hq = np.array([], dtype=np.int16) # questions\n self.he = np.array([], dtype=np.float32) # elapsed time\n self.hx = np.array([], dtype=np.bool) # prior question had explanation\n self.hl = np.array([], dtype=np.float32) # lag\n\n self.hat = np.array([], dtype=np.int8) # attempts\n\n # Last timestamps\n self.q = 0 # Questions\n self.timestamp_u = []\n self.timestamp_u_correct = []\n\n # content_id seen \n self.cseen0 = bit_array()\n self.cseen1 = bit_array()\n self.cseen2 = bit_array()\n\n\ndef add_attempt(custom, cid):\n new_attempt = get_attempts(custom, cid) + 1\n if new_attempt < 8:\n if new_attempt == 1: \n custom.cseen0[cid] = True\n custom.cseen1[cid] = False\n custom.cseen2[cid] = False\n elif new_attempt == 2: \n custom.cseen0[cid] = False\n custom.cseen1[cid] = True\n custom.cseen2[cid] = False\n elif new_attempt == 3: \n custom.cseen0[cid] = True\n custom.cseen1[cid] = True\n custom.cseen2[cid] = False\n elif new_attempt == 4: \n custom.cseen0[cid] = False\n custom.cseen1[cid] = False\n custom.cseen2[cid] = True\n elif new_attempt == 5: \n custom.cseen0[cid] = True\n custom.cseen1[cid] = False\n custom.cseen2[cid] = True\n elif new_attempt == 6: \n custom.cseen0[cid] = False\n custom.cseen1[cid] = True\n custom.cseen2[cid] = True\n else: \n custom.cseen0[cid] = True\n custom.cseen1[cid] = True\n custom.cseen2[cid] = True\n\ndef get_attempts(custom, cid):\n return custom.cseen0[cid] + 2*custom.cseen1[cid] + 4*custom.cseen2[cid]\n@MPWARE, thank you very much for being kindness to share, really appreciated. I haven't read it in detail to fully understand, but seems a very elegant yet efficient approach to the problem.
\nPersonally, I was able to fix this memory issue ( Per user and per content performance history ) in a short time (yet not a very clean way) by the following idea:
\nQuite a lot of (user, question) pairs are in fact with (n_attempt, n_correctness) being (1, 1) and (1, 0). The pairs with attempt >= 2 is quite rare. Therefore, I have 2 nested dictionaries:
1. The 1st dict is: user_id -> question_id -> the performances for those n_attempt >= 2 (quite small dict)\n\n2. The 2nd dict: (for those pairs with `n_attemp = 1`)\n\n {\n user_id: \n { \n 'correct': [q_id_x, q_id_y, ...]\n 'incorrect': [q_id_m, q_in_, ...]\n }\n }\nIf (user, question) can't be found in the above 2 dictionaries, it means that question is not seen by the user.
\nThe idea is kind similar to sparse matrix, where we only store the content with values.
\nDespite this effort, it still take 4G or 5G memory when loading into memory. I will try to compare your approach later.
\nComments to my approaches are appreciated (if any). Thanks again for sharing.
", "rawMarkdown": "@MPWARE, thank you very much for being kindness to share, really appreciated. I haven't read it in detail to fully understand, but seems a very elegant yet efficient approach to the problem.\n\nPersonally, I was able to fix this memory issue ( Per user and per content performance history ) in a short time (yet not a very clean way) by the following idea:\n\nQuite a lot of (user, question) pairs are in fact with (n_attempt, n_correctness) being (1, 1) and (1, 0). The pairs with `attempt >= 2` is quite rare. Therefore, I have 2 nested dictionaries:\n\n 1. The 1st dict is: user_id -> question_id -> the performances for those n_attempt >= 2 (quite small dict)\n \n 2. The 2nd dict: (for those pairs with `n_attemp = 1`)\n\n {\n user_id: \n { \n 'correct': [q_id_x, q_id_y, ...]\n 'incorrect': [q_id_m, q_in_, ...]\n }\n }\n\nIf (user, question) can't be found in the above 2 dictionaries, it means that question is not seen by the user.\n\nThe idea is kind similar to sparse matrix, where we only store the content with values.\n\nDespite this effort, it still take 4G or 5G memory when loading into memory. I will try to compare your approach later.\n\nComments to my approaches are appreciated (if any). Thanks again for sharing.", "votes": 2 }, { "id": 1149507, "postDate": "2021-01-11T22:23:04.897Z", "content": "@mpware I understand your way of memory management now. Great.
\nI am wondering why your team don't make a feature of the nb of correctness per user per content, or better, the ratio, if you already have such good way to manage memory. It think you might even get better by doing so, although in my case, I can't see the positive effect on LB due to the input inconsistency bug in my pipeline. I am just wondering the reason your team don't use it
", "rawMarkdown": "@mpware I understand your way of memory management now. Great.\n\nI am wondering why your team don't make a feature of the nb of correctness per user per content, or better, the ratio, if you already have such good way to manage memory. It think you might even get better by doing so, although in my case, I can't see the positive effect on LB due to the input inconsistency bug in my pipeline. I am just wondering the reason your team don't use it", "votes": 1 }, { "id": 1149648, "postDate": "2021-01-12T03:27:22.690Z", "content": "Really cool use of bit arrays! I didn't extend it to do the counting part as well! Sweet!
\ndef get_attempts(custom, cid):\n return custom.cseen0[cid] + 2*custom.cseen1[cid] + 4*custom.cseen2[cid]\nThis is basically bit shifting, correct? (Converting the bits back to decimal radix)
\n # content_id seen \n self.cseen0 = bit_array()\n self.cseen1 = bit_array()\n self.cseen2 = bit_array()\nThis basically represents the 3 bits needed to count till 7 i believe.[ with cseen2 as the MSB]
Another use of the same bitarray's is here @yihdarshieh! Basically a bool flag to detect whether use has seen that content in past or not.
", "rawMarkdown": "Really cool use of bit arrays! I didn't extend it to do the counting part as well! Sweet!\n\n```\ndef get_attempts(custom, cid):\n return custom.cseen0[cid] + 2*custom.cseen1[cid] + 4*custom.cseen2[cid]\n```\n\nThis is basically bit shifting, correct? (Converting the bits back to decimal radix)\n\n\n```\n # content_id seen \n self.cseen0 = bit_array()\n self.cseen1 = bit_array()\n self.cseen2 = bit_array()\n```\nThis basically represents the 3 bits needed to count till 7 i believe.[ with `cseen2` as the MSB]\n\nAnother use of the same bitarray's is [here](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/194266#1069347) @yihdarshieh! Basically a bool flag to detect whether use has seen that content in past or not.", "votes": 2 }, { "id": 1149990, "postDate": "2021-01-12T09:31:09.773Z", "content": "We've tried it but results were close (little bit lower) than without.
", "rawMarkdown": "We've tried it but results were close (little bit lower) than without.", "votes": 1 }, { "id": 1150173, "postDate": "2021-01-12T12:30:57.373Z", "content": "Alright, that is quite strange. Do anyone in your team have some insight about why this is not working?
", "rawMarkdown": "Alright, that is quite strange. Do anyone in your team have some insight about why this is not working?" }, { "id": 1150174, "postDate": "2021-01-12T12:32:56.903Z", "content": "@adityaecdrid I saw that thread, but didn't take the time to read it carefully and find the shown code, I was so depressed about my inference bug at that time. Anyway, now I know there is bitarray.
", "rawMarkdown": "@adityaecdrid I saw that thread, but didn't take the time to read it carefully and find the shown code, I was so depressed about my inference bug at that time. Anyway, now I know there is bitarray. " } ] }, { "id": 1145911, "postDate": "2021-01-09T12:23:13.433Z", "content": "Congratz to you all !
", "rawMarkdown": "Congratz to you all !", "votes": 2 }, { "id": 1145432, "postDate": "2021-01-09T06:24:04.433Z", "content": "thanks for sharing! I am a beginer to transformer model. I think I need to learn the model before seeing your share.
", "rawMarkdown": "thanks for sharing! I am a beginer to transformer model. I think I need to learn the model before seeing your share.", "votes": 2 }, { "id": 1145294, "postDate": "2021-01-09T03:52:07.940Z", "content": "Thanks for your sharing and congratulations to your team.
\nWould you like to explain how to add the \"Number of attempts\" feature into encoder?
\nIs it convert to embedding?
\nThank you!!
Yes as an embedding and add it to other encoder input embeddings.
\nThe encoder input begins as x = tf.keras.layers.Embedding(number of unique content id, 288)(CONTENT ID SEQUENCE). Then you add positional encoding, x += positional_encoding(window size, 288). And last you add \"Number of attempts\" encoding. x += tf.keras.layers.Embedding(max number of attempts + 1, 288)(NUMBER OF ATTEMPTS SEQUENCE).
The maximum number of attempts is clipped with NUMBER OF ATTEMPTS SEQUENCE = np.clip(NUMBER OF ATTEMPTS SEQUENCE, 0, 7).
And @mpware has a brilliant way of storing each user's attempt history using only 3 bits per time step to save memory. (3 bits can represent 0 thru 7 in binary).
", "rawMarkdown": "Yes as an embedding and add it to other encoder input embeddings. \n\nThe encoder input begins as `x = tf.keras.layers.Embedding(number of unique content id, 288)(CONTENT ID SEQUENCE)`. Then you add positional encoding, `x += positional_encoding(window size, 288)`. And last you add \"Number of attempts\" encoding. `x += tf.keras.layers.Embedding(max number of attempts + 1, 288)(NUMBER OF ATTEMPTS SEQUENCE)`.\n\nThe maximum number of attempts is clipped with `NUMBER OF ATTEMPTS SEQUENCE = np.clip(NUMBER OF ATTEMPTS SEQUENCE, 0, 7)`.\n\nAnd @mpware has a brilliant way of storing each user's attempt history using only 3 bits per time step to save memory. (3 bits can represent 0 thru 7 in binary).", "votes": 3 }, { "id": 1145376, "postDate": "2021-01-09T05:17:44.343Z", "content": "Very impressive. Thanks for your reply.
", "rawMarkdown": "Very impressive. Thanks for your reply.", "votes": 2 }, { "id": 1145897, "postDate": "2021-01-09T12:17:48.453Z", "content": "@cdeotte I can see that 99% of of attempts on content_id is <=7. Is that the reason of clipping the max value to 7?
x += tf.keras.layers.Embedding(max number of attempts + 1, 288)(NUMBER OF ATTEMPTS SEQUENCE)
so max number of attempts will be 7 then and why are you adding 1 more to number of attempts ?
@abdurrehman245 The real reason was to limit memory usage for inference. We limit attempts from 0 to 7, everything above 7 falls in the last category which is indeed 7 and beyond.
", "rawMarkdown": "@abdurrehman245 The real reason was to limit memory usage for inference. We limit attempts from 0 to 7, everything above 7 falls in the last category which is indeed 7 and beyond.", "votes": 1 }, { "id": 1148196, "postDate": "2021-01-11T01:43:04.583Z", "content": "\n\n\nwhy are you adding 1 more to number of attempts?
\n
Since the max is 7, there are 8 unique values, i.e. 0,1,2,3,4,5,6,7. So the embedding needs to handle a categorical variable of cardinality 8.
", "rawMarkdown": "@abdurrehman245\n>why are you adding 1 more to number of attempts?\n\nSince the max is 7, there are 8 unique values, i.e. 0,1,2,3,4,5,6,7. So the embedding needs to handle a categorical variable of cardinality 8.", "votes": 1 } ] }, { "id": 1145155, "postDate": "2021-01-09T00:10:55.967Z", "content": "@mpware Congrats on the gold medal and nice single model solution !
", "rawMarkdown": "@mpware Congrats on the gold medal and nice single model solution !", "votes": 2, "replies": [ { "id": 1145680, "postDate": "2021-01-09T09:39:16.443Z", "content": "@tikutiku Thank you!
", "rawMarkdown": " @tikutiku Thank you! " } ] }, { "id": 1145144, "postDate": "2021-01-08T23:44:57.310Z", "content": "@mpware Congratulations to your team, very impressed that single RAINT+ with these 5 features can reach such a high score. My best RAINT+ model is only 0.802 on private
\nWould like to know more details about which of these features are on the encoder side and which are on the decoder side?
\nContent id\nLag time\nPrior question elapsed time\nPrevious responses\nNumber of attempts\n
\nThere's no doubt on the content id and response. For other features, could you please explain a little bit about which parts did you put the feature in?
Thanks. Encoder had Content id and Number of attempts. Decoder had Lag time, Prior question elapsed time, and Previous responses.
", "rawMarkdown": "Thanks. Encoder had Content id and Number of attempts. Decoder had Lag time, Prior question elapsed time, and Previous responses.", "votes": 2 }, { "id": 1145175, "postDate": "2021-01-09T00:31:15.080Z", "content": "Got it, thanks!
", "rawMarkdown": "Got it, thanks! ", "votes": 2 } ] }, { "id": 1144738, "postDate": "2021-01-08T16:32:19.443Z", "content": "Congratulations on 16th place and gold medal @mpware @cdeotte and team. Good work and strongly transformer model
", "rawMarkdown": "Congratulations on 16th place and gold medal @mpware @cdeotte and team. Good work and strongly transformer model", "votes": 2 }, { "id": 1147557, "postDate": "2021-01-10T15:21:41.433Z", "content": "How about the question's tags, it's a VarLen feature. Do you have any idea of adding VarLen feature in the RAINT+? What I did is sorting the tags in each question then combine them and map to an integer. But it's not an optimal way, since the information of each individual tag is lost.
", "rawMarkdown": "How about the question's tags, it's a VarLen feature. Do you have any idea of adding VarLen feature in the RAINT+? What I did is sorting the tags in each question then combine them and map to an integer. But it's not an optimal way, since the information of each individual tag is lost.", "replies": [ { "id": 1148508, "postDate": "2021-01-11T07:28:17.583Z", "content": "I use a tag embedding, i.e. map each integer in [-1, 188) to vectors of a fixed dim (in my case 256). Then these embedding are averaged (the embedding of -1 is for padding for the tags, as you said, it is varlen, and it is ignored during the averaging). I don't know how the team of @mpware did it though.
", "rawMarkdown": "I use a tag embedding, i.e. map each integer in [-1, 188) to vectors of a fixed dim (in my case 256). Then these embedding are averaged (the embedding of -1 is for padding for the tags, as you said, it is varlen, and it is ignored during the averaging). I don't know how the team of @mpware did it though." } ] }, { "id": 1145433, "postDate": "2021-01-09T06:24:04.433Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1146920, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2021-01-10T07:05:16.133000", "content": "@MPWare Thanks for sharing the \"Simple\" Pytorch code, but why does it score so much less than Tensorflow?
\nAre there any other differences? Would the Pytorch model be as good if you changed it to use all these features? Ty, I will really cherish that \"simple\" pytorch demo notebook you provided.
", "votes": 3, "replies": [ { "id": 1147039, "author_name": "MPWARE", "author_url": "", "post_date": "2021-01-10T08:50:14.530000", "content": "@returnofsputnik As you've noticed, the main difference is that I'm not using exactly the same features. I'm not using attempts at all. The configuration to get LB=0.795 with Pytorch model is below, it requires seq_len=256. We've noticed that seq_len is a quite important to get a better model. The training procedure is also important, I'm not using the smart window described by @cdeotte. I've spent some time to fix the NaN loss issue with Pytorch MultiHeadAttention + padding masks. Root cause was right padding instead of left padding. I've stopped to try improving it once we got better results with @rafiko1 TensorFlow model which uses seq_len=512. Other difference is sigmoid vs softmax but I don't think it's really important.
class raw_conf:\n\n pad_mode = \"token\"\n pad_right = False\n flatten = True\n sampler = None # \"prob\" # Option to give long user's sequence higher probability\n\n seq_len = 256 # 100\n embedding_dim = 256\n exercices_id_size = 13523\n exercices_part_size = 7\n response_size = 2 \n elapsed_time_cat = True\n elapsed_time_size = 73 # Categories after binning\n lag_time_cat = True\n lag_time_size = 366 # Categories after binning\n explanation_size = 2\n\n # Model\n nhead = 8 \n num_encoder_layers = 4\n num_decoder_layers = 4\n dim_feedforward = 2048\n dropout = 0.1\n activation = None\n num_classes = 1\n loss = MaskedBCEWithLogitsLoss(num_classes)\n post_activation = \"sigmoid\"\n\n optimizer = \"Noam\" # \"Adam\"\n scheduler = \"Cosine\" if optimizer == \"Adam\" else None\n lr = 0.0001\n min_lr = 0.00005\n beta1 = 0.9\n\n BATCH_SIZE = 128\nWe're going to upload the full pytorch model on GitHub with the weights.
", "votes": 3, "replies": [] }, { "id": 1147512, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2021-01-10T14:59:40.747000", "content": "Thank you. Since I know PT better than TF, I want to use your PT Transformer model for many future competitions since to me it is the simplest since it invokes nn.Transformer.
", "votes": 1, "replies": [] }, { "id": 1149848, "author_name": "MPWARE", "author_url": "", "post_date": "2021-01-12T07:50:08.993000", "content": "@returnofsputnik We've uploaded Pytorch full code here:
\nhttps://github.com/rafiko1/Riiid-sharing/tree/main/pytorch
\nIt's MODEL-PT-v14
@mpware thanks. I will go through it and will try to reproduce your solution.
", "votes": 0, "replies": [] }, { "id": 1149996, "author_name": "MPWARE", "author_url": "", "post_date": "2021-01-12T09:33:39.397000", "content": "The best model is with TensorFlow (seq_len=512), Public LB=0.812, Pytorch (seq_len=256) one is only Public LB=0.795.
Thanks for the complete src!
", "votes": 1, "replies": [] } ] }, { "id": 1146671, "author_name": "u++", "author_url": "", "post_date": "2021-01-09T23:46:57.540000", "content": "Congrats & thank you for sharing your approach and codes.
", "votes": 3, "replies": [] }, { "id": 1146361, "author_name": "Darren Lahr", "author_url": "", "post_date": "2021-01-09T18:15:33.017000", "content": "Congratulations and thanks for sharing your solution, I and many others will be able to learn a lot from it. I also used Tensorflow and implemented Saint+ however my code to sample sequences/users was not robust so I intend to see how the model would have performed using the sampling code you have shared.
\nI have read through the code in demo_riiid_train.ipynb and think I understand how you have set up training and validation sets but was hoping you could confirm my understanding?
\nYou sample using N_SELECT_PER_EPOCH = 100000, was this just a hyperparameter you tuned?
\nThank you
", "votes": 3, "replies": [ { "id": 1146978, "author_name": "Rafi Hai", "author_url": "", "post_date": "2021-01-10T08:15:12.450000", "content": "select_window_size as the equivalent of WeightedRandomSampler in PytorchN_SELECT_PER_EPOCH is defining how many samples to pass within each epoch. It's not really a hyperparameter to tune. Just how long you'd like each epoch to be by specifying number of samples.
Thank you, will do a late submission and see how my score would have changed.
", "votes": 1, "replies": [] } ] }, { "id": 1146392, "author_name": "mamas", "author_url": "", "post_date": "2021-01-09T18:35:27.793000", "content": "Congrats, single 0.812/0.815 with SAINT is really great!
", "votes": 4, "replies": [ { "id": 1146980, "author_name": "Rafi Hai", "author_url": "", "post_date": "2021-01-10T08:17:43.870000", "content": "Congrats on your amazing 2nd place @mamas, well-deserved!
", "votes": 1, "replies": [] } ] }, { "id": 1144929, "author_name": "Abdur Rehman", "author_url": "", "post_date": "2021-01-08T18:49:27.457000", "content": "@mpware congrats on gold medal and thanks for sharing your approach. Would you like to share your code ?
", "votes": 1, "replies": [ { "id": 1144950, "author_name": "MPWARE", "author_url": "", "post_date": "2021-01-08T19:12:24.127000", "content": "Thanks! Our best single model + features is with TensorFlow. @rafiko1 if you get a chance to share it …
\nIn the other thread, it was a Pytorch model (with simple features) that only reached public LB 0.795. Here it is if it can help.
\n# Model\nclass RIIIDModel(nn.Module):\n def __init__(self, cfg, verbose=False):\n super().__init__()\n self.response_size = cfg.response_size\n self.lag_time_size = cfg.lag_time_size\n self.elapsed_time_size = cfg.elapsed_time_size\n self.explanation_size = cfg.explanation_size\n self.attempt_size = cfg.attempt_size\n self.seq_len = cfg.seq_len\n self.embedding_dim = cfg.embedding_dim\n self.elapsed_time_cat = cfg.elapsed_time_cat\n self.lag_time_cat = cfg.lag_time_cat\n self.num_classes = cfg.num_classes\n self.verbose = verbose\n self.pad_mode = cfg.pad_mode\n\n self.pos_encoder1 = None\n self.pos_encoder2 = None\n\n additional_token_dim = 1 if self.pad_mode == \"token\" else 0\n\n # Exercices embeddings\n self.exercices_id_embedding = nn.Embedding(cfg.exercices_id_size + additional_token_dim, self.embedding_dim)\n self.exercices_part_embedding = nn.Embedding(cfg.exercices_part_size + additional_token_dim, self.embedding_dim) if cfg.exercices_part_size is not None else None\n\n # Response embeddings\n self.response_embedding = nn.Embedding(cfg.response_size + 1 + additional_token_dim, self.embedding_dim) # +1 to include start token\n\n if self.elapsed_time_cat is True:\n self.elapsed_time_embedding = nn.Embedding(cfg.elapsed_time_size + 1 + additional_token_dim, self.embedding_dim) if cfg.elapsed_time_size is not None else None # +1 to include start token\n else:\n self.elapsed_time_embedding = nn.Linear(1, self.embedding_dim, bias=False) if cfg.elapsed_time_size is not None else None # Continuous embedding\n\n if self.lag_time_cat is True:\n self.lag_time_embedding = nn.Embedding(cfg.lag_time_size + 1 + additional_token_dim, self.embedding_dim) if cfg.lag_time_size is not None else None # +1 to include start token\n else:\n self.lag_time_embedding = nn.Linear(1, self.embedding_dim, bias=False) if cfg.lag_time_size is not None else None # Continuous embedding\n\n self.explanation_embedding = nn.Embedding(cfg.explanation_size + 1 + additional_token_dim, self.embedding_dim) if cfg.explanation_size is not None else None # +1 to include start token\n\n input_features_dim = self.embedding_dim\n\n # Position encoder (relative or absolute position of the tokens in the sequence)\n if cfg.position_encoding_enabled is True:\n self.pos_encoder1 = PositionalEncoding(input_features_dim, cfg.dropout)\n self.pos_encoder2 = self.pos_encoder1\n\n # Transformer with default encoder/decoder \n self.transformer = nn.Transformer(d_model=input_features_dim, \n nhead=cfg.nhead, \n num_encoder_layers=cfg.num_encoder_layers,\n num_decoder_layers=cfg.num_decoder_layers, \n dim_feedforward=cfg.dim_feedforward, \n dropout=cfg.dropout, \n activation='relu', \n custom_encoder = None, \n custom_decoder = None)\n\n # Decoder\n self.fc = nn.Linear(input_features_dim, self.num_classes)\n\n\n # If a BoolTensor is provided, the positions with the value of True will be ignored while the position with the value of False will be unchanged.\n # tensor([[False, True, True, True],\n # [False, False, True, True],\n # [False, False, False, True],\n # [False, False, False, False]]) \n def generate_mask(self, size, diagonal=1): \n return torch.triu(torch.ones(size, size)==1, diagonal=diagonal)\n\n def forward(self, data, src_mask=None, tgt_mask=None, mem_mask=None, src_key_padding_mask=None, tgt_key_padding_mask=None, memory_key_padding_mask=None):\n\n # Each input is (BS, seq_len)\n # Content\n data_content_id = data[CONTENT_ID].long()\n # Answers\n data_response = data[TARGET].long() \n\n # Optional features\n data_part = data[PART].long() if self.exercices_part_embedding is not None else None\n if self.elapsed_time_cat is True:\n data_elapsed_time = data[PRIOR_QUESTION_ELAPSED_TIME].long() if self.elapsed_time_embedding is not None else None\n else:\n data_elapsed_time = data[PRIOR_QUESTION_ELAPSED_TIME].float().unsqueeze(2) if self.elapsed_time_embedding is not None else None\n if self.lag_time_cat is True:\n data_lag_time = data[LAG].long() if self.lag_time_embedding is not None else None\n else:\n data_lag_time = data[LAG].float().unsqueeze(2) if self.lag_time_embedding is not None else None\n\n data_explanation = data[PRIOR_QUESTION_HAD_EXPLANATION].long() if self.explanation_embedding is not None else None\n\n # Start token(s)\n\n # Add start token to correctness\n data_response = torch.roll(data_response, shifts=(0, 1), dims=(0, 1)) # Shift right the sequence\n data_response[:,0] = self.response_size # Start token (2)\n\n # Add start token to lag time\n if data_lag_time is not None:\n data_lag_time = torch.roll(data_lag_time, shifts=(0, 1), dims=(0, 1)) # Shift right the sequence\n data_lag_time[:,0] = self.lag_time_size # Start token\n\n # Add start token to elapsed time\n if data_elapsed_time is not None:\n data_elapsed_time = torch.roll(data_elapsed_time, shifts=(0, 1), dims=(0, 1)) # Shift right the sequence\n if self.elapsed_time_cat is True:\n data_elapsed_time[:,0] = self.elapsed_time_size # Start token\n else:\n data_elapsed_time[:,0] = 0.0\n\n # Add start token to explanation\n if data_explanation is not None:\n data_explanation = torch.roll(data_explanation, shifts=(0, 1), dims=(0, 1)) # Shift right the sequence\n data_explanation[:,0] = self.explanation_size # Start token\n\n # Questions, Part, Elapsed time, Lag embeddings\n x_content_id = self.exercices_id_embedding(data_content_id) # (BS, seq_len, embedding_dim)\n\n x_exercices_part = self.exercices_part_embedding(data_part) if self.exercices_part_embedding is not None else None # (BS, seq_len, embedding_dim)\n x_elapsed_time = self.elapsed_time_embedding(data_elapsed_time) if self.elapsed_time_embedding is not None else None # (BS, seq_len, embedding_dim)\n x_lag_time = self.lag_time_embedding(data_lag_time) if self.lag_time_embedding is not None else None # (BS, seq_len, embedding_dim)\n x_explanation = self.explanation_embedding(data_explanation) if self.explanation_embedding is not None else None # (BS, seq_len, embedding_dim)\n\n # Response embeddings\n x_correctness = self.response_embedding(data_response) # (BS, seq_len, embedding_dim)\n\n x_position = None\n\n # Ei (sum of embeddings)\n x_exercices = x_content_id\n\n if x_exercices_part is not None:\n x_exercices = x_exercices + x_exercices_part # (BS, seq_len, embedding_dim)\n\n x_position_exercices = self.pos_encoder1(x_exercices) if self.pos_encoder1 is not None else x_exercices # (BS, seq_len, embedding_dim)\n\n # Ri (sum of embeddings) [S, R1, Rk-1], S is start token\n\n x_responses = x_correctness \n\n if x_lag_time is not None:\n x_responses = x_responses + x_lag_time\n\n if x_elapsed_time is not None:\n x_responses = x_responses + x_elapsed_time # (BS, seq_len, embedding_dim)\n\n if x_explanation is not None:\n x_responses = x_responses + x_explanation # (BS, seq_len, embedding_dim)\n\n if x_attempt is not None:\n x_responses = x_responses + x_attempt # (BS, seq_len, embedding_dim) \n\n if x_exercices_task is not None:\n x_responses = x_responses + x_exercices_task # (BS, seq_len, embedding_dim)\n\n\n x_position_responses = self.pos_encoder2(x_responses) if self.pos_encoder2 is not None else x_responses # (BS, seq_len, embedding_dim)\n\n # Transformer src: (S,N,E), tgt:(T,N,E), src_mask:(S,S), tgt_mask:(T,T)\n # where S is the source sequence length, T is the target sequence length, N is the batch size, E is the feature number\n # output: (T,N,E)\n # src_key_padding_mask: (N,S), tgt_key_padding_mask: (N,T), memory_key_padding_mask: (N,S) \n x_position_exercices = x_position_exercices.transpose(1,0) # (seq_len, BS, embedding_dim)\n x_position_responses = x_position_responses.transpose(1,0) # (seq_len, BS, embedding_dim)\n\n x_transformer = self.transformer(src=x_position_exercices, tgt=x_position_responses, src_mask=src_mask, tgt_mask=tgt_mask, memory_mask=mem_mask, \n src_key_padding_mask=src_key_padding_mask, tgt_key_padding_mask=tgt_key_padding_mask, memory_key_padding_mask=memory_key_padding_mask) # (seq_len, BS, embedding_dim)\n x_transformer = x_transformer.transpose(1,0) # (BS, seq_len, embedding_dim)\n\n output = self.fc(x_transformer)\n output = output.squeeze(dim=2)\n\n return output\n@abdurrehman245 You can find the code for our Tensorflow model here
", "votes": 2, "replies": [] }, { "id": 1145005, "author_name": "Abdur Rehman", "author_url": "", "post_date": "2021-01-08T19:57:33.353000", "content": "@mpware Thanks for sharing.
\nDid your best single model in tensorflow have same architecture as of pytorch model above apart from the features which achieved Private LB 0.815?
Also, what type of embedding (continuous or categorical)did you used for lag_time and Prior question elapsed time ?
Did you guys used any other feature in your best single model (tensorflow model) other than which you mentioned above as ?
\nContent id\nLag time\nPrior question elapsed time\nPrevious responses\nNumber of attempts \nIt would be great if you can share some training details like lr_scheduler, epochs for model convergence and anything special if you guys used.
", "votes": 1, "replies": [] }, { "id": 1145009, "author_name": "Abdur Rehman", "author_url": "", "post_date": "2021-01-08T20:00:11.827000", "content": "@rafiko1 thanks for sharing the solution.
", "votes": 2, "replies": [] }, { "id": 1145017, "author_name": "Rafi Hai", "author_url": "", "post_date": "2021-01-08T20:11:19.707000", "content": "For the Tensorflow model that achieved the private LB 0.815:
\nWe tried other features as well, but those were by far the most important for our model and others didn't really improve our CV much.
\nEmbedding were all categorical, as continuous didn't work out well for us.
\nWe transformed lag_time to 173 categories, which gave a nice boost to the model. We used the help of pandas qcut, like so:
N_ltg = 400 # number of groups 400 -> 173\ntrain[\"ltg\"] = train.groupby(\"user_id\")[\"timestamp\"].shift()\n\n# Lag in seconds\ntrain[\"ltg\"] = ((train[\"timestamp\"] - train[\"ltg\"])/(1000.0))\ntrain[\"ltg\"] = round(train[\"ltg\"])\n\ntrain[\"ltg\"], bins = pd.qcut(train[\"ltg\"], N_ltg, duplicates=\"drop\", retbins=True) # duplicated -> reduce about 1/2 of N_ltg\ntrain[\"ltg\"] = train[\"ltg\"].cat.codes # codes\n\n# Replace values\nN_ltg = train[\"ltg\"].nunique() # 173\ntrain[\"ltg\"] = train[\"ltg\"].replace(-1, N_ltg) # Replace -1 = NaN id \nFor the lr scheduler, we used Noam like in SAINT papers, and trained until convergence of the model, can be around 40-50 epochs of full training data per epoch.
", "votes": 3, "replies": [] }, { "id": 1145072, "author_name": "Abdur Rehman", "author_url": "", "post_date": "2021-01-08T21:22:49.230000", "content": "@rafiko1 thanks.
\nThe model in the train notebook have achieved AUC 0.8001 even after 68 epochs. What's the reason of that even the model is using all features ?
", "votes": 1, "replies": [] }, { "id": 1145099, "author_name": "Rafi Hai", "author_url": "", "post_date": "2021-01-08T22:10:22.567000", "content": "Mainly because this is the \"demo\" version. You'll need to pass in the parameters of the transformers written in the description in order to get the higher score.
", "votes": 2, "replies": [] }, { "id": 1145115, "author_name": "Abdur Rehman", "author_url": "", "post_date": "2021-01-08T22:49:33.730000", "content": "yeah you are right. I did not notice the hyperparams.
", "votes": 1, "replies": [] }, { "id": 1145264, "author_name": "Hiep Nguyen", "author_url": "", "post_date": "2021-01-09T03:04:37.637000", "content": "@mpware @rafiko1 thanks for sharing, your team did a great job.
\nDid you try to calculate lagged time by task_container_id? I did something like below to calculate lagged time. I wonder what is the good way for calculating lagged time? If you have tried it, could you tell me what is better and how much does it boost your model?
\nclass UserLaggedTimeEncoder:\n def __init__(self):\n self.last_timestamp = defaultdict(float)\n self.last_task_ts = defaultdict(float)\n self.last_task_id = defaultdict(int)\n\n def update(self, user_id, timestamp, task_id):\n if task_id > self.last_task_id[user_id]:\n self.last_task_id[user_id] = task_id\n self.last_task_ts[user_id] = self.last_timestamp[user_id]\n self.last_timestamp[user_id] = timestamp\n\n def encode(self, df):\n lagged_time = np.zeros((df.shape[0],), dtype=np.float32)\n for i, row in enumerate(tqdm(zip(df['user_id'].to_numpy(),\n df['timestamp'].to_numpy(),\n df['task_container_id'].to_numpy()),\n total=df.shape[0])):\n self.update(row[0], row[1], row[2])\n lagged_time[i] = row[1] - self.last_task_ts[row[0]]\n return lagged_time\nThank you @nvhbk16k53. I noticed it's better to have a specific category for questions within a bundle. The lag time for questions within a bundle becomes 0 when you subtract.
\nYou can see how lag time was calculated in my code above.
Thank you for your reply @rafiko1 . I saw the way you calculate lag time, it's be all zeros for all questions that have the same task_container_id except the first one. Mine is questions have the same task_container_id will have the same lag time value. What is the advantage of your compare to mine?
I also experiment with LGBM that when calculate lag time like I did will give higher score.
I see. I haven't tried the same for lag time like you did. So I can't tell what has more advantage, your method might work better also for transformers.
@rafiko1 thanks for sharing all the code and thoughts.
\nI am asking a bit irrelevant question but it would be great if you can share your thoughts. Since you have done awesome with transformers so I just want to know that can we apply transformers on any time-series data or it depends on some characteristics of data to apply transformers on them.
\nFor example, if we want to apply the transformers to predict the fraud transaction of a user given its history of transactions(similarly we have history of user interactions in riid dataset), do you think transformers will be a good approach to apply on that dataset. I will appreciate your suggestions.
", "votes": 2, "replies": [] }, { "id": 1146960, "author_name": "Rafi Hai", "author_url": "", "post_date": "2021-01-10T08:00:26.043000", "content": "@abdurrehman245
\nI have a feeling transformers can be applied to more use-cases of time series, including fraud detection.
\nCan't tell with uncertainty until one experiments with it.
About
\nAnd @mpware has a brilliant way of storing each user's attempt history using only 3 bits per time step to save memory. (3 bits can represent 0 thru 7 in binary).\nWould it be possible to share how you perform this? At the very end of this competition, I needed to deal with the time/memory issue once I introduced the performance features. And it took time, and I didn't have much time to fix the other inference bugs due to the new features.
\nIt would be great to learn from your great management of the memory, hope I could hear from you, at least a bit.
", "votes": 2, "replies": [ { "id": 1148710, "author_name": "MPWARE", "author_url": "", "post_date": "2021-01-11T10:33:09.883000", "content": "@yihdarshieh I'm maintaining the following stucture in memory. I've been inspired from different threads in forum and benchmarked multiple solutions before keeping this one. Per user and per content attempts dictionary costs too much memory if stored as integers (even np.int8) because it could be up to 393k x 13k. To lower memory usage I'm using 3xbitarray (index is content_id) to address 2x2x2=8 attempts. As we're using attempts are categories we have 0-7 values and all attempts beyond 7 fall into last category which is indeed 7 or higher.
We also need to store per user history for:
\nAttempts history per user is computed on-fly based on the cseen values containing only the total (that's the trick).
Our different models (LGBM, Transfomer) can share this dictionary.
\nper_user_dict = defaultdict(CustomDictNP)
def bit_array():\n b = bitarray(13530, endian='little') # Higher than totals question if any new question in test set\n b.setall(False)\n return b\n\nclass CustomDictNP:\n def __init__(self):\n # Count/sum for further average\n self.qc = 0 # answers count\n self.qs = 0 # answers sum (correct)\n self.s = 0 # session id\n self.sc = 1 # session count\n self.ss = 0 # total sessions\n self.qes = 0.0 # total elapsed time\n\n # History\n self.ha = np.array([], dtype=np.bool) # answers\n self.hp = np.array([], dtype=np.int8) # parts\n self.hq = np.array([], dtype=np.int16) # questions\n self.he = np.array([], dtype=np.float32) # elapsed time\n self.hx = np.array([], dtype=np.bool) # prior question had explanation\n self.hl = np.array([], dtype=np.float32) # lag\n\n self.hat = np.array([], dtype=np.int8) # attempts\n\n # Last timestamps\n self.q = 0 # Questions\n self.timestamp_u = []\n self.timestamp_u_correct = []\n\n # content_id seen \n self.cseen0 = bit_array()\n self.cseen1 = bit_array()\n self.cseen2 = bit_array()\n\n\ndef add_attempt(custom, cid):\n new_attempt = get_attempts(custom, cid) + 1\n if new_attempt < 8:\n if new_attempt == 1: \n custom.cseen0[cid] = True\n custom.cseen1[cid] = False\n custom.cseen2[cid] = False\n elif new_attempt == 2: \n custom.cseen0[cid] = False\n custom.cseen1[cid] = True\n custom.cseen2[cid] = False\n elif new_attempt == 3: \n custom.cseen0[cid] = True\n custom.cseen1[cid] = True\n custom.cseen2[cid] = False\n elif new_attempt == 4: \n custom.cseen0[cid] = False\n custom.cseen1[cid] = False\n custom.cseen2[cid] = True\n elif new_attempt == 5: \n custom.cseen0[cid] = True\n custom.cseen1[cid] = False\n custom.cseen2[cid] = True\n elif new_attempt == 6: \n custom.cseen0[cid] = False\n custom.cseen1[cid] = True\n custom.cseen2[cid] = True\n else: \n custom.cseen0[cid] = True\n custom.cseen1[cid] = True\n custom.cseen2[cid] = True\n\ndef get_attempts(custom, cid):\n return custom.cseen0[cid] + 2*custom.cseen1[cid] + 4*custom.cseen2[cid]\n@MPWARE, thank you very much for being kindness to share, really appreciated. I haven't read it in detail to fully understand, but seems a very elegant yet efficient approach to the problem.
\nPersonally, I was able to fix this memory issue ( Per user and per content performance history ) in a short time (yet not a very clean way) by the following idea:
\nQuite a lot of (user, question) pairs are in fact with (n_attempt, n_correctness) being (1, 1) and (1, 0). The pairs with attempt >= 2 is quite rare. Therefore, I have 2 nested dictionaries:
1. The 1st dict is: user_id -> question_id -> the performances for those n_attempt >= 2 (quite small dict)\n\n2. The 2nd dict: (for those pairs with `n_attemp = 1`)\n\n {\n user_id: \n { \n 'correct': [q_id_x, q_id_y, ...]\n 'incorrect': [q_id_m, q_in_, ...]\n }\n }\nIf (user, question) can't be found in the above 2 dictionaries, it means that question is not seen by the user.
\nThe idea is kind similar to sparse matrix, where we only store the content with values.
\nDespite this effort, it still take 4G or 5G memory when loading into memory. I will try to compare your approach later.
\nComments to my approaches are appreciated (if any). Thanks again for sharing.
", "votes": 2, "replies": [] }, { "id": 1149507, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2021-01-11T22:23:04.897000", "content": "@mpware I understand your way of memory management now. Great.
\nI am wondering why your team don't make a feature of the nb of correctness per user per content, or better, the ratio, if you already have such good way to manage memory. It think you might even get better by doing so, although in my case, I can't see the positive effect on LB due to the input inconsistency bug in my pipeline. I am just wondering the reason your team don't use it
", "votes": 1, "replies": [] }, { "id": 1149648, "author_name": "Aditya Soni", "author_url": "", "post_date": "2021-01-12T03:27:22.690000", "content": "Really cool use of bit arrays! I didn't extend it to do the counting part as well! Sweet!
\ndef get_attempts(custom, cid):\n return custom.cseen0[cid] + 2*custom.cseen1[cid] + 4*custom.cseen2[cid]\nThis is basically bit shifting, correct? (Converting the bits back to decimal radix)
\n # content_id seen \n self.cseen0 = bit_array()\n self.cseen1 = bit_array()\n self.cseen2 = bit_array()\nThis basically represents the 3 bits needed to count till 7 i believe.[ with cseen2 as the MSB]
Another use of the same bitarray's is here @yihdarshieh! Basically a bool flag to detect whether use has seen that content in past or not.
", "votes": 2, "replies": [] }, { "id": 1149990, "author_name": "MPWARE", "author_url": "", "post_date": "2021-01-12T09:31:09.773000", "content": "We've tried it but results were close (little bit lower) than without.
", "votes": 1, "replies": [] }, { "id": 1150173, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2021-01-12T12:30:57.373000", "content": "Alright, that is quite strange. Do anyone in your team have some insight about why this is not working?
", "votes": 0, "replies": [] }, { "id": 1150174, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2021-01-12T12:32:56.903000", "content": "@adityaecdrid I saw that thread, but didn't take the time to read it carefully and find the shown code, I was so depressed about my inference bug at that time. Anyway, now I know there is bitarray.
", "votes": 0, "replies": [] } ] }, { "id": 1145911, "author_name": "Theo Viel", "author_url": "", "post_date": "2021-01-09T12:23:13.433000", "content": "Congratz to you all !
", "votes": 2, "replies": [] }, { "id": 1145432, "author_name": "2981", "author_url": "", "post_date": "2021-01-09T06:24:04.433000", "content": "thanks for sharing! I am a beginer to transformer model. I think I need to learn the model before seeing your share.
", "votes": 2, "replies": [] }, { "id": 1145294, "author_name": "Dean", "author_url": "", "post_date": "2021-01-09T03:52:07.940000", "content": "Thanks for your sharing and congratulations to your team.
\nWould you like to explain how to add the \"Number of attempts\" feature into encoder?
\nIs it convert to embedding?
\nThank you!!
Yes as an embedding and add it to other encoder input embeddings.
\nThe encoder input begins as x = tf.keras.layers.Embedding(number of unique content id, 288)(CONTENT ID SEQUENCE). Then you add positional encoding, x += positional_encoding(window size, 288). And last you add \"Number of attempts\" encoding. x += tf.keras.layers.Embedding(max number of attempts + 1, 288)(NUMBER OF ATTEMPTS SEQUENCE).
The maximum number of attempts is clipped with NUMBER OF ATTEMPTS SEQUENCE = np.clip(NUMBER OF ATTEMPTS SEQUENCE, 0, 7).
And @mpware has a brilliant way of storing each user's attempt history using only 3 bits per time step to save memory. (3 bits can represent 0 thru 7 in binary).
", "votes": 3, "replies": [] }, { "id": 1145376, "author_name": "Dean", "author_url": "", "post_date": "2021-01-09T05:17:44.343000", "content": "Very impressive. Thanks for your reply.
", "votes": 2, "replies": [] }, { "id": 1145897, "author_name": "Abdur Rehman", "author_url": "", "post_date": "2021-01-09T12:17:48.453000", "content": "@cdeotte I can see that 99% of of attempts on content_id is <=7. Is that the reason of clipping the max value to 7?
x += tf.keras.layers.Embedding(max number of attempts + 1, 288)(NUMBER OF ATTEMPTS SEQUENCE)
so max number of attempts will be 7 then and why are you adding 1 more to number of attempts ?
@abdurrehman245 The real reason was to limit memory usage for inference. We limit attempts from 0 to 7, everything above 7 falls in the last category which is indeed 7 and beyond.
", "votes": 1, "replies": [] }, { "id": 1148196, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-01-11T01:43:04.583000", "content": "\n\n\nwhy are you adding 1 more to number of attempts?
\n
Since the max is 7, there are 8 unique values, i.e. 0,1,2,3,4,5,6,7. So the embedding needs to handle a categorical variable of cardinality 8.
", "votes": 1, "replies": [] } ] }, { "id": 1145155, "author_name": "Tom", "author_url": "", "post_date": "2021-01-09T00:10:55.967000", "content": "@mpware Congrats on the gold medal and nice single model solution !
", "votes": 2, "replies": [ { "id": 1145680, "author_name": "MPWARE", "author_url": "", "post_date": "2021-01-09T09:39:16.443000", "content": "@tikutiku Thank you!
", "votes": 0, "replies": [] } ] }, { "id": 1145144, "author_name": "william.wu", "author_url": "", "post_date": "2021-01-08T23:44:57.310000", "content": "@mpware Congratulations to your team, very impressed that single RAINT+ with these 5 features can reach such a high score. My best RAINT+ model is only 0.802 on private
\nWould like to know more details about which of these features are on the encoder side and which are on the decoder side?
\nContent id\nLag time\nPrior question elapsed time\nPrevious responses\nNumber of attempts\n
\nThere's no doubt on the content id and response. For other features, could you please explain a little bit about which parts did you put the feature in?
Thanks. Encoder had Content id and Number of attempts. Decoder had Lag time, Prior question elapsed time, and Previous responses.
", "votes": 2, "replies": [] }, { "id": 1145175, "author_name": "william.wu", "author_url": "", "post_date": "2021-01-09T00:31:15.080000", "content": "Got it, thanks!
", "votes": 2, "replies": [] } ] }, { "id": 1144738, "author_name": "KhanhVD", "author_url": "", "post_date": "2021-01-08T16:32:19.443000", "content": "Congratulations on 16th place and gold medal @mpware @cdeotte and team. Good work and strongly transformer model
", "votes": 2, "replies": [] }, { "id": 1147557, "author_name": "william.wu", "author_url": "", "post_date": "2021-01-10T15:21:41.433000", "content": "How about the question's tags, it's a VarLen feature. Do you have any idea of adding VarLen feature in the RAINT+? What I did is sorting the tags in each question then combine them and map to an integer. But it's not an optimal way, since the information of each individual tag is lost.
", "votes": 0, "replies": [ { "id": 1148508, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2021-01-11T07:28:17.583000", "content": "I use a tag embedding, i.e. map each integer in [-1, 188) to vectors of a fixed dim (in my case 256). Then these embedding are averaged (the embedding of -1 is for padding for the tags, as you said, it is varlen, and it is ignored during the averaging). I don't know how the team of @mpware did it though.
", "votes": 0, "replies": [] } ] }, { "id": 1145433, "author_name": "", "author_url": "", "post_date": "2021-01-09T06:24:04.433000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1144696": "Hi all,\n\n\nHere is an insight of our 2 solutions that both scored public 0.812/private 0.815 and that reached 16th gold place.\n\n\nThis competition was both ML and engineering optimization to make everything work in 9h with 13GB RAM/16GB GPU. We spent almost 30% of time on optimization to keep the last 512 interactions per users + per content attempts in memory + required for our features.\n\nWe would like to thank Kaggle and RIIID organizers for this great competition! Congratulations to the top teams and all competitors for their motivation all along the challenge.\nI would like to thank my teammates @rafiko1, @cdeotte @titericz and @matthiasanderer. You've been amazing, I've learnt a lot from you. I really enjoyed this competition.\n\n\n## Solution 1: Single transformer model\n\nThe SAINT+ model is described here https://arxiv.org/pdf/2010.12042.pdf\nThe code for our SAINT+ adaptation is available here https://github.com/rafiko1/Riiid-sharing.\n\n\n\n\n\n\nOur single model SAINT++ achieved, CV: 0.812 Public LB: 0.812, Private LB: 0.815\n\nWe trained with 95% of users first then fine tuned with all data using smart window technique (see diagram below for SAKT). The model is simple in terms of features. It only contains the four features of SAINT+ (pictured above), with one additional feature - the number of attempts of a user for specific content (hence SAINT++):\n\n\n\n* Content id\n* Lag time\n* Prior question elapsed time\n* Previous responses\n* Number of attempts \n\nThe greatest improvement in features compared to SAINT+ came from grouping lag time into seconds, unlike minutes as done in the paper. \nThen, we went bigger and bigger on the architecture and burned some GPU power 🔥. We increased on parameters of the model, most importantly the sequence length and number of layers. Final parameters of the model are as follows: \n\n\n\n\n1. Input Sequence length: 512\n2. Encoding layers: 4\n3. Decoding Layers: 4\n4. Embedding size: 288\n5. Dense Layer: 768\n6. heads: 8\n7. Dropout: 0.20\n\nWe used the Noam learning rate scheduler: with initial warmup and exponential decrease down to 2e-5. \n \nFinal improvement came from our **_recursive trick_** during inference. Here, we rounded predictions that came from the same bundle to **_0 or 1 _**- as their true response is unknown in time yet. The rounded predictions are then fed back to the model to predict the next response within the same bundle. This trick boosts CV LB +0.0025, but requires a batch size of 1, so we couldn’t ensemble multiple transformer models.\n\n\n### Solution 2: Ensemble of transformer, modified SAKT and LGB\n\n\n\n* LightGBM model scored CV=0.793, public LB=0.792 with 44 features. \nOur main features: \nQuestion correctness per content and per user, tags 1 and 2, part, elapsed time, had explanation, number of attempts, multiple lags, running average (answer), multiple rolling means/median (answers, lags, elapsed time) + weighted mean, mean after/before 30 interactions, multiple momentums (lag, answers), per part correctness, per session (8 hours split) running average. Only 3 categories: part, tags1, tags2. Train/valid split from [Tito](https://www.kaggle.com/its7171/cv-strategy). \n\n\n* Pytorch SAKT modified scored CV=0.786, LB=0.789 with additional features. Training procedure with a smart window. \n* When a user’s sequence length is larger than model input, i.e. N>W, then using random crops gives +0.002 CV versus tiled crops. And using smart random crops gives +0.003 CV versus tiled crops. Basic random crops have a low probability of selecting the early or late questions from a user’s sequence whereas smart random crops have an equally likely probability of selecting all questions from a user.\n\n \n\n\n \n\n\n* TensorFlow Transformer model alone scored CV=0.811, LB=0.811\n\n Same as solution#1 but with sequence length = 256\n\n\n\n### What did not work:\n\n\n\n* TabNet\n* Features with lectures for LGB. It worked on CV but not on LB (might be an issue in inference).\n* Post processing using absolute position of question aka. question sequence number. Plotting mean(answered_correctly) vs question number looked like the image below. We can see that the 30 first questions have a different distribution compared with the rest. Also looks like there are subsequently batches of 30 questions (becomes visible if we zoom in the plot below). PP using that information worked in CV improving by around 0.0009, but didn’t work on LB. \n\n \n\n\n\n\n\n### What worked partially:\n\n \nBut was not applicable for us within the 9h runtime limit:\n\n\n\n* More than 3 models ensemble\n* Level 2 model (XGB) could boost by +0.001\n\n\n### Lessons learnt:\n\n\n\n* Start inference Kernel as soon as possible when you need to deal with an API.\n* Try to simulate API locally to understand how data will be handled. [Tito](https://www.kaggle.com/its7171/time-series-api-iter-test-emulator)’s simulator was perfect for that purpose.\n* Push your inference (with the simulator) to the limits to debug it, it will avoid the frustrating “submission scoring error”. \n* Team-up at some point, your teammates always have good ideas.\n\nOne additional word to Kaggle @sohier I loved your API and the way it hides private data, it’s more realistic as in real world usage/production and it avoided chaotic blending. Congratulations for that, however, even if I guess you want to prevent probing, you should find a solution to provide better error feedback. If it is not possible (the more error codes the more probing) then you need to provide a simulator and guidelines allowing competitors to troubleshoot locally.\n", "1146920": "@MPWare Thanks for sharing the \"Simple\" Pytorch code, but why does it score so much less than Tensorflow?\n\n1. The Pytorch model uses 2 layers instead of Tensorflow uses 4 layers\n2. The Pytorch model uses 100 sequence length instead of Tensorflow uses 512 sequence length\n3. The Pytorch model uses 256 embedding size instead of Tensorflow uses 288\n4. The Pytorch model uses dropout = 0.1 but Tensorflow uses dropout = 0.2\n5. The Pytorch model only uses exercise_id in encoder, but Tensorflow uses and response, but Tensorflow uses exercise_id, Lag time, prior question elapsed time, previous responses, number of attempts. Both models just use response_correct embedding in the decoder\n\nAre there any other differences? Would the Pytorch model be as good if you changed it to use all these features? Ty, I will really cherish that \"simple\" pytorch demo notebook you provided.", "1146671": "Congrats & thank you for sharing your approach and codes.", "1146361": "Congratulations and thanks for sharing your solution, I and many others will be able to learn a lot from it. I also used Tensorflow and implemented Saint+ however my code to sample sequences/users was not robust so I intend to see how the model would have performed using the sampling code you have shared.\n\nI have read through the code in demo_riiid_train.ipynb and think I understand how you have set up training and validation sets but was hoping you could confirm my understanding?\n\n1. You allocate distinct users to both the training and validation set\n2. You then assign a probability to each user based on sequence length. For say user 115 in the training set their probability is their sequence length divided by the sum of all sequence lengths in training set.\n3. You then generate a one off validation set where you randomly sample users with replacement and take **random training crops** for each row. Based on the 'select_window_size' function\n4. For the training dataset you do the same but repeat step 3 each epoch\n\nYou sample using N_SELECT_PER_EPOCH = 100000, was this just a hyperparameter you tuned?\n\n\nThank you ", "1146392": "Congrats, single 0.812/0.815 with SAINT is really great!", "1144929": "@mpware congrats on gold medal and thanks for sharing your approach. Would you like to share your code ?", "1147263": "@mpware @cdeotte \n\nAbout\n\n```\nAnd @mpware has a brilliant way of storing each user's attempt history using only 3 bits per time step to save memory. (3 bits can represent 0 thru 7 in binary).\n```\n\nWould it be possible to share how you perform this? At the very end of this competition, I needed to deal with the time/memory issue once I introduced the performance features. And it took time, and I didn't have much time to fix the other inference bugs due to the new features.\n\nIt would be great to learn from your great management of the memory, hope I could hear from you, at least a bit.", "1145911": "Congratz to you all !", "1145432": "thanks for sharing! I am a beginer to transformer model. I think I need to learn the model before seeing your share.", "1145294": "Thanks for your sharing and congratulations to your team.\n\nWould you like to explain how to add the \"Number of attempts\" feature into encoder?\nIs it convert to embedding?\nThank you!!", "1145155": "@mpware Congrats on the gold medal and nice single model solution !", "1145144": "@mpware Congratulations to your team, very impressed that single RAINT+ with these 5 features can reach such a high score. My [best RAINT+ model](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/209596) is only 0.802 on private\n\nWould like to know more details about which of these features are on the encoder side and which are on the decoder side?\n```\nContent id\nLag time\nPrior question elapsed time\nPrevious responses\nNumber of attempts\n``` \nThere's no doubt on the content id and response. For other features, could you please explain a little bit about which parts did you put the feature in?", "1144738": "Congratulations on 16th place and gold medal @mpware @cdeotte and team. Good work and strongly transformer model", "1147557": "How about the question's tags, it's a VarLen feature. Do you have any idea of adding VarLen feature in the RAINT+? What I did is sorting the tags in each question then combine them and map to an integer. But it's not an optimal way, since the information of each individual tag is lost.", "1145433": "" } }