{ "id": 193276, "title": "Any sequence models around there?", "url": "/competitions/riiid-test-answer-prediction/discussion/193276", "author_name": "Claudio Verdú Ruiz", "post_date": "2020-10-26T09:01:30.476000", "votes": 27, "comment_count": 57, "views": 0, "content": "

Hello guys,

\n

I'm currently developing a sequence model: everything fine and working but the inference phase seems to be exceeding the time limit.

\n

The thing is: for every new row in every group in the test set, you have to go for the existing user's rows, concat the new row, compute cummulative features, tail it (or pad it), etc. Those who are working on this will understand. Also add the new confirmed user rows with the previous_correct_answers (this is almost free in time though).

\n

Is there any obvious trick that I'm missing?

\n

Thank you mates.

\n

EDIT:

\n

After 20 days of twerking the code, I finally got my submission with a strong baseline (0.76) in ~2h of inferencing time. I wanted to share what I did for others to be able to reproduce:

\n\n

I still had to drop some features that I'll be recovering now little by little. The whole thing made me improve from ~1.30s to ~0.25s every 100 rows.

", "messages": [ { "id": 1060504, "postDate": "2020-10-26T09:01:30.477Z", "content": "

Hello guys,

\n

I'm currently developing a sequence model: everything fine and working but the inference phase seems to be exceeding the time limit.

\n

The thing is: for every new row in every group in the test set, you have to go for the existing user's rows, concat the new row, compute cummulative features, tail it (or pad it), etc. Those who are working on this will understand. Also add the new confirmed user rows with the previous_correct_answers (this is almost free in time though).

\n

Is there any obvious trick that I'm missing?

\n

Thank you mates.

\n

EDIT:

\n

After 20 days of twerking the code, I finally got my submission with a strong baseline (0.76) in ~2h of inferencing time. I wanted to share what I did for others to be able to reproduce:

\n\n

I still had to drop some features that I'll be recovering now little by little. The whole thing made me improve from ~1.30s to ~0.25s every 100 rows.

", "rawMarkdown": "Hello guys,\n\nI'm currently developing a sequence model: everything fine and working but the inference phase seems to be exceeding the time limit.\n\nThe thing is: for every new row in every group in the test set, you have to go for the existing user's rows, concat the new row, compute cummulative features, tail it (or pad it), etc. Those who are working on this will understand. Also add the new confirmed user rows with the previous_correct_answers (this is almost free in time though).\n\nIs there any obvious trick that I'm missing?\n\nThank you mates.\n\nEDIT:\n\nAfter 20 days of twerking the code, I finally got my submission with a strong baseline (0.76) in ~2h of inferencing time. I wanted to share what I did for others to be able to reproduce:\n\n- I was joining questions' features on the fly when training the model, and also in inference time. This is specially slow, so what I did is catching these features in a separated run for the last [windows_size] interactions of every user. I only need those last interactions in inference time to append new interactions from the test set.\n- Avoid pandas if possible. Given that for every new row you will be creating a whole 2D input, you will have to compute everything you can on numpy directly. To be able to track every column, you can just create a hashmap with columns and their indices. Tailing is also super slow, you can change that for a slice in numpy to really speed it up.\n\nI still had to drop some features that I'll be recovering now little by little. The whole thing made me improve from ~1.30s to ~0.25s every 100 rows. ", "votes": 27 }, { "id": 1062674, "postDate": "2020-10-28T04:59:24.463Z", "content": "

Finally got a competitive score with a sequential model -> 77.1 public lb. I am just glad the pipeline is working smooth. @abdurrafae It takes my pipeline about an hour + for inference on the entire test set as well.

", "rawMarkdown": "Finally got a competitive score with a sequential model -> 77.1 public lb. I am just glad the pipeline is working smooth. @abdurrafae It takes my pipeline about an hour + for inference on the entire test set as well. ", "votes": 10, "replies": [ { "id": 1062681, "postDate": "2020-10-28T05:09:20.757Z", "content": "

Congrats 🥳 ! Quite efficient pipeline you got there! So it's in typical RNN style or it's Transformers?

", "rawMarkdown": "Congrats 🥳 ! Quite efficient pipeline you got there! So it's in typical RNN style or it's Transformers?", "votes": 1, "replies": [ { "id": 1062692, "postDate": "2020-10-28T05:24:27.547Z", "content": "

Thanks Aditya! RNNs were overfitting a lot and I just couldn't figure out the right regularization there. Transformers I believe are better in every way to RNNs(including LSTMs and GRUs) except when it comes to memory constraints of handling very large( > 500 length) sequences.

", "rawMarkdown": "Thanks Aditya! RNNs were overfitting a lot and I just couldn't figure out the right regularization there. Transformers I believe are better in every way to RNNs(including LSTMs and GRUs) except when it comes to memory constraints of handling very large( > 500 length) sequences.", "votes": 4 }, { "id": 1062748, "postDate": "2020-10-28T06:41:42.957Z", "content": "

Great to have a transformer working then!

", "rawMarkdown": "Great to have a transformer working then!" } ] }, { "id": 1062744, "postDate": "2020-10-28T06:35:37.857Z", "content": "

That's really nice, are you trimming the length of sequence to 100. It's taking quiet a lot longer if I don't limit it.

", "rawMarkdown": "That's really nice, are you trimming the length of sequence to 100. It's taking quiet a lot longer if I don't limit it." }, { "id": 1062771, "postDate": "2020-10-28T06:58:45.257Z", "content": "

Some users have thousands of sessions. Therefore the sequence length will have to be limited to a certain threshold

", "rawMarkdown": "Some users have thousands of sessions. Therefore the sequence length will have to be limited to a certain threshold" }, { "id": 1063399, "postDate": "2020-10-28T21:03:32.907Z", "content": "

Congratulations mate! The most impressive part IMO is the inference time. I'm still struggling to make it work in 8 hours. Would you provide any tricks on this?

", "rawMarkdown": "Congratulations mate! The most impressive part IMO is the inference time. I'm still struggling to make it work in 8 hours. Would you provide any tricks on this?", "votes": 1 }, { "id": 1063548, "postDate": "2020-10-29T04:07:16.320Z", "content": "

That is probably because I am not using the encoder decoder architecture as described in the SAINT paper. I am using a simpler transformer variant to get a quicker inference time. Will update the inference pipeline time taken for SAINT model as well.

\n

Tips : Ditch pandas.

", "rawMarkdown": "That is probably because I am not using the encoder decoder architecture as described in the SAINT paper. I am using a simpler transformer variant to get a quicker inference time. Will update the inference pipeline time taken for SAINT model as well.\n\nTips : Ditch pandas.", "votes": 4 }, { "id": 1063661, "postDate": "2020-10-29T07:28:27.873Z", "content": "

I guess that you are not using the information that we get from test-set API about the correct labels to kinda re-fit; otherwise it shouldn't be less than an hour. Or you are and it's still an hour?

", "rawMarkdown": "I guess that you are not using the information that we get from test-set API about the correct labels to kinda re-fit; otherwise it shouldn't be less than an hour. Or you are and it's still an hour?", "votes": 1 }, { "id": 1064655, "postDate": "2020-10-30T11:33:31.353Z", "content": "

Refitting during inference is a bad idea imho

", "rawMarkdown": "Refitting during inference is a bad idea imho", "votes": 1 }, { "id": 1064668, "postDate": "2020-10-30T11:47:36.823Z", "content": "
\n

Refitting during inference is a bad idea imho

\n
\n

@abhimanyud You say that because the test set represents only 3.15% of the entire dataset?

", "rawMarkdown": "> Refitting during inference is a bad idea imho\n\n@abhimanyud You say that because the test set represents only 3.15% of the entire dataset?" }, { "id": 1064741, "postDate": "2020-10-30T13:26:33.090Z", "content": "

My comment was more specific to Neural networks, since incremental data wont make much difference to model performance while being expensive(which becomes really important given the nature of this competition). Other methods like FTRL which are suited for incremental learning do not fall in the same category.

", "rawMarkdown": "My comment was more specific to Neural networks, since incremental data wont make much difference to model performance while being expensive(which becomes really important given the nature of this competition). Other methods like FTRL which are suited for incremental learning do not fall in the same category. " }, { "id": 1065272, "postDate": "2020-10-31T05:33:11.167Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1067841, "postDate": "2020-11-02T18:49:59.373Z", "content": "

@abhimanyud @claverru Have you monitored the prediction loss / accuracy during training? I got only about 61 % accuracy for training, haven't tested on the validation dataset. But my loss calculation is based on the whole batch (i.e. the whole sequence for each users - randomly selected with a fixed WINDOW size).

", "rawMarkdown": "@abhimanyud @claverru Have you monitored the prediction loss / accuracy during training? I got only about 61 % accuracy for training, haven't tested on the validation dataset. But my loss calculation is based on the whole batch (i.e. the whole sequence for each users - randomly selected with a fixed WINDOW size).", "votes": 1 }, { "id": 1068064, "postDate": "2020-11-03T03:04:39.030Z", "content": "

Yes accuracy values are low. Possible reasons -

\n
    \n
  1. Accuracy is being calculated on 0 padded targets as well.
    2. \n
  2. Since the accuracy is being calculated on the prediction of all time steps, its bound to be lower for the earlier time steps where there is very little information for the model to work with because of causality.
    3. \n
", "rawMarkdown": "Yes accuracy values are low. Possible reasons -\n1. Accuracy is being calculated on 0 padded targets as well.\n2. Since the accuracy is being calculated on the prediction of all time steps, its bound to be lower for the earlier time steps where there is very little information for the model to work with because of causality.", "votes": 2 }, { "id": 1068222, "postDate": "2020-11-03T07:30:22.250Z", "content": "

@abhimanyud holy fk you are first mate congrats!!

", "rawMarkdown": "@abhimanyud holy fk you are first mate congrats!!", "votes": 2 }, { "id": 1068283, "postDate": "2020-11-03T08:36:50.757Z", "content": "

I am as shocked as you are 😂

", "rawMarkdown": "I am as shocked as you are 😂", "votes": 2 }, { "id": 1068314, "postDate": "2020-11-03T09:13:36.487Z", "content": "

@abhimanyud Could you share your local validation - LB relation? It's been a while since my last submission because even if my intuition tells me I'm doing better, my local validation is dropping. I just want to know if they are even in your case with a solid sequence implementation or there is a huge gap.

", "rawMarkdown": "@abhimanyud Could you share your local validation - LB relation? It's been a while since my last submission because even if my intuition tells me I'm doing better, my local validation is dropping. I just want to know if they are even in your case with a solid sequence implementation or there is a huge gap.", "votes": 1 }, { "id": 1068334, "postDate": "2020-11-03T09:43:28.980Z", "content": "

Validation and lb more or less increase and decrease in sync. However my local AUC is 86.4 (I know, I know 🤕). Its because I was unable to properly propagate padding masks to the loss.

", "rawMarkdown": "Validation and lb more or less increase and decrease in sync. However my local AUC is 86.4 (I know, I know 🤕). Its because I was unable to properly propagate padding masks to the loss.", "votes": 2 }, { "id": 1068377, "postDate": "2020-11-03T10:45:29.240Z", "content": "

Doesn't this work for you?

\n
loss_object = tf.keras.losses.SparseCategoricalCrossentropy(\n    from_logits=True, reduction='none')\n\ndef loss_function(real, pred):\n  mask = tf.math.logical_not(tf.math.equal(real, 0))\n  loss_ = loss_object(real, pred)\n\n  mask = tf.cast(mask, dtype=loss_.dtype)\n  loss_ *= mask\n\n  return tf.reduce_sum(loss_)/tf.reduce_sum(mask)\n\n\ndef accuracy_function(real, pred):\n  accuracies = tf.equal(real, tf.argmax(pred, axis=2))\n\n  mask = tf.math.logical_not(tf.math.equal(real, 0))\n  accuracies = tf.math.logical_and(mask, accuracies)\n\n  accuracies = tf.cast(accuracies, dtype=tf.float32)\n  mask = tf.cast(mask, dtype=tf.float32)\n  return tf.reduce_sum(accuracies)/tf.reduce_sum(mask)\n
\n

(from https://www.tensorflow.org/tutorials/text/transformer)

\n

If you wanted to use AUC just mask it as in the example above.

", "rawMarkdown": "Doesn't this work for you?\n\n```python\nloss_object = tf.keras.losses.SparseCategoricalCrossentropy(\n from_logits=True, reduction='none')\n\ndef loss_function(real, pred):\n mask = tf.math.logical_not(tf.math.equal(real, 0))\n loss_ = loss_object(real, pred)\n\n mask = tf.cast(mask, dtype=loss_.dtype)\n loss_ *= mask\n\n return tf.reduce_sum(loss_)/tf.reduce_sum(mask)\n\n\ndef accuracy_function(real, pred):\n accuracies = tf.equal(real, tf.argmax(pred, axis=2))\n\n mask = tf.math.logical_not(tf.math.equal(real, 0))\n accuracies = tf.math.logical_and(mask, accuracies)\n\n accuracies = tf.cast(accuracies, dtype=tf.float32)\n mask = tf.cast(mask, dtype=tf.float32)\n return tf.reduce_sum(accuracies)/tf.reduce_sum(mask)\n```\n(from [https://www.tensorflow.org/tutorials/text/transformer](https://www.tensorflow.org/tutorials/text/transformer))\n\nIf you wanted to use AUC just mask it as in the example above.", "votes": 1 }, { "id": 1068379, "postDate": "2020-11-03T10:48:56.463Z", "content": "

It does work. But only if the masks reach the final layer. In my case, I get DenseToDenseSet Operation not implemented error. So I decided to ditch getting too deep into this error and thought lets just work with unmasked losses.

", "rawMarkdown": "It does work. But only if the masks reach the final layer. In my case, I get DenseToDenseSet Operation not implemented error. So I decided to ditch getting too deep into this error and thought lets just work with unmasked losses. ", "votes": 1 }, { "id": 1068449, "postDate": "2020-11-03T12:33:48.317Z", "content": "

Well mate I'd say you're doing good :)

", "rawMarkdown": "Well mate I'd say you're doing good :)", "votes": 1 }, { "id": 1071146, "postDate": "2020-11-06T14:54:31.647Z", "content": "

@abhimanyud would you mind to share how you use encoder only for training and prediction with transformer model? I think decoder would be too slow for inference, so I want to use only encoder also. But in this case, the problem is that I think I need to mask some input places and only predict on those places. Something like masked language modeling as Bert. Unlike encoder decoder approach, only a small portion of input places will be used for training, otherwise masking too many places will confuse the model. Is your approach similar to this?

", "rawMarkdown": "@abhimanyud would you mind to share how you use encoder only for training and prediction with transformer model? I think decoder would be too slow for inference, so I want to use only encoder also. But in this case, the problem is that I think I need to mask some input places and only predict on those places. Something like masked language modeling as Bert. Unlike encoder decoder approach, only a small portion of input places will be used for training, otherwise masking too many places will confuse the model. Is your approach similar to this?", "votes": 3 }, { "id": 1071161, "postDate": "2020-11-06T15:16:38.097Z", "content": "

@yihdarshieh Masked language modeling will not be useful in my opinion, because there is no sense of causality there. That is why BERT (utilizes bidirectional context) is useful for generating good word part embeddings but not for language modeling. Causal modeling (of the kind employed in GPT or Universal transformer) should be the way to go. You want your output at every time step to be dependent on the past but not the future. The encoder only approach uses interaction id as query and response as key value in a self attention layer. That's about it.

", "rawMarkdown": "@yihdarshieh Masked language modeling will not be useful in my opinion, because there is no sense of causality there. That is why BERT (utilizes bidirectional context) is useful for generating good word part embeddings but not for language modeling. Causal modeling (of the kind employed in GPT or Universal transformer) should be the way to go. You want your output at every time step to be dependent on the past but not the future. The encoder only approach uses interaction id as query and response as key value in a self attention layer. That's about it.", "votes": 3 }, { "id": 1071180, "postDate": "2020-11-06T15:31:54.137Z", "content": "

@abhimanyud Thanks! It helps. Actually, we can apply causal masking in encoder while doing MLM, and this will make the encoder only see the previous position while predict the masked places. (Maybe in this case, the name of encoder is not very proper).
\nDo you switch to encoder-decoder already, or still build on encoder-only archeticture?

", "rawMarkdown": "@abhimanyud Thanks! It helps. Actually, we can apply causal masking in encoder while doing MLM, and this will make the encoder only see the previous position while predict the masked places. (Maybe in this case, the name of encoder is not very proper).\nDo you switch to encoder-decoder already, or still build on encoder-only archeticture?" }, { "id": 1072254, "postDate": "2020-11-08T01:26:21.850Z", "content": "

So, if we use only ENCODERs, at the output level, you will have output for every time_stamp in the sequence and with only padding masks if any?

", "rawMarkdown": "So, if we use only ENCODERs, at the output level, you will have output for every time_stamp in the sequence and with only padding masks if any?" } ] }, { "id": 1075033, "postDate": "2020-11-11T10:36:34.163Z", "content": "

I have opened a Notebook with a light version of a Transformer Encoder in TensorFlow. Feel free to check it out to gather some ideas. NOTEBOOK.

", "rawMarkdown": "I have opened a Notebook with a light version of a Transformer Encoder in TensorFlow. Feel free to check it out to gather some ideas. [NOTEBOOK](https://www.kaggle.com/claverru/demystifying-transformers-let-s-make-it-public).", "votes": 1 }, { "id": 1062047, "postDate": "2020-10-27T14:28:28.443Z", "content": "

I am working on it too. The best I have come up with is ditching pandas dataframes altogether and using a hash table of numpy arrays.

", "rawMarkdown": "I am working on it too. The best I have come up with is ditching pandas dataframes altogether and using a hash table of numpy arrays.", "votes": 1, "replies": [ { "id": 1062065, "postDate": "2020-10-27T14:43:23.917Z", "content": "

Something similar here, I have my users in a table with user_id as key.

", "rawMarkdown": "Something similar here, I have my users in a table with user_id as key." }, { "id": 1062086, "postDate": "2020-10-27T15:02:41.063Z", "content": "

That seems to be the only working solution that I have found as well. Though it still takes about an hour at inference time just to manage the data pipeline. How are you faring in this regard?

", "rawMarkdown": "That seems to be the only working solution that I have found as well. Though it still takes about an hour at inference time just to manage the data pipeline. How are you faring in this regard?" } ] }, { "id": 1060573, "postDate": "2020-10-26T10:37:30.030Z", "content": "

@claverru we have a large training dataset, so:

\n\n

Another suggestion is to start with fewer features / lesser number of computations at inference time, then measure the time taken and keep adding code while staying below the kernel time limits.

", "rawMarkdown": "@claverru we have a large training dataset, so:\n- compute features etc using only training dataset (offline) and train your sequence model\n- inference / test using pre-computed rows (and avoid recomputing for every row)\n\nAnother suggestion is to start with fewer features / lesser number of computations at inference time, then measure the time taken and keep adding code while staying below the kernel time limits.", "votes": 1, "replies": [ { "id": 1060679, "postDate": "2020-10-26T13:11:49.513Z", "content": "

Hello! thanks for answering. I've trained my model asside, and the weights are loaded on the fly when submitting. I tried to precompute some things to be able to drop user interactions older than my windows size. Though this approach is so prune to fail (hard to addecuate). Also I dont have many features, (less than 30), with a relatively small windows size (lower than 100).

", "rawMarkdown": "Hello! thanks for answering. I've trained my model asside, and the weights are loaded on the fly when submitting. I tried to precompute some things to be able to drop user interactions older than my windows size. Though this approach is so prune to fail (hard to addecuate). Also I dont have many features, (less than 30), with a relatively small windows size (lower than 100). " } ] }, { "id": 1060558, "postDate": "2020-10-26T10:15:14.667Z", "content": "

I am working on it as well, but , and if it is acceptable in performance (both time and score) share it. Edit: no luck :P

\n

PS: about time, try to see if different options work better (i.e., join vs merge).

", "rawMarkdown": "I am working on it as well, but ~~I am dealing with an error that gives me repeated indexes. Hopefully I can dedicate some time to it to fix it~~, and if it is acceptable in performance (both time and score) share it. Edit: no luck :P\n\nPS: about time, try to see if different options work better (i.e., join vs merge).", "votes": 1, "replies": [ { "id": 1060680, "postDate": "2020-10-26T13:12:19.410Z", "content": "

Try .reset_index(drop=True) before that.

", "rawMarkdown": "Try .reset_index(drop=True) before that. " }, { "id": 1060808, "postDate": "2020-10-26T14:51:34.977Z", "content": "

Maybe if you share a snippet I can try to help. No need to be real code, just something reproducible.

", "rawMarkdown": "Maybe if you share a snippet I can try to help. No need to be real code, just something reproducible." } ] }, { "id": 1060538, "postDate": "2020-10-26T09:39:02.560Z", "content": "

That's why you don't see seq2seq models yet as the tradeoff for using lgbm let's say vs a DL based model as seq2seq is not so easy. The complexity really increases for DL models and might not be needed as it seems people are using lgbms as of now on top of the lb. Plus it might work for old users, but for new users it's gonna take time to warm-up as you won't have content_ids etc and need to pad them etc etc. As compared to lgbm's, they are pretty good in reducing this complexity but track user history efficiently is also complicated. Plus creating a strong lgbm model means quite strong FE's; Hence there's a tradeoff on both sides…

", "rawMarkdown": "That's why you don't see seq2seq models yet as the tradeoff for using lgbm let's say vs a DL based model as seq2seq is not so easy. The complexity really increases for DL models and might not be needed as it seems people are using lgbms as of now on top of the lb. Plus it might work for old users, but for new users it's gonna take time to warm-up as you won't have content_ids etc and need to pad them etc etc. As compared to lgbm's, they are pretty good in reducing this complexity but track user history efficiently is also complicated. Plus creating a strong lgbm model means quite strong FE's; Hence there's a tradeoff on both sides...", "votes": 1, "replies": [ { "id": 1060685, "postDate": "2020-10-26T13:17:10.023Z", "content": "

I'm too committed already xD. I don't really want to enter in a super close feature engineering fight to feed a tree. The only way I see to hit gold is by doing something different, and sequences can give that little advantage. If I can get my current model to work, I think it will surpass the 0.76 mark with room for improvement.

", "rawMarkdown": "I'm too committed already xD. I don't really want to enter in a super close feature engineering fight to feed a tree. The only way I see to hit gold is by doing something different, and sequences can give that little advantage. If I can get my current model to work, I think it will surpass the 0.76 mark with room for improvement.", "votes": 1 }, { "id": 1060842, "postDate": "2020-10-26T15:05:49.013Z", "content": "

Another thought is to use a sequential model to extract features that you then pass to a non sequential model, like a simple neural net or LGBM. While it is hard to infer with a sequential model in this comp, it is easy to train one.

", "rawMarkdown": "Another thought is to use a sequential model to extract features that you then pass to a non sequential model, like a simple neural net or LGBM. While it is hard to infer with a sequential model in this comp, it is easy to train one. ", "votes": 2 }, { "id": 1061109, "postDate": "2020-10-26T18:46:56.847Z", "content": "

I appreciate your comment. I've got to train the model, and I'd say it's in a good track. At commiting time I just load the data and weights and start the inference. The inference process though is what is driving me crazy. I need to shorten times to be able to fit in the 9 hours. Also, I can't cache too much to not run OOM.

", "rawMarkdown": "I appreciate your comment. I've got to train the model, and I'd say it's in a good track. At commiting time I just load the data and weights and start the inference. The inference process though is what is driving me crazy. I need to shorten times to be able to fit in the 9 hours. Also, I can't cache too much to not run OOM.", "votes": 1 } ] }, { "id": 1065406, "postDate": "2020-10-31T08:56:58.907Z", "content": "

I updated the post with some changes I did to have it finally working.

", "rawMarkdown": "I updated the post with some changes I did to have it finally working.", "votes": 2, "replies": [ { "id": 1065412, "postDate": "2020-10-31T09:04:46.460Z", "content": "

Excellent tips! I second all of them. Although you might want to change 'twerking' :P

", "rawMarkdown": "Excellent tips! I second all of them. Although you might want to change 'twerking' :P", "votes": 1 }, { "id": 1065419, "postDate": "2020-10-31T09:11:18.360Z", "content": "

ha! Twerking is always good!

", "rawMarkdown": "ha! Twerking is always good!", "votes": 1 }, { "id": 1065444, "postDate": "2020-10-31T09:40:31.493Z", "content": "

Just curious, it's independent of the user_id, right?

", "rawMarkdown": "Just curious, it's independent of the user_id, right?" }, { "id": 1065460, "postDate": "2020-10-31T10:06:13.510Z", "content": "

Yes, never used it, I'm betting for a higher percent of new users on the private test set.

", "rawMarkdown": "Yes, never used it, I'm betting for a higher percent of new users on the private test set. ", "votes": 1 }, { "id": 1065476, "postDate": "2020-10-31T10:24:17.350Z", "content": "

@adityaecdrid Yea no user_id. The sequence of user interactions can serve as an identifier for the user in a sequence model.

", "rawMarkdown": "@adityaecdrid Yea no user_id. The sequence of user interactions can serve as an identifier for the user in a sequence model. ", "votes": 1 }, { "id": 1065544, "postDate": "2020-10-31T12:23:43.117Z", "content": "

Cool! Thanks for the info.
\nWould be great, if you can add the info to the CV/LB discussion for your Val scores Vs LB! Ty!

", "rawMarkdown": "Cool! Thanks for the info.\nWould be great, if you can add the info to the CV/LB discussion for your Val scores Vs LB! Ty!" }, { "id": 1065601, "postDate": "2020-10-31T13:50:19.003Z", "content": "

That's tough actually because I am having some issues with propagating masks to my loss layer. As a result all my CV val aucs are in the 90 + range 👀. But at least changes in local CV match with LB.

", "rawMarkdown": "That's tough actually because I am having some issues with propagating masks to my loss layer. As a result all my CV val aucs are in the 90 + range 👀. But at least changes in local CV match with LB." }, { "id": 1065780, "postDate": "2020-10-31T18:19:55.973Z", "content": "

I added a comment on that discussion. My score increases a lot from V to LB since I'm validating only against unseen users.

", "rawMarkdown": "I added a comment on that discussion. My score increases a lot from V to LB since I'm validating only against unseen users.", "votes": 1 } ] }, { "id": 1078873, "postDate": "2020-11-15T11:46:20.070Z", "content": "

how to make batch during test inference ?
\nmeans when group has two responses for a user how to track two or more sequence for one iter_test()
\nwithout predicting one row at a time ?

", "rawMarkdown": "how to make batch during test inference ?\nmeans when group has two responses for a user how to track two or more sequence for one iter_test()\nwithout predicting one row at a time ?\n" }, { "id": 1061579, "postDate": "2020-10-27T05:17:12.553Z", "content": "

Are you using transformer architecture or RNN architecture? RNNs have a longer inference time overall compared to transformers so may be you can shift to transformers. However the issue of dealing with the large test data still remains.

", "rawMarkdown": "Are you using transformer architecture or RNN architecture? RNNs have a longer inference time overall compared to transformers so may be you can shift to transformers. However the issue of dealing with the large test data still remains.", "replies": [ { "id": 1062064, "postDate": "2020-10-27T14:42:50.067Z", "content": "

I'm using a Transformer based architecture, the model inference itself is pretty fast, the problem is cummulating features, concatenating new rows, etc.

", "rawMarkdown": "I'm using a Transformer based architecture, the model inference itself is pretty fast, the problem is cummulating features, concatenating new rows, etc." }, { "id": 1062078, "postDate": "2020-10-27T14:59:49.670Z", "content": "

Are you using the full size model with 512 units and 4 layers? I have been trying to implement it but it doesn't work when I train the model, it doesn't learn anything. If I use lower dimensions I don't get results which are competitive (hovers around ~0.76 for training).

", "rawMarkdown": "Are you using the full size model with 512 units and 4 layers? I have been trying to implement it but it doesn't work when I train the model, it doesn't learn anything. If I use lower dimensions I don't get results which are competitive (hovers around ~0.76 for training). " }, { "id": 1062094, "postDate": "2020-10-27T15:09:10.970Z", "content": "

Though I m just using the given data as inputs with no FE.

", "rawMarkdown": "Though I m just using the given data as inputs with no FE." }, { "id": 1062115, "postDate": "2020-10-27T15:29:49.087Z", "content": "

Well here's what we can do. Keep a hash table with a deque so that all old users will have the sequence in it with a capped max-len and always keep the Len of the queue accessible as well. So now when you get a new user, you can easily pad etc as needed.

\n

This is what I am trying to develop as well, so just shared. Also I think we don't have to keep it in mem, use an inverted index on top and wrote to a file via seeking to a line-no etc (Just my current state of the mind, and will test it only a handful of users first, so take it with a grain of salt)

\n

I might be over complicating things for sure, as top scores which people have are from lgbms, NNs etc I believe as of now.

", "rawMarkdown": "Well here's what we can do. Keep a hash table with a deque so that all old users will have the sequence in it with a capped max-len and always keep the Len of the queue accessible as well. So now when you get a new user, you can easily pad etc as needed.\n\nThis is what I am trying to develop as well, so just shared. Also I think we don't have to keep it in mem, use an inverted index on top and wrote to a file via seeking to a line-no etc (Just my current state of the mind, and will test it only a handful of users first, so take it with a grain of salt)\n\nI might be over complicating things for sure, as top scores which people have are from lgbms, NNs etc I believe as of now.", "votes": 2 }, { "id": 1062238, "postDate": "2020-10-27T17:16:29.273Z", "content": "

@abdurrafae I'm using just two Encoder Layers, with 12 as model dimension and 3 heads (pretty small but data dim is also small).

", "rawMarkdown": "@abdurrafae I'm using just two Encoder Layers, with 12 as model dimension and 3 heads (pretty small but data dim is also small)." }, { "id": 1062326, "postDate": "2020-10-27T18:26:46.237Z", "content": "

I think that would work. Cause the complete model isn't going anywhere. 😄

", "rawMarkdown": "I think that would work. Cause the complete model isn't going anywhere. 😄" }, { "id": 1062368, "postDate": "2020-10-27T19:04:34.633Z", "content": "

@adityaecdrid how much time would the memory access takes when keeping data on disk using this approach? I think it might become a bottleneck due to timing constraint.

", "rawMarkdown": "@adityaecdrid how much time would the memory access takes when keeping data on disk using this approach? I think it might become a bottleneck due to timing constraint." } ] } ], "comments": [ { "id": 1062674, "author_name": "Abhimanyu Dikshit", "author_url": "", "post_date": "2020-10-28T04:59:24.463000", "content": "

Finally got a competitive score with a sequential model -> 77.1 public lb. I am just glad the pipeline is working smooth. @abdurrafae It takes my pipeline about an hour + for inference on the entire test set as well.

", "votes": 10, "replies": [ { "id": 1062681, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-10-28T05:09:20.757000", "content": "

Congrats 🥳 ! Quite efficient pipeline you got there! So it's in typical RNN style or it's Transformers?

", "votes": 1, "replies": [ { "id": 1062692, "author_name": "Abhimanyu Dikshit", "author_url": "", "post_date": "2020-10-28T05:24:27.547000", "content": "

Thanks Aditya! RNNs were overfitting a lot and I just couldn't figure out the right regularization there. Transformers I believe are better in every way to RNNs(including LSTMs and GRUs) except when it comes to memory constraints of handling very large( > 500 length) sequences.

", "votes": 4, "replies": [] }, { "id": 1062748, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-10-28T06:41:42.957000", "content": "

Great to have a transformer working then!

", "votes": 0, "replies": [] } ] }, { "id": 1062744, "author_name": "AbdurRafae", "author_url": "", "post_date": "2020-10-28T06:35:37.857000", "content": "

That's really nice, are you trimming the length of sequence to 100. It's taking quiet a lot longer if I don't limit it.

", "votes": 0, "replies": [] }, { "id": 1062771, "author_name": "Abhimanyu Dikshit", "author_url": "", "post_date": "2020-10-28T06:58:45.257000", "content": "

Some users have thousands of sessions. Therefore the sequence length will have to be limited to a certain threshold

", "votes": 0, "replies": [] }, { "id": 1063399, "author_name": "Claudio Verdú Ruiz", "author_url": "", "post_date": "2020-10-28T21:03:32.907000", "content": "

Congratulations mate! The most impressive part IMO is the inference time. I'm still struggling to make it work in 8 hours. Would you provide any tricks on this?

", "votes": 1, "replies": [] }, { "id": 1063548, "author_name": "Abhimanyu Dikshit", "author_url": "", "post_date": "2020-10-29T04:07:16.320000", "content": "

That is probably because I am not using the encoder decoder architecture as described in the SAINT paper. I am using a simpler transformer variant to get a quicker inference time. Will update the inference pipeline time taken for SAINT model as well.

\n

Tips : Ditch pandas.

", "votes": 4, "replies": [] }, { "id": 1063661, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-10-29T07:28:27.873000", "content": "

I guess that you are not using the information that we get from test-set API about the correct labels to kinda re-fit; otherwise it shouldn't be less than an hour. Or you are and it's still an hour?

", "votes": 1, "replies": [] }, { "id": 1064655, "author_name": "Abhimanyu Dikshit", "author_url": "", "post_date": "2020-10-30T11:33:31.353000", "content": "

Refitting during inference is a bad idea imho

", "votes": 1, "replies": [] }, { "id": 1064668, "author_name": "Nya 🚀", "author_url": "", "post_date": "2020-10-30T11:47:36.823000", "content": "
\n

Refitting during inference is a bad idea imho

\n
\n

@abhimanyud You say that because the test set represents only 3.15% of the entire dataset?

", "votes": 0, "replies": [] }, { "id": 1064741, "author_name": "Abhimanyu Dikshit", "author_url": "", "post_date": "2020-10-30T13:26:33.090000", "content": "

My comment was more specific to Neural networks, since incremental data wont make much difference to model performance while being expensive(which becomes really important given the nature of this competition). Other methods like FTRL which are suited for incremental learning do not fall in the same category.

", "votes": 0, "replies": [] }, { "id": 1065272, "author_name": "", "author_url": "", "post_date": "2020-10-31T05:33:11.167000", "content": "", "votes": 0, "replies": [] }, { "id": 1067841, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-11-02T18:49:59.373000", "content": "

@abhimanyud @claverru Have you monitored the prediction loss / accuracy during training? I got only about 61 % accuracy for training, haven't tested on the validation dataset. But my loss calculation is based on the whole batch (i.e. the whole sequence for each users - randomly selected with a fixed WINDOW size).

", "votes": 1, "replies": [] }, { "id": 1068064, "author_name": "Abhimanyu Dikshit", "author_url": "", "post_date": "2020-11-03T03:04:39.030000", "content": "

Yes accuracy values are low. Possible reasons -

\n
    \n
  1. Accuracy is being calculated on 0 padded targets as well.
    2. \n
  2. Since the accuracy is being calculated on the prediction of all time steps, its bound to be lower for the earlier time steps where there is very little information for the model to work with because of causality.
    3. \n
", "votes": 2, "replies": [] }, { "id": 1068222, "author_name": "Claudio Verdú Ruiz", "author_url": "", "post_date": "2020-11-03T07:30:22.250000", "content": "

@abhimanyud holy fk you are first mate congrats!!

", "votes": 2, "replies": [] }, { "id": 1068283, "author_name": "Abhimanyu Dikshit", "author_url": "", "post_date": "2020-11-03T08:36:50.757000", "content": "

I am as shocked as you are 😂

", "votes": 2, "replies": [] }, { "id": 1068314, "author_name": "Claudio Verdú Ruiz", "author_url": "", "post_date": "2020-11-03T09:13:36.487000", "content": "

@abhimanyud Could you share your local validation - LB relation? It's been a while since my last submission because even if my intuition tells me I'm doing better, my local validation is dropping. I just want to know if they are even in your case with a solid sequence implementation or there is a huge gap.

", "votes": 1, "replies": [] }, { "id": 1068334, "author_name": "Abhimanyu Dikshit", "author_url": "", "post_date": "2020-11-03T09:43:28.980000", "content": "

Validation and lb more or less increase and decrease in sync. However my local AUC is 86.4 (I know, I know 🤕). Its because I was unable to properly propagate padding masks to the loss.

", "votes": 2, "replies": [] }, { "id": 1068377, "author_name": "Claudio Verdú Ruiz", "author_url": "", "post_date": "2020-11-03T10:45:29.240000", "content": "

Doesn't this work for you?

\n
loss_object = tf.keras.losses.SparseCategoricalCrossentropy(\n    from_logits=True, reduction='none')\n\ndef loss_function(real, pred):\n  mask = tf.math.logical_not(tf.math.equal(real, 0))\n  loss_ = loss_object(real, pred)\n\n  mask = tf.cast(mask, dtype=loss_.dtype)\n  loss_ *= mask\n\n  return tf.reduce_sum(loss_)/tf.reduce_sum(mask)\n\n\ndef accuracy_function(real, pred):\n  accuracies = tf.equal(real, tf.argmax(pred, axis=2))\n\n  mask = tf.math.logical_not(tf.math.equal(real, 0))\n  accuracies = tf.math.logical_and(mask, accuracies)\n\n  accuracies = tf.cast(accuracies, dtype=tf.float32)\n  mask = tf.cast(mask, dtype=tf.float32)\n  return tf.reduce_sum(accuracies)/tf.reduce_sum(mask)\n
\n

(from https://www.tensorflow.org/tutorials/text/transformer)

\n

If you wanted to use AUC just mask it as in the example above.

", "votes": 1, "replies": [] }, { "id": 1068379, "author_name": "Abhimanyu Dikshit", "author_url": "", "post_date": "2020-11-03T10:48:56.463000", "content": "

It does work. But only if the masks reach the final layer. In my case, I get DenseToDenseSet Operation not implemented error. So I decided to ditch getting too deep into this error and thought lets just work with unmasked losses.

", "votes": 1, "replies": [] }, { "id": 1068449, "author_name": "Claudio Verdú Ruiz", "author_url": "", "post_date": "2020-11-03T12:33:48.317000", "content": "

Well mate I'd say you're doing good :)

", "votes": 1, "replies": [] }, { "id": 1071146, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-11-06T14:54:31.647000", "content": "

@abhimanyud would you mind to share how you use encoder only for training and prediction with transformer model? I think decoder would be too slow for inference, so I want to use only encoder also. But in this case, the problem is that I think I need to mask some input places and only predict on those places. Something like masked language modeling as Bert. Unlike encoder decoder approach, only a small portion of input places will be used for training, otherwise masking too many places will confuse the model. Is your approach similar to this?

", "votes": 3, "replies": [] }, { "id": 1071161, "author_name": "Abhimanyu Dikshit", "author_url": "", "post_date": "2020-11-06T15:16:38.097000", "content": "

@yihdarshieh Masked language modeling will not be useful in my opinion, because there is no sense of causality there. That is why BERT (utilizes bidirectional context) is useful for generating good word part embeddings but not for language modeling. Causal modeling (of the kind employed in GPT or Universal transformer) should be the way to go. You want your output at every time step to be dependent on the past but not the future. The encoder only approach uses interaction id as query and response as key value in a self attention layer. That's about it.

", "votes": 3, "replies": [] }, { "id": 1071180, "author_name": "Yih-Dar SHIEH", "author_url": "", "post_date": "2020-11-06T15:31:54.137000", "content": "

@abhimanyud Thanks! It helps. Actually, we can apply causal masking in encoder while doing MLM, and this will make the encoder only see the previous position while predict the masked places. (Maybe in this case, the name of encoder is not very proper).
\nDo you switch to encoder-decoder already, or still build on encoder-only archeticture?

", "votes": 0, "replies": [] }, { "id": 1072254, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-11-08T01:26:21.850000", "content": "

So, if we use only ENCODERs, at the output level, you will have output for every time_stamp in the sequence and with only padding masks if any?

", "votes": 0, "replies": [] } ] }, { "id": 1075033, "author_name": "Claudio Verdú Ruiz", "author_url": "", "post_date": "2020-11-11T10:36:34.163000", "content": "

I have opened a Notebook with a light version of a Transformer Encoder in TensorFlow. Feel free to check it out to gather some ideas. NOTEBOOK.

", "votes": 1, "replies": [] }, { "id": 1062047, "author_name": "Abhimanyu Dikshit", "author_url": "", "post_date": "2020-10-27T14:28:28.443000", "content": "

I am working on it too. The best I have come up with is ditching pandas dataframes altogether and using a hash table of numpy arrays.

", "votes": 1, "replies": [ { "id": 1062065, "author_name": "Claudio Verdú Ruiz", "author_url": "", "post_date": "2020-10-27T14:43:23.917000", "content": "

Something similar here, I have my users in a table with user_id as key.

", "votes": 0, "replies": [] }, { "id": 1062086, "author_name": "AbdurRafae", "author_url": "", "post_date": "2020-10-27T15:02:41.063000", "content": "

That seems to be the only working solution that I have found as well. Though it still takes about an hour at inference time just to manage the data pipeline. How are you faring in this regard?

", "votes": 0, "replies": [] } ] }, { "id": 1060573, "author_name": "Sirish Somanchi", "author_url": "", "post_date": "2020-10-26T10:37:30.030000", "content": "

@claverru we have a large training dataset, so:

\n\n

Another suggestion is to start with fewer features / lesser number of computations at inference time, then measure the time taken and keep adding code while staying below the kernel time limits.

", "votes": 1, "replies": [ { "id": 1060679, "author_name": "Claudio Verdú Ruiz", "author_url": "", "post_date": "2020-10-26T13:11:49.513000", "content": "

Hello! thanks for answering. I've trained my model asside, and the weights are loaded on the fly when submitting. I tried to precompute some things to be able to drop user interactions older than my windows size. Though this approach is so prune to fail (hard to addecuate). Also I dont have many features, (less than 30), with a relatively small windows size (lower than 100).

", "votes": 0, "replies": [] } ] }, { "id": 1060558, "author_name": "curious1", "author_url": "", "post_date": "2020-10-26T10:15:14.667000", "content": "

I am working on it as well, but , and if it is acceptable in performance (both time and score) share it. Edit: no luck :P

\n

PS: about time, try to see if different options work better (i.e., join vs merge).

", "votes": 1, "replies": [ { "id": 1060680, "author_name": "Claudio Verdú Ruiz", "author_url": "", "post_date": "2020-10-26T13:12:19.410000", "content": "

Try .reset_index(drop=True) before that.

", "votes": 0, "replies": [] }, { "id": 1060808, "author_name": "Claudio Verdú Ruiz", "author_url": "", "post_date": "2020-10-26T14:51:34.977000", "content": "

Maybe if you share a snippet I can try to help. No need to be real code, just something reproducible.

", "votes": 0, "replies": [] } ] }, { "id": 1060538, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-10-26T09:39:02.560000", "content": "

That's why you don't see seq2seq models yet as the tradeoff for using lgbm let's say vs a DL based model as seq2seq is not so easy. The complexity really increases for DL models and might not be needed as it seems people are using lgbms as of now on top of the lb. Plus it might work for old users, but for new users it's gonna take time to warm-up as you won't have content_ids etc and need to pad them etc etc. As compared to lgbm's, they are pretty good in reducing this complexity but track user history efficiently is also complicated. Plus creating a strong lgbm model means quite strong FE's; Hence there's a tradeoff on both sides…

", "votes": 1, "replies": [ { "id": 1060685, "author_name": "Claudio Verdú Ruiz", "author_url": "", "post_date": "2020-10-26T13:17:10.023000", "content": "

I'm too committed already xD. I don't really want to enter in a super close feature engineering fight to feed a tree. The only way I see to hit gold is by doing something different, and sequences can give that little advantage. If I can get my current model to work, I think it will surpass the 0.76 mark with room for improvement.

", "votes": 1, "replies": [] }, { "id": 1060842, "author_name": "Tucker Arrants", "author_url": "", "post_date": "2020-10-26T15:05:49.013000", "content": "

Another thought is to use a sequential model to extract features that you then pass to a non sequential model, like a simple neural net or LGBM. While it is hard to infer with a sequential model in this comp, it is easy to train one.

", "votes": 2, "replies": [] }, { "id": 1061109, "author_name": "Claudio Verdú Ruiz", "author_url": "", "post_date": "2020-10-26T18:46:56.847000", "content": "

I appreciate your comment. I've got to train the model, and I'd say it's in a good track. At commiting time I just load the data and weights and start the inference. The inference process though is what is driving me crazy. I need to shorten times to be able to fit in the 9 hours. Also, I can't cache too much to not run OOM.

", "votes": 1, "replies": [] } ] }, { "id": 1065406, "author_name": "Claudio Verdú Ruiz", "author_url": "", "post_date": "2020-10-31T08:56:58.907000", "content": "

I updated the post with some changes I did to have it finally working.

", "votes": 2, "replies": [ { "id": 1065412, "author_name": "Abhimanyu Dikshit", "author_url": "", "post_date": "2020-10-31T09:04:46.460000", "content": "

Excellent tips! I second all of them. Although you might want to change 'twerking' :P

", "votes": 1, "replies": [] }, { "id": 1065419, "author_name": "Claudio Verdú Ruiz", "author_url": "", "post_date": "2020-10-31T09:11:18.360000", "content": "

ha! Twerking is always good!

", "votes": 1, "replies": [] }, { "id": 1065444, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-10-31T09:40:31.493000", "content": "

Just curious, it's independent of the user_id, right?

", "votes": 0, "replies": [] }, { "id": 1065460, "author_name": "Claudio Verdú Ruiz", "author_url": "", "post_date": "2020-10-31T10:06:13.510000", "content": "

Yes, never used it, I'm betting for a higher percent of new users on the private test set.

", "votes": 1, "replies": [] }, { "id": 1065476, "author_name": "Abhimanyu Dikshit", "author_url": "", "post_date": "2020-10-31T10:24:17.350000", "content": "

@adityaecdrid Yea no user_id. The sequence of user interactions can serve as an identifier for the user in a sequence model.

", "votes": 1, "replies": [] }, { "id": 1065544, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-10-31T12:23:43.117000", "content": "

Cool! Thanks for the info.
\nWould be great, if you can add the info to the CV/LB discussion for your Val scores Vs LB! Ty!

", "votes": 0, "replies": [] }, { "id": 1065601, "author_name": "Abhimanyu Dikshit", "author_url": "", "post_date": "2020-10-31T13:50:19.003000", "content": "

That's tough actually because I am having some issues with propagating masks to my loss layer. As a result all my CV val aucs are in the 90 + range 👀. But at least changes in local CV match with LB.

", "votes": 0, "replies": [] }, { "id": 1065780, "author_name": "Claudio Verdú Ruiz", "author_url": "", "post_date": "2020-10-31T18:19:55.973000", "content": "

I added a comment on that discussion. My score increases a lot from V to LB since I'm validating only against unseen users.

", "votes": 1, "replies": [] } ] }, { "id": 1078873, "author_name": "Dhyey_Patel1234", "author_url": "", "post_date": "2020-11-15T11:46:20.070000", "content": "

how to make batch during test inference ?
\nmeans when group has two responses for a user how to track two or more sequence for one iter_test()
\nwithout predicting one row at a time ?

", "votes": 0, "replies": [] }, { "id": 1061579, "author_name": "AbdurRafae", "author_url": "", "post_date": "2020-10-27T05:17:12.553000", "content": "

Are you using transformer architecture or RNN architecture? RNNs have a longer inference time overall compared to transformers so may be you can shift to transformers. However the issue of dealing with the large test data still remains.

", "votes": 0, "replies": [ { "id": 1062064, "author_name": "Claudio Verdú Ruiz", "author_url": "", "post_date": "2020-10-27T14:42:50.067000", "content": "

I'm using a Transformer based architecture, the model inference itself is pretty fast, the problem is cummulating features, concatenating new rows, etc.

", "votes": 0, "replies": [] }, { "id": 1062078, "author_name": "AbdurRafae", "author_url": "", "post_date": "2020-10-27T14:59:49.670000", "content": "

Are you using the full size model with 512 units and 4 layers? I have been trying to implement it but it doesn't work when I train the model, it doesn't learn anything. If I use lower dimensions I don't get results which are competitive (hovers around ~0.76 for training).

", "votes": 0, "replies": [] }, { "id": 1062094, "author_name": "AbdurRafae", "author_url": "", "post_date": "2020-10-27T15:09:10.970000", "content": "

Though I m just using the given data as inputs with no FE.

", "votes": 0, "replies": [] }, { "id": 1062115, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-10-27T15:29:49.087000", "content": "

Well here's what we can do. Keep a hash table with a deque so that all old users will have the sequence in it with a capped max-len and always keep the Len of the queue accessible as well. So now when you get a new user, you can easily pad etc as needed.

\n

This is what I am trying to develop as well, so just shared. Also I think we don't have to keep it in mem, use an inverted index on top and wrote to a file via seeking to a line-no etc (Just my current state of the mind, and will test it only a handful of users first, so take it with a grain of salt)

\n

I might be over complicating things for sure, as top scores which people have are from lgbms, NNs etc I believe as of now.

", "votes": 2, "replies": [] }, { "id": 1062238, "author_name": "Claudio Verdú Ruiz", "author_url": "", "post_date": "2020-10-27T17:16:29.273000", "content": "

@abdurrafae I'm using just two Encoder Layers, with 12 as model dimension and 3 heads (pretty small but data dim is also small).

", "votes": 0, "replies": [] }, { "id": 1062326, "author_name": "AbdurRafae", "author_url": "", "post_date": "2020-10-27T18:26:46.237000", "content": "

I think that would work. Cause the complete model isn't going anywhere. 😄

", "votes": 0, "replies": [] }, { "id": 1062368, "author_name": "AbdurRafae", "author_url": "", "post_date": "2020-10-27T19:04:34.633000", "content": "

@adityaecdrid how much time would the memory access takes when keeping data on disk using this approach? I think it might become a bottleneck due to timing constraint.

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1060504": "Hello guys,\n\nI'm currently developing a sequence model: everything fine and working but the inference phase seems to be exceeding the time limit.\n\nThe thing is: for every new row in every group in the test set, you have to go for the existing user's rows, concat the new row, compute cummulative features, tail it (or pad it), etc. Those who are working on this will understand. Also add the new confirmed user rows with the previous_correct_answers (this is almost free in time though).\n\nIs there any obvious trick that I'm missing?\n\nThank you mates.\n\nEDIT:\n\nAfter 20 days of twerking the code, I finally got my submission with a strong baseline (0.76) in ~2h of inferencing time. I wanted to share what I did for others to be able to reproduce:\n\n- I was joining questions' features on the fly when training the model, and also in inference time. This is specially slow, so what I did is catching these features in a separated run for the last [windows_size] interactions of every user. I only need those last interactions in inference time to append new interactions from the test set.\n- Avoid pandas if possible. Given that for every new row you will be creating a whole 2D input, you will have to compute everything you can on numpy directly. To be able to track every column, you can just create a hashmap with columns and their indices. Tailing is also super slow, you can change that for a slice in numpy to really speed it up.\n\nI still had to drop some features that I'll be recovering now little by little. The whole thing made me improve from ~1.30s to ~0.25s every 100 rows. ", "1062674": "Finally got a competitive score with a sequential model -> 77.1 public lb. I am just glad the pipeline is working smooth. @abdurrafae It takes my pipeline about an hour + for inference on the entire test set as well. ", "1075033": "I have opened a Notebook with a light version of a Transformer Encoder in TensorFlow. Feel free to check it out to gather some ideas. [NOTEBOOK](https://www.kaggle.com/claverru/demystifying-transformers-let-s-make-it-public).", "1062047": "I am working on it too. The best I have come up with is ditching pandas dataframes altogether and using a hash table of numpy arrays.", "1060573": "@claverru we have a large training dataset, so:\n- compute features etc using only training dataset (offline) and train your sequence model\n- inference / test using pre-computed rows (and avoid recomputing for every row)\n\nAnother suggestion is to start with fewer features / lesser number of computations at inference time, then measure the time taken and keep adding code while staying below the kernel time limits.", "1060558": "I am working on it as well, but ~~I am dealing with an error that gives me repeated indexes. Hopefully I can dedicate some time to it to fix it~~, and if it is acceptable in performance (both time and score) share it. Edit: no luck :P\n\nPS: about time, try to see if different options work better (i.e., join vs merge).", "1060538": "That's why you don't see seq2seq models yet as the tradeoff for using lgbm let's say vs a DL based model as seq2seq is not so easy. The complexity really increases for DL models and might not be needed as it seems people are using lgbms as of now on top of the lb. Plus it might work for old users, but for new users it's gonna take time to warm-up as you won't have content_ids etc and need to pad them etc etc. As compared to lgbm's, they are pretty good in reducing this complexity but track user history efficiently is also complicated. Plus creating a strong lgbm model means quite strong FE's; Hence there's a tradeoff on both sides...", "1065406": "I updated the post with some changes I did to have it finally working.", "1078873": "how to make batch during test inference ?\nmeans when group has two responses for a user how to track two or more sequence for one iter_test()\nwithout predicting one row at a time ?\n", "1061579": "Are you using transformer architecture or RNN architecture? RNNs have a longer inference time overall compared to transformers so may be you can shift to transformers. However the issue of dealing with the large test data still remains." } }