![\"enter](\"https://i.imgur.com/zUY9tVN.png\")
First of all, we want to thank Kaggle for hosting the competition and Quora for providing such a large dataset. Last 3 months were quite exhausting for us with a steep learning curve and tons of the ideas we wanted to try out. In the following we try to summarize some of the main points of our solution.
\n\nModel Structure\nWe played around with a variety of different model structures, but in the end resorted to a quite simple one that is very similar to those posted here https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/79824. It’s basically a Single Bi-LSTM 128 followed by a Conv1D with kernel size 1 only and GlobalMaxPooling afterwards plus additional dropout layers with minimal dropout. We additionally use a few statistical features.
\n\n
Embeddings\nFirst, we use all tokens from both train and test data for our vocabulary. We do the simple pre-cleaning that was posted in a kernel at the start of the competition and split by space afterwards (spacy and nltk resulted in similar performance). We do not lowercase, but keep uppercase, and do not limit the vocab at all. For embeddings we use glove and para where we weight glove a bit higher. The most important thing now is to find as many embeddings as possible for our vocabulary. We had a few steps to achieve this, like checking singular and plural of the word, checking lowercase embeddings, removing special tokens, etc. For public test data we had around 50k of vocab tokens we did not find in the embeddings afterwards. Even though we tried a few different strategies for handling the OOV tokens, we resorted to a single OOV token with a single random embedding vector.
\n\nThreshold\nWe spent a lot of time trying to figure out good strategies for choosing a good threshold for classification. Over time, we saw that estimating the threshold on validation data and then applying it on test data does not really work. There is a large variation on optimal thresholds. So what we did instead is to try to find a fixed threshold on CV that produces the least deviation for the f1 score from the optimal threshold. We saw that we can get more stable results when we produce ranks on the predicted probability and average the ranks instead of averaging probabilities. For final submission we then chose the best CV threshold. This also allowed us to fit the model on the complete data without the need to rely on a random split and less training data. The visualization below shows that in action (not necessarily our final eval). On the x-axis we plot the different fixed thresholds and on the y-axis we see the deviation from the optimal F1 score across folds using this fixed threshold (see CV chapter below). The blue line is the mean, green is median, purple is minimum, red is maximum, and bars are stds. So for example here, if we choose a threshold in the range of 0.927 we expect the F1 score to be not much worse (around 0.001) compared to choosing the optimal threshold (which we can't do for test data). In practise, this might of course deviate further and we could also see larger deviations on PLB. For further elaboration, please check the comments.
\n\n![\"enter](\"https://i.imgur.com/2NPwBIR.png\")
Runtime tricks\nWe aimed at combining as many models as possible. To do this, we needed to improve runtime and the most important thing to achieve this was the following. We do not pad sequences to the same length based on the whole data, but just on a batch level. That means we conduct padding and truncation on the data generator level for each batch separately, so that length of the sentences in a batch can vary in size. Additionally, we further improved this by not truncating based on the length of the longest sequence in the batch, but based on the 95% percentile of lengths within the sequence. This improved runtime heavily and kept accuracy quite robust on single model level, and improved it by being able to average more models.
\n\nFitting\nWe use a one cycle policy with Nadam optimizer (you can do this with the typical CyclicLearningRate implementations by just changing the step size to half your total iterations). We chose a batch size of 512. We could achieve similar results by even taking 10 or 20 times higher batch sizes, which goes hand in hand with recent research on fast convergence. With these larger batch sizes we could even fit close to 20 models, but results stabilized close to 10 models which is why we chose to go with the smaller batch size in the end. However, there might still be some room left here if one properly tunes this.
\n\nMultiple models\nIn the end, we managed to fit more than 10 models on the complete training dataset with help of the runtime tricks mentioned before. Our best final private score even had only a runtime of 6000 seconds (I think they used a bit better hardware for running), so there would be space for 1-2 more models. With larger batch sizes even much more might be feasible. As mentioned, we then average the rank predictions of each model and use our specified threshold for prediction.
\n\nEmbrace the randomness\nAs it was necessary to utilize CUDNN Layers in this competition, there was some randomness involved that could be quite frustrating from time to time. I saw many people trying to fix seeds etc. and some claiming they could completely remove the randomness by using Pytorch (I still don’t believe this BTW as CUDNN has atomic operations). However, as mentioned before, a well working strategy in this competition was to combine multiple models and to end up with a good ensemble, those models should be a bit different to each other. So having different random initializations etc. can be helpful. Seeing people setting the seed as a hyperparameter is weird.
\n\nCV Evaluation\nWhat I saw many people doing wrongly in this competition, and we also only figured this out after a while, is to trust their single out-of-fold evaluation. However, in this competition, it is crucial to combine (average) multiple models (in our case the same model). That means that our CV evaluation looks like the following. We do a k-fold split (mostly 10-fold) and fit the same model up to v-times on the same training split and then successively evaluate it on the single out of fold. So for the first split, we first fit one model and evaluate it, then a second one and evaluate the average and so forth. We repeat this for all 10 folds, landing us with e.g., 100 model fits overall, and then we can take a look at the median or mean over all folds for v-model-ensembles. The reason for doing this is that f1 scores are very different on the split you have. For one 10% split you might end up with a maximum of 0.72 and for the other you might end up at 0.705 or similar. So repeating the split 10 times, fitting the same model v-times for each split, and then looking at the grand picture gave us the best overall evaluation. This routine helped us to compare individual solutions with each other. BTW our final scores are exactly what we would expect from our CV evaluation, but again this might be lucky :)
\n\nRobustness and over/underfitting\nAround 2 weeks before final submission, our results became so stable that changing things did not alter results much. Things like finding more OOV embedding vectors resulted in same results, using slightly different layers ended up being similar, and other things. This was a bit frustrating, but in the end things worked out. In the end, it was important to find a good balance between over and underfitting (as always). Underfitting too much led to good single model performances, but was worse for combining models, and the other way around. For example, if your model overfits, there can be many different solutions to tackle this, e.g., add dropouts, or reduce the vocab size, or reduce model complexity, etc. So if someone says on kaggle that one things works for him/her, that does not necessarily mean that it will work for you as you might already be doing something similar that has similar effects (a good example is the Gaussian noise discussion).
\n\nWhat did not work for us\nMostly you only read what worked, but here is an incomplete shortlist of what did not work for us. This does not mean that it doesn’t work at all, but rather that it was worse for our specific solution.
\n\nFirst of all, we want to thank Kaggle for hosting the competition and Quora for providing such a large dataset. Last 3 months were quite exhausting for us with a steep learning curve and tons of the ideas we wanted to try out. In the following we try to summarize some of the main points of our solution.
\n\nModel Structure\nWe played around with a variety of different model structures, but in the end resorted to a quite simple one that is very similar to those posted here https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/79824. It’s basically a Single Bi-LSTM 128 followed by a Conv1D with kernel size 1 only and GlobalMaxPooling afterwards plus additional dropout layers with minimal dropout. We additionally use a few statistical features.
\n\n
Embeddings\nFirst, we use all tokens from both train and test data for our vocabulary. We do the simple pre-cleaning that was posted in a kernel at the start of the competition and split by space afterwards (spacy and nltk resulted in similar performance). We do not lowercase, but keep uppercase, and do not limit the vocab at all. For embeddings we use glove and para where we weight glove a bit higher. The most important thing now is to find as many embeddings as possible for our vocabulary. We had a few steps to achieve this, like checking singular and plural of the word, checking lowercase embeddings, removing special tokens, etc. For public test data we had around 50k of vocab tokens we did not find in the embeddings afterwards. Even though we tried a few different strategies for handling the OOV tokens, we resorted to a single OOV token with a single random embedding vector.
\n\nThreshold\nWe spent a lot of time trying to figure out good strategies for choosing a good threshold for classification. Over time, we saw that estimating the threshold on validation data and then applying it on test data does not really work. There is a large variation on optimal thresholds. So what we did instead is to try to find a fixed threshold on CV that produces the least deviation for the f1 score from the optimal threshold. We saw that we can get more stable results when we produce ranks on the predicted probability and average the ranks instead of averaging probabilities. For final submission we then chose the best CV threshold. This also allowed us to fit the model on the complete data without the need to rely on a random split and less training data. The visualization below shows that in action (not necessarily our final eval). On the x-axis we plot the different fixed thresholds and on the y-axis we see the deviation from the optimal F1 score across folds using this fixed threshold (see CV chapter below). The blue line is the mean, green is median, purple is minimum, red is maximum, and bars are stds. So for example here, if we choose a threshold in the range of 0.927 we expect the F1 score to be not much worse (around 0.001) compared to choosing the optimal threshold (which we can't do for test data). In practise, this might of course deviate further and we could also see larger deviations on PLB. For further elaboration, please check the comments.
\n\n
Runtime tricks\nWe aimed at combining as many models as possible. To do this, we needed to improve runtime and the most important thing to achieve this was the following. We do not pad sequences to the same length based on the whole data, but just on a batch level. That means we conduct padding and truncation on the data generator level for each batch separately, so that length of the sentences in a batch can vary in size. Additionally, we further improved this by not truncating based on the length of the longest sequence in the batch, but based on the 95% percentile of lengths within the sequence. This improved runtime heavily and kept accuracy quite robust on single model level, and improved it by being able to average more models.
\n\nFitting\nWe use a one cycle policy with Nadam optimizer (you can do this with the typical CyclicLearningRate implementations by just changing the step size to half your total iterations). We chose a batch size of 512. We could achieve similar results by even taking 10 or 20 times higher batch sizes, which goes hand in hand with recent research on fast convergence. With these larger batch sizes we could even fit close to 20 models, but results stabilized close to 10 models which is why we chose to go with the smaller batch size in the end. However, there might still be some room left here if one properly tunes this.
\n\nMultiple models\nIn the end, we managed to fit more than 10 models on the complete training dataset with help of the runtime tricks mentioned before. Our best final private score even had only a runtime of 6000 seconds (I think they used a bit better hardware for running), so there would be space for 1-2 more models. With larger batch sizes even much more might be feasible. As mentioned, we then average the rank predictions of each model and use our specified threshold for prediction.
\n\nEmbrace the randomness\nAs it was necessary to utilize CUDNN Layers in this competition, there was some randomness involved that could be quite frustrating from time to time. I saw many people trying to fix seeds etc. and some claiming they could completely remove the randomness by using Pytorch (I still don’t believe this BTW as CUDNN has atomic operations). However, as mentioned before, a well working strategy in this competition was to combine multiple models and to end up with a good ensemble, those models should be a bit different to each other. So having different random initializations etc. can be helpful. Seeing people setting the seed as a hyperparameter is weird.
\n\nCV Evaluation\nWhat I saw many people doing wrongly in this competition, and we also only figured this out after a while, is to trust their single out-of-fold evaluation. However, in this competition, it is crucial to combine (average) multiple models (in our case the same model). That means that our CV evaluation looks like the following. We do a k-fold split (mostly 10-fold) and fit the same model up to v-times on the same training split and then successively evaluate it on the single out of fold. So for the first split, we first fit one model and evaluate it, then a second one and evaluate the average and so forth. We repeat this for all 10 folds, landing us with e.g., 100 model fits overall, and then we can take a look at the median or mean over all folds for v-model-ensembles. The reason for doing this is that f1 scores are very different on the split you have. For one 10% split you might end up with a maximum of 0.72 and for the other you might end up at 0.705 or similar. So repeating the split 10 times, fitting the same model v-times for each split, and then looking at the grand picture gave us the best overall evaluation. This routine helped us to compare individual solutions with each other. BTW our final scores are exactly what we would expect from our CV evaluation, but again this might be lucky :)
\n\nRobustness and over/underfitting\nAround 2 weeks before final submission, our results became so stable that changing things did not alter results much. Things like finding more OOV embedding vectors resulted in same results, using slightly different layers ended up being similar, and other things. This was a bit frustrating, but in the end things worked out. In the end, it was important to find a good balance between over and underfitting (as always). Underfitting too much led to good single model performances, but was worse for combining models, and the other way around. For example, if your model overfits, there can be many different solutions to tackle this, e.g., add dropouts, or reduce the vocab size, or reduce model complexity, etc. So if someone says on kaggle that one things works for him/her, that does not necessarily mean that it will work for you as you might already be doing something similar that has similar effects (a good example is the Gaussian noise discussion).
\n\nWhat did not work for us\nMostly you only read what worked, but here is an incomplete shortlist of what did not work for us. This does not mean that it doesn’t work at all, but rather that it was worse for our specific solution.
\n\ncongrats, can you open source your kernel so we know exactly what you did? look forward to it!
", "rawMarkdown": "congrats, can you open source your kernel so we know exactly what you did? look forward to it!", "votes": 34 }, { "id": 1255628, "postDate": "2021-03-29T02:10:10.067Z", "content": "Order the train data by the length of the sentences - This approach gave a dramatic improvement in the fitting time because each batch contained only sentences with similar sizes, but it hurt the accuracy of the model too much.
\nRegarding this point @Psi, what can be the reason for this? I have also tried this in some other sequence model, the accuracy dropped dramatically.
", "rawMarkdown": "Order the train data by the length of the sentences - This approach gave a dramatic improvement in the fitting time because each batch contained only sentences with similar sizes, but it hurt the accuracy of the model too much.\n\nRegarding this point @Psi, what can be the reason for this? I have also tried this in some other sequence model, the accuracy dropped dramatically.", "votes": 1, "replies": [ { "id": 1261809, "postDate": "2021-04-03T13:26:15.617Z", "content": "The model focuses only on certain sequence lengths in each batch and has no diversity.
", "rawMarkdown": "The model focuses only on certain sequence lengths in each batch and has no diversity.", "votes": 1 } ] }, { "id": 472122, "postDate": "2019-02-15T11:48:01.430Z", "content": "Could you elaborate about\n 1. Which statistical features did you use?\n 2. On the model architecture the 1D conv follows a BiLSTM, what was done there exactly? Did you use all the states (from all the steps)? Or did you concat the last outputs of each LSTM?
", "rawMarkdown": "Could you elaborate about\n 1. Which statistical features did you use?\n 2. On the model architecture the 1D conv follows a BiLSTM, what was done there exactly? Did you use all the states (from all the steps)? Or did you concat the last outputs of each LSTM?", "votes": 5, "replies": [ { "id": 472178, "postDate": "2019-02-15T13:01:34.987Z", "content": "Can you kindly elaborate on how did you come up with the idea of applying convolution layer here (Why did you use it)?
", "rawMarkdown": "Can you kindly elaborate on how did you come up with the idea of applying convolution layer here (Why did you use it)?" } ] }, { "id": 471482, "postDate": "2019-02-14T14:19:13.843Z", "content": "Congratulation, and thank you for sharing the solution which is truly educative!!\nYou have done much work during these 3 months and totally deserve it.
", "rawMarkdown": "Congratulation, and thank you for sharing the solution which is truly educative!!\nYou have done much work during these 3 months and totally deserve it.\n", "votes": 4 }, { "id": 474345, "postDate": "2019-02-19T08:46:05.543Z", "content": "i like your EDA
", "rawMarkdown": "i like your EDA", "votes": 3 }, { "id": 471629, "postDate": "2019-02-14T17:28:00.223Z", "content": "It's very interesting that you guys did the padding on the fly. We tried the same thing, but our batchsizes were much larger so we ended up with almost exactly the same runtime as if we just prepadded. Did not consider reducing batch_size and only doing the top 95 percent. Very clever trick. I wonder with both of our trick combined how many models could be fit.
", "rawMarkdown": "It's very interesting that you guys did the padding on the fly. We tried the same thing, but our batchsizes were much larger so we ended up with almost exactly the same runtime as if we just prepadded. Did not consider reducing batch_size and only doing the top 95 percent. Very clever trick. I wonder with both of our trick combined how many models could be fit. ", "votes": 4, "replies": [ { "id": 471633, "postDate": "2019-02-14T17:33:42.830Z", "content": "As written, we actually tried much larger batch sizes up to 10240 allowing us to fit close to 20 models. Results looked good as well, but we saw convergence after combining around 10 models or so and would have needed to further fine-tune the higher batch size models which is why we submitted the 512 batch size models. But as you say, the higher the batch size to choose, the larger the max sequence length is, ending up in exactly what you observed.
", "rawMarkdown": "As written, we actually tried much larger batch sizes up to 10240 allowing us to fit close to 20 models. Results looked good as well, but we saw convergence after combining around 10 models or so and would have needed to further fine-tune the higher batch size models which is why we submitted the 512 batch size models. But as you say, the higher the batch size to choose, the larger the max sequence length is, ending up in exactly what you observed. ", "votes": 5 }, { "id": 471852, "postDate": "2019-02-15T01:40:05.200Z", "content": "Padding on the fly is standard practice. Those who are familiar with Tensorflow, padding happens only dynamically (not only, mostly). This is the problem when people start using keras and some default options. I have shared my kernel here https://www.kaggle.com/s4sarath/cudnngru-best-final . With a max length of 150 words per sentences, I was able to train a model in 3600 seconds ( 5 fold with overall 23 epochs), As @psi told, I have also created 10-12 models roughly, but was no helping in improving PB. Thanks, @psi, and team for detailed information.
", "rawMarkdown": "Padding on the fly is standard practice. Those who are familiar with Tensorflow, padding happens only dynamically (not only, mostly). This is the problem when people start using keras and some default options. I have shared my kernel here https://www.kaggle.com/s4sarath/cudnngru-best-final . With a max length of 150 words per sentences, I was able to train a model in 3600 seconds ( 5 fold with overall 23 epochs), As @psi told, I have also created 10-12 models roughly, but was no helping in improving PB. Thanks, @psi, and team for detailed information.", "votes": 5 }, { "id": 530061, "postDate": "2019-05-11T15:54:35.523Z", "content": "@psi, @ryches What do we mean by \"padding on fly\"? Right now, I am padding sequences as data preparation step, like \nX_train = sequence.pad_sequences(X_train,maxlen=MAX_LEN)\nX_valid = sequence.pad_sequences(X_valid,maxlen=MAX_LEN)
", "rawMarkdown": "@psi, @ryches What do we mean by \"padding on fly\"? Right now, I am padding sequences as data preparation step, like \nX_train = sequence.pad_sequences(X_train,maxlen=MAX_LEN)\nX_valid = sequence.pad_sequences(X_valid,maxlen=MAX_LEN)\n", "votes": 1 } ] }, { "id": 476532, "postDate": "2019-02-22T09:59:06.270Z", "content": "awesome
", "rawMarkdown": "awesome", "votes": 1 }, { "id": 474510, "postDate": "2019-02-19T13:28:46.623Z", "content": "Impressive
", "rawMarkdown": "Impressive ", "votes": 1 }, { "id": 473273, "postDate": "2019-02-17T17:41:27.870Z", "content": "What an impressive way. Congratulations and thanks for sharing!
", "rawMarkdown": "What an impressive way. Congratulations and thanks for sharing!", "votes": 1 }, { "id": 472947, "postDate": "2019-02-17T00:35:32.077Z", "content": "Congrats on winning and fantastic explanation throughout !!
", "rawMarkdown": "Congrats on winning and fantastic explanation throughout !! ", "votes": 1 }, { "id": 472515, "postDate": "2019-02-16T04:33:21.767Z", "content": "Congrats! and thanks for sharing your insights.
", "rawMarkdown": "Congrats! and thanks for sharing your insights.", "votes": 1 }, { "id": 472436, "postDate": "2019-02-15T22:59:43.150Z", "content": "Many interesting points here, practical advise and great writeup. Thanks a lot for sharing!
", "rawMarkdown": "Many interesting points here, practical advise and great writeup. Thanks a lot for sharing!", "votes": 1 }, { "id": 472390, "postDate": "2019-02-15T20:11:10.663Z", "content": "Congrats and thanks for sharing your solution. Your solution is really excellent!
", "rawMarkdown": "Congrats and thanks for sharing your solution. Your solution is really excellent!", "votes": 1 }, { "id": 472377, "postDate": "2019-02-15T19:39:21.477Z", "content": "Very interesting model structure.Thanks for sharing!
", "rawMarkdown": "Very interesting model structure.Thanks for sharing!", "votes": 1 }, { "id": 471917, "postDate": "2019-02-15T04:41:40.363Z", "content": "Congrast and thanks for sharing your work.
", "rawMarkdown": "Congrast and thanks for sharing your work.", "votes": 1 }, { "id": 471883, "postDate": "2019-02-15T03:10:16.867Z", "content": "Thank you for the write ups, and congratulations on the strong finish!
", "rawMarkdown": "Thank you for the write ups, and congratulations on the strong finish! ", "votes": 1 }, { "id": 471875, "postDate": "2019-02-15T02:50:27.690Z", "content": "Congrats! Really helped, and thanks for sharing.
", "rawMarkdown": "Congrats! Really helped, and thanks for sharing.", "votes": 1 }, { "id": 471834, "postDate": "2019-02-15T00:53:56.420Z", "content": "Congratulations @philippsinger and team for wining this amazing competition. Thank you for sharing your solution.
", "rawMarkdown": "Congratulations @philippsinger and team for wining this amazing competition. Thank you for sharing your solution.", "votes": 1 }, { "id": 471643, "postDate": "2019-02-14T17:57:48.483Z", "content": "Congratulations. @psi. Thanks for sharing your work and I am really impressed with the idea of bucketing.
", "rawMarkdown": "Congratulations. @psi. Thanks for sharing your work and I am really impressed with the idea of bucketing.", "votes": 1 }, { "id": 471619, "postDate": "2019-02-14T16:57:18.133Z", "content": "Congrats! Really nice work!
", "rawMarkdown": "Congrats! Really nice work!", "votes": 1 }, { "id": 471617, "postDate": "2019-02-14T16:50:42.257Z", "content": "Congratulations and well done! Thank you for the explanation and what a great idea with the padding!
", "rawMarkdown": "Congratulations and well done! Thank you for the explanation and what a great idea with the padding!", "votes": 1 }, { "id": 471549, "postDate": "2019-02-14T15:42:50.430Z", "content": "Congrats, you really deserve it. Thanks for detailed explanation.
", "rawMarkdown": "Congrats, you really deserve it. Thanks for detailed explanation.", "votes": 1 }, { "id": 474782, "postDate": "2019-02-19T19:39:08.670Z", "content": "Certainly pointers can be taken from involved data
", "rawMarkdown": "Certainly pointers can be taken from involved data", "votes": 2 }, { "id": 472794, "postDate": "2019-02-16T17:21:04.663Z", "content": "Congrats...
", "rawMarkdown": "Congrats...", "votes": 2 }, { "id": 471796, "postDate": "2019-02-14T22:21:58.670Z", "content": "Very nice. Congratulations!
\n\nWhat do you mean by:\n\"So what we did instead is to try to find a fixed threshold on CV that produces the least deviation for the f1 score from the optimal threshold. We saw that we can get more stable results when we produce ranks on the predicted probability and average the ranks instead of averaging probabilities. For final submission we then chose the best CV threshold. This also allowed us to fit the model on the complete data without the need to rely on a random split and less training data.\"
\n\nCould you explain it with an example maybe?
", "rawMarkdown": "Very nice. Congratulations!\n\nWhat do you mean by:\n\"So what we did instead is to try to find a fixed threshold on CV that produces the least deviation for the f1 score from the optimal threshold. We saw that we can get more stable results when we produce ranks on the predicted probability and average the ranks instead of averaging probabilities. For final submission we then chose the best CV threshold. This also allowed us to fit the model on the complete data without the need to rely on a random split and less training data.\"\n\nCould you explain it with an example maybe?", "votes": 2, "replies": [ { "id": 471998, "postDate": "2019-02-15T07:40:59.497Z", "content": "Regarding rank prediction: Assume you have predicted probabilities for a single model, you then transform them into ranks (e.g., rankdata in numpy). Then you average the ranks when combining individual models and divide by the length so your final predictions again end up between zero and one. For then finding a fixed threshold a simple strategy is to just take the mean best threshold on multiple CV runs or similar simulations. However, there are still outliers in both directions depending on the split so if you are really unlucky your fixed threshold is \"far\" away from the optimal. So what we tried is testing various fixed thresholds and evaluate how far the resulting F1 score is compared to if you would take the optimal threshold for this fold. We then finally chose that threshold that had the least average deviation from the optimal. Of course, you can still get unlucky, but to a lesser extend. Does that make it a bit clearer? I will try to add an image example to the top post.
", "rawMarkdown": "Regarding rank prediction: Assume you have predicted probabilities for a single model, you then transform them into ranks (e.g., rankdata in numpy). Then you average the ranks when combining individual models and divide by the length so your final predictions again end up between zero and one. For then finding a fixed threshold a simple strategy is to just take the mean best threshold on multiple CV runs or similar simulations. However, there are still outliers in both directions depending on the split so if you are really unlucky your fixed threshold is \"far\" away from the optimal. So what we tried is testing various fixed thresholds and evaluate how far the resulting F1 score is compared to if you would take the optimal threshold for this fold. We then finally chose that threshold that had the least average deviation from the optimal. Of course, you can still get unlucky, but to a lesser extend. Does that make it a bit clearer? I will try to add an image example to the top post.", "votes": 14 }, { "id": 474494, "postDate": "2019-02-19T12:46:47.230Z", "content": "That clarified it a lot, thanks! Interesting approach.
", "rawMarkdown": "That clarified it a lot, thanks! Interesting approach." } ] }, { "id": 471618, "postDate": "2019-02-14T16:54:44.343Z", "content": "Excellent work!\nCongrats!
", "rawMarkdown": "Excellent work!\nCongrats! ", "votes": 2 }, { "id": 471594, "postDate": "2019-02-14T16:19:16.807Z", "content": "How much of an improvement did you get by weigthing glove and para rather than just using glove?
", "rawMarkdown": "How much of an improvement did you get by weigthing glove and para rather than just using glove?", "votes": 2, "replies": [ { "id": 471624, "postDate": "2019-02-14T17:12:04.463Z", "content": "I can't say for sure how much as we decided on using both quite early in the competition, but according to CV it was definitely worth it. We tried a few other things like concatenating etc. which led to similar results, but worse runtime. In the end, this is another aspect of the overfit/underfit discussion above and utilizing other things might lead to the need for doing the combination of embeddings a bit differently.
", "rawMarkdown": "I can't say for sure how much as we decided on using both quite early in the competition, but according to CV it was definitely worth it. We tried a few other things like concatenating etc. which led to similar results, but worse runtime. In the end, this is another aspect of the overfit/underfit discussion above and utilizing other things might lead to the need for doing the combination of embeddings a bit differently.", "votes": 2 }, { "id": 471657, "postDate": "2019-02-14T18:16:55.013Z", "content": "As far as I remember averaged embeddings gave relatively small boost, but quite a stable one. I would say around 0.002 or a bit more.
", "rawMarkdown": "As far as I remember averaged embeddings gave relatively small boost, but quite a stable one. I would say around 0.002 or a bit more.", "votes": 2 } ] }, { "id": 471587, "postDate": "2019-02-14T16:13:08.157Z", "content": "Awesome! Those are some great ideas that I can't wait to try out. Thanks a lot for sharing and congrats!!
", "rawMarkdown": "Awesome! Those are some great ideas that I can't wait to try out. Thanks a lot for sharing and congrats!!", "votes": 2 }, { "id": 3211178, "postDate": "2025-05-28T06:51:11.123Z", "content": "cheers man
\ngreat job
Hi, I have a doubt. If padding is done batch-wise, the sequence length will differ for different batches. In that case, input neurons will be different for different batches. How is it possible practically?
", "rawMarkdown": "Hi, I have a doubt. If padding is done batch-wise, the sequence length will differ for different batches. In that case, input neurons will be different for different batches. How is it possible practically? " }, { "id": 698766, "postDate": "2019-12-19T17:13:25.843Z", "content": "great efforts
", "rawMarkdown": "great efforts" }, { "id": 681462, "postDate": "2019-11-26T06:25:40.753Z", "content": "Wow. This is very informative. Thanks for sharing this.
", "rawMarkdown": "Wow. This is very informative. Thanks for sharing this." }, { "id": 530025, "postDate": "2019-05-11T14:16:20.423Z", "content": "@psi Even few minutes faster counts a lot when we have to finish everything within two hours. According to what I have read online Pytorch is faster than Keras. So just wondering if you used Pytorch and if this helped you to get things done faster?
", "rawMarkdown": "@psi Even few minutes faster counts a lot when we have to finish everything within two hours. According to what I have read online Pytorch is faster than Keras. So just wondering if you used Pytorch and if this helped you to get things done faster?", "replies": [ { "id": 534025, "postDate": "2019-05-20T13:22:57.090Z", "content": "We used Keras in this competition. I shortly tested Pytorch and did not see any speed gains.
", "rawMarkdown": "We used Keras in this competition. I shortly tested Pytorch and did not see any speed gains.", "votes": 1 } ] }, { "id": 529935, "postDate": "2019-05-11T07:39:55.567Z", "content": "Congratulations! This was a great learning experience for me.
", "rawMarkdown": "Congratulations! This was a great learning experience for me." }, { "id": 524655, "postDate": "2019-04-29T09:23:36.523Z", "content": "\n\n\nWe use a one cycle policy with Nadam optimizer (you can do this with the typical CyclicLearningRate implementations by just changing the step size to half your total iterations). We chose a batch size of 512.
\n
Hi @philippsinger. Could you please help me understand this. I'm not quite sure that total refers to here. Is it based upon the batch size \"total\" or the total considering all the iterations needed to go through all epochs?
", "rawMarkdown": "> We use a one cycle policy with Nadam optimizer (you can do this with the typical CyclicLearningRate implementations by just changing the step size to half your total iterations). We chose a batch size of 512.\n\nHi @philippsinger. Could you please help me understand this. I'm not quite sure that total refers to here. Is it based upon the batch size \"total\" or the total considering all the iterations needed to go through all epochs?\n", "replies": [ { "id": 524660, "postDate": "2019-04-29T09:31:32.363Z", "content": "@learnmower For half of the epochs you increase the LR, and for half of the epochs you decrease it. So basically step_size is half of the epochs and you only do one full cycle.
", "rawMarkdown": "@learnmower For half of the epochs you increase the LR, and for half of the epochs you decrease it. So basically step_size is half of the epochs and you only do one full cycle.", "votes": 3 }, { "id": 524861, "postDate": "2019-04-29T16:57:49.620Z", "content": "Ah. Thanks, Psi, for that explanation. +1 up'd!
", "rawMarkdown": "Ah. Thanks, Psi, for that explanation. +1 up'd!" } ] }, { "id": 516428, "postDate": "2019-04-14T06:54:34.517Z", "content": "Great work! Thanks for your detail explanation.
\n\nA question about padding sentence:\nYou said in Runtime tricks: Additionally, we further improved this by not truncating based on the length of the longest sequence in the batch, but based on the 95% percentile of lengths within the sequence. \nand What did not work for us: Order the train data by the length of the sentences - This approach gave a dramatic improvement in the fitting time because each batch contained only sentences with similar sizes, but it hurt the accuracy of the model too much.
Is that means: \n1. Don't sort the sentences by length in whole dataset, even more, we can shuffle it?\n2. On a batch , pad and truncate the sentence by 95% length.
\n\nCongratulation!
", "rawMarkdown": "Great work! Thanks for your detail explanation.\n\nA question about padding sentence:\nYou said in `Runtime tricks`: Additionally, we further improved this by not truncating based on the length of the longest sequence in the batch, but based on the 95% percentile of lengths within the sequence. \nand `What did not work for us`: Order the train data by the length of the sentences - This approach gave a dramatic improvement in the fitting time because each batch contained only sentences with similar sizes, but it hurt the accuracy of the model too much.\n\nIs that means: \n1. Don't sort the sentences by length in whole dataset, even more, we can shuffle it?\n2. On a batch , pad and truncate the sentence by 95% length.\n\nCongratulation!", "replies": [ { "id": 516532, "postDate": "2019-04-14T11:21:14.907Z", "content": "Exactly and thanks :)
", "rawMarkdown": "Exactly and thanks :)" } ] }, { "id": 510114, "postDate": "2019-04-08T17:01:10.163Z", "content": "Congratulations ! And this is an incredible discussion. Thanks for sharing.
", "rawMarkdown": "Congratulations ! And this is an incredible discussion. Thanks for sharing." }, { "id": 479682, "postDate": "2019-02-27T10:38:05.340Z", "content": "Congratulations, tough competition!
", "rawMarkdown": "Congratulations, tough competition! " }, { "id": 477449, "postDate": "2019-02-24T15:19:47.620Z", "content": "good idea!
", "rawMarkdown": "good idea!" }, { "id": 477282, "postDate": "2019-02-24T09:26:34.160Z", "content": "Congrats!
", "rawMarkdown": "Congrats!" }, { "id": 477099, "postDate": "2019-02-23T21:54:32.517Z", "content": "Your EDA is good i have learned a lot.
", "rawMarkdown": "Your EDA is good i have learned a lot." }, { "id": 477047, "postDate": "2019-02-23T18:52:38.773Z", "content": "Congrats !!
", "rawMarkdown": "Congrats !!" }, { "id": 476894, "postDate": "2019-02-23T12:11:47.800Z", "content": "Looking forward to the kernel
", "rawMarkdown": "Looking forward to the kernel" }, { "id": 476871, "postDate": "2019-02-23T11:06:37Z", "content": "Perfect!
", "rawMarkdown": "Perfect!" }, { "id": 476853, "postDate": "2019-02-23T10:17:13.590Z", "rawMarkdown": "" }, { "id": 476030, "postDate": "2019-02-21T14:16:53.100Z", "content": "Congrats and thk u shake. My understanding of your CV Evaluation is like this. You have 10 models and splits training set 10-folds. So each fold of training set was used to train 10 models. You will get the predictions with shape of (n_model, n_sample), and then get average of models result or the average of ranks on the predictions as your ensemble results. Is my understanding correct? BTW, how do u consider to use ranks on the predictions to find threshold to instead of probability?
", "rawMarkdown": "Congrats and thk u shake. My understanding of your CV Evaluation is like this. You have 10 models and splits training set 10-folds. So each fold of training set was used to train 10 models. You will get the predictions with shape of (n_model, n_sample), and then get average of models result or the average of ranks on the predictions as your ensemble results. Is my understanding correct? BTW, how do u consider to use ranks on the predictions to find threshold to instead of probability?" }, { "id": 476025, "postDate": "2019-02-21T14:08:09.757Z", "content": "Awesome
", "rawMarkdown": "Awesome" }, { "id": 475983, "postDate": "2019-02-21T13:10:42.347Z", "content": "Cool
", "rawMarkdown": "Cool" }, { "id": 475716, "postDate": "2019-02-21T05:43:49.370Z", "content": "Brilliant work !
", "rawMarkdown": "Brilliant work !" }, { "id": 475407, "postDate": "2019-02-20T17:59:11.830Z", "content": "Fantastic!
", "rawMarkdown": "Fantastic!" }, { "id": 475375, "postDate": "2019-02-20T17:28:07.597Z", "content": "good idea
", "rawMarkdown": "good idea" }, { "id": 475225, "postDate": "2019-02-20T13:25:03.893Z", "content": "Congrats on your win m8! An incredibly useful follow-up.
", "rawMarkdown": "Congrats on your win m8! An incredibly useful follow-up." }, { "id": 475182, "postDate": "2019-02-20T11:58:52.370Z", "content": "Congrats and thanks for your kindly share. I can learn a lot from it.
", "rawMarkdown": "Congrats and thanks for your kindly share. I can learn a lot from it." }, { "id": 475038, "postDate": "2019-02-20T06:42:34.607Z", "content": "Congrats and thank you for sharing this!
", "rawMarkdown": "Congrats and thank you for sharing this!" }, { "id": 474952, "postDate": "2019-02-20T03:20:37.583Z", "content": "Thanks for sharing, this is great for a beginner like me to learn from.
", "rawMarkdown": "Thanks for sharing, this is great for a beginner like me to learn from. " }, { "id": 474850, "postDate": "2019-02-19T22:13:46.957Z", "content": "Congratulations! Great work!!
", "rawMarkdown": "Congratulations! Great work!!" }, { "id": 474792, "postDate": "2019-02-19T20:03:11.667Z", "content": "Congratulations and thanks for sharing !
", "rawMarkdown": "Congratulations and thanks for sharing !" }, { "id": 474471, "postDate": "2019-02-19T11:49:43.977Z", "content": "Great Approach. Informative.
", "rawMarkdown": "Great Approach. Informative." }, { "id": 473765, "postDate": "2019-02-18T13:32:01.103Z", "content": "Good job!
", "rawMarkdown": "Good job!" }, { "id": 473694, "postDate": "2019-02-18T11:44:53.697Z", "content": "Congrats! That's fantastic.
", "rawMarkdown": "Congrats! That's fantastic." }, { "id": 473648, "postDate": "2019-02-18T10:10:29.627Z", "content": "congratulations
", "rawMarkdown": "congratulations " }, { "id": 473495, "postDate": "2019-02-18T05:07:43.420Z", "content": "ขอบคุณค่ะ:pn\nhttps://slot.im/สล็อต
", "rawMarkdown": "ขอบคุณค่ะ:pn\nhttps://slot.im/สล็อต" }, { "id": 472381, "postDate": "2019-02-15T19:53:45.193Z", "content": "Congrats, impressive to achieve this result with a quite simple model (no attention or capsule layers)\nyou said that different models used for ensembling, is it a similar structure of the diagram or totally different ?
", "rawMarkdown": "Congrats, impressive to achieve this result with a quite simple model (no attention or capsule layers)\nyou said that different models used for ensembling, is it a similar structure of the diagram or totally different ?", "replies": [ { "id": 472385, "postDate": "2019-02-15T20:04:41.633Z", "content": "Same model fit several times.
", "rawMarkdown": "Same model fit several times.", "votes": 1 } ] }, { "id": 472369, "postDate": "2019-02-15T19:22:24.817Z", "content": "Congrats
", "rawMarkdown": "Congrats " }, { "id": 472248, "postDate": "2019-02-15T15:12:45.160Z", "content": "Congrats! Great job, thank you for sharing.
", "rawMarkdown": "Congrats! Great job, thank you for sharing." }, { "id": 471962, "postDate": "2019-02-15T06:36:35.817Z", "content": "Congrats! @psi\nHow did you decide the weights for embeddings?
", "rawMarkdown": "Congrats! @psi\nHow did you decide the weights for embeddings?", "replies": [ { "id": 471988, "postDate": "2019-02-15T07:24:34.313Z", "content": "As my guess, They test different weights to select best(This is my way). :)
", "rawMarkdown": "As my guess, They test different weights to select best(This is my way). :)", "votes": 1 }, { "id": 472000, "postDate": "2019-02-15T07:46:40.520Z", "content": "Thanks! Hyperparameter tuning based on CV.
", "rawMarkdown": "Thanks! Hyperparameter tuning based on CV.", "votes": 1 } ] }, { "id": 471634, "postDate": "2019-02-14T17:34:05.200Z", "content": "Congrats! You mention that you have 10 models on complete training data. Are these models all similar to the one you describe in Model Structure Section?
", "rawMarkdown": "Congrats! You mention that you have 10 models on complete training data. Are these models all similar to the one you describe in Model Structure Section?", "replies": [ { "id": 471635, "postDate": "2019-02-14T17:37:41.190Z", "content": "Yeah, 10 times the same model.
", "rawMarkdown": "Yeah, 10 times the same model.", "votes": 3 } ] }, { "id": 1297775, "postDate": "2021-05-08T09:52:43.873Z", "rawMarkdown": "", "isDeleted": true }, { "id": 486712, "postDate": "2019-03-09T09:18:51.330Z", "rawMarkdown": "", "isDeleted": true }, { "id": 476336, "postDate": "2019-02-22T01:48:27.290Z", "rawMarkdown": "", "isDeleted": true }, { "id": 476238, "postDate": "2019-02-21T21:05:55.960Z", "rawMarkdown": "", "isDeleted": true }, { "id": 474410, "postDate": "2019-02-19T10:03:49.997Z", "rawMarkdown": "", "isDeleted": true }, { "id": 473184, "postDate": "2019-02-17T13:57:46.653Z", "rawMarkdown": "", "isDeleted": true }, { "id": 472514, "postDate": "2019-02-16T04:31:19.573Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 472242, "postDate": "2019-02-15T15:05:49.193Z", "rawMarkdown": "", "isDeleted": true }, { "id": 471753, "postDate": "2019-02-14T20:47:05.130Z", "content": "Congrats! and thanks for sharing your insights. \nI am impressed that you managed to train 10 models; I was stuck at 5 and that was a serious limitation.
", "rawMarkdown": "Congrats! and thanks for sharing your insights. \nI am impressed that you managed to train 10 models; I was stuck at 5 and that was a serious limitation.", "votes": 2, "isDeleted": true, "replies": [ { "id": 471784, "postDate": "2019-02-14T21:55:56.400Z", "content": "Same for me! I really have to try that batch padding trick.
", "rawMarkdown": "Same for me! I really have to try that batch padding trick.", "votes": 1 } ] }, { "id": 473203, "postDate": "2019-02-17T14:39:28.843Z", "content": "Congrats and Thanks !!!
", "rawMarkdown": "Congrats and Thanks !!!", "votes": 1 }, { "id": 473199, "postDate": "2019-02-17T14:28:39.737Z", "content": "Congrats and thanks for sharing!
", "rawMarkdown": "Congrats and thanks for sharing!", "votes": 1 }, { "id": 472963, "postDate": "2019-02-17T02:03:48.380Z", "content": "Congrats and thanks for sharing!
", "rawMarkdown": "Congrats and thanks for sharing!", "votes": 1 }, { "id": 472669, "postDate": "2019-02-16T12:50:09.620Z", "content": "Congrats and thanks for sharing!
", "rawMarkdown": "Congrats and thanks for sharing!", "votes": 1 }, { "id": 472665, "postDate": "2019-02-16T12:43:14.360Z", "content": "Congrats and Thanks for sharing!
", "rawMarkdown": "Congrats and Thanks for sharing!", "votes": 1 }, { "id": 472616, "postDate": "2019-02-16T10:01:06.670Z", "content": "Congrats and Thanks for sharing!
", "rawMarkdown": "Congrats and Thanks for sharing!", "votes": 1 }, { "id": 472539, "postDate": "2019-02-16T05:58:32.957Z", "content": "Congrats and Thanks for sharing!
", "rawMarkdown": "Congrats and Thanks for sharing!", "votes": 1 }, { "id": 472495, "postDate": "2019-02-16T03:23:22.573Z", "content": "Thanks for sharing and congratulations!
", "rawMarkdown": "Thanks for sharing and congratulations!", "votes": 1 }, { "id": 471810, "postDate": "2019-02-14T22:52:07.163Z", "content": "Congrats! and thanks for sharing.
", "rawMarkdown": "Congrats! and thanks for sharing. ", "votes": 1 }, { "id": 1201938, "postDate": "2021-02-15T19:01:54.350Z", "content": "Very useful. Thanks for sharing.
", "rawMarkdown": "Very useful. Thanks for sharing." }, { "id": 570793, "postDate": "2019-07-08T19:06:16.077Z", "content": "Congrats, and Thanks for sharing :)
", "rawMarkdown": "Congrats, and Thanks for sharing :)" }, { "id": 564470, "postDate": "2019-06-29T13:32:13.210Z", "content": "GOOD!Thanks for your work,i love it.
", "rawMarkdown": "GOOD!Thanks for your work,i love it." }, { "id": 552845, "postDate": "2019-06-14T16:27:35.167Z", "content": "Thanks for the post
", "rawMarkdown": "Thanks for the post" }, { "id": 487092, "postDate": "2019-03-10T05:42:03.127Z", "content": "Thank you for sharing this! Congrats!
", "rawMarkdown": "Thank you for sharing this! Congrats!" }, { "id": 478489, "postDate": "2019-02-26T08:06:49.257Z", "content": "thanks for sharing
", "rawMarkdown": "thanks for sharing" }, { "id": 477831, "postDate": "2019-02-25T10:32:09.057Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" }, { "id": 477540, "postDate": "2019-02-24T20:06:06.937Z", "content": "learn a lot. thank you!
", "rawMarkdown": "learn a lot. thank you!" }, { "id": 477064, "postDate": "2019-02-23T19:28:27.337Z", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing." }, { "id": 476883, "postDate": "2019-02-23T11:42:17.853Z", "content": "Congrats and thanks for sharing!
", "rawMarkdown": "Congrats and thanks for sharing!" }, { "id": 476826, "postDate": "2019-02-23T09:15:18.917Z", "content": "Thanks for sharing and congratulations!
", "rawMarkdown": "Thanks for sharing and congratulations!" }, { "id": 476325, "postDate": "2019-02-22T01:30:01.993Z", "content": "Congrats and Thanks for sharing!!!
", "rawMarkdown": "Congrats and Thanks for sharing!!!" }, { "id": 476310, "postDate": "2019-02-22T00:28:46.983Z", "content": "Thanks
", "rawMarkdown": "Thanks" }, { "id": 476078, "postDate": "2019-02-21T15:29:00.587Z", "content": "congrats and thanks for sharing
", "rawMarkdown": "congrats and thanks for sharing" }, { "id": 475750, "postDate": "2019-02-21T06:32:17.937Z", "content": "Congrats and thanks for sharing!
", "rawMarkdown": "Congrats and thanks for sharing!" }, { "id": 475215, "postDate": "2019-02-20T13:03:24.250Z", "content": "Thanks for sharing this!
", "rawMarkdown": "Thanks for sharing this!" }, { "id": 475177, "postDate": "2019-02-20T11:50:46.710Z", "content": "Congrats and thanks!!!
", "rawMarkdown": "Congrats and thanks!!!" }, { "id": 475065, "postDate": "2019-02-20T07:36:58.293Z", "content": "Congrats and thanks!
", "rawMarkdown": "Congrats and thanks!" }, { "id": 474385, "postDate": "2019-02-19T09:33:26.977Z", "content": "Congrats and thanks for the insights
", "rawMarkdown": "Congrats and thanks for the insights" }, { "id": 473970, "postDate": "2019-02-18T19:12:37.290Z", "content": "This taught me a lot! Thank you
", "rawMarkdown": "This taught me a lot! Thank you" }, { "id": 473954, "postDate": "2019-02-18T18:39:09.623Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" }, { "id": 473337, "postDate": "2019-02-17T19:48:52.430Z", "content": "Congrats! Thanks for the write-up
", "rawMarkdown": "Congrats! Thanks for the write-up" }, { "id": 472192, "postDate": "2019-02-15T13:30:49.227Z", "content": "congrats and thanks for sharing !
", "rawMarkdown": "congrats and thanks for sharing !" } ], "comments": [ { "id": 471814, "author_name": "Shubin", "author_url": "", "post_date": "2019-02-14T23:13:19.793000", "content": "congrats, can you open source your kernel so we know exactly what you did? look forward to it!
", "votes": 34, "replies": [] }, { "id": 1255628, "author_name": "Abhishek Verma", "author_url": "", "post_date": "2021-03-29T02:10:10.067000", "content": "Order the train data by the length of the sentences - This approach gave a dramatic improvement in the fitting time because each batch contained only sentences with similar sizes, but it hurt the accuracy of the model too much.
\nRegarding this point @Psi, what can be the reason for this? I have also tried this in some other sequence model, the accuracy dropped dramatically.
", "votes": 1, "replies": [ { "id": 1261809, "author_name": "Psi", "author_url": "", "post_date": "2021-04-03T13:26:15.617000", "content": "The model focuses only on certain sequence lengths in each batch and has no diversity.
", "votes": 1, "replies": [] } ] }, { "id": 472122, "author_name": "Elior Cohen", "author_url": "", "post_date": "2019-02-15T11:48:01.430000", "content": "Could you elaborate about\n 1. Which statistical features did you use?\n 2. On the model architecture the 1D conv follows a BiLSTM, what was done there exactly? Did you use all the states (from all the steps)? Or did you concat the last outputs of each LSTM?
", "votes": 5, "replies": [ { "id": 472178, "author_name": "dott", "author_url": "", "post_date": "2019-02-15T13:01:34.987000", "content": "Can you kindly elaborate on how did you come up with the idea of applying convolution layer here (Why did you use it)?
", "votes": 0, "replies": [] } ] }, { "id": 471482, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-02-14T14:19:13.843000", "content": "Congratulation, and thank you for sharing the solution which is truly educative!!\nYou have done much work during these 3 months and totally deserve it.
", "votes": 4, "replies": [] }, { "id": 474345, "author_name": "Rohith Mohite", "author_url": "", "post_date": "2019-02-19T08:46:05.543000", "content": "i like your EDA
", "votes": 3, "replies": [] }, { "id": 471629, "author_name": "ryches", "author_url": "", "post_date": "2019-02-14T17:28:00.223000", "content": "It's very interesting that you guys did the padding on the fly. We tried the same thing, but our batchsizes were much larger so we ended up with almost exactly the same runtime as if we just prepadded. Did not consider reducing batch_size and only doing the top 95 percent. Very clever trick. I wonder with both of our trick combined how many models could be fit.
", "votes": 4, "replies": [ { "id": 471633, "author_name": "Psi", "author_url": "", "post_date": "2019-02-14T17:33:42.830000", "content": "As written, we actually tried much larger batch sizes up to 10240 allowing us to fit close to 20 models. Results looked good as well, but we saw convergence after combining around 10 models or so and would have needed to further fine-tune the higher batch size models which is why we submitted the 512 batch size models. But as you say, the higher the batch size to choose, the larger the max sequence length is, ending up in exactly what you observed.
", "votes": 5, "replies": [] }, { "id": 471852, "author_name": "aintnosunshine", "author_url": "", "post_date": "2019-02-15T01:40:05.200000", "content": "Padding on the fly is standard practice. Those who are familiar with Tensorflow, padding happens only dynamically (not only, mostly). This is the problem when people start using keras and some default options. I have shared my kernel here https://www.kaggle.com/s4sarath/cudnngru-best-final . With a max length of 150 words per sentences, I was able to train a model in 3600 seconds ( 5 fold with overall 23 epochs), As @psi told, I have also created 10-12 models roughly, but was no helping in improving PB. Thanks, @psi, and team for detailed information.
", "votes": 5, "replies": [] }, { "id": 530061, "author_name": "egdelwonKsIataD", "author_url": "", "post_date": "2019-05-11T15:54:35.523000", "content": "@psi, @ryches What do we mean by \"padding on fly\"? Right now, I am padding sequences as data preparation step, like \nX_train = sequence.pad_sequences(X_train,maxlen=MAX_LEN)\nX_valid = sequence.pad_sequences(X_valid,maxlen=MAX_LEN)
", "votes": 1, "replies": [] } ] }, { "id": 476532, "author_name": "Mohamed Sheik Ibrahim", "author_url": "", "post_date": "2019-02-22T09:59:06.270000", "content": "awesome
", "votes": 1, "replies": [] }, { "id": 474510, "author_name": "anil777", "author_url": "", "post_date": "2019-02-19T13:28:46.623000", "content": "Impressive
", "votes": 1, "replies": [] }, { "id": 473273, "author_name": "Gunhyuk Park", "author_url": "", "post_date": "2019-02-17T17:41:27.870000", "content": "What an impressive way. Congratulations and thanks for sharing!
", "votes": 1, "replies": [] }, { "id": 472947, "author_name": "Vishy", "author_url": "", "post_date": "2019-02-17T00:35:32.077000", "content": "Congrats on winning and fantastic explanation throughout !!
", "votes": 1, "replies": [] }, { "id": 472515, "author_name": "Shabbir", "author_url": "", "post_date": "2019-02-16T04:33:21.767000", "content": "Congrats! and thanks for sharing your insights.
", "votes": 1, "replies": [] }, { "id": 472436, "author_name": "averagemn", "author_url": "", "post_date": "2019-02-15T22:59:43.150000", "content": "Many interesting points here, practical advise and great writeup. Thanks a lot for sharing!
", "votes": 1, "replies": [] }, { "id": 472390, "author_name": "Nancy Guo", "author_url": "", "post_date": "2019-02-15T20:11:10.663000", "content": "Congrats and thanks for sharing your solution. Your solution is really excellent!
", "votes": 1, "replies": [] }, { "id": 472377, "author_name": "Ilya Bakalets", "author_url": "", "post_date": "2019-02-15T19:39:21.477000", "content": "Very interesting model structure.Thanks for sharing!
", "votes": 1, "replies": [] }, { "id": 471917, "author_name": "pbcquoc", "author_url": "", "post_date": "2019-02-15T04:41:40.363000", "content": "Congrast and thanks for sharing your work.
", "votes": 1, "replies": [] }, { "id": 471883, "author_name": "yukiya", "author_url": "", "post_date": "2019-02-15T03:10:16.867000", "content": "Thank you for the write ups, and congratulations on the strong finish!
", "votes": 1, "replies": [] }, { "id": 471875, "author_name": "Yi Chen Huang", "author_url": "", "post_date": "2019-02-15T02:50:27.690000", "content": "Congrats! Really helped, and thanks for sharing.
", "votes": 1, "replies": [] }, { "id": 471834, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-02-15T00:53:56.420000", "content": "Congratulations @philippsinger and team for wining this amazing competition. Thank you for sharing your solution.
", "votes": 1, "replies": [] }, { "id": 471643, "author_name": "Karthik Chowdary Tsaliki", "author_url": "", "post_date": "2019-02-14T17:57:48.483000", "content": "Congratulations. @psi. Thanks for sharing your work and I am really impressed with the idea of bucketing.
", "votes": 1, "replies": [] }, { "id": 471619, "author_name": "XY", "author_url": "", "post_date": "2019-02-14T16:57:18.133000", "content": "Congrats! Really nice work!
", "votes": 1, "replies": [] }, { "id": 471617, "author_name": "Hamish", "author_url": "", "post_date": "2019-02-14T16:50:42.257000", "content": "Congratulations and well done! Thank you for the explanation and what a great idea with the padding!
", "votes": 1, "replies": [] }, { "id": 471549, "author_name": "Murat Korkmaz", "author_url": "", "post_date": "2019-02-14T15:42:50.430000", "content": "Congrats, you really deserve it. Thanks for detailed explanation.
", "votes": 1, "replies": [] }, { "id": 474782, "author_name": "Anuj Garg", "author_url": "", "post_date": "2019-02-19T19:39:08.670000", "content": "Certainly pointers can be taken from involved data
", "votes": 2, "replies": [] }, { "id": 472794, "author_name": "S T MOHAMMED", "author_url": "", "post_date": "2019-02-16T17:21:04.663000", "content": "Congrats...
", "votes": 2, "replies": [] }, { "id": 471796, "author_name": "ManuelSH", "author_url": "", "post_date": "2019-02-14T22:21:58.670000", "content": "Very nice. Congratulations!
\n\nWhat do you mean by:\n\"So what we did instead is to try to find a fixed threshold on CV that produces the least deviation for the f1 score from the optimal threshold. We saw that we can get more stable results when we produce ranks on the predicted probability and average the ranks instead of averaging probabilities. For final submission we then chose the best CV threshold. This also allowed us to fit the model on the complete data without the need to rely on a random split and less training data.\"
\n\nCould you explain it with an example maybe?
", "votes": 2, "replies": [ { "id": 471998, "author_name": "Psi", "author_url": "", "post_date": "2019-02-15T07:40:59.497000", "content": "Regarding rank prediction: Assume you have predicted probabilities for a single model, you then transform them into ranks (e.g., rankdata in numpy). Then you average the ranks when combining individual models and divide by the length so your final predictions again end up between zero and one. For then finding a fixed threshold a simple strategy is to just take the mean best threshold on multiple CV runs or similar simulations. However, there are still outliers in both directions depending on the split so if you are really unlucky your fixed threshold is \"far\" away from the optimal. So what we tried is testing various fixed thresholds and evaluate how far the resulting F1 score is compared to if you would take the optimal threshold for this fold. We then finally chose that threshold that had the least average deviation from the optimal. Of course, you can still get unlucky, but to a lesser extend. Does that make it a bit clearer? I will try to add an image example to the top post.
", "votes": 14, "replies": [] }, { "id": 474494, "author_name": "ManuelSH", "author_url": "", "post_date": "2019-02-19T12:46:47.230000", "content": "That clarified it a lot, thanks! Interesting approach.
", "votes": 0, "replies": [] } ] }, { "id": 471618, "author_name": "Guanshuo Xu", "author_url": "", "post_date": "2019-02-14T16:54:44.343000", "content": "Excellent work!\nCongrats!
", "votes": 2, "replies": [] }, { "id": 471594, "author_name": "João Duro", "author_url": "", "post_date": "2019-02-14T16:19:16.807000", "content": "How much of an improvement did you get by weigthing glove and para rather than just using glove?
", "votes": 2, "replies": [ { "id": 471624, "author_name": "Psi", "author_url": "", "post_date": "2019-02-14T17:12:04.463000", "content": "I can't say for sure how much as we decided on using both quite early in the competition, but according to CV it was definitely worth it. We tried a few other things like concatenating etc. which led to similar results, but worse runtime. In the end, this is another aspect of the overfit/underfit discussion above and utilizing other things might lead to the need for doing the combination of embeddings a bit differently.
", "votes": 2, "replies": [] }, { "id": 471657, "author_name": "dott", "author_url": "", "post_date": "2019-02-14T18:16:55.013000", "content": "As far as I remember averaged embeddings gave relatively small boost, but quite a stable one. I would say around 0.002 or a bit more.
", "votes": 2, "replies": [] } ] }, { "id": 471587, "author_name": "Max Schumacher", "author_url": "", "post_date": "2019-02-14T16:13:08.157000", "content": "Awesome! Those are some great ideas that I can't wait to try out. Thanks a lot for sharing and congrats!!
", "votes": 2, "replies": [] }, { "id": 3211178, "author_name": "Hoai Vo", "author_url": "", "post_date": "2025-05-28T06:51:11.123000", "content": "cheers man
\ngreat job