{ "id": 71946, "title": "What is your single best model doing ?", "url": "/competitions/quora-insincere-questions-classification/discussion/71946", "author_name": "AcademycalBastard", "post_date": "2018-11-18T16:26:38.867000", "votes": 50, "comment_count": 124, "views": 0, "content": "

It will be great to share insights about how is your best single model performming.

", "messages": [ { "id": 423589, "postDate": "2018-11-18T16:26:38.867Z", "content": "

It will be great to share insights about how is your best single model performming.

", "rawMarkdown": "It will be great to share insights about how is your best single model performming.", "votes": 49 }, { "id": 437792, "postDate": "2018-12-12T13:52:43.977Z", "content": "

Migrated to Pytorch. Single model - 0.700 on LB 4 Folds 5 epochs

", "rawMarkdown": "Migrated to Pytorch. Single model - 0.700 on LB 4 Folds 5 epochs", "votes": 11, "replies": [ { "id": 437814, "postDate": "2018-12-12T14:45:16.840Z", "content": "

So your Pytorch results are better than your Keras ones?

", "rawMarkdown": "So your Pytorch results are better than your Keras ones?" }, { "id": 437818, "postDate": "2018-12-12T15:00:55.950Z", "content": "

Yes, and the training time is reduced to. And, honestly I am loving the documentation. Its lit and clean

", "rawMarkdown": "Yes, and the training time is reduced to. And, honestly I am loving the documentation. Its lit and clean", "votes": 3 }, { "id": 437822, "postDate": "2018-12-12T15:10:21.080Z", "content": "

Pytorch is the way to go..

\n\n

Keras is sometimes limited and Tensorflow is a nightmare.

\n\n

I decided to switch to Pytorch since TGS competition, but I'm still too lazy for that.

", "rawMarkdown": "Pytorch is the way to go..\n\nKeras is sometimes limited and Tensorflow is a nightmare. \n\nI decided to switch to Pytorch since TGS competition, but I'm still too lazy for that. ", "votes": 2 }, { "id": 437866, "postDate": "2018-12-12T16:56:32.443Z", "content": "

Any tutorial recommendation to get into it? Also not sure how much changed with the new 1.0 Pytorch release, which I guess won't be in the docker image anyways though.

", "rawMarkdown": "Any tutorial recommendation to get into it? Also not sure how much changed with the new 1.0 Pytorch release, which I guess won't be in the docker image anyways though." }, { "id": 437934, "postDate": "2018-12-12T19:32:28.980Z", "content": "

@shahebaz, does your pytorch model give consistent result? To me that is the most important thing I am trying to figure out here. Models with inconsistent results is pretty much useless imho.

", "rawMarkdown": "@shahebaz, does your pytorch model give consistent result? To me that is the most important thing I am trying to figure out here. Models with inconsistent results is pretty much useless imho." }, { "id": 438282, "postDate": "2018-12-13T12:05:14.330Z", "content": "

hey Shahebaz/serigne can you mark down your pytorch building blocks?\ni am using double lstm + attention +2 dense layers, relu and dropout(0.1),batch size is 512, getting 0.68 single model 3 fold(unable to run more then that in 2 hours). i am not reaching 0.69+.\npreprossing : special char spacing and lower casing all sentences.\nwhat am i doing wrong?

", "rawMarkdown": "hey Shahebaz/serigne can you mark down your pytorch building blocks?\ni am using double lstm + attention +2 dense layers, relu and dropout(0.1),batch size is 512, getting 0.68 single model 3 fold(unable to run more then that in 2 hours). i am not reaching 0.69+.\npreprossing : special char spacing and lower casing all sentences.\nwhat am i doing wrong?\n" }, { "id": 438427, "postDate": "2018-12-13T16:37:33.090Z", "content": "

I'm not using Pytorch, at least not for now. But it's my aim to switch (at least for next images competitions I enter )

\n\n

And my building blocks is different from those typical LSTM/GRU+ Attention/Pooled layers + Dense layers I see on kernels. But you can get very good score with these architectures, try to tune more your network and check for overfitting.

", "rawMarkdown": "I'm not using Pytorch, at least not for now. But it's my aim to switch (at least for next images competitions I enter ) \n\nAnd my building blocks is different from those typical `LSTM/GRU+ Attention/Pooled layers + Dense layers` I see on kernels. But you can get very good score with these architectures, try to tune more your network and check for overfitting. ", "votes": 2 }, { "id": 438584, "postDate": "2018-12-13T23:04:30.110Z", "content": "

@Shahebaz - when you migrated your model from keras to pytorch did you get similar validation results? I just migrated my model to pytorch but the results are far worse than my keras model. As far as I can tell just about everything is identical.

", "rawMarkdown": "@Shahebaz - when you migrated your model from keras to pytorch did you get similar validation results? I just migrated my model to pytorch but the results are far worse than my keras model. As far as I can tell just about everything is identical." }, { "id": 438720, "postDate": "2018-12-14T04:16:03.387Z", "content": "

@rdizzl3 My architecture for 0.700 in Pytorch is differnet that of keras. But, I observed including cyclic CLR made the results worse in Pytorch. I am yet to implement the same architecture that gave me best score with keras. (I hate my exams)

", "rawMarkdown": "@rdizzl3 My architecture for 0.700 in Pytorch is differnet that of keras. But, I observed including cyclic CLR made the results worse in Pytorch. I am yet to implement the same architecture that gave me best score with keras. (I hate my exams) " }, { "id": 438721, "postDate": "2018-12-14T04:27:35.757Z", "content": "

@Shahebaz - thanks for the info! This is my first go with pytorch so I wasn't quite sure what to expect. I will tinker around some more with it :)

", "rawMarkdown": "@Shahebaz - thanks for the info! This is my first go with pytorch so I wasn't quite sure what to expect. I will tinker around some more with it :)" }, { "id": 459514, "postDate": "2019-01-21T22:26:58.513Z", "content": "

Udemy has a good an free course on Pytorch: https://www.udacity.com/course/deep-learning-pytorch--ud188

", "rawMarkdown": "Udemy has a good an free course on Pytorch: https://www.udacity.com/course/deep-learning-pytorch--ud188", "votes": 1 } ] }, { "id": 426445, "postDate": "2018-11-23T09:11:55.917Z", "content": "

I have 0.505 with naïve Bayes :-)

", "rawMarkdown": "I have 0.505 with naïve Bayes :-)", "votes": 10 }, { "id": 437086, "postDate": "2018-12-11T11:06:14.427Z", "content": "

Edit : LB 0.701, I still use single model , 4 folds and some processing

", "rawMarkdown": "Edit : LB 0.701, I still use single model , 4 folds and some processing", "votes": 8, "replies": [ { "id": 437095, "postDate": "2018-12-11T11:21:12.927Z", "content": "

What is your execution time?\nI got 0.701 with minor changes to one of the public kernels but it exceeds execution time limit (8220.5s). Not sure if I did something wrong but didn't get any error when submitting.

", "rawMarkdown": "What is your execution time?\nI got 0.701 with minor changes to one of the public kernels but it exceeds execution time limit (8220.5s). Not sure if I did something wrong but didn't get any error when submitting." }, { "id": 437098, "postDate": "2018-12-11T11:23:31.557Z", "rawMarkdown": "", "votes": -1 }, { "id": 437267, "postDate": "2018-12-11T16:39:27.330Z", "content": "

It run 7177s (a bit less than 2hrs :) )\nAnyway , optimisation will be necessary , 2nd stage will have bigger test data

", "rawMarkdown": "It run 7177s (a bit less than 2hrs :) )\nAnyway , optimisation will be necessary , 2nd stage will have bigger test data ", "votes": 1 }, { "id": 437273, "postDate": "2018-12-11T16:47:26.213Z", "content": "

Nice, Serigne. Is your local f1 score pretty close to what you're seeing on the public LB?

", "rawMarkdown": "Nice, Serigne. Is your local f1 score pretty close to what you're seeing on the public LB?", "votes": 1 }, { "id": 437288, "postDate": "2018-12-11T17:22:01.570Z", "content": "

Congrats on breaking 0.7!

", "rawMarkdown": "Congrats on breaking 0.7!", "votes": 1 }, { "id": 437297, "postDate": "2018-12-11T17:29:04.537Z", "content": "
\n

Following the final submission deadline for the competition, your kernel code will be re-run on a privately-held test set that is not provided to you. It is your model's score against this private test set that will determine your ranking on the private leaderboard and final standing in the competition.

\n
\n\n

@serigne I saw that there was bigger test in last kernel competition. But, I came across no mention of bigger test for this competition. Can you point at the source which says bigger or how big? Thanks!

", "rawMarkdown": "> Following the final submission deadline for the competition, your kernel code will be re-run on a privately-held test set that is not provided to you. It is your model's score against this private test set that will determine your ranking on the private leaderboard and final standing in the competition.\n\n@serigne I saw that there was bigger test in last kernel competition. But, I came across no mention of bigger test for this competition. Can you point at the source which says bigger or how big? Thanks!\n\n" }, { "id": 437378, "postDate": "2018-12-11T19:41:34.520Z", "content": "

@Shahebaz -

\n\n
\n

What will be available in the 2nd stage of the competition? In the\n second stage of the competition, we will re-run your selected Kernels.\n The following files will be swapped with new data:

\n \n

test.csv - This will be swapped with the complete public and private\n test dataset. This file will have ~56k rows in stage 1 and ~376k rows\n in stage 2. The public leaderboard data remains the same for both\n versions. The file name will be the same (both test.csv) to ensure\n that your code will run. sample_submission.csv - similar to test.csv,\n this will be changed from ~56k in stage 1 to ~376k rows in stage 2 .\n The file name will remain the same.

\n
\n\n

From the data page so 2nd stage will be roughly ~7 times larger.

", "rawMarkdown": "@Shahebaz -\n\n> What will be available in the 2nd stage of the competition? In the\n> second stage of the competition, we will re-run your selected Kernels.\n> The following files will be swapped with new data:\n> \n> test.csv - This will be swapped with the complete public and private\n> test dataset. This file will have ~56k rows in stage 1 and ~376k rows\n> in stage 2. The public leaderboard data remains the same for both\n> versions. The file name will be the same (both test.csv) to ensure\n> that your code will run. sample_submission.csv - similar to test.csv,\n> this will be changed from ~56k in stage 1 to ~376k rows in stage 2 .\n> The file name will remain the same.\n\nFrom the data page so 2nd stage will be roughly ~7 times larger. ", "votes": 3 }, { "id": 437535, "postDate": "2018-12-12T05:12:28.550Z", "content": "

Oh, I missed it. Thanks a bunch.

", "rawMarkdown": "Oh, I missed it. Thanks a bunch. " }, { "id": 437544, "postDate": "2018-12-12T05:35:40.773Z", "content": "

@Learnmower I didn't compute f1 in callback in my latest kernels, but my CV loss 0.0955 ( it scores now LB 0.702)

\n\n

Thank @Shujian, No doubt you'll do it very soon :)

", "rawMarkdown": "@Learnmower I didn't compute f1 in callback in my latest kernels, but my CV loss 0.0955 ( it scores now LB 0.702) \n\nThank @Shujian, No doubt you'll do it very soon :)\n", "votes": 4 }, { "id": 437612, "postDate": "2018-12-12T07:34:28.620Z", "content": "

@Serigne Thanks! Your loss looks very good.

", "rawMarkdown": "@Serigne Thanks! Your loss looks very good. ", "votes": 1 }, { "id": 440816, "postDate": "2018-12-18T01:52:29.090Z", "content": "

I got my 4-fold cv down to 0.0955 loss and 0.693 f1-score locally but it still doesn't score well on the LB. It only scores 0.690. I am going to run it multiple times to see how the score changes over each run.

", "rawMarkdown": "I got my 4-fold cv down to 0.0955 loss and 0.693 f1-score locally but it still doesn't score well on the LB. It only scores 0.690. I am going to run it multiple times to see how the score changes over each run.", "votes": 1 } ] }, { "id": 428628, "postDate": "2018-11-27T16:05:11.790Z", "content": "

Adding to the CV vs. LB train:

\n\n

5-fold avg of 1 model, LB .687

\n\n

5 fold CV (with best threshold):

\n\n

Fold 0: .697\nFold 1: .687\nFold 2: .684\nFold 3: .688\nFold 4: .688

\n\n

I guess I should be happy that the LB score is a plausible CV fold score? I'd say I'm relatively confident that the leaderboard isn't super reliable though.

\n\n

Edit: also, same model trained on all data in one run scores .685

", "rawMarkdown": "Adding to the CV vs. LB train:\n\n5-fold avg of 1 model, LB .687\n\n5 fold CV (with best threshold):\n\nFold 0: .697\nFold 1: .687\nFold 2: .684\nFold 3: .688\nFold 4: .688\n\nI guess I should be happy that the LB score is a plausible CV fold score? I'd say I'm relatively confident that the leaderboard isn't super reliable though.\n\nEdit: also, same model trained on all data in one run scores .685", "votes": 8, "replies": [ { "id": 428712, "postDate": "2018-11-27T18:43:31.717Z", "content": "

Do these 5 folds have a separate F1 threshold each, or do you lump all the validation data into a single set and estimate a single F1 threshold which you then apply to the 5 folds to get a fold-wise F1?

\n\n

If you are indeed using a separate threshold for each fold , doesn't the variance in the threshold scare you a bit (In some of my runs, the thresholds range from 0.19 to 0.39 across folds).

", "rawMarkdown": "Do these 5 folds have a separate F1 threshold each, or do you lump all the validation data into a single set and estimate a single F1 threshold which you then apply to the 5 folds to get a fold-wise F1?\n\nIf you are indeed using a separate threshold for each fold , doesn't the variance in the threshold scare you a bit (In some of my runs, the thresholds range from 0.19 to 0.39 across folds). ", "votes": 2 }, { "id": 428740, "postDate": "2018-11-27T19:49:30.557Z", "content": "

This was looking at separate thresholds for each fold, but I also don't see as much variance in thresholds across folds as you've seen - mine range between around .31 and .37. Checking for a global threshold after across the training data (using out of sample preds) also seems to give me something roughly near the average of thresholds (.35). I've also been ensembling using your voting strategy with different thresholds instead of just averaging together the probabilities, though they've seemed to work about the same for me.

\n\n

But yes, the variance in thresholds does scare me, and I'm really wondering what the proper way to ensemble is. In an ideal world I would want to just stack with something like logreg to get a single set of reasonably calibrated probabilistic outputs, but stacking is probably too slow and so far it's unclear to me that blending based on a single out-of-sample set works better than averaging different runs of the same model (like in the k-fold strategy). It's tricky!

", "rawMarkdown": "This was looking at separate thresholds for each fold, but I also don't see as much variance in thresholds across folds as you've seen - mine range between around .31 and .37. Checking for a global threshold after across the training data (using out of sample preds) also seems to give me something roughly near the average of thresholds (.35). I've also been ensembling using your voting strategy with different thresholds instead of just averaging together the probabilities, though they've seemed to work about the same for me.\n\nBut yes, the variance in thresholds does scare me, and I'm really wondering what the proper way to ensemble is. In an ideal world I would want to just stack with something like logreg to get a single set of reasonably calibrated probabilistic outputs, but stacking is probably too slow and so far it's unclear to me that blending based on a single out-of-sample set works better than averaging different runs of the same model (like in the k-fold strategy). It's tricky! ", "votes": 3 } ] }, { "id": 436288, "postDate": "2018-12-10T04:12:02.293Z", "content": "

0.699 single model with 5 folds. Slightly modified my public kernels according to some posts above.

", "rawMarkdown": "0.699 single model with 5 folds. Slightly modified my public kernels according to some posts above.", "votes": 6 }, { "id": 424011, "postDate": "2018-11-19T12:32:02.523Z", "content": "

LB : 0.694 ( single model with k-fold)

", "rawMarkdown": "LB : 0.694 ( single model with k-fold)", "votes": 6, "replies": [ { "id": 424016, "postDate": "2018-11-19T12:47:23.780Z", "content": "

Same for me: 0.693 single model k-fold. It varies though, with the exact same model I got 0.685 too (I have not investigated yet)

", "rawMarkdown": "Same for me: 0.693 single model k-fold. It varies though, with the exact same model I got 0.685 too (I have not investigated yet)", "votes": 3 }, { "id": 424041, "postDate": "2018-11-19T13:20:29.483Z", "content": "

Wow , without any change ?

\n\n

My score varies between 0.691 and 0.694 when I do some changes either on the network or on the data ( I didn't try to submit the same exact model twice to check such variation )

", "rawMarkdown": "Wow , without any change ? \n\nMy score varies between 0.691 and 0.694 when I do some changes either on the network or on the data ( I didn't try to submit the same exact model twice to check such variation ) ", "votes": 2 }, { "id": 424076, "postDate": "2018-11-19T14:30:38.543Z", "content": "

Congratz, pretty good for a single model. How is the runtime of that model approximately?

", "rawMarkdown": "Congratz, pretty good for a single model. How is the runtime of that model approximately?" }, { "id": 424081, "postDate": "2018-11-19T14:38:24.253Z", "content": "

Yes, without any change. My best was an earlier version, I modified it, ran a few other trials, but there was no improvement. So I reverted back to the best version and I accidentally ran it again. I did not notice until I compared the two version with the diff-tool. I only changed some comments (the results: 0.693 / 0.685). I will look at this problem closer, it is weird.

", "rawMarkdown": "Yes, without any change. My best was an earlier version, I modified it, ran a few other trials, but there was no improvement. So I reverted back to the best version and I accidentally ran it again. I did not notice until I compared the two version with the diff-tool. I only changed some comments (the results: 0.693 / 0.685). I will look at this problem closer, it is weird." }, { "id": 424082, "postDate": "2018-11-19T14:39:52.207Z", "content": "

@philippsinger, runtime for me was ~6000-6500s

", "rawMarkdown": "@philippsinger, runtime for me was ~6000-6500s" }, { "id": 424202, "postDate": "2018-11-19T18:18:50.357Z", "content": "

@pestipeti Are you using Keras or pure Tensorflow or Pytorch ?

\n\n

I think Keras may have some very random behavior compared to the two others . I tried to fix all seeds , even for kernel_initializers.

\n\n

@philippsinger, for my best it's about 5800s

", "rawMarkdown": "@pestipeti Are you using Keras or pure Tensorflow or Pytorch ? \n\nI think Keras may have some very random behavior compared to the two others . I tried to fix all seeds , even for kernel_initializers. \n\n@philippsinger, for my best it's about 5800s", "votes": 1 }, { "id": 424205, "postDate": "2018-11-19T18:23:53.920Z", "content": "

@serigne, I use keras, but I will switch to pytorch. I remember earlier posts (TGS competition) about the same weird behavior, they blamed keras too..

", "rawMarkdown": "@serigne, I use keras, but I will switch to pytorch. I remember earlier posts (TGS competition) about the same weird behavior, they blamed keras too.." }, { "id": 424366, "postDate": "2018-11-20T01:35:06.577Z", "content": "

Are you doing some kind of text preprocessing with single models? I understand if you don't want to answer it :)

", "rawMarkdown": "Are you doing some kind of text preprocessing with single models? I understand if you don't want to answer it :)" }, { "id": 424399, "postDate": "2018-11-20T03:56:51.867Z", "content": "

My best single model scores 0.676 on the LB, and it is a single run not k-Folds.

\n\n

@Serigne, @Peter, what is the score for your single model for one run? I mean before you did k-Folds?

\n\n

I have also noticed inconsistencies in scores with another model of mine. That one is an ensemble model so I thought may be I have a bug in my code but have not had time to debug it yet. I am using keras.

", "rawMarkdown": "My best single model scores 0.676 on the LB, and it is a single run not k-Folds. \n\n@Serigne, @Peter, what is the score for your single model for one run? I mean before you did k-Folds?\n\nI have also noticed inconsistencies in scores with another model of mine. That one is an ensemble model so I thought may be I have a bug in my code but have not had time to debug it yet. I am using keras.", "votes": 1 }, { "id": 424424, "postDate": "2018-11-20T05:23:40.750Z", "content": "

@serigne, \nI have been pretty frustrated with the non-reproducibility aspect of Keras based kernels as well. This is not very well documented either.

\n\n

I currently use \nnp.random.seed(1)\ntf.set_random_seed(1)

\n\n

Some Kaggle and Stackoverflow thread also suggest that using CuDNNLSTM or CuDNNGRU can lead to non-deterministic behavior. I have not explore this further.

\n\n

@serigne, Would you mind sharing the random intializations that you use?

", "rawMarkdown": "@serigne, \nI have been pretty frustrated with the non-reproducibility aspect of Keras based kernels as well. This is not very well documented either.\n\n\nI currently use \nnp.random.seed(1)\ntf.set_random_seed(1)\n\nSome Kaggle and Stackoverflow thread also suggest that using CuDNNLSTM or CuDNNGRU can lead to non-deterministic behavior. I have not explore this further.\n\n@serigne, Would you mind sharing the random intializations that you use?" }, { "id": 424446, "postDate": "2018-11-20T06:14:15.157Z", "content": "

@sheriytm, My single best was 0.679, but because of this keras/cudnn issue it could be 0.688 (or 0.670). If I can not reproduce then it is worthless.

", "rawMarkdown": "@sheriytm, My single best was 0.679, but because of this keras/cudnn issue it could be 0.688 (or 0.670). If I can not reproduce then it is worthless.", "votes": 1 }, { "id": 424500, "postDate": "2018-11-20T08:02:47.677Z", "content": "

I feel the same way. Running experiments right now before I rule those models out.

\n\n

One thing I know is that I have used both CuDNNLSTM and CuDNNGRU in the toxic comment competition but with k-fold and did not observe any inconsistencies in my results. I am hoping doing k-fold will eliminate that. My current single models are too heavy for k-folds at the moment, so I need to work on streamlining that first.

", "rawMarkdown": "I feel the same way. Running experiments right now before I rule those models out. \n\nOne thing I know is that I have used both CuDNNLSTM and CuDNNGRU in the toxic comment competition but with k-fold and did not observe any inconsistencies in my results. I am hoping doing k-fold will eliminate that. My current single models are too heavy for k-folds at the moment, so I need to work on streamlining that first.", "votes": 1 }, { "id": 424569, "postDate": "2018-11-20T10:32:39.297Z", "content": "

@mihajlot ....No cleaning and no special processing for now

\n\n

I think the architecture of the network is important here ...Some others architectures I tried, failed to improve after 3 or 4 epoches.

", "rawMarkdown": "@mihajlot ....No cleaning and no special processing for now \n\nI think the architecture of the network is important here ...Some others architectures I tried, failed to improve after 3 or 4 epoches.", "votes": 4 }, { "id": 424571, "postDate": "2018-11-20T10:36:05.723Z", "content": "

@sheriytm I didn't try it. Actually I built k-folds since my first kernel , before starting experimenting models :)

\n\n

But I will later to see.

", "rawMarkdown": "@sheriytm I didn't try it. Actually I built k-folds since my first kernel , before starting experimenting models :) \n\nBut I will later to see. ", "votes": 1 }, { "id": 424573, "postDate": "2018-11-20T10:43:32.240Z", "content": "

@kagsen \nI fixed those and the kernel_initializers features like this :

\n\n
from keras.initializers import he_normal, he_uniform,  glorot_normal,  glorot_uniform\n\nBidirectional(CuDNNLSTM(self.lstm_num_feat, \n                        kernel_initializer=glorot_uniform(seed = 123),\n                         return_sequences=True)) \n\nDense(self.dense_num_feat, \n          kernel_initializer=he_uniform(seed=123), \n          activation='relu')\n
", "rawMarkdown": "@kagsen \nI fixed those and the kernel_initializers features like this : \n\n from keras.initializers import he_normal, he_uniform, glorot_normal, glorot_uniform\n\n Bidirectional(CuDNNLSTM(self.lstm_num_feat, \n kernel_initializer=glorot_uniform(seed = 123),\n return_sequences=True)) \n\n Dense(self.dense_num_feat, \n kernel_initializer=he_uniform(seed=123), \n activation='relu')\n\n", "votes": 10 }, { "id": 424966, "postDate": "2018-11-21T00:09:28.743Z", "content": "

@Serigne, @Peter - how many folds are you guys using?

", "rawMarkdown": "@Serigne, @Peter - how many folds are you guys using?" }, { "id": 424975, "postDate": "2018-11-21T00:31:54.127Z", "content": "

@rdizzl3, I used 5 folds for my best result (6 epochs/fold)

", "rawMarkdown": "@rdizzl3, I used 5 folds for my best result (6 epochs/fold)", "votes": 2 }, { "id": 425034, "postDate": "2018-11-21T03:22:04.963Z", "content": "

Hey Serigne. How many folds are you using? I saw your post in another thread and with the timings, I had assumed 5. I'm also seeing convergence around 3-4 epochs. Though I am wondering if I should be basing my decision on loss or accuracy. Accuracy starts to drop about 3-4, but loss still improves.

", "rawMarkdown": "Hey Serigne. How many folds are you using? I saw your post in another thread and with the timings, I had assumed 5. I'm also seeing convergence around 3-4 epochs. Though I am wondering if I should be basing my decision on loss or accuracy. Accuracy starts to drop about 3-4, but loss still improves.\n" }, { "id": 425048, "postDate": "2018-11-21T03:56:58.250Z", "content": "

@Serigne, I am really curious about what is k in your k-fold. Seems if k is small, the training set is too small so the results are not good. If k is large, the model has to be really simple so it can finish in 2 hrs.

", "rawMarkdown": "@Serigne, I am really curious about what is k in your k-fold. Seems if k is small, the training set is too small so the results are not good. If k is large, the model has to be really simple so it can finish in 2 hrs.", "votes": 1 }, { "id": 425158, "postDate": "2018-11-21T08:00:16.887Z", "rawMarkdown": "", "isDeleted": true }, { "id": 425320, "postDate": "2018-11-21T12:47:44.883Z", "content": "

@rdizzl3 and @learnmower I splitted the data in 4 folds but use only 3 folds for now. So there are three runs and still a part of the data not used for validation. However I plan to reintegrate the last fold as soon as I optimize better my code .

\n\n

@shujian I suppose k small or large depends also on the size of the test set ( and we don't have many choices here , as you say, due to 2hrs limitation ) . Anyway I did not split the data randomly :)

", "rawMarkdown": "@rdizzl3 and @learnmower I splitted the data in 4 folds but use only 3 folds for now. So there are three runs and still a part of the data not used for validation. However I plan to reintegrate the last fold as soon as I optimize better my code . \n\n@shujian I suppose k small or large depends also on the size of the test set ( and we don't have many choices here , as you say, due to 2hrs limitation ) . Anyway I did not split the data randomly :) ", "votes": 3 }, { "id": 425897, "postDate": "2018-11-22T09:17:41.313Z", "content": "

@Peter, @Serigne - if you don't mind saying, do you know your average K-Fold scores and overall score on OOF predictions?

", "rawMarkdown": "@Peter, @Serigne - if you don't mind saying, do you know your average K-Fold scores and overall score on OOF predictions?" }, { "id": 425916, "postDate": "2018-11-22T09:55:29.153Z", "content": "

@rdizzl3, For some reason kaggle cut my logs after ~300 lines, so I don't have all of it, but here are my local f1 scores for my best model:

\n\n\n\n

My public score for this (average of 5 folds) was 0.693, but read this too.

", "rawMarkdown": "@rdizzl3, For some reason kaggle cut my logs after ~300 lines, so I don't have all of it, but here are my local f1 scores for my best model:\n\n- Fold 0 - 0.6771\n- Fold 1 - 0.6740\n- Fold 2 - 0.6812\n- Fold 3 - 0.6751\n\nMy public score for this (average of 5 folds) was 0.693, but read [this](https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/72040) too.", "votes": 2 }, { "id": 425952, "postDate": "2018-11-22T10:56:10.937Z", "content": "

@RDizzl3 my f1 val_scores :

\n\n

Fold0 : 0.6668\nFold1 : 0.6761\nFold2 : (skipped for now and will be reintegrated later)\nFold3 : 0.6846

", "rawMarkdown": "@RDizzl3 my f1 val_scores : \n\nFold0 : 0.6668\nFold1 : 0.6761\nFold2 : (skipped for now and will be reintegrated later)\nFold3 : 0.6846\n", "votes": 2 }, { "id": 425956, "postDate": "2018-11-22T11:00:44.553Z", "content": "

@pestipeti you can append them on a list and print them at the next cell

\n\n

I use to run my kernels in interactive session too, just after committing, so I can check out all the output later.

", "rawMarkdown": "@pestipeti you can append them on a list and print them at the next cell\n\nI use to run my kernels in interactive session too, just after committing, so I can check out all the output later. ", "votes": 1 }, { "id": 426206, "postDate": "2018-11-22T21:40:53.123Z", "content": "

Thank you guys for the replies! Check out my 5-folds:

\n\n

fold-0: 0.6779\nfold-1: 0.6821\nfold-2: 0.6860\nfold-3: 0.6883\nfold-4: 0.6817

\n\n

only scores 0.683 on the LB - average of 5-folds.

", "rawMarkdown": "Thank you guys for the replies! Check out my 5-folds:\n\nfold-0: 0.6779\nfold-1: 0.6821\nfold-2: 0.6860\nfold-3: 0.6883\nfold-4: 0.6817\n\nonly scores 0.683 on the LB - average of 5-folds.", "votes": 1 }, { "id": 426284, "postDate": "2018-11-23T02:23:24.630Z", "content": "

@serigne, hi, I am getting closer to your model :)

", "rawMarkdown": "@serigne, hi, I am getting closer to your model :) ", "votes": 2 }, { "id": 426443, "postDate": "2018-11-23T09:10:16.157Z", "content": "

@Peter how many epochs?

", "rawMarkdown": "@Peter how many epochs?" }, { "id": 426737, "postDate": "2018-11-23T18:26:02.617Z", "content": "

@Shujian I have no doubt you will do it very soon :)

", "rawMarkdown": "@Shujian I have no doubt you will do it very soon :)", "votes": 1 }, { "id": 426739, "postDate": "2018-11-23T18:29:57.150Z", "content": "

Thanks ;)

", "rawMarkdown": "Thanks ;)" }, { "id": 427108, "postDate": "2018-11-24T15:53:27.407Z", "content": "

@rupeshs, 6 epochs/fold

", "rawMarkdown": "@rupeshs, 6 epochs/fold" }, { "id": 427340, "postDate": "2018-11-25T08:50:45.697Z", "content": "

A few people have mentioned the randomness that happens in keras CuDNNs - I've was curious enough to do some digging and found this comment in the tensorflow gh

\n\n
\n

On GPU, small amount of non-deterministic results is expected. TensorFlow uses the Eigen library, which uses Cuda atomics to implement reduction operations, such as tf.reduce_sum etc. Those operations are non-determnistical. Each operation can introduce a small difference. If your model is not stable, it could accumulate into large errors, after many steps.

\n
\n\n

Wonder if this is just \"a GPU thing\" in general?

", "rawMarkdown": "A few people have mentioned the randomness that happens in keras CuDNNs - I've was curious enough to do some digging and found [this comment](https://github.com/tensorflow/tensorflow/issues/2732#issuecomment-224661591) in the tensorflow gh\n\n> On GPU, small amount of non-deterministic results is expected. TensorFlow uses the Eigen library, which uses Cuda atomics to implement reduction operations, such as tf.reduce_sum etc. Those operations are non-determnistical. Each operation can introduce a small difference. If your model is not stable, it could accumulate into large errors, after many steps.\n\nWonder if this is just \"a GPU thing\" in general?", "votes": 2 }, { "id": 429707, "postDate": "2018-11-29T08:14:41.880Z", "content": "

My single model is at LB 0.696, CV 0.683 with 5 epochs and 4 folds

", "rawMarkdown": "My single model is at LB 0.696, CV 0.683 with 5 epochs and 4 folds" } ] }, { "id": 452606, "postDate": "2019-01-09T00:06:44.217Z", "content": "

single model with 5 Stratified KFold, CV log_loss 0.09908, f1_score 0.67972, LB 0.684

", "rawMarkdown": "single model with 5 Stratified KFold, CV log_loss 0.09908, f1_score 0.67972, LB 0.684", "votes": 4 }, { "id": 426817, "postDate": "2018-11-23T23:29:09.667Z", "content": "

Currently have a LB of 0.688 with one model, no k fold and only GloVe embeddings

", "rawMarkdown": "Currently have a LB of 0.688 with one model, no k fold and only GloVe embeddings", "votes": 3 }, { "id": 436389, "postDate": "2018-12-10T08:17:50.007Z", "content": "

Adding on my kernel the cleaning from this kernel improved my score from 0.696 to 0.699

\n\n

I didn't clean anything before.

", "rawMarkdown": "Adding on my kernel the cleaning from [this kernel][1] improved my score from 0.696 to 0.699\n\nI didn't clean anything before. \n\n [1]: https://www.kaggle.com/hung96ad/magic-numbers-is-all-you-need-0-696-lb", "votes": 1, "replies": [ { "id": 436396, "postDate": "2018-12-10T08:23:51.033Z", "content": "

I noted the same. Btw, are you sure improvement is cause of cleaning and not randomness. I use different processing than that of kernel and my average score of 10-15 runtimes is slower while clearing punctuations. Thanks for reporting though!

", "rawMarkdown": "I noted the same. Btw, are you sure improvement is cause of cleaning and not randomness. I use different processing than that of kernel and my average score of 10-15 runtimes is slower while clearing punctuations. Thanks for reporting though!" }, { "id": 436399, "postDate": "2018-12-10T08:29:30.980Z", "content": "

I don't know...likely randomness, given I didn't notice huge improvement on CV :)

", "rawMarkdown": "I don't know...likely randomness, given I didn't notice huge improvement on CV :)\n" }, { "id": 436401, "postDate": "2018-12-10T08:33:54.237Z", "content": "

I am -1 for any new experiments until I migrate to Pytorch. I haven't used it before. Never say never :)

", "rawMarkdown": "I am -1 for any new experiments until I migrate to Pytorch. I haven't used it before. Never say never :) " }, { "id": 436425, "postDate": "2018-12-10T09:23:31.390Z", "content": "

@Serigne, same here. Adding the cleaning plus a couple of tweaks, my score improved from 0.693 to 0.698 but I am still concerned about the randomness/ inconsistencies.

", "rawMarkdown": "@Serigne, same here. Adding the cleaning plus a couple of tweaks, my score improved from 0.693 to 0.698 but I am still concerned about the randomness/ inconsistencies.", "votes": 1 }, { "id": 437153, "postDate": "2018-12-11T13:10:12.433Z", "content": "

I wonder how that cleaning makes sense for most of the characters in it as keras tokenizer is removing them anyways by default.

", "rawMarkdown": "I wonder how that cleaning makes sense for most of the characters in it as keras tokenizer is removing them anyways by default." }, { "id": 437561, "postDate": "2018-12-12T06:01:15.457Z", "content": "

I also got scores in the ranges of 0.692 - 0.697 with cleaning. Even the same kernel when run again is giving different results. It's frustrating in the sense that we don't really know what improved our results and what not. @Serigne, can you run your highest scoring kernel again and check if the score remains same? If you don't mind wasting a submission.

", "rawMarkdown": "I also got scores in the ranges of 0.692 - 0.697 with cleaning. Even the same kernel when run again is giving different results. It's frustrating in the sense that we don't really know what improved our results and what not. @Serigne, can you run your highest scoring kernel again and check if the score remains same? If you don't mind wasting a submission.", "votes": 1 } ] }, { "id": 433084, "postDate": "2018-12-04T17:07:53.580Z", "content": "

I've got 0.691 on 4-folds with RMSprop optimizer and mean-combined embedding. I've tried perhaps 10 different parameter configurations but I can't get past this score on LB.

", "rawMarkdown": "I've got 0.691 on 4-folds with RMSprop optimizer and mean-combined embedding. I've tried perhaps 10 different parameter configurations but I can't get past this score on LB.", "votes": 1, "replies": [ { "id": 433088, "postDate": "2018-12-04T17:18:00.680Z", "content": "

Hi Radek, did RMSprop optimizer boost LB? Thanks.

", "rawMarkdown": "Hi Radek, did RMSprop optimizer boost LB? Thanks." }, { "id": 433113, "postDate": "2018-12-04T18:03:03.787Z", "content": "

It's hard to tell due to the LB score variance, but I think it helped in this case. I've used your network from https://www.kaggle.com/shujian/single-rnn-with-4-folds-clr but went with one less epoch and second LSTM layer instead of GRU. I've tried tweaking learning rate, number of neurons in layers, dropout and so on, but nothing seems to help getting better LB score although it gets better locally on CV.

", "rawMarkdown": "It's hard to tell due to the LB score variance, but I think it helped in this case. I've used your network from https://www.kaggle.com/shujian/single-rnn-with-4-folds-clr but went with one less epoch and second LSTM layer instead of GRU. I've tried tweaking learning rate, number of neurons in layers, dropout and so on, but nothing seems to help getting better LB score although it gets better locally on CV.", "votes": 1 }, { "id": 433119, "postDate": "2018-12-04T18:10:57.147Z", "content": "

You can take a look at it here: https://www.kaggle.com/rasvob/let-s-try-clr-v3\nLB with 0.691 is in Version 2 and 7, have fun :)

", "rawMarkdown": "You can take a look at it here: https://www.kaggle.com/rasvob/let-s-try-clr-v3\nLB with 0.691 is in Version 2 and 7, have fun :) ", "votes": 1 }, { "id": 433120, "postDate": "2018-12-04T18:10:59.323Z", "content": "

Thanks for your feedback!

", "rawMarkdown": "Thanks for your feedback!" } ] }, { "id": 433803, "postDate": "2018-12-05T13:49:43.647Z", "content": "

0.697 - 4 Kfold , cyclic LR, mean combined embedding. But can't go above this even if I were able to improve CV. Which is very frustrating as I can't get any idea about what is what.

", "rawMarkdown": "0.697 - 4 Kfold , cyclic LR, mean combined embedding. But can't go above this even if I were able to improve CV. Which is very frustrating as I can't get any idea about what is what.", "votes": 2 }, { "id": 432970, "postDate": "2018-12-04T14:29:03.687Z", "content": "

79 mins - 4fold - CV 0.685 - LB 0.691

", "rawMarkdown": "`79 mins` - `4fold` - `CV 0.685` - `LB 0.691`", "votes": 2, "replies": [ { "id": 433672, "postDate": "2018-12-05T10:26:03.960Z", "content": "

thank you

", "rawMarkdown": "thank you" }, { "id": 434998, "postDate": "2018-12-07T09:33:08.087Z", "content": "

small old brother , please raise me up

", "rawMarkdown": "small old brother , please raise me up ", "votes": -1 }, { "id": 441123, "postDate": "2018-12-18T09:33:02.767Z", "content": "

83 min - 5fold - CV 0.683 - LB 0.697 - pytorch

", "rawMarkdown": "`83 min` - `5fold` - `CV 0.683` - `LB 0.697` - `pytorch`", "votes": 4 }, { "id": 441702, "postDate": "2018-12-18T23:57:07.230Z", "content": "

Did you verify how many minutes can pytorch save you ?

", "rawMarkdown": "Did you verify how many minutes can pytorch save you ?" }, { "id": 441744, "postDate": "2018-12-19T02:15:21.667Z", "content": "

@Neuron

\n\n

No, in the process of migrating to pytorch, data loading and model structure will have some necessary adjustments.

", "rawMarkdown": "@Neuron\n\nNo, in the process of migrating to pytorch, data loading and model structure will have some necessary adjustments." }, { "id": 443849, "postDate": "2018-12-22T15:14:21.850Z", "content": "

100 min - 5fold - CV 0.685 - LB 0.698 - pytorch

", "rawMarkdown": "`100 min` - `5fold` - `CV 0.685` - `LB 0.698` - `pytorch ` ", "votes": 2 }, { "id": 447802, "postDate": "2018-12-30T15:34:11.177Z", "content": "

110 min - 5fold - CV 0.683 - LB 0.699 - pytorch

", "rawMarkdown": "`110 min` - `5fold` - `CV 0.683` - `LB 0.699` - `pytorch `" }, { "id": 448327, "postDate": "2018-12-31T19:37:01.823Z", "content": "

regular structure? (2 bi lstm +attention?)

", "rawMarkdown": "regular structure? (2 bi lstm +attention?)" } ] }, { "id": 423760, "postDate": "2018-11-19T02:37:36.537Z", "content": "

My Best Single Model Public Leaderboard Score: 0.675

", "rawMarkdown": "My Best Single Model Public Leaderboard Score: 0.675", "votes": 2 }, { "id": 423869, "postDate": "2018-11-19T06:57:52.167Z", "content": "

0.685

", "rawMarkdown": "0.685" }, { "id": 446459, "postDate": "2018-12-28T05:22:53.320Z", "content": "

change parameters improve LB score to 0.702, it may be overfit on testset,so,I'm worried about the second stage

", "rawMarkdown": "change parameters improve LB score to 0.702, it may be overfit on testset,so,I'm worried about the second stage", "votes": 1 }, { "id": 426442, "postDate": "2018-11-23T09:07:00.243Z", "content": "

LB: 66.3 (Single model ,no kfold)

", "rawMarkdown": "LB: 66.3 (Single model ,no kfold)", "votes": -1 }, { "id": 460246, "postDate": "2019-01-23T09:30:24.160Z", "content": "

Change parameters of public kernel\nLB: 0.701 CV 0.6823

", "rawMarkdown": "Change parameters of public kernel\nLB: 0.701 CV 0.6823" }, { "id": 439764, "postDate": "2018-12-16T09:58:22.037Z", "content": "

Hi, is anybody able to go beyond both “LB 0.69” and “Local CV 0.69” , using only ‘single classifier’ ? (single model with no KFold, no ensemble) . In my case, even though my ensemble get LB0.697, my single only get LB0.675 (with 0.693 Local CV)

", "rawMarkdown": "Hi, is anybody able to go beyond both “LB 0.69” and “Local CV 0.69” , using only ‘single classifier’ ? (single model with no KFold, no ensemble) . In my case, even though my ensemble get LB0.697, my single only get LB0.675 (with 0.693 Local CV)", "replies": [ { "id": 440268, "postDate": "2018-12-17T10:01:04.153Z", "content": "

I think your model might be overfitting...

", "rawMarkdown": "I think your model might be overfitting...", "votes": 1 }, { "id": 440281, "postDate": "2018-12-17T10:27:25.433Z", "content": "

You are right. However, I just would like to know if anybody can do >= 0.69 on both LB and Local CV. Can you ?

", "rawMarkdown": "You are right. However, I just would like to know if anybody can do >= 0.69 on both LB and Local CV. Can you ?" }, { "id": 440285, "postDate": "2018-12-17T10:46:43.780Z", "content": "

What do you mean with Local CV no KFold?

", "rawMarkdown": "What do you mean with Local CV no KFold?", "votes": 1 }, { "id": 440297, "postDate": "2018-12-17T11:05:03.713Z", "content": "

Hi Philipp,

\n\n

Since there might be confusion about prediction using 'a single model with KFold' and 'a single classifier'.

\n\n

When we do KFold (although with a single model), in the end we have K classifiers (one for each validation fold), and usually we predict the final result using this ensemble of K classifiers.

\n\n

What I want to know is the performance of a single classifier (no average with other classifiers). Previously, I see one of the kaggler (top 10 in the current LB) claimed that he could get his very high performance by not using ensemble at all (but with some trick), but now his comment is deleted.

", "rawMarkdown": "Hi Philipp, \n\nSince there might be confusion about prediction using 'a single model with KFold' and 'a single classifier'.\n\nWhen we do KFold (although with a single model), in the end we have K classifiers (one for each validation fold), and usually we predict the final result using this ensemble of K classifiers. \n\nWhat I want to know is the performance of a single classifier (no average with other classifiers). Previously, I see one of the kaggler (top 10 in the current LB) claimed that he could get his very high performance by not using ensemble at all (but with some trick), but now his comment is deleted. ", "votes": 1 }, { "id": 440306, "postDate": "2018-12-17T11:25:57.840Z", "content": "

@ratthachat, I think I may have also read the comment you mentioned in the forum. I think by single model, he meant KFold but not a kernel like SKR's that had 3-different types of models averaging the result.

", "rawMarkdown": "@ratthachat, I think I may have also read the comment you mentioned in the forum. I think by single model, he meant KFold but not a kernel like SKR's that had 3-different types of models averaging the result.", "votes": 1 }, { "id": 440320, "postDate": "2018-12-17T11:57:25.627Z", "content": "

That's also how I understood it.

", "rawMarkdown": "That's also how I understood it." }, { "id": 440382, "postDate": "2018-12-17T13:28:29.993Z", "content": "

Thanks YaGana @sheriytm ; But I remember vividly that he wrote ‘Single Model, No KFold, with some tricks’. Perhaps my memory is wrong :D

", "rawMarkdown": "Thanks YaGana @sheriytm ; But I remember vividly that he wrote ‘Single Model, No KFold, with some tricks’. Perhaps my memory is wrong :D" }, { "id": 440636, "postDate": "2018-12-17T20:08:45.577Z", "content": "

@ratthachat, may be you remembered correctly and I missed that comment. I usually start with a single run model, get it to a good score before I go to KFold which is why I also searched the discussion section thoroughly for such results when I started. But I found most people were either doing ensembles or KFold you can see an example from my exchanges with Peter above.

", "rawMarkdown": "@ratthachat, may be you remembered correctly and I missed that comment. I usually start with a single run model, get it to a good score before I go to KFold which is why I also searched the discussion section thoroughly for such results when I started. But I found most people were either doing ensembles or KFold you can see an example from my exchanges with Peter above.", "votes": 1 }, { "id": 440915, "postDate": "2018-12-18T04:57:05.850Z", "content": "

Thanks YaGana.

", "rawMarkdown": "Thanks YaGana.", "votes": 1 }, { "id": 441663, "postDate": "2018-12-18T22:53:25.947Z", "content": "

@Neuron Engineer - I have finally managed to get my local CV at 0.692 F1 score and on the LB it is 0.691. The only thing is this is with K-fold but the validation is extremely close now and translating on the LB (so far). In the coming days I think I will check performance without K-fold and i'll report any findings here.

", "rawMarkdown": "@Neuron Engineer - I have finally managed to get my local CV at 0.692 F1 score and on the LB it is 0.691. The only thing is this is with K-fold but the validation is extremely close now and translating on the LB (so far). In the coming days I think I will check performance without K-fold and i'll report any findings here.", "votes": 1 }, { "id": 441700, "postDate": "2018-12-18T23:54:35.787Z", "content": "

Thanks @rdizzl3! Looking forward to hear that!

", "rawMarkdown": "Thanks @rdizzl3! Looking forward to hear that!" }, { "id": 441979, "postDate": "2018-12-19T10:10:11.730Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 434126, "postDate": "2018-12-05T23:31:15.243Z", "content": "

Hm... some people have now reported having large differences between their local and public LB score... especially where these lower scores result in very high public LB scores. I am going to have to side with Eddy in that I'm probably much more dubious about standings on the public LB at this point.

", "rawMarkdown": "Hm... some people have now reported having large differences between their local and public LB score... especially where these lower scores result in very high public LB scores. I am going to have to side with Eddy in that I'm probably much more dubious about standings on the public LB at this point.", "replies": [ { "id": 435702, "postDate": "2018-12-08T15:36:59.880Z", "content": "

It is very weird, I have some ensembles that score much higher on local CV and then suck badly on public LB, and then there are single models which score much worse on local CV, but get a boost on public LB. Not really sure which direction to go from here...

", "rawMarkdown": "It is very weird, I have some ensembles that score much higher on local CV and then suck badly on public LB, and then there are single models which score much worse on local CV, but get a boost on public LB. Not really sure which direction to go from here...", "votes": 1 }, { "id": 437275, "postDate": "2018-12-11T16:50:15.853Z", "content": "

Yeah. I think there's a lot of noise... It feels like this competition will be a toss-up.

", "rawMarkdown": "Yeah. I think there's a lot of noise... It feels like this competition will be a toss-up.", "votes": 1 } ] }, { "id": 429622, "postDate": "2018-11-29T04:38:37.280Z", "content": "

Update: .690 LB for a super simple RNN with data that I briefly preprocessed beforehand. Still looking into data augmentation, folds, post-processing, and hyper parameters to improve my solo model score even more. My validation score for this was .6782.

", "rawMarkdown": "**Update:** .690 LB for a super simple RNN with data that I briefly preprocessed beforehand. Still looking into data augmentation, folds, post-processing, and hyper parameters to improve my solo model score even more. My validation score for this was .6782." }, { "id": 427339, "postDate": "2018-11-25T08:44:31.513Z", "content": "

with a GRU I managed to get 0.677 ... I've only seen this once due to CuDNN randomness

\n\n

I'm really impressed some people are getting over 0.68!!!

", "rawMarkdown": "with a GRU I managed to get 0.677 ... I've only seen this once due to CuDNN randomness\n\nI'm really impressed some people are getting over 0.68!!!" }, { "id": 425512, "postDate": "2018-11-21T17:58:24.650Z", "content": "

0.669 no kfold

", "rawMarkdown": "0.669 no kfold" }, { "id": 425052, "postDate": "2018-11-21T04:04:21.527Z", "content": "

LB 0.679 in 1310 sec (GPU): https://www.kaggle.com/shujian/single-rnn-model-with-meta-features?scriptVersionId=7593124

", "rawMarkdown": "LB 0.679 in 1310 sec (GPU): https://www.kaggle.com/shujian/single-rnn-model-with-meta-features?scriptVersionId=7593124\n", "replies": [ { "id": 425718, "postDate": "2018-11-22T02:24:33.100Z", "content": "

Update:

\n\n

0.685 with 5 folds: https://www.kaggle.com/shujian/single-rnn-with-5-folds

", "rawMarkdown": "**Update:**\n\n0.685 with 5 folds: https://www.kaggle.com/shujian/single-rnn-with-5-folds" }, { "id": 426208, "postDate": "2018-11-22T21:42:58.660Z", "content": "

interesting! you’re local score is much lower than public...

\n\n

my f1 scores for a fold are as follows: \n1: 0.68603\n2: 0.68225\n3: 0.68375\n4: 0.68435\n5: 0.68521

\n\n

I trained up to 6 epochs.

\n\n

I tested the blend used by @kagSen and it scores a 0.687. I will try your blending method.

", "rawMarkdown": "interesting! you’re local score is much lower than public...\n\nmy f1 scores for a fold are as follows: \n1: 0.68603\n2: 0.68225\n3: 0.68375\n4: 0.68435\n5: 0.68521\n\nI trained up to 6 epochs.\n\nI tested the blend used by @kagSen and it scores a 0.687. I will try your blending method.\n", "votes": 2 }, { "id": 426209, "postDate": "2018-11-22T21:44:07.803Z", "content": "

Cool!

", "rawMarkdown": "Cool!" }, { "id": 426234, "postDate": "2018-11-23T00:02:30.547Z", "content": "

Update:

\n\n

LB 0.691 single model with 5 folds and 6 epochs each: https://www.kaggle.com/shujian/single-rnn-with-5-folds-v1-8

\n\n

LB 0.692 single model with 4 folds and 8 epochs each: https://www.kaggle.com/shujian/single-rnn-with-5-folds-v1-9

", "rawMarkdown": "**Update:**\n\nLB 0.691 single model with 5 folds and 6 epochs each: https://www.kaggle.com/shujian/single-rnn-with-5-folds-v1-8\n\n\nLB 0.692 single model with 4 folds and 8 epochs each: https://www.kaggle.com/shujian/single-rnn-with-5-folds-v1-9", "votes": 3 }, { "id": 426309, "postDate": "2018-11-23T03:28:36.837Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 445650, "postDate": "2018-12-26T21:34:47Z", "content": "

LB: 0.701, Keras, single model, 5-fold CV, 5 epochs.\nF1 score from CV: 0.677!

\n\n

Edit: LB: 0.702 same model, different optim. parameters ...

", "rawMarkdown": "LB: 0.701, Keras, single model, 5-fold CV, 5 epochs.\nF1 score from CV: 0.677!\n\nEdit: LB: 0.702 same model, different optim. parameters ...", "votes": 3, "isDeleted": true, "replies": [ { "id": 446091, "postDate": "2018-12-27T13:33:07.310Z", "content": "

Nice! Are you using an attention mechanism or capsule like I have seen others try?

", "rawMarkdown": "Nice! Are you using an attention mechanism or capsule like I have seen others try?" }, { "id": 446320, "postDate": "2018-12-27T21:45:48.587Z", "content": "

No, only what comes with Keras: mostly fully-connected, convolutional, and LSTM layers.\nI try to keep it simple to allow reasonable Nb CV folds

", "rawMarkdown": "No, only what comes with Keras: mostly fully-connected, convolutional, and LSTM layers.\nI try to keep it simple to allow reasonable Nb CV folds ", "votes": 2, "isDeleted": true }, { "id": 448440, "postDate": "2019-01-01T07:33:51.297Z", "content": "

What is the execution time @Annabelle ?

", "rawMarkdown": "What is the execution time @Annabelle ? " }, { "id": 448477, "postDate": "2019-01-01T09:37:48Z", "content": "

Too much to add another model: approx 6500 s total kernel execution time

", "rawMarkdown": "Too much to add another model: approx 6500 s total kernel execution time ", "isDeleted": true } ] }, { "id": 429623, "postDate": "2018-11-29T04:44:05.307Z", "rawMarkdown": "", "isDeleted": true }, { "id": 426610, "postDate": "2018-11-23T14:37:30.877Z", "content": "

F1 score = 0.622 , word2vec embedding, single XGB model

", "rawMarkdown": "F1 score = 0.622 , word2vec embedding, single XGB model", "isDeleted": true } ], "comments": [ { "id": 437792, "author_name": "Shahebaz Mohammad", "author_url": "", "post_date": "2018-12-12T13:52:43.977000", "content": "

Migrated to Pytorch. Single model - 0.700 on LB 4 Folds 5 epochs

", "votes": 11, "replies": [ { "id": 437814, "author_name": "Psi", "author_url": "", "post_date": "2018-12-12T14:45:16.840000", "content": "

So your Pytorch results are better than your Keras ones?

", "votes": 0, "replies": [] }, { "id": 437818, "author_name": "Shahebaz Mohammad", "author_url": "", "post_date": "2018-12-12T15:00:55.950000", "content": "

Yes, and the training time is reduced to. And, honestly I am loving the documentation. Its lit and clean

", "votes": 3, "replies": [] }, { "id": 437822, "author_name": "Serigne ", "author_url": "", "post_date": "2018-12-12T15:10:21.080000", "content": "

Pytorch is the way to go..

\n\n

Keras is sometimes limited and Tensorflow is a nightmare.

\n\n

I decided to switch to Pytorch since TGS competition, but I'm still too lazy for that.

", "votes": 2, "replies": [] }, { "id": 437866, "author_name": "Psi", "author_url": "", "post_date": "2018-12-12T16:56:32.443000", "content": "

Any tutorial recommendation to get into it? Also not sure how much changed with the new 1.0 Pytorch release, which I guess won't be in the docker image anyways though.

", "votes": 0, "replies": [] }, { "id": 437934, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-12-12T19:32:28.980000", "content": "

@shahebaz, does your pytorch model give consistent result? To me that is the most important thing I am trying to figure out here. Models with inconsistent results is pretty much useless imho.

", "votes": 0, "replies": [] }, { "id": 438282, "author_name": "Alkimay", "author_url": "", "post_date": "2018-12-13T12:05:14.330000", "content": "

hey Shahebaz/serigne can you mark down your pytorch building blocks?\ni am using double lstm + attention +2 dense layers, relu and dropout(0.1),batch size is 512, getting 0.68 single model 3 fold(unable to run more then that in 2 hours). i am not reaching 0.69+.\npreprossing : special char spacing and lower casing all sentences.\nwhat am i doing wrong?

", "votes": 0, "replies": [] }, { "id": 438427, "author_name": "Serigne ", "author_url": "", "post_date": "2018-12-13T16:37:33.090000", "content": "

I'm not using Pytorch, at least not for now. But it's my aim to switch (at least for next images competitions I enter )

\n\n

And my building blocks is different from those typical LSTM/GRU+ Attention/Pooled layers + Dense layers I see on kernels. But you can get very good score with these architectures, try to tune more your network and check for overfitting.

", "votes": 2, "replies": [] }, { "id": 438584, "author_name": "RDizzl3", "author_url": "", "post_date": "2018-12-13T23:04:30.110000", "content": "

@Shahebaz - when you migrated your model from keras to pytorch did you get similar validation results? I just migrated my model to pytorch but the results are far worse than my keras model. As far as I can tell just about everything is identical.

", "votes": 0, "replies": [] }, { "id": 438720, "author_name": "Shahebaz Mohammad", "author_url": "", "post_date": "2018-12-14T04:16:03.387000", "content": "

@rdizzl3 My architecture for 0.700 in Pytorch is differnet that of keras. But, I observed including cyclic CLR made the results worse in Pytorch. I am yet to implement the same architecture that gave me best score with keras. (I hate my exams)

", "votes": 0, "replies": [] }, { "id": 438721, "author_name": "RDizzl3", "author_url": "", "post_date": "2018-12-14T04:27:35.757000", "content": "

@Shahebaz - thanks for the info! This is my first go with pytorch so I wasn't quite sure what to expect. I will tinker around some more with it :)

", "votes": 0, "replies": [] }, { "id": 459514, "author_name": "JM100", "author_url": "", "post_date": "2019-01-21T22:26:58.513000", "content": "

Udemy has a good an free course on Pytorch: https://www.udacity.com/course/deep-learning-pytorch--ud188

", "votes": 1, "replies": [] } ] }, { "id": 426445, "author_name": "Andrzej Kuro", "author_url": "", "post_date": "2018-11-23T09:11:55.917000", "content": "

I have 0.505 with naïve Bayes :-)

", "votes": 10, "replies": [] }, { "id": 437086, "author_name": "Serigne ", "author_url": "", "post_date": "2018-12-11T11:06:14.427000", "content": "

Edit : LB 0.701, I still use single model , 4 folds and some processing

", "votes": 8, "replies": [ { "id": 437095, "author_name": "Mihajlo T.", "author_url": "", "post_date": "2018-12-11T11:21:12.927000", "content": "

What is your execution time?\nI got 0.701 with minor changes to one of the public kernels but it exceeds execution time limit (8220.5s). Not sure if I did something wrong but didn't get any error when submitting.

", "votes": 0, "replies": [] }, { "id": 437098, "author_name": "Link", "author_url": "", "post_date": "2018-12-11T11:23:31.557000", "content": "", "votes": -1, "replies": [] }, { "id": 437267, "author_name": "Serigne ", "author_url": "", "post_date": "2018-12-11T16:39:27.330000", "content": "

It run 7177s (a bit less than 2hrs :) )\nAnyway , optimisation will be necessary , 2nd stage will have bigger test data

", "votes": 1, "replies": [] }, { "id": 437273, "author_name": "Thomas Yokota", "author_url": "", "post_date": "2018-12-11T16:47:26.213000", "content": "

Nice, Serigne. Is your local f1 score pretty close to what you're seeing on the public LB?

", "votes": 1, "replies": [] }, { "id": 437288, "author_name": "Shujian Liu", "author_url": "", "post_date": "2018-12-11T17:22:01.570000", "content": "

Congrats on breaking 0.7!

", "votes": 1, "replies": [] }, { "id": 437297, "author_name": "Shahebaz Mohammad", "author_url": "", "post_date": "2018-12-11T17:29:04.537000", "content": "
\n

Following the final submission deadline for the competition, your kernel code will be re-run on a privately-held test set that is not provided to you. It is your model's score against this private test set that will determine your ranking on the private leaderboard and final standing in the competition.

\n
\n\n

@serigne I saw that there was bigger test in last kernel competition. But, I came across no mention of bigger test for this competition. Can you point at the source which says bigger or how big? Thanks!

", "votes": 0, "replies": [] }, { "id": 437378, "author_name": "RDizzl3", "author_url": "", "post_date": "2018-12-11T19:41:34.520000", "content": "

@Shahebaz -

\n\n
\n

What will be available in the 2nd stage of the competition? In the\n second stage of the competition, we will re-run your selected Kernels.\n The following files will be swapped with new data:

\n \n

test.csv - This will be swapped with the complete public and private\n test dataset. This file will have ~56k rows in stage 1 and ~376k rows\n in stage 2. The public leaderboard data remains the same for both\n versions. The file name will be the same (both test.csv) to ensure\n that your code will run. sample_submission.csv - similar to test.csv,\n this will be changed from ~56k in stage 1 to ~376k rows in stage 2 .\n The file name will remain the same.

\n
\n\n

From the data page so 2nd stage will be roughly ~7 times larger.

", "votes": 3, "replies": [] }, { "id": 437535, "author_name": "Shahebaz Mohammad", "author_url": "", "post_date": "2018-12-12T05:12:28.550000", "content": "

Oh, I missed it. Thanks a bunch.

", "votes": 0, "replies": [] }, { "id": 437544, "author_name": "Serigne ", "author_url": "", "post_date": "2018-12-12T05:35:40.773000", "content": "

@Learnmower I didn't compute f1 in callback in my latest kernels, but my CV loss 0.0955 ( it scores now LB 0.702)

\n\n

Thank @Shujian, No doubt you'll do it very soon :)

", "votes": 4, "replies": [] }, { "id": 437612, "author_name": "Thomas Yokota", "author_url": "", "post_date": "2018-12-12T07:34:28.620000", "content": "

@Serigne Thanks! Your loss looks very good.

", "votes": 1, "replies": [] }, { "id": 440816, "author_name": "RDizzl3", "author_url": "", "post_date": "2018-12-18T01:52:29.090000", "content": "

I got my 4-fold cv down to 0.0955 loss and 0.693 f1-score locally but it still doesn't score well on the LB. It only scores 0.690. I am going to run it multiple times to see how the score changes over each run.

", "votes": 1, "replies": [] } ] }, { "id": 428628, "author_name": "Joe Eddy", "author_url": "", "post_date": "2018-11-27T16:05:11.790000", "content": "

Adding to the CV vs. LB train:

\n\n

5-fold avg of 1 model, LB .687

\n\n

5 fold CV (with best threshold):

\n\n

Fold 0: .697\nFold 1: .687\nFold 2: .684\nFold 3: .688\nFold 4: .688

\n\n

I guess I should be happy that the LB score is a plausible CV fold score? I'd say I'm relatively confident that the leaderboard isn't super reliable though.

\n\n

Edit: also, same model trained on all data in one run scores .685

", "votes": 8, "replies": [ { "id": 428712, "author_name": "kagSen", "author_url": "", "post_date": "2018-11-27T18:43:31.717000", "content": "

Do these 5 folds have a separate F1 threshold each, or do you lump all the validation data into a single set and estimate a single F1 threshold which you then apply to the 5 folds to get a fold-wise F1?

\n\n

If you are indeed using a separate threshold for each fold , doesn't the variance in the threshold scare you a bit (In some of my runs, the thresholds range from 0.19 to 0.39 across folds).

", "votes": 2, "replies": [] }, { "id": 428740, "author_name": "Joe Eddy", "author_url": "", "post_date": "2018-11-27T19:49:30.557000", "content": "

This was looking at separate thresholds for each fold, but I also don't see as much variance in thresholds across folds as you've seen - mine range between around .31 and .37. Checking for a global threshold after across the training data (using out of sample preds) also seems to give me something roughly near the average of thresholds (.35). I've also been ensembling using your voting strategy with different thresholds instead of just averaging together the probabilities, though they've seemed to work about the same for me.

\n\n

But yes, the variance in thresholds does scare me, and I'm really wondering what the proper way to ensemble is. In an ideal world I would want to just stack with something like logreg to get a single set of reasonably calibrated probabilistic outputs, but stacking is probably too slow and so far it's unclear to me that blending based on a single out-of-sample set works better than averaging different runs of the same model (like in the k-fold strategy). It's tricky!

", "votes": 3, "replies": [] } ] }, { "id": 436288, "author_name": "Shujian Liu", "author_url": "", "post_date": "2018-12-10T04:12:02.293000", "content": "

0.699 single model with 5 folds. Slightly modified my public kernels according to some posts above.

", "votes": 6, "replies": [] }, { "id": 424011, "author_name": "Serigne ", "author_url": "", "post_date": "2018-11-19T12:32:02.523000", "content": "

LB : 0.694 ( single model with k-fold)

", "votes": 6, "replies": [ { "id": 424016, "author_name": "Peter", "author_url": "", "post_date": "2018-11-19T12:47:23.780000", "content": "

Same for me: 0.693 single model k-fold. It varies though, with the exact same model I got 0.685 too (I have not investigated yet)

", "votes": 3, "replies": [] }, { "id": 424041, "author_name": "Serigne ", "author_url": "", "post_date": "2018-11-19T13:20:29.483000", "content": "

Wow , without any change ?

\n\n

My score varies between 0.691 and 0.694 when I do some changes either on the network or on the data ( I didn't try to submit the same exact model twice to check such variation )

", "votes": 2, "replies": [] }, { "id": 424076, "author_name": "Psi", "author_url": "", "post_date": "2018-11-19T14:30:38.543000", "content": "

Congratz, pretty good for a single model. How is the runtime of that model approximately?

", "votes": 0, "replies": [] }, { "id": 424081, "author_name": "Peter", "author_url": "", "post_date": "2018-11-19T14:38:24.253000", "content": "

Yes, without any change. My best was an earlier version, I modified it, ran a few other trials, but there was no improvement. So I reverted back to the best version and I accidentally ran it again. I did not notice until I compared the two version with the diff-tool. I only changed some comments (the results: 0.693 / 0.685). I will look at this problem closer, it is weird.

", "votes": 0, "replies": [] }, { "id": 424082, "author_name": "Peter", "author_url": "", "post_date": "2018-11-19T14:39:52.207000", "content": "

@philippsinger, runtime for me was ~6000-6500s

", "votes": 0, "replies": [] }, { "id": 424202, "author_name": "Serigne ", "author_url": "", "post_date": "2018-11-19T18:18:50.357000", "content": "

@pestipeti Are you using Keras or pure Tensorflow or Pytorch ?

\n\n

I think Keras may have some very random behavior compared to the two others . I tried to fix all seeds , even for kernel_initializers.

\n\n

@philippsinger, for my best it's about 5800s

", "votes": 1, "replies": [] }, { "id": 424205, "author_name": "Peter", "author_url": "", "post_date": "2018-11-19T18:23:53.920000", "content": "

@serigne, I use keras, but I will switch to pytorch. I remember earlier posts (TGS competition) about the same weird behavior, they blamed keras too..

", "votes": 0, "replies": [] }, { "id": 424366, "author_name": "Mihajlo T.", "author_url": "", "post_date": "2018-11-20T01:35:06.577000", "content": "

Are you doing some kind of text preprocessing with single models? I understand if you don't want to answer it :)

", "votes": 0, "replies": [] }, { "id": 424399, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-11-20T03:56:51.867000", "content": "

My best single model scores 0.676 on the LB, and it is a single run not k-Folds.

\n\n

@Serigne, @Peter, what is the score for your single model for one run? I mean before you did k-Folds?

\n\n

I have also noticed inconsistencies in scores with another model of mine. That one is an ensemble model so I thought may be I have a bug in my code but have not had time to debug it yet. I am using keras.

", "votes": 1, "replies": [] }, { "id": 424424, "author_name": "kagSen", "author_url": "", "post_date": "2018-11-20T05:23:40.750000", "content": "

@serigne, \nI have been pretty frustrated with the non-reproducibility aspect of Keras based kernels as well. This is not very well documented either.

\n\n

I currently use \nnp.random.seed(1)\ntf.set_random_seed(1)

\n\n

Some Kaggle and Stackoverflow thread also suggest that using CuDNNLSTM or CuDNNGRU can lead to non-deterministic behavior. I have not explore this further.

\n\n

@serigne, Would you mind sharing the random intializations that you use?

", "votes": 0, "replies": [] }, { "id": 424446, "author_name": "Peter", "author_url": "", "post_date": "2018-11-20T06:14:15.157000", "content": "

@sheriytm, My single best was 0.679, but because of this keras/cudnn issue it could be 0.688 (or 0.670). If I can not reproduce then it is worthless.

", "votes": 1, "replies": [] }, { "id": 424500, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-11-20T08:02:47.677000", "content": "

I feel the same way. Running experiments right now before I rule those models out.

\n\n

One thing I know is that I have used both CuDNNLSTM and CuDNNGRU in the toxic comment competition but with k-fold and did not observe any inconsistencies in my results. I am hoping doing k-fold will eliminate that. My current single models are too heavy for k-folds at the moment, so I need to work on streamlining that first.

", "votes": 1, "replies": [] }, { "id": 424569, "author_name": "Serigne ", "author_url": "", "post_date": "2018-11-20T10:32:39.297000", "content": "

@mihajlot ....No cleaning and no special processing for now

\n\n

I think the architecture of the network is important here ...Some others architectures I tried, failed to improve after 3 or 4 epoches.

", "votes": 4, "replies": [] }, { "id": 424571, "author_name": "Serigne ", "author_url": "", "post_date": "2018-11-20T10:36:05.723000", "content": "

@sheriytm I didn't try it. Actually I built k-folds since my first kernel , before starting experimenting models :)

\n\n

But I will later to see.

", "votes": 1, "replies": [] }, { "id": 424573, "author_name": "Serigne ", "author_url": "", "post_date": "2018-11-20T10:43:32.240000", "content": "

@kagsen \nI fixed those and the kernel_initializers features like this :

\n\n
from keras.initializers import he_normal, he_uniform,  glorot_normal,  glorot_uniform\n\nBidirectional(CuDNNLSTM(self.lstm_num_feat, \n                        kernel_initializer=glorot_uniform(seed = 123),\n                         return_sequences=True)) \n\nDense(self.dense_num_feat, \n          kernel_initializer=he_uniform(seed=123), \n          activation='relu')\n
", "votes": 10, "replies": [] }, { "id": 424966, "author_name": "RDizzl3", "author_url": "", "post_date": "2018-11-21T00:09:28.743000", "content": "

@Serigne, @Peter - how many folds are you guys using?

", "votes": 0, "replies": [] }, { "id": 424975, "author_name": "Peter", "author_url": "", "post_date": "2018-11-21T00:31:54.127000", "content": "

@rdizzl3, I used 5 folds for my best result (6 epochs/fold)

", "votes": 2, "replies": [] }, { "id": 425034, "author_name": "Thomas Yokota", "author_url": "", "post_date": "2018-11-21T03:22:04.963000", "content": "

Hey Serigne. How many folds are you using? I saw your post in another thread and with the timings, I had assumed 5. I'm also seeing convergence around 3-4 epochs. Though I am wondering if I should be basing my decision on loss or accuracy. Accuracy starts to drop about 3-4, but loss still improves.

", "votes": 0, "replies": [] }, { "id": 425048, "author_name": "Shujian Liu", "author_url": "", "post_date": "2018-11-21T03:56:58.250000", "content": "

@Serigne, I am really curious about what is k in your k-fold. Seems if k is small, the training set is too small so the results are not good. If k is large, the model has to be really simple so it can finish in 2 hrs.

", "votes": 1, "replies": [] }, { "id": 425158, "author_name": "", "author_url": "", "post_date": "2018-11-21T08:00:16.887000", "content": "", "votes": 0, "replies": [] }, { "id": 425320, "author_name": "Serigne ", "author_url": "", "post_date": "2018-11-21T12:47:44.883000", "content": "

@rdizzl3 and @learnmower I splitted the data in 4 folds but use only 3 folds for now. So there are three runs and still a part of the data not used for validation. However I plan to reintegrate the last fold as soon as I optimize better my code .

\n\n

@shujian I suppose k small or large depends also on the size of the test set ( and we don't have many choices here , as you say, due to 2hrs limitation ) . Anyway I did not split the data randomly :)

", "votes": 3, "replies": [] }, { "id": 425897, "author_name": "RDizzl3", "author_url": "", "post_date": "2018-11-22T09:17:41.313000", "content": "

@Peter, @Serigne - if you don't mind saying, do you know your average K-Fold scores and overall score on OOF predictions?

", "votes": 0, "replies": [] }, { "id": 425916, "author_name": "Peter", "author_url": "", "post_date": "2018-11-22T09:55:29.153000", "content": "

@rdizzl3, For some reason kaggle cut my logs after ~300 lines, so I don't have all of it, but here are my local f1 scores for my best model:

\n\n\n\n

My public score for this (average of 5 folds) was 0.693, but read this too.

", "votes": 2, "replies": [] }, { "id": 425952, "author_name": "Serigne ", "author_url": "", "post_date": "2018-11-22T10:56:10.937000", "content": "

@RDizzl3 my f1 val_scores :

\n\n

Fold0 : 0.6668\nFold1 : 0.6761\nFold2 : (skipped for now and will be reintegrated later)\nFold3 : 0.6846

", "votes": 2, "replies": [] }, { "id": 425956, "author_name": "Serigne ", "author_url": "", "post_date": "2018-11-22T11:00:44.553000", "content": "

@pestipeti you can append them on a list and print them at the next cell

\n\n

I use to run my kernels in interactive session too, just after committing, so I can check out all the output later.

", "votes": 1, "replies": [] }, { "id": 426206, "author_name": "RDizzl3", "author_url": "", "post_date": "2018-11-22T21:40:53.123000", "content": "

Thank you guys for the replies! Check out my 5-folds:

\n\n

fold-0: 0.6779\nfold-1: 0.6821\nfold-2: 0.6860\nfold-3: 0.6883\nfold-4: 0.6817

\n\n

only scores 0.683 on the LB - average of 5-folds.

", "votes": 1, "replies": [] }, { "id": 426284, "author_name": "Shujian Liu", "author_url": "", "post_date": "2018-11-23T02:23:24.630000", "content": "

@serigne, hi, I am getting closer to your model :)

", "votes": 2, "replies": [] }, { "id": 426443, "author_name": "Rupesh Sreeraman", "author_url": "", "post_date": "2018-11-23T09:10:16.157000", "content": "

@Peter how many epochs?

", "votes": 0, "replies": [] }, { "id": 426737, "author_name": "Serigne ", "author_url": "", "post_date": "2018-11-23T18:26:02.617000", "content": "

@Shujian I have no doubt you will do it very soon :)

", "votes": 1, "replies": [] }, { "id": 426739, "author_name": "Shujian Liu", "author_url": "", "post_date": "2018-11-23T18:29:57.150000", "content": "

Thanks ;)

", "votes": 0, "replies": [] }, { "id": 427108, "author_name": "Peter", "author_url": "", "post_date": "2018-11-24T15:53:27.407000", "content": "

@rupeshs, 6 epochs/fold

", "votes": 0, "replies": [] }, { "id": 427340, "author_name": "Hamish", "author_url": "", "post_date": "2018-11-25T08:50:45.697000", "content": "

A few people have mentioned the randomness that happens in keras CuDNNs - I've was curious enough to do some digging and found this comment in the tensorflow gh

\n\n
\n

On GPU, small amount of non-deterministic results is expected. TensorFlow uses the Eigen library, which uses Cuda atomics to implement reduction operations, such as tf.reduce_sum etc. Those operations are non-determnistical. Each operation can introduce a small difference. If your model is not stable, it could accumulate into large errors, after many steps.

\n
\n\n

Wonder if this is just \"a GPU thing\" in general?

", "votes": 2, "replies": [] }, { "id": 429707, "author_name": "Shahebaz Mohammad", "author_url": "", "post_date": "2018-11-29T08:14:41.880000", "content": "

My single model is at LB 0.696, CV 0.683 with 5 epochs and 4 folds

", "votes": 0, "replies": [] } ] }, { "id": 452606, "author_name": "Qing Liu", "author_url": "", "post_date": "2019-01-09T00:06:44.217000", "content": "

single model with 5 Stratified KFold, CV log_loss 0.09908, f1_score 0.67972, LB 0.684

", "votes": 4, "replies": [] }, { "id": 426817, "author_name": "Tim Johnson", "author_url": "", "post_date": "2018-11-23T23:29:09.667000", "content": "

Currently have a LB of 0.688 with one model, no k fold and only GloVe embeddings

", "votes": 3, "replies": [] }, { "id": 436389, "author_name": "Serigne ", "author_url": "", "post_date": "2018-12-10T08:17:50.007000", "content": "

Adding on my kernel the cleaning from this kernel improved my score from 0.696 to 0.699

\n\n

I didn't clean anything before.

", "votes": 1, "replies": [ { "id": 436396, "author_name": "Shahebaz Mohammad", "author_url": "", "post_date": "2018-12-10T08:23:51.033000", "content": "

I noted the same. Btw, are you sure improvement is cause of cleaning and not randomness. I use different processing than that of kernel and my average score of 10-15 runtimes is slower while clearing punctuations. Thanks for reporting though!

", "votes": 0, "replies": [] }, { "id": 436399, "author_name": "Serigne ", "author_url": "", "post_date": "2018-12-10T08:29:30.980000", "content": "

I don't know...likely randomness, given I didn't notice huge improvement on CV :)

", "votes": 0, "replies": [] }, { "id": 436401, "author_name": "Shahebaz Mohammad", "author_url": "", "post_date": "2018-12-10T08:33:54.237000", "content": "

I am -1 for any new experiments until I migrate to Pytorch. I haven't used it before. Never say never :)

", "votes": 0, "replies": [] }, { "id": 436425, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-12-10T09:23:31.390000", "content": "

@Serigne, same here. Adding the cleaning plus a couple of tweaks, my score improved from 0.693 to 0.698 but I am still concerned about the randomness/ inconsistencies.

", "votes": 1, "replies": [] }, { "id": 437153, "author_name": "Psi", "author_url": "", "post_date": "2018-12-11T13:10:12.433000", "content": "

I wonder how that cleaning makes sense for most of the characters in it as keras tokenizer is removing them anyways by default.

", "votes": 0, "replies": [] }, { "id": 437561, "author_name": "Ankit Sati", "author_url": "", "post_date": "2018-12-12T06:01:15.457000", "content": "

I also got scores in the ranges of 0.692 - 0.697 with cleaning. Even the same kernel when run again is giving different results. It's frustrating in the sense that we don't really know what improved our results and what not. @Serigne, can you run your highest scoring kernel again and check if the score remains same? If you don't mind wasting a submission.

", "votes": 1, "replies": [] } ] }, { "id": 433084, "author_name": "Radek Svoboda", "author_url": "", "post_date": "2018-12-04T17:07:53.580000", "content": "

I've got 0.691 on 4-folds with RMSprop optimizer and mean-combined embedding. I've tried perhaps 10 different parameter configurations but I can't get past this score on LB.

", "votes": 1, "replies": [ { "id": 433088, "author_name": "Shujian Liu", "author_url": "", "post_date": "2018-12-04T17:18:00.680000", "content": "

Hi Radek, did RMSprop optimizer boost LB? Thanks.

", "votes": 0, "replies": [] }, { "id": 433113, "author_name": "Radek Svoboda", "author_url": "", "post_date": "2018-12-04T18:03:03.787000", "content": "

It's hard to tell due to the LB score variance, but I think it helped in this case. I've used your network from https://www.kaggle.com/shujian/single-rnn-with-4-folds-clr but went with one less epoch and second LSTM layer instead of GRU. I've tried tweaking learning rate, number of neurons in layers, dropout and so on, but nothing seems to help getting better LB score although it gets better locally on CV.

", "votes": 1, "replies": [] }, { "id": 433119, "author_name": "Radek Svoboda", "author_url": "", "post_date": "2018-12-04T18:10:57.147000", "content": "

You can take a look at it here: https://www.kaggle.com/rasvob/let-s-try-clr-v3\nLB with 0.691 is in Version 2 and 7, have fun :)

", "votes": 1, "replies": [] }, { "id": 433120, "author_name": "Shujian Liu", "author_url": "", "post_date": "2018-12-04T18:10:59.323000", "content": "

Thanks for your feedback!

", "votes": 0, "replies": [] } ] }, { "id": 433803, "author_name": "Ankit Sati", "author_url": "", "post_date": "2018-12-05T13:49:43.647000", "content": "

0.697 - 4 Kfold , cyclic LR, mean combined embedding. But can't go above this even if I were able to improve CV. Which is very frustrating as I can't get any idea about what is what.

", "votes": 2, "replies": [] }, { "id": 432970, "author_name": "Hikkiiiiiiiii", "author_url": "", "post_date": "2018-12-04T14:29:03.687000", "content": "

79 mins - 4fold - CV 0.685 - LB 0.691

", "votes": 2, "replies": [ { "id": 433672, "author_name": "jetou Xu", "author_url": "", "post_date": "2018-12-05T10:26:03.960000", "content": "

thank you

", "votes": 0, "replies": [] }, { "id": 434998, "author_name": "AlexYung", "author_url": "", "post_date": "2018-12-07T09:33:08.087000", "content": "

small old brother , please raise me up

", "votes": -1, "replies": [] }, { "id": 441123, "author_name": "Hikkiiiiiiiii", "author_url": "", "post_date": "2018-12-18T09:33:02.767000", "content": "

83 min - 5fold - CV 0.683 - LB 0.697 - pytorch

", "votes": 4, "replies": [] }, { "id": 441702, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2018-12-18T23:57:07.230000", "content": "

Did you verify how many minutes can pytorch save you ?

", "votes": 0, "replies": [] }, { "id": 441744, "author_name": "Hikkiiiiiiiii", "author_url": "", "post_date": "2018-12-19T02:15:21.667000", "content": "

@Neuron

\n\n

No, in the process of migrating to pytorch, data loading and model structure will have some necessary adjustments.

", "votes": 0, "replies": [] }, { "id": 443849, "author_name": "Hikkiiiiiiiii", "author_url": "", "post_date": "2018-12-22T15:14:21.850000", "content": "

100 min - 5fold - CV 0.685 - LB 0.698 - pytorch

", "votes": 2, "replies": [] }, { "id": 447802, "author_name": "Hikkiiiiiiiii", "author_url": "", "post_date": "2018-12-30T15:34:11.177000", "content": "

110 min - 5fold - CV 0.683 - LB 0.699 - pytorch

", "votes": 0, "replies": [] }, { "id": 448327, "author_name": "Alkimay", "author_url": "", "post_date": "2018-12-31T19:37:01.823000", "content": "

regular structure? (2 bi lstm +attention?)

", "votes": 0, "replies": [] } ] }, { "id": 423760, "author_name": "Shreyans Dhankhar", "author_url": "", "post_date": "2018-11-19T02:37:36.537000", "content": "

My Best Single Model Public Leaderboard Score: 0.675

", "votes": 2, "replies": [] }, { "id": 423869, "author_name": "Si Feng", "author_url": "", "post_date": "2018-11-19T06:57:52.167000", "content": "

0.685

", "votes": 0, "replies": [] }, { "id": 446459, "author_name": "GuanQun Wu", "author_url": "", "post_date": "2018-12-28T05:22:53.320000", "content": "

change parameters improve LB score to 0.702, it may be overfit on testset,so,I'm worried about the second stage

", "votes": 1, "replies": [] }, { "id": 426442, "author_name": "Rupesh Sreeraman", "author_url": "", "post_date": "2018-11-23T09:07:00.243000", "content": "

LB: 66.3 (Single model ,no kfold)

", "votes": -1, "replies": [] }, { "id": 460246, "author_name": "tamanyan", "author_url": "", "post_date": "2019-01-23T09:30:24.160000", "content": "

Change parameters of public kernel\nLB: 0.701 CV 0.6823

", "votes": 0, "replies": [] }, { "id": 439764, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2018-12-16T09:58:22.037000", "content": "

Hi, is anybody able to go beyond both “LB 0.69” and “Local CV 0.69” , using only ‘single classifier’ ? (single model with no KFold, no ensemble) . In my case, even though my ensemble get LB0.697, my single only get LB0.675 (with 0.693 Local CV)

", "votes": 0, "replies": [ { "id": 440268, "author_name": "Link", "author_url": "", "post_date": "2018-12-17T10:01:04.153000", "content": "

I think your model might be overfitting...

", "votes": 1, "replies": [] }, { "id": 440281, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2018-12-17T10:27:25.433000", "content": "

You are right. However, I just would like to know if anybody can do >= 0.69 on both LB and Local CV. Can you ?

", "votes": 0, "replies": [] }, { "id": 440285, "author_name": "Psi", "author_url": "", "post_date": "2018-12-17T10:46:43.780000", "content": "

What do you mean with Local CV no KFold?

", "votes": 1, "replies": [] }, { "id": 440297, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2018-12-17T11:05:03.713000", "content": "

Hi Philipp,

\n\n

Since there might be confusion about prediction using 'a single model with KFold' and 'a single classifier'.

\n\n

When we do KFold (although with a single model), in the end we have K classifiers (one for each validation fold), and usually we predict the final result using this ensemble of K classifiers.

\n\n

What I want to know is the performance of a single classifier (no average with other classifiers). Previously, I see one of the kaggler (top 10 in the current LB) claimed that he could get his very high performance by not using ensemble at all (but with some trick), but now his comment is deleted.

", "votes": 1, "replies": [] }, { "id": 440306, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-12-17T11:25:57.840000", "content": "

@ratthachat, I think I may have also read the comment you mentioned in the forum. I think by single model, he meant KFold but not a kernel like SKR's that had 3-different types of models averaging the result.

", "votes": 1, "replies": [] }, { "id": 440320, "author_name": "Psi", "author_url": "", "post_date": "2018-12-17T11:57:25.627000", "content": "

That's also how I understood it.

", "votes": 0, "replies": [] }, { "id": 440382, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2018-12-17T13:28:29.993000", "content": "

Thanks YaGana @sheriytm ; But I remember vividly that he wrote ‘Single Model, No KFold, with some tricks’. Perhaps my memory is wrong :D

", "votes": 0, "replies": [] }, { "id": 440636, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-12-17T20:08:45.577000", "content": "

@ratthachat, may be you remembered correctly and I missed that comment. I usually start with a single run model, get it to a good score before I go to KFold which is why I also searched the discussion section thoroughly for such results when I started. But I found most people were either doing ensembles or KFold you can see an example from my exchanges with Peter above.

", "votes": 1, "replies": [] }, { "id": 440915, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2018-12-18T04:57:05.850000", "content": "

Thanks YaGana.

", "votes": 1, "replies": [] }, { "id": 441663, "author_name": "RDizzl3", "author_url": "", "post_date": "2018-12-18T22:53:25.947000", "content": "

@Neuron Engineer - I have finally managed to get my local CV at 0.692 F1 score and on the LB it is 0.691. The only thing is this is with K-fold but the validation is extremely close now and translating on the LB (so far). In the coming days I think I will check performance without K-fold and i'll report any findings here.

", "votes": 1, "replies": [] }, { "id": 441700, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2018-12-18T23:54:35.787000", "content": "

Thanks @rdizzl3! Looking forward to hear that!

", "votes": 0, "replies": [] }, { "id": 441979, "author_name": "", "author_url": "", "post_date": "2018-12-19T10:10:11.730000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 434126, "author_name": "Thomas Yokota", "author_url": "", "post_date": "2018-12-05T23:31:15.243000", "content": "

Hm... some people have now reported having large differences between their local and public LB score... especially where these lower scores result in very high public LB scores. I am going to have to side with Eddy in that I'm probably much more dubious about standings on the public LB at this point.

", "votes": 0, "replies": [ { "id": 435702, "author_name": "Psi", "author_url": "", "post_date": "2018-12-08T15:36:59.880000", "content": "

It is very weird, I have some ensembles that score much higher on local CV and then suck badly on public LB, and then there are single models which score much worse on local CV, but get a boost on public LB. Not really sure which direction to go from here...

", "votes": 1, "replies": [] }, { "id": 437275, "author_name": "Thomas Yokota", "author_url": "", "post_date": "2018-12-11T16:50:15.853000", "content": "

Yeah. I think there's a lot of noise... It feels like this competition will be a toss-up.

", "votes": 1, "replies": [] } ] }, { "id": 429622, "author_name": "Matthew Anderson", "author_url": "", "post_date": "2018-11-29T04:38:37.280000", "content": "

Update: .690 LB for a super simple RNN with data that I briefly preprocessed beforehand. Still looking into data augmentation, folds, post-processing, and hyper parameters to improve my solo model score even more. My validation score for this was .6782.

", "votes": 0, "replies": [] }, { "id": 427339, "author_name": "Hamish", "author_url": "", "post_date": "2018-11-25T08:44:31.513000", "content": "

with a GRU I managed to get 0.677 ... I've only seen this once due to CuDNN randomness

\n\n

I'm really impressed some people are getting over 0.68!!!

", "votes": 0, "replies": [] }, { "id": 425512, "author_name": "Ankit Sati", "author_url": "", "post_date": "2018-11-21T17:58:24.650000", "content": "

0.669 no kfold

", "votes": 0, "replies": [] }, { "id": 425052, "author_name": "Shujian Liu", "author_url": "", "post_date": "2018-11-21T04:04:21.527000", "content": "

LB 0.679 in 1310 sec (GPU): https://www.kaggle.com/shujian/single-rnn-model-with-meta-features?scriptVersionId=7593124

", "votes": 0, "replies": [ { "id": 425718, "author_name": "Shujian Liu", "author_url": "", "post_date": "2018-11-22T02:24:33.100000", "content": "

Update:

\n\n

0.685 with 5 folds: https://www.kaggle.com/shujian/single-rnn-with-5-folds

", "votes": 0, "replies": [] }, { "id": 426208, "author_name": "Thomas Yokota", "author_url": "", "post_date": "2018-11-22T21:42:58.660000", "content": "

interesting! you’re local score is much lower than public...

\n\n

my f1 scores for a fold are as follows: \n1: 0.68603\n2: 0.68225\n3: 0.68375\n4: 0.68435\n5: 0.68521

\n\n

I trained up to 6 epochs.

\n\n

I tested the blend used by @kagSen and it scores a 0.687. I will try your blending method.

", "votes": 2, "replies": [] }, { "id": 426209, "author_name": "Shujian Liu", "author_url": "", "post_date": "2018-11-22T21:44:07.803000", "content": "

Cool!

", "votes": 0, "replies": [] }, { "id": 426234, "author_name": "Shujian Liu", "author_url": "", "post_date": "2018-11-23T00:02:30.547000", "content": "

Update:

\n\n

LB 0.691 single model with 5 folds and 6 epochs each: https://www.kaggle.com/shujian/single-rnn-with-5-folds-v1-8

\n\n

LB 0.692 single model with 4 folds and 8 epochs each: https://www.kaggle.com/shujian/single-rnn-with-5-folds-v1-9

", "votes": 3, "replies": [] }, { "id": 426309, "author_name": "", "author_url": "", "post_date": "2018-11-23T03:28:36.837000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 445650, "author_name": "", "author_url": "", "post_date": "2018-12-26T21:34:47", "content": "

LB: 0.701, Keras, single model, 5-fold CV, 5 epochs.\nF1 score from CV: 0.677!

\n\n

Edit: LB: 0.702 same model, different optim. parameters ...

", "votes": 3, "replies": [ { "id": 446091, "author_name": "Bjenk Ellefsen", "author_url": "", "post_date": "2018-12-27T13:33:07.310000", "content": "

Nice! Are you using an attention mechanism or capsule like I have seen others try?

", "votes": 0, "replies": [] }, { "id": 446320, "author_name": "", "author_url": "", "post_date": "2018-12-27T21:45:48.587000", "content": "

No, only what comes with Keras: mostly fully-connected, convolutional, and LSTM layers.\nI try to keep it simple to allow reasonable Nb CV folds

", "votes": 2, "replies": [] }, { "id": 448440, "author_name": "Rajesh Shreedhar", "author_url": "", "post_date": "2019-01-01T07:33:51.297000", "content": "

What is the execution time @Annabelle ?

", "votes": 0, "replies": [] }, { "id": 448477, "author_name": "", "author_url": "", "post_date": "2019-01-01T09:37:48", "content": "

Too much to add another model: approx 6500 s total kernel execution time

", "votes": 0, "replies": [] } ] }, { "id": 429623, "author_name": "", "author_url": "", "post_date": "2018-11-29T04:44:05.307000", "content": "", "votes": 0, "replies": [] }, { "id": 426610, "author_name": "", "author_url": "", "post_date": "2018-11-23T14:37:30.877000", "content": "

F1 score = 0.622 , word2vec embedding, single XGB model

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "423589": "It will be great to share insights about how is your best single model performming.", "437792": "Migrated to Pytorch. Single model - 0.700 on LB 4 Folds 5 epochs", "426445": "I have 0.505 with naïve Bayes :-)", "437086": "Edit : LB 0.701, I still use single model , 4 folds and some processing", "428628": "Adding to the CV vs. LB train:\n\n5-fold avg of 1 model, LB .687\n\n5 fold CV (with best threshold):\n\nFold 0: .697\nFold 1: .687\nFold 2: .684\nFold 3: .688\nFold 4: .688\n\nI guess I should be happy that the LB score is a plausible CV fold score? I'd say I'm relatively confident that the leaderboard isn't super reliable though.\n\nEdit: also, same model trained on all data in one run scores .685", "436288": "0.699 single model with 5 folds. Slightly modified my public kernels according to some posts above.", "424011": "LB : 0.694 ( single model with k-fold)", "452606": "single model with 5 Stratified KFold, CV log_loss 0.09908, f1_score 0.67972, LB 0.684", "426817": "Currently have a LB of 0.688 with one model, no k fold and only GloVe embeddings", "436389": "Adding on my kernel the cleaning from [this kernel][1] improved my score from 0.696 to 0.699\n\nI didn't clean anything before. \n\n [1]: https://www.kaggle.com/hung96ad/magic-numbers-is-all-you-need-0-696-lb", "433084": "I've got 0.691 on 4-folds with RMSprop optimizer and mean-combined embedding. I've tried perhaps 10 different parameter configurations but I can't get past this score on LB.", "433803": "0.697 - 4 Kfold , cyclic LR, mean combined embedding. But can't go above this even if I were able to improve CV. Which is very frustrating as I can't get any idea about what is what.", "432970": "`79 mins` - `4fold` - `CV 0.685` - `LB 0.691`", "423760": "My Best Single Model Public Leaderboard Score: 0.675", "423869": "0.685", "446459": "change parameters improve LB score to 0.702, it may be overfit on testset,so,I'm worried about the second stage", "426442": "LB: 66.3 (Single model ,no kfold)", "460246": "Change parameters of public kernel\nLB: 0.701 CV 0.6823", "439764": "Hi, is anybody able to go beyond both “LB 0.69” and “Local CV 0.69” , using only ‘single classifier’ ? (single model with no KFold, no ensemble) . In my case, even though my ensemble get LB0.697, my single only get LB0.675 (with 0.693 Local CV)", "434126": "Hm... some people have now reported having large differences between their local and public LB score... especially where these lower scores result in very high public LB scores. I am going to have to side with Eddy in that I'm probably much more dubious about standings on the public LB at this point.", "429622": "**Update:** .690 LB for a super simple RNN with data that I briefly preprocessed beforehand. Still looking into data augmentation, folds, post-processing, and hyper parameters to improve my solo model score even more. My validation score for this was .6782.", "427339": "with a GRU I managed to get 0.677 ... I've only seen this once due to CuDNN randomness\n\nI'm really impressed some people are getting over 0.68!!!", "425512": "0.669 no kfold", "425052": "LB 0.679 in 1310 sec (GPU): https://www.kaggle.com/shujian/single-rnn-model-with-meta-features?scriptVersionId=7593124\n", "445650": "LB: 0.701, Keras, single model, 5-fold CV, 5 epochs.\nF1 score from CV: 0.677!\n\nEdit: LB: 0.702 same model, different optim. parameters ...", "429623": "", "426610": "F1 score = 0.622 , word2vec embedding, single XGB model" } }