I wonder what the best single model score is. I got my NN to 0.2240, but havn't included any image features nor the active and period files yet. How much is the improvement with image and co.?
", "messages": [ { "id": 328917, "postDate": "2018-05-15T11:17:09.200Z", "content": "I wonder what the best single model score is. I got my NN to 0.2240, but havn't included any image features nor the active and period files yet. How much is the improvement with image and co.?
", "rawMarkdown": "I wonder what the best single model score is. I got my NN to 0.2240, but havn't included any image features nor the active and period files yet. How much is the improvement with image and co.?", "votes": 47 }, { "id": 340360, "postDate": "2018-06-09T02:28:20.813Z", "content": "I got 0.2181 with a NN network with 10 folds (UPDATED)
", "rawMarkdown": "I got 0.2181 with a NN network with 10 folds (UPDATED)\n", "votes": 17, "replies": [ { "id": 340367, "postDate": "2018-06-09T02:45:00.490Z", "content": "Wow, that's great!\nIs the feature amount effective? Or is the network special?
", "rawMarkdown": "Wow, that's great!\nIs the feature amount effective? Or is the network special?" }, { "id": 340372, "postDate": "2018-06-09T03:01:40.660Z", "content": "Wow,can I ask that using MLP or RNN or CNN?
", "rawMarkdown": "Wow,can I ask that using MLP or RNN or CNN?" }, { "id": 340378, "postDate": "2018-06-09T03:23:17.113Z", "content": "Nothing special. I use four kind features: continous (contains ImageNet score), category, text with embedding, raw image pixel. Four submodels for four kinds feature, concatenated to one tensor, serval Dense Layers. Heavily use BatchNormalization and Dropout. I believe there still have some room to improve because I rarely did feature engineering.
", "rawMarkdown": "Nothing special. I use four kind features: continous (contains ImageNet score), category, text with embedding, raw image pixel. Four submodels for four kinds feature, concatenated to one tensor, serval Dense Layers. Heavily use BatchNormalization and Dropout. I believe there still have some room to improve because I rarely did feature engineering.", "votes": 9 }, { "id": 340381, "postDate": "2018-06-09T03:32:07.747Z", "content": "Thank you for sharing!\nMy nn is still 0.2222(TT) I will do my best from here!
", "rawMarkdown": "Thank you for sharing!\nMy nn is still 0.2222(TT) I will do my best from here!" }, { "id": 340382, "postDate": "2018-06-09T03:42:58.173Z", "content": "My LGB is still 0.2205 and I have no idea how to improve it. Poor feature engineer skills.
", "rawMarkdown": "My LGB is still 0.2205 and I have no idea how to improve it. Poor feature engineer skills.", "votes": 3 }, { "id": 340388, "postDate": "2018-06-09T04:12:33.540Z", "content": "Wah!!! Great.. R u using self-embedding ? \"Raw image pixel\" - is it like using the raw pixel value of the image..How r u using it in the NN model?
", "rawMarkdown": "Wah!!! Great.. R u using self-embedding ? \"Raw image pixel\" - is it like using the raw pixel value of the image..How r u using it in the NN model?\n" }, { "id": 340389, "postDate": "2018-06-09T04:13:28.803Z", "content": "Congrats @Liu Jilong,
\n\n\n\n\nraw image pixel
\n
Do you mean the raw image without through any pretrained model?
\nIf yes, that is impressive. How long does it take to train?
Thanks for sharing, I have confused that you train four model for for four kinds feature,then a final model based on the four predictions?
", "rawMarkdown": "Thanks for sharing, I have confused that you train four model for for four kinds feature,then a final model based on the four predictions? " }, { "id": 340400, "postDate": "2018-06-09T04:58:48.003Z", "content": "Just like any other image network, serval Conv2D layers.
", "rawMarkdown": "Just like any other image network, serval Conv2D layers." }, { "id": 340401, "postDate": "2018-06-09T04:59:20.403Z", "content": "UPDATED: about 15min per epoch, 2.5h per fold.
", "rawMarkdown": "UPDATED: about 15min per epoch, 2.5h per fold.", "votes": 1 }, { "id": 340403, "postDate": "2018-06-09T05:08:52.107Z", "content": "Thanks Liu... Any insight on text pre-processing / embedding?
", "rawMarkdown": "Thanks Liu... Any insight on text pre-processing / embedding?" }, { "id": 340405, "postDate": "2018-06-09T05:19:17.890Z", "content": "Just as many kagglers shared, I use self-trained word2vec embedding. I have no idea about russian, so no text pre-processing is used.
", "rawMarkdown": "Just as many kagglers shared, I use self-trained word2vec embedding. I have no idea about russian, so no text pre-processing is used." }, { "id": 340407, "postDate": "2018-06-09T05:26:39.460Z", "content": "Thank you for sharing.
\n\nMy situation is the opposite of you. I just started trying NN today, and it didn't look good.
\n\nVery valuable experience, thanks :).
", "rawMarkdown": "Thank you for sharing.\n\nMy situation is the opposite of you. I just started trying NN today, and it didn't look good. \n\nVery valuable experience, thanks :)." }, { "id": 340408, "postDate": "2018-06-09T05:31:06.123Z", "content": "Many tricks to make NN work well. Good luck.
", "rawMarkdown": "Many tricks to make NN work well. Good luck.", "votes": 1 }, { "id": 340410, "postDate": "2018-06-09T05:32:25.727Z", "content": "Thanks. Appreciate your help...
", "rawMarkdown": "Thanks. Appreciate your help..." }, { "id": 340430, "postDate": "2018-06-09T06:21:50.623Z", "content": "Thanks. wish to see experience sharing for NN model after this competition.I have no idea to train a good NN model.My NN model is poor score.
", "rawMarkdown": "Thanks. wish to see experience sharing for NN model after this competition.I have no idea to train a good NN model.My NN model is poor score." }, { "id": 340688, "postDate": "2018-06-10T02:41:29.237Z", "content": "Thanks for your sharing. I have a memory problem of training a CNN on images. Since I don't have enough memory to load all images, currently I use a generator to read images batch by batch from disk and feed them to the CNN, which makes the training process very slow (1 hour per epoch). Is there any better way to make it faster?
", "rawMarkdown": "Thanks for your sharing. I have a memory problem of training a CNN on images. Since I don't have enough memory to load all images, currently I use a generator to read images batch by batch from disk and feed them to the CNN, which makes the training process very slow (1 hour per epoch). Is there any better way to make it faster?" }, { "id": 340691, "postDate": "2018-06-10T03:20:04.987Z", "content": "I also tried fit_generator, but fit_generator's result is worse than fit, which is strange and I did not find the reason. As for time-consuming problem, you can try fit_generator's multi-processing parameter.
", "rawMarkdown": "I also tried fit_generator, but fit_generator's result is worse than fit, which is strange and I did not find the reason. As for time-consuming problem, you can try fit_generator's multi-processing parameter." }, { "id": 340699, "postDate": "2018-06-10T03:47:12.887Z", "content": "Wondering 15min/epoch for what kind of GPU?
", "rawMarkdown": "Wondering 15min/epoch for what kind of GPU?" }, { "id": 340702, "postDate": "2018-06-10T04:02:07.473Z", "content": "gtx1070
", "rawMarkdown": "gtx1070" }, { "id": 341040, "postDate": "2018-06-10T21:38:04.990Z", "content": "Hi Liu,\nThanks for your sharing, it's very encouraging. I wonder if you did not use fit_generator, how can you load 1300k images to memory? Thanks!
", "rawMarkdown": "Hi Liu,\nThanks for your sharing, it's very encouraging. I wonder if you did not use fit_generator, how can you load 1300k images to memory? Thanks!" }, { "id": 341651, "postDate": "2018-06-12T03:03:38.347Z", "content": "Hi Jilong, is your NN model end2end training? Did you use keras concatenate four kinds feature layers? \nThx for your sharing.
", "rawMarkdown": "Hi Jilong, is your NN model end2end training? Did you use keras concatenate four kinds feature layers? \nThx for your sharing." }, { "id": 342079, "postDate": "2018-06-12T19:46:52.283Z", "content": "\n\n\nI also tried fit_generator, but fit_generator's result is worse than fit, which is strange and I did not find the reason. As for time-consuming problem, you can try fit_generator's multi-processing parameter.
\n
Fit automatically does a few things that are critical for NNs ( such as randomizing input per epoch) that you have to explicitly do with your own fit_generator. Fit_generator automatically should load info on a new thread- so unless you have something crazy expensive in your generator (like a garbage collecting call) it shouldn't be a bottleneck at all-loading is faster than training.
", "rawMarkdown": "> I also tried fit_generator, but fit_generator's result is worse than fit, which is strange and I did not find the reason. As for time-consuming problem, you can try fit_generator's multi-processing parameter.\n\nFit automatically does a few things that are critical for NNs ( such as randomizing input per epoch) that you have to explicitly do with your own fit_generator. Fit_generator automatically should load info on a new thread- so unless you have something crazy expensive in your generator (like a garbage collecting call) it shouldn't be a bottleneck at all-loading is faster than training.\n\n", "votes": 3 }, { "id": 343689, "postDate": "2018-06-15T20:30:19.343Z", "content": "I've had several issues with using the image data. I think one of the most major ones is that it turns out keras cannot do multiprocessing on windows so it isnt actually loading in images properly beforehand. Second hand is I am trying to do the image resize and padding in line. This ends up costing roughly .05 per image. I worked a little around this using concurrent. Futures, but a full epoch still takes much much more than 15 minutes. Closer to 2-3 hours.. One avenue I spent some time researching is rewriting the images as a hdf5 dataset, but I'm unsure how to load it in alongside my other features, but that has various issues as well regarding shuffling and getting everything aligned
", "rawMarkdown": "I've had several issues with using the image data. I think one of the most major ones is that it turns out keras cannot do multiprocessing on windows so it isnt actually loading in images properly beforehand. Second hand is I am trying to do the image resize and padding in line. This ends up costing roughly .05 per image. I worked a little around this using concurrent. Futures, but a full epoch still takes much much more than 15 minutes. Closer to 2-3 hours.. One avenue I spent some time researching is rewriting the images as a hdf5 dataset, but I'm unsure how to load it in alongside my other features, but that has various issues as well regarding shuffling and getting everything aligned" }, { "id": 344203, "postDate": "2018-06-17T07:59:08.747Z", "content": "Train with image pixels end to end is indeed tough, considering the million count. I did some trick to make it works, also i encountered some strange questions. I will share my workflow after competition ends and hoping for answers.
", "rawMarkdown": "Train with image pixels end to end is indeed tough, considering the million count. I did some trick to make it works, also i encountered some strange questions. I will share my workflow after competition ends and hoping for answers.", "votes": 1 }, { "id": 344290, "postDate": "2018-06-17T14:33:53.243Z", "content": "thank you! I am looking forward to your model
", "rawMarkdown": "thank you! I am looking forward to your model" }, { "id": 344749, "postDate": "2018-06-18T16:10:00.313Z", "content": "\n\n\nI think one of the most major ones is that it turns out Keras cannot do multiprocessing on Windows
\n
This is interesting. Can you do your own threading process? maybe something with threading and queue like:
\n\n def data_loader(q,):\n for start in range(0, len(train_imgs), batch_size):\n #do your batch creation code here \n q.put([feat_batch, y_batch])\n q = queue.Queue(maxsize=q_size)\n t1 = threading.Thread(target=data_loader, name='DataLoader', args=(q,))\n t1.start()\n t1.join()\nAnd then just read the generator batches from the queue? I can't really check ( I'm running Linux atm).\nIf that's not possible, I recommend Linux...
", "rawMarkdown": "> I think one of the most major ones is that it turns out Keras cannot do multiprocessing on Windows\n\nThis is interesting. Can you do your own threading process? maybe something with threading and queue like:\n\n def data_loader(q,):\n for start in range(0, len(train_imgs), batch_size):\n #do your batch creation code here \n q.put([feat_batch, y_batch])\n q = queue.Queue(maxsize=q_size)\n t1 = threading.Thread(target=data_loader, name='DataLoader', args=(q,))\n t1.start()\n t1.join()\n\n\nAnd then just read the generator batches from the queue? I can't really check ( I'm running Linux atm).\nIf that's not possible, I recommend Linux...", "votes": 2 }, { "id": 344921, "postDate": "2018-06-18T23:26:44.383Z", "content": "I really should move over to linux at some point, but I am eventually planning on dedicating one system to work like this and the other for games and less demanding functions so dont want to dual boot right away. That being said, I have found some way to kind of work around this using the concurrent.futures builtin python to load in the images with threading and I also preprocessed the images with padding and downscaling so I dont have to do that function within the generator.
", "rawMarkdown": "I really should move over to linux at some point, but I am eventually planning on dedicating one system to work like this and the other for games and less demanding functions so dont want to dual boot right away. That being said, I have found some way to kind of work around this using the concurrent.futures builtin python to load in the images with threading and I also preprocessed the images with padding and downscaling so I dont have to do that function within the generator. ", "votes": 1 }, { "id": 345074, "postDate": "2018-06-19T06:41:18.443Z", "content": "Hi Jilong, In my efforts to include text + image + tabular features, I find that it heavily overfits even before it finishes one epoch, does heavy use of batch normalization help to tackle overfitting? Thanks
", "rawMarkdown": "Hi Jilong, In my efforts to include text + image + tabular features, I find that it heavily overfits even before it finishes one epoch, does heavy use of batch normalization help to tackle overfitting? Thanks" }, { "id": 345152, "postDate": "2018-06-19T09:28:40.913Z", "content": "Yes, i heavily use BN and dropout, you may try add a BN before every Dense Layer.
", "rawMarkdown": "Yes, i heavily use BN and dropout, you may try add a BN before every Dense Layer.", "votes": 1 }, { "id": 345417, "postDate": "2018-06-19T20:57:11.850Z", "content": "@Liu Jilong, there are few rows in test set where images are missing, how are you handling those when you say you are using imagent features?
", "rawMarkdown": "@Liu Jilong, there are few rows in test set where images are missing, how are you handling those when you say you are using imagent features?\n" }, { "id": 345512, "postDate": "2018-06-20T01:14:15.283Z", "content": "I just give them an all-black image for row image pixels and zero score for image net scores. It may have some approaches to improve, but i do not have enough time to handle this.
", "rawMarkdown": "I just give them an all-black image for row image pixels and zero score for image net scores. It may have some approaches to improve, but i do not have enough time to handle this.", "votes": 1 }, { "id": 346569, "postDate": "2018-06-22T01:17:55.993Z", "content": "Have you tried training a model purely on images? Do you know roughly what training or validation loss that can get down to? I have been working on this for a while and cant seem to get much out of the images.
", "rawMarkdown": "Have you tried training a model purely on images? Do you know roughly what training or validation loss that can get down to? I have been working on this for a while and cant seem to get much out of the images. " }, { "id": 346622, "postDate": "2018-06-22T04:09:39.683Z", "content": "I tried several CNN models to learn only on images to predict deal probably. I used relu on last several dense layer, and I found out last dense layer (before Dense (1)) predict all zeros, which give me 0.133 rmse loss...
", "rawMarkdown": "I tried several CNN models to learn only on images to predict deal probably. I used relu on last several dense layer, and I found out last dense layer (before Dense (1)) predict all zeros, which give me 0.133 rmse loss...", "votes": 1 }, { "id": 346640, "postDate": "2018-06-22T04:56:09.560Z", "content": "Mine are consistently getting to .066ish MSE but dont seem to contribute anything to my combined model. Havent inspected what the outputs or the weights areFor a while I thought there was something wrong with my generator but it seems to work fine without the images just fine so dont think that is the issues
", "rawMarkdown": "Mine are consistently getting to .066ish MSE but dont seem to contribute anything to my combined model. Havent inspected what the outputs or the weights areFor a while I thought there was something wrong with my generator but it seems to work fine without the images just fine so dont think that is the issues" } ] }, { "id": 338921, "postDate": "2018-06-06T01:33:30.123Z", "content": "I'm hitting .2190 with a 5-fold averaged LGBM
", "rawMarkdown": "I'm hitting .2190 with a 5-fold averaged LGBM", "votes": 9, "replies": [ { "id": 338925, "postDate": "2018-06-06T01:49:48.870Z", "content": "That's impressive, Congratulation!
", "rawMarkdown": "That's impressive, Congratulation!", "votes": 1 }, { "id": 338928, "postDate": "2018-06-06T01:55:08.110Z", "content": "Wow,you must find some amazing features~
", "rawMarkdown": "Wow,you must find some amazing features~", "votes": 1 }, { "id": 338929, "postDate": "2018-06-06T01:57:05.233Z", "content": "Seems like there is a magic feature for the big jump.
", "rawMarkdown": "Seems like there is a magic feature for the big jump.", "votes": 1 }, { "id": 338930, "postDate": "2018-06-06T02:09:09.757Z", "content": "Thanks! :)
\n\nI'm using 100+ engineered tabular features, but there are definitely some that have a particularly substantial contribution.
", "rawMarkdown": "Thanks! :)\n\nI'm using 100+ engineered tabular features, but there are definitely some that have a particularly substantial contribution.", "votes": 1 }, { "id": 338939, "postDate": "2018-06-06T02:34:12.020Z", "content": "good job,did you use target encoding?
", "rawMarkdown": "good job,did you use target encoding?", "votes": 1 }, { "id": 339017, "postDate": "2018-06-06T06:43:56.893Z", "content": "Wow, do you mean you didn't use text features at all?
", "rawMarkdown": "Wow, do you mean you didn't use text features at all?" }, { "id": 339162, "postDate": "2018-06-06T12:37:55.033Z", "content": "Sorry, I do use text features in addition to the tabular ones - could have been more clear
", "rawMarkdown": "Sorry, I do use text features in addition to the tabular ones - could have been more clear", "votes": 3 }, { "id": 339181, "postDate": "2018-06-06T13:13:56.817Z", "content": "Thank you a lot for a clarification!
", "rawMarkdown": "Thank you a lot for a clarification!", "votes": 2 }, { "id": 340151, "postDate": "2018-06-08T13:24:03.263Z", "content": "great work! did you use the train_active csv ?
", "rawMarkdown": "great work! did you use the train_active csv ?" } ] }, { "id": 333988, "postDate": "2018-05-26T08:13:34.680Z", "content": "LGB (Categorical, Numerical, TFIDF, Images meta-data + ImageNet + NIMA scoring): 0.2200 (updated)
\n\nXGB (Categorical, Numerical, TFIDF, Images meta-data + ImageNet + NIMA scoring): 0.2216 (updated)
\n\nNN (Categorical, Numerical, W2V, Images meta-data + ImageNet scoring): 0.2198 (updated)
\n\nAll with CV5.
", "rawMarkdown": "LGB (Categorical, Numerical, TFIDF, Images meta-data + ImageNet + NIMA scoring): 0.2200 (updated)\n\nXGB (Categorical, Numerical, TFIDF, Images meta-data + ImageNet + NIMA scoring): 0.2216 (updated)\n\nNN (Categorical, Numerical, W2V, Images meta-data + ImageNet scoring): 0.2198 (updated)\n\nAll with CV5.", "votes": 8, "replies": [ { "id": 334098, "postDate": "2018-05-26T14:27:42.823Z", "content": "may i ask what is image meta?
", "rawMarkdown": "may i ask what is image meta?" }, { "id": 334132, "postDate": "2018-05-26T16:01:46.743Z", "content": "What numerical features are working for you?
", "rawMarkdown": "What numerical features are working for you?" }, { "id": 334182, "postDate": "2018-05-26T17:42:30.647Z", "content": "Images meta-data is width, height. I've also included the average of VGG16, VGG19, InceptionV3, Xception, ResNet50 ImageNet scoring for each image but it does not really help. It only improves by 0.0002. And for numerical, only the basic ones, log1p(item_seq_number) and log1p(price). I'm trying different target-encoding but currently, as already reported below, it overfits too much.
", "rawMarkdown": "Images meta-data is width, height. I've also included the average of VGG16, VGG19, InceptionV3, Xception, ResNet50 ImageNet scoring for each image but it does not really help. It only improves by 0.0002. And for numerical, only the basic ones, log1p(item_seq_number) and log1p(price). I'm trying different target-encoding but currently, as already reported below, it overfits too much.", "votes": 8 }, { "id": 336600, "postDate": "2018-06-01T02:28:34.793Z", "content": "does NIMA stand for Neural Image Assessment?
", "rawMarkdown": "does NIMA stand for Neural Image Assessment?" }, { "id": 336711, "postDate": "2018-06-01T06:40:25.510Z", "content": "Yes, I used this implementation:\nhttps://github.com/titu1994/neural-image-assessment
\n\nIt improves LGB LB by 0.0002 and XGB by 0.0003 (only). I included mean/std as features from MobileNet and InceptionResNetv2. It does not improve NN (which should mean the NLP part is better).
", "rawMarkdown": "Yes, I used this implementation:\nhttps://github.com/titu1994/neural-image-assessment\n\nIt improves LGB LB by 0.0002 and XGB by 0.0003 (only). I included mean/std as features from MobileNet and InceptionResNetv2. It does not improve NN (which should mean the NLP part is better)." } ] }, { "id": 344107, "postDate": "2018-06-17T02:10:41.900Z", "content": "LGB 10-fold 0.2179
", "rawMarkdown": "LGB 10-fold 0.2179", "votes": 5, "replies": [ { "id": 344108, "postDate": "2018-06-17T02:14:32.873Z", "content": "Excellent achieve！Can you show some black magic you used？
", "rawMarkdown": "Excellent achieve！Can you show some black magic you used？" }, { "id": 344128, "postDate": "2018-06-17T03:33:02.557Z", "content": "no magic,features are almost from kernels like aggregations,images,tfidf,tsvd,impute...just try to use more features.
", "rawMarkdown": "no magic,features are almost from kernels like aggregations,images,tfidf,tsvd,impute...just try to use more features.", "votes": 1 }, { "id": 344129, "postDate": "2018-06-17T03:36:12.897Z", "content": "@senkin13: Is that a CV score or a LB score? Does your model contain a Ridge?
", "rawMarkdown": "@senkin13: Is that a CV score or a LB score? Does your model contain a Ridge?" }, { "id": 344130, "postDate": "2018-06-17T03:39:13.767Z", "content": "LB score without ridge,I prefer to use ridge for stacking instead of lgb features.
", "rawMarkdown": "LB score without ridge,I prefer to use ridge for stacking instead of lgb features.", "votes": 1 }, { "id": 344132, "postDate": "2018-06-17T03:52:27.573Z", "content": "What do you mean by tsvd and impute? Any reference kernels or information?
", "rawMarkdown": "What do you mean by tsvd and impute? Any reference kernels or information?" }, { "id": 344134, "postDate": "2018-06-17T04:07:04.590Z", "content": "here: https://www.kaggle.com/krithi07/baseline-model-with-new-features
", "rawMarkdown": "here: https://www.kaggle.com/krithi07/baseline-model-with-new-features", "votes": 4 }, { "id": 344146, "postDate": "2018-06-17T04:52:48.563Z", "content": "Congrats @Senkin13, which image features did you use. The ones I've got makes my scores worse... both CV and LB.
", "rawMarkdown": "Congrats @Senkin13, which image features did you use. The ones I've got makes my scores worse... both CV and LB." }, { "id": 344172, "postDate": "2018-06-17T06:21:42.553Z", "content": "A few simple image meta features,not too much improvement compared to Peter Hurford(0.002 improvment)
", "rawMarkdown": "A few simple image meta features,not too much improvement compared to Peter Hurford(0.002 improvment)" }, { "id": 344178, "postDate": "2018-06-17T06:47:15.090Z", "content": "I know you called my improvement from images impressive, but I'm not coming anywhere close to 0.2179 single model LGB... Hey, I wonder what would happen if we teamed up and pooled our feature engineering?
", "rawMarkdown": "I know you called my improvement from images impressive, but I'm not coming anywhere close to 0.2179 single model LGB... Hey, I wonder what would happen if we teamed up and pooled our feature engineering?" }, { "id": 344179, "postDate": "2018-06-17T06:52:11.893Z", "content": "Hello, I have tried tsvd to tfidf of description and title,but my score became to worse,I choose 32 component tsvd.
", "rawMarkdown": "Hello, I have tried tsvd to tfidf of description and title,but my score became to worse,I choose 32 component tsvd.", "votes": 1 }, { "id": 344185, "postDate": "2018-06-17T07:13:08.823Z", "content": "Maybe I dont choose the right n_ component
", "rawMarkdown": "Maybe I dont choose the right n_ component " }, { "id": 344187, "postDate": "2018-06-17T07:14:05.520Z", "content": "@Peter Hurford Your proposal is attractive ,there still have 4 days ,let's wait and see.
", "rawMarkdown": "@Peter Hurford Your proposal is attractive ,there still have 4 days ,let's wait and see." }, { "id": 344188, "postDate": "2018-06-17T07:15:09.567Z", "content": "@Johnny You can always check sum of explained_variance_ratio_. If it is very low, it means you not only lose noise but also information
", "rawMarkdown": "@Johnny You can always check sum of explained_variance_ratio_. If it is very low, it means you not only lose noise but also information", "votes": 1 }, { "id": 344189, "postDate": "2018-06-17T07:16:47.913Z", "content": "@Johnny Liu I only get a little improvement from tsvd,yes should try different n_component and tfidf parameters.
", "rawMarkdown": "@Johnny Liu I only get a little improvement from tsvd,yes should try different n_component and tfidf parameters.", "votes": 1 }, { "id": 344191, "postDate": "2018-06-17T07:18:52.707Z", "content": "very solid work!
", "rawMarkdown": "very solid work! " }, { "id": 344201, "postDate": "2018-06-17T07:47:13.687Z", "content": "thanks @AhmetErdem @senkin13, now I sum explained_variance_ratio_ to find the best component
", "rawMarkdown": "thanks @AhmetErdem @senkin13, now I sum explained_variance_ratio_ to find the best component\n\n" } ] }, { "id": 338049, "postDate": "2018-06-04T10:21:58.890Z", "content": "LGBM 10-fold avg. - 0.2203 (categorical, numerical, target encoding, text and features predicted using all csv data)
", "rawMarkdown": "LGBM 10-fold avg. - 0.2203 (categorical, numerical, target encoding, text and features predicted using all csv data)", "votes": 5, "replies": [ { "id": 338078, "postDate": "2018-06-04T11:43:39.543Z", "content": "Hello, does target encoding contribute your score a lot?
", "rawMarkdown": "Hello, does target encoding contribute your score a lot?" }, { "id": 338083, "postDate": "2018-06-04T12:00:17.947Z", "content": "When you say \"features predicted using all csv data\", does that mean that you found some interesting features leveraging train_active/test_active/periods_train/periods_test beyond the ones identified by Benjamin Minixhofer?
", "rawMarkdown": "When you say \"features predicted using all csv data\", does that mean that you found some interesting features leveraging train_active/test_active/periods_train/periods_test beyond the ones identified by Benjamin Minixhofer?", "votes": 1 }, { "id": 338100, "postDate": "2018-06-04T12:28:06.580Z", "content": "Previous lgbm was at 0.2215 but TE was not the only features added so it is hard to say how much TE contributed to score. Still half of features in top-100 are TEs.
", "rawMarkdown": "Previous lgbm was at 0.2215 but TE was not the only features added so it is hard to say how much TE contributed to score. Still half of features in top-100 are TEs. ", "votes": 2 }, { "id": 338104, "postDate": "2018-06-04T12:32:30.757Z", "content": "@Larry Currently I am using only 'times up' and 'days up' predictions, but I think it is worth trying to fill other missing values by predicting them from all csv data.
", "rawMarkdown": "@Larry Currently I am using only 'times up' and 'days up' predictions, but I think it is worth trying to fill other missing values by predicting them from all csv data." }, { "id": 338111, "postDate": "2018-06-04T12:48:16.607Z", "content": "@Andrii Do you mean you trained a model to estimate 'times up' and 'days up' from the other features? Or you just used the features described in Benjamin's kernel?
", "rawMarkdown": "@Andrii Do you mean you trained a model to estimate 'times up' and 'days up' from the other features? Or you just used the features described in Benjamin's kernel?" }, { "id": 338116, "postDate": "2018-06-04T12:59:20.747Z", "content": "@Dmitriy I am using both - averages from kernel and model predictions.
", "rawMarkdown": "@Dmitriy I am using both - averages from kernel and model predictions.", "votes": 2 }, { "id": 338427, "postDate": "2018-06-05T03:24:17.920Z", "content": "wow, thanks for your replying.I have not try to using target encoding now
", "rawMarkdown": "wow, thanks for your replying.I have not try to using target encoding now" } ] }, { "id": 333337, "postDate": "2018-05-24T23:45:27.317Z", "content": "NN:
\n0.2218 Fasttext
\n0.2215 Self-training embedding
Thanks @Dieter
", "rawMarkdown": "NN: \n0.2218 Fasttext \n0.2215 Self-training embedding \n\nThanks @Dieter", "votes": 5, "replies": [ { "id": 333362, "postDate": "2018-05-25T02:50:23.843Z", "content": "Wow your fasttext result is great. Are u using the crawl one or wiki one? Did you play a lot with different NN structures?
", "rawMarkdown": "Wow your fasttext result is great. Are u using the crawl one or wiki one? Did you play a lot with different NN structures?" }, { "id": 333366, "postDate": "2018-05-25T03:09:20.937Z", "content": "I use wiki 300D.
\nI tried some NLP parts such as: RNN, Bi-GRU, Bi-LSTM, CNN, ... (with Attention and without Attention).
@Ghost can you tell us how much do you think contributes the NLP part to your score? Let's say you take a naive approach with a single gru layer, how much would that worsen your result?
", "rawMarkdown": "@Ghost can you tell us how much do you think contributes the NLP part to your score? Let's say you take a naive approach with a single gru layer, how much would that worsen your result?" }, { "id": 333372, "postDate": "2018-05-25T03:35:58.927Z", "content": "@Dieter
\nI am not sure that I understand your question correctly. Please feel free to ask.
\nWithout NLP, my model is really bad. It is arround 0.2240. Then I tried to add NLP.
I have not ran all of the NLP parts above, so I don't have results of each part.
\nTo determine which NLP part is good or not. I have a training plan as follows:
You will have list of cross-validation of model in different NLP architectures (GRU, CNN, LSTM,...).\nSelect the best one and go fully training.
\n\nMy top 3 best NLP parts are:\nCapsule --> CNN --> Bi-GRU --> ...
", "rawMarkdown": "@Dieter \nI am not sure that I understand your question correctly. Please feel free to ask. \nWithout NLP, my model is really bad. It is arround 0.2240. Then I tried to add NLP.\n\nI have not ran all of the NLP parts above, so I don't have results of each part. \nTo determine which NLP part is good or not. I have a training plan as follows: \n\n 1. Select fixed training set and validation set. Ex: First fold of 5-Fold split\n 2. Replace NLP part and run for some epochs (Ex: 5 epochs) \n\nYou will have list of cross-validation of model in different NLP architectures (GRU, CNN, LSTM,...).\nSelect the best one and go fully training. \n\nMy top 3 best NLP parts are:\nCapsule --> CNN --> Bi-GRU --> ... \n\n", "votes": 3 }, { "id": 333377, "postDate": "2018-05-25T03:52:36.307Z", "content": "\n\n\nWithout NLP
\n
you mean skipping text completely?
", "rawMarkdown": "> Without NLP\n\nyou mean skipping text completely?" }, { "id": 333379, "postDate": "2018-05-25T03:56:39.793Z", "content": "Yeap.
\nWithout NLP, I only have numeric and categorical features.
Thank you, that helps :)
", "rawMarkdown": "Thank you, that helps :) " }, { "id": 333499, "postDate": "2018-05-25T09:17:35.427Z", "content": "0.2218 is this a single model or average of 5 fold?
", "rawMarkdown": "0.2218 is this a single model or average of 5 fold?" }, { "id": 334030, "postDate": "2018-05-26T10:47:38.253Z", "content": "@Ghost you must be very talented. You joined kaggle 2 days ago, and already have models that score top 50 in LB. I can only salute.
", "rawMarkdown": "@Ghost you must be very talented. You joined kaggle 2 days ago, and already have models that score top 50 in LB. I can only salute.", "votes": 1 }, { "id": 334099, "postDate": "2018-05-26T14:32:58.730Z", "content": "0.2240 without Text is a strong model I think. \nDid you use image meta features in this?
", "rawMarkdown": "0.2240 without Text is a strong model I think. \nDid you use image meta features in this?" }, { "id": 334592, "postDate": "2018-05-27T23:48:41.247Z", "content": "Sorry for the late reply. I have not tried with image meta-features. The result is after CV.
", "rawMarkdown": "Sorry for the late reply. I have not tried with image meta-features. The result is after CV." } ] }, { "id": 329320, "postDate": "2018-05-16T08:25:08.463Z", "content": "I am working exclusively with NN for now..
\n\nSo far, best single NN model : LB = 0.2246 ( 5 folds CV RMSE : 0.2215xxx ), no image features either
\n\nThe model was scoring 0.244xxx on LB when I started it , 5 days ago...(Some ideas from kernels and discussion topics helped me in the improvement)
\n\nEdit : The NN model is now scoring 0.2223 on LB
", "rawMarkdown": "I am working exclusively with NN for now..\n\nSo far, best single NN model : LB = 0.2246 ( 5 folds CV RMSE : 0.2215xxx ), no image features either\n\nThe model was scoring 0.244xxx on LB when I started it , 5 days ago...(Some ideas from kernels and discussion topics helped me in the improvement) \n\n**Edit :** The NN model is now scoring 0.2223 on LB ", "votes": 6, "replies": [ { "id": 330506, "postDate": "2018-05-19T01:47:12.347Z", "content": "Nice.
\nMy model now is 0.2222 LB. Can I know which pretrained embedding are you using?
Thanks :)
\n\nI am using self-trained embeddings (following the idea suggested by Dieter )
", "rawMarkdown": "Thanks :)\n\nI am using self-trained embeddings (following the idea suggested by Dieter )" }, { "id": 330561, "postDate": "2018-05-19T05:32:38.377Z", "content": "Amazing guy.
\nI could not get the high result with this embedding :D
@ Totoro - Did you try both options ( trainable = True and Trainable = False )??
", "rawMarkdown": "@ Totoro - Did you try both options ( trainable = True and Trainable = False )??" }, { "id": 330567, "postDate": "2018-05-19T05:51:58.893Z", "content": "I use pretrained embedding, so trainble = False. I tried trainable = True, but it is worse for me
", "rawMarkdown": "I use pretrained embedding, so trainble = False. I tried trainable = True, but it is worse for me" }, { "id": 330568, "postDate": "2018-05-19T05:54:44.660Z", "content": "If it's any consolation...the biggest improvements in my model are not from the embedding itself :)
", "rawMarkdown": "If it's any consolation...the biggest improvements in my model are not from the embedding itself :)" }, { "id": 330571, "postDate": "2018-05-19T05:59:16.977Z", "content": "Interesting
\nHope to see your model when the competition ends :D
Thank you
", "rawMarkdown": "Thank you" } ] }, { "id": 342176, "postDate": "2018-06-13T03:26:59.310Z", "content": "Single model LightGBM 0.2199, with 5seed-avg 0.2195
", "rawMarkdown": "Single model LightGBM 0.2199, with 5seed-avg 0.2195", "votes": 3, "replies": [ { "id": 343951, "postDate": "2018-06-16T15:11:08.827Z", "content": "could you show some hints about how you achieve this?
", "rawMarkdown": "could you show some hints about how you achieve this?\n " }, { "id": 345827, "postDate": "2018-06-20T14:44:08.317Z", "content": "Sure, it will be in my write up if I get gold :)
", "rawMarkdown": "Sure, it will be in my write up if I get gold :)", "votes": 3 } ] }, { "id": 339507, "postDate": "2018-06-07T04:21:41.537Z", "content": "Finally! Able to achieve 2210-XGB and 2217-LGB. No image features though.\nI was stuck earlier and started from scratch, seems the right decision. Best of luck guys.
", "rawMarkdown": "Finally! Able to achieve 2210-XGB and 2217-LGB. No image features though.\nI was stuck earlier and started from scratch, seems the right decision. Best of luck guys.", "votes": 3, "replies": [ { "id": 339528, "postDate": "2018-06-07T04:58:37.717Z", "content": "Wow,Congratulation! Please sharing your solution after competition,I'm every interested how to achieve the score.
", "rawMarkdown": "Wow,Congratulation! Please sharing your solution after competition,I'm every interested how to achieve the score.", "votes": 3 }, { "id": 339547, "postDate": "2018-06-07T05:56:38.403Z", "content": "Is your XGB model using the same features as LGBM? I am finding that my LGBM models are both faster and perform better. I am able to get to 2217-LGB but not even 2217-XGB.
", "rawMarkdown": "Is your XGB model using the same features as LGBM? I am finding that my LGBM models are both faster and perform better. I am able to get to 2217-LGB but not even 2217-XGB. ", "votes": 2 }, { "id": 339568, "postDate": "2018-06-07T06:27:00.560Z", "content": "Congratulations @Nooh on getting those scores. Any hints you can share?
\n\n@Larry Freeman, I have the same issue with you. My best LGBM is at 0.2215 but xgb is 0.2232. Both with single runs. The speed difference is not surprising as LGBM is always faster than xgb but in my experience their performance is comparable. Hence my surprise that I am not able to get xgb to match lgb performance here.
", "rawMarkdown": "Congratulations @Nooh on getting those scores. Any hints you can share?\n\n@Larry Freeman, I have the same issue with you. My best LGBM is at 0.2215 but xgb is 0.2232. Both with single runs. The speed difference is not surprising as LGBM is always faster than xgb but in my experience their performance is comparable. Hence my surprise that I am not able to get xgb to match lgb performance here.", "votes": 2 }, { "id": 339648, "postDate": "2018-06-07T09:18:25.063Z", "content": "Thanks Liu, though I'm really bad in documentation and my code is a mess right now but will surely post a brief if I don't end up over-fitting.
", "rawMarkdown": "Thanks Liu, though I'm really bad in documentation and my code is a mess right now but will surely post a brief if I don't end up over-fitting. ", "votes": 1 }, { "id": 339649, "postDate": "2018-06-07T09:20:37.430Z", "content": "Yes Larry, I can say XGB has 98% same features as of my LGB. \nMy XGB took around 15-hours to train. :/
", "rawMarkdown": "Yes Larry, I can say XGB has 98% same features as of my LGB. \nMy XGB took around 15-hours to train. :/", "votes": 2 }, { "id": 339651, "postDate": "2018-06-07T09:28:44.263Z", "content": "Thanks a lot YaGana Sheriff-Hussaini!
\n\nNothing special in features though. I'm using nearly all the features used by this kernel. Played with some statistical features around deal_prob and Price. And another tip, try to thoroughly analyze image_top_1 and item_seq_no; I handcrafted something using them. After all that I tuned the hyperparams.
\n\nBest of luck buddy!
", "rawMarkdown": "Thanks a lot YaGana Sheriff-Hussaini!\n\nNothing special in features though. I'm using nearly all the features used by [this kernel][1]. Played with some statistical features around deal_prob and Price. And another tip, try to thoroughly analyze image_top_1 and item_seq_no; I handcrafted something using them. After all that I tuned the hyperparams.\n\nBest of luck buddy!\n\n\n [1]: https://www.kaggle.com/bminixhofer/aggregated-features-lightgbm", "votes": 7 }, { "id": 339863, "postDate": "2018-06-07T19:44:29.180Z", "content": "Added OOF ridge in LGB and now hitting 2211. :)
", "rawMarkdown": "Added OOF ridge in LGB and now hitting 2211. :) ", "votes": 3 }, { "id": 340353, "postDate": "2018-06-09T01:29:08.370Z", "content": "Thanks @Nooh and best of luck to you too.
\n\nI just managed to get my lgb to LB=0.2212 but still have some optimization to do. You are right the xgb takes a long time for me as well, hence I am trying to optimize my features with LGB before running the xgb model again. Like you, I use the same features for both.
", "rawMarkdown": "Thanks @Nooh and best of luck to you too.\n\nI just managed to get my lgb to LB=0.2212 but still have some optimization to do. You are right the xgb takes a long time for me as well, hence I am trying to optimize my features with LGB before running the xgb model again. Like you, I use the same features for both.", "votes": 1 }, { "id": 340584, "postDate": "2018-06-09T17:43:15.437Z", "content": "Okay, so I'm convinced now that text features are bringing in some improvements in my models. Text normalization seems promising in my case. I just did some very basic normalization on title+desc and it is progressive. I should quote the original author here i.e; ololo (who stood 5th in the previous avito competition). I just stumbled upon his code and there is a lot to scavenge from his repo here
\n\nI used this script to get some text features. Hope this helps.
", "rawMarkdown": "Okay, so I'm convinced now that text features are bringing in some improvements in my models. Text normalization seems promising in my case. I just did some very basic normalization on title+desc and it is progressive. I should quote the original author here i.e; ololo (who stood 5th in the previous avito competition). I just stumbled upon his code and there is a lot to scavenge from his repo [here][1] \n\nI used [this][2] script to get some text features. Hope this helps.\n\n\n [1]: https://github.com/alexeygrigorev/avito-duplicates-kaggle\n [2]: https://raw.githubusercontent.com/alexeygrigorev/avito-duplicates-kaggle/master/prepare_text_features.py", "votes": 7 }, { "id": 340759, "postDate": "2018-06-10T08:21:03.113Z", "content": "Thanks @Nooh. I have a simple text normalization but after reading your comment, I tried ololo's i.e. \"def normalize(text):\" and I get the following error:
\n\n\n", "rawMarkdown": "Thanks @Nooh. I have a simple text normalization but after reading your comment, I tried ololo's i.e. \"def normalize(text):\" and I get the following error:\n\n> File \"../src/script.py\", line 98\n text = re.sub(ur'(?<=[^а-я])(' + shortenings + ')[.]', ur'\\1 ', text)\n ^\nSyntaxError: invalid syntax", "votes": 2 }, { "id": 340775, "postDate": "2018-06-10T09:24:03.940Z", "content": "File \"../src/script.py\", line 98\n text = re.sub(ur'(?<=[^а-я])(' + shortenings + ')[.]', ur'\\1 ', text)\n ^\n SyntaxError: invalid syntax
\n
Hi YaGana Sheriff-Hussaini. His code is for Python 2.x\nAre you on Python3x? If so, make sure you port it accordingly.\nI didn't used it as is actually instead get ideas from his code and made my own normalizations.
", "rawMarkdown": "Hi YaGana Sheriff-Hussaini. His code is for Python 2.x\nAre you on Python3x? If so, make sure you port it accordingly.\nI didn't used it as is actually instead get ideas from his code and made my own normalizations.", "votes": 2 }, { "id": 341220, "postDate": "2018-06-11T08:22:41.477Z", "content": "@Nooh you are right Ololo's code is for python2.x. I am using python3 and wrote a better text normalization than the basic one I had. This one improved both my CV and LB scores.
", "rawMarkdown": "@Nooh you are right Ololo's code is for python2.x. I am using python3 and wrote a better text normalization than the basic one I had. This one improved both my CV and LB scores.", "votes": 1 }, { "id": 341293, "postDate": "2018-06-11T11:25:21.387Z", "content": "Great bro!, happy to hear that. I'm making up some image meta features' code. I hope that will bring in some improvement. If so, I'll share the code+data. \nFingers crossed. :)
", "rawMarkdown": "Great bro!, happy to hear that. I'm making up some image meta features' code. I hope that will bring in some improvement. If so, I'll share the code+data. \nFingers crossed. :)", "votes": 1 }, { "id": 341351, "postDate": "2018-06-11T13:25:45.630Z", "content": "I hope you get a better results with image features than I do. I have a weak machine hence using a different LGBM model with reduced features that includes image ones but only scores 0.21777 CV and 0.2233 LB.
", "rawMarkdown": "I hope you get a better results with image features than I do. I have a weak machine hence using a different LGBM model with reduced features that includes image ones but only scores 0.21777 CV and 0.2233 LB." }, { "id": 341522, "postDate": "2018-06-11T18:12:12.117Z", "content": "Hopefully yeah. I'm done with test set images and training set is running for 1 day now. I think it will take a couple of more hours.\nHow many models have you ensembled to get the current lb score you have? I have 4 of them, 3 lgbms and 1 xgb.\nTry training different lgbms with different features and different hyperparams. You can get all your features in that way i guess.
", "rawMarkdown": "Hopefully yeah. I'm done with test set images and training set is running for 1 day now. I think it will take a couple of more hours.\nHow many models have you ensembled to get the current lb score you have? I have 4 of them, 3 lgbms and 1 xgb.\nTry training different lgbms with different features and different hyperparams. You can get all your features in that way i guess. " }, { "id": 341598, "postDate": "2018-06-11T21:21:31.863Z", "content": "Six.
", "rawMarkdown": "Six." }, { "id": 344448, "postDate": "2018-06-18T03:05:43.033Z", "content": "which image meta features will you care for？
", "rawMarkdown": "which image meta features will you care for？" } ] }, { "id": 337277, "postDate": "2018-06-02T12:40:10.413Z", "content": "We are at
\n\nImpressive @Dieter. Are your results for single or N-Folds. If so is it 3 or 5 folds?
\n\nMy best single model is LGBM with LB = 0.2217 single run. I have not run it for N-Folds yet.
", "rawMarkdown": "Impressive @Dieter. Are your results for single or N-Folds. If so is it 3 or 5 folds?\n\nMy best single model is LGBM with LB = 0.2217 single run. I have not run it for N-Folds yet.", "votes": 1 }, { "id": 337916, "postDate": "2018-06-04T03:47:50.800Z", "content": "5 fold
", "rawMarkdown": "5 fold", "votes": 2 }, { "id": 338044, "postDate": "2018-06-04T10:04:21.810Z", "content": "What features are you using for LGB .2206? That is very impressive.
", "rawMarkdown": "What features are you using for LGB .2206? That is very impressive. " }, { "id": 338101, "postDate": "2018-06-04T12:29:12.633Z", "content": "@Dieter, if you can share , what is the difference in score between a single run and 5-Fold?
", "rawMarkdown": "@Dieter, if you can share , what is the difference in score between a single run and 5-Fold?" }, { "id": 338426, "postDate": "2018-06-05T03:15:44.950Z", "content": "@Larry, I guess some part of the good score are image features
\n\n@YaGana Sheriff-Hussaini difference is about 0.001
", "rawMarkdown": "@Larry, I guess some part of the good score are image features\n\n@YaGana Sheriff-Hussaini difference is about 0.001", "votes": 2 }, { "id": 342435, "postDate": "2018-06-13T14:10:32.643Z", "content": "@Dieter, the difference for me is 0.0003
", "rawMarkdown": "@Dieter, the difference for me is 0.0003" } ] }, { "id": 335359, "postDate": "2018-05-29T16:31:37.513Z", "content": "My best so far is an LGB model with .2216 (recently improved) after a 10-fold average.
", "rawMarkdown": "My best so far is an LGB model with .2216 (recently improved) after a 10-fold average.", "votes": 3, "replies": [ { "id": 335485, "postDate": "2018-05-29T20:05:05.627Z", "content": "Good job! Your public score is 0.2195, how did you earn those 0.002 on your score ? with some blending ?\nSame thing @AlexTru there is a significant gap between your LB and this LGB model. I'd understand if you don't want to share...
", "rawMarkdown": "Good job! Your public score is 0.2195, how did you earn those 0.002 on your score ? with some blending ?\nSame thing @AlexTru there is a significant gap between your LB and this LGB model. I'd understand if you don't want to share..." }, { "id": 335487, "postDate": "2018-05-29T20:09:31.143Z", "content": "I used lessons learned from here:\nhttps://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/52557
", "rawMarkdown": "I used lessons learned from here:\nhttps://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/52557", "votes": 2 }, { "id": 335510, "postDate": "2018-05-29T20:50:30.400Z", "content": "Thanks for sharing this link !
", "rawMarkdown": "Thanks for sharing this link !" }, { "id": 335518, "postDate": "2018-05-29T21:21:05.710Z", "content": "@larryfreeman Are you using train/test-time augments? I assume you're not doing psuedo-labeling?
", "rawMarkdown": "@larryfreeman Are you using train/test-time augments? I assume you're not doing psuedo-labeling?" }, { "id": 335542, "postDate": "2018-05-29T22:17:33.987Z", "content": "I am making attempts to use multiple methods (train/test augmentation, pseudo-labeling, and (less robust) cv + stacking framework). I think its mostly stacking that's contributing to score.
\n\n@PeterHurford, congrats to you on your excellent score. Any reason why you assume that I'm not doing pseudo-labeling? Is this something that you have tried? I have not gotten too much mileage from it yet but I am working on it. :-)
", "rawMarkdown": "I am making attempts to use multiple methods (train/test augmentation, pseudo-labeling, and (less robust) cv + stacking framework). I think its mostly stacking that's contributing to score. \n\n@PeterHurford, congrats to you on your excellent score. Any reason why you assume that I'm not doing pseudo-labeling? Is this something that you have tried? I have not gotten too much mileage from it yet but I am working on it. :-)" }, { "id": 335546, "postDate": "2018-05-29T22:23:42.073Z", "content": "@larryfreeman Thanks for the congrats - I hope it holds! :)
\n\n...My assumption is that pseudo-labeling is only worthwhile when you have really accurate models. We were lucky to be in that case in the toxic competition, though I stupidly rejected the idea and didn't try it there. With RMSE in the >0.2 range, I don't think our models are accurate enough, so I'm rejecting the idea again. I have not tried it. I hope it's the right call this time. ;)
", "rawMarkdown": "@larryfreeman Thanks for the congrats - I hope it holds! :)\n\n...My assumption is that pseudo-labeling is only worthwhile when you have really accurate models. We were lucky to be in that case in the toxic competition, though I stupidly rejected the idea and didn't try it there. With RMSE in the >0.2 range, I don't think our models are accurate enough, so I'm rejecting the idea again. I have not tried it. I hope it's the right call this time. ;)", "votes": 3 }, { "id": 335549, "postDate": "2018-05-29T22:27:27.060Z", "content": "additionally train and test distributions are quite close, which reduces the benefit from pseudo labeling
", "rawMarkdown": "additionally train and test distributions are quite close, which reduces the benefit from pseudo labeling", "votes": 6 }, { "id": 335553, "postDate": "2018-05-29T22:30:46.693Z", "content": "Thanks, @Peter and @Dieter. I am still developing my intuition about pseudo labeling. Cheers.
", "rawMarkdown": "Thanks, @Peter and @Dieter. I am still developing my intuition about pseudo labeling. Cheers." }, { "id": 335555, "postDate": "2018-05-29T22:31:39.537Z", "content": "@larryfreeman I'd also be really curious to hear if augmentation is helpful. My assumption is that with 10x the data here compared to toxic, plus the impact of categoricals/images in addition to text, the benefits of adding translated data would be much lower now. ...But I could also be wrong about that.
", "rawMarkdown": "@larryfreeman I'd also be really curious to hear if augmentation is helpful. My assumption is that with 10x the data here compared to toxic, plus the impact of categoricals/images in addition to text, the benefits of adding translated data would be much lower now. ...But I could also be wrong about that.", "votes": 1 }, { "id": 335556, "postDate": "2018-05-29T22:35:47.767Z", "content": "I share your assumption on data augmentation. At this point, I'm trying everything that I can think of just to see what happens and to see if it gives me any other ideas to try.
", "rawMarkdown": "I share your assumption on data augmentation. At this point, I'm trying everything that I can think of just to see what happens and to see if it gives me any other ideas to try.", "votes": 1 }, { "id": 335559, "postDate": "2018-05-29T22:38:52.247Z", "content": "@larryfreeman It's awesome to try stuff. This competition is overwhelming in the amount of things there are to try. I hope you get lucky and one of these ideas pans out for you. :)
", "rawMarkdown": "@larryfreeman It's awesome to try stuff. This competition is overwhelming in the amount of things there are to try. I hope you get lucky and one of these ideas pans out for you. :)", "votes": 1 }, { "id": 335584, "postDate": "2018-05-30T00:50:49.670Z", "content": "Thanks, @Peter. Best of luck to you too. :-)
", "rawMarkdown": "Thanks, @Peter. Best of luck to you too. :-)" } ] }, { "id": 328983, "postDate": "2018-05-15T13:29:53.330Z", "content": "Got 0.2222 with NN just train and test
", "rawMarkdown": "Got 0.2222 with NN just train and test", "votes": 3, "replies": [ { "id": 329135, "postDate": "2018-05-15T20:13:18.727Z", "content": "So the NN is for the NLP of description or more like for the classifier for different features just like LGBM
", "rawMarkdown": "So the NN is for the NLP of description or more like for the classifier for different features just like LGBM" }, { "id": 329244, "postDate": "2018-05-16T04:40:09.050Z", "content": "I guess both. At least I use it for both. It makes it much easier for me to connect different types of information.
", "rawMarkdown": "I guess both. At least I use it for both. It makes it much easier for me to connect different types of information. " } ] }, { "id": 341626, "postDate": "2018-06-12T00:36:43.723Z", "content": "After a lot of hard work I got my single LGB (with 5 fold) down to 0.2200 on LB. Yay !
", "rawMarkdown": "After a lot of hard work I got my single LGB (with 5 fold) down to 0.2200 on LB. Yay !", "votes": 4, "replies": [ { "id": 341667, "postDate": "2018-06-12T04:12:58.910Z", "content": "Good going buddy!
", "rawMarkdown": "Good going buddy!" }, { "id": 342197, "postDate": "2018-06-13T04:31:23.570Z", "content": "Good work, buddy!
", "rawMarkdown": "Good work, buddy!" } ] }, { "id": 340365, "postDate": "2018-06-09T02:38:39.960Z", "content": "0.2190 10-fold LGB
", "rawMarkdown": "0.2190 10-fold LGB", "votes": 4, "replies": [ { "id": 340375, "postDate": "2018-06-09T03:15:45.817Z", "content": "Do you known what is lb difference between 5-fold and 10-fold?
", "rawMarkdown": "Do you known what is lb difference between 5-fold and 10-fold?", "votes": 1 }, { "id": 340379, "postDate": "2018-06-09T03:23:30.757Z", "content": "Approximately 0.0001-0.0002
", "rawMarkdown": "Approximately 0.0001-0.0002", "votes": 1 }, { "id": 340406, "postDate": "2018-06-09T05:22:56.690Z", "content": "@Liu Jilong, I've only tried 1fold and 10fold.
\n\nBut I think it should be around 0.0002
Congratulation!! you guys are so cool. upvote for you guys.
", "rawMarkdown": "Congratulation!! you guys are so cool. upvote for you guys." }, { "id": 340689, "postDate": "2018-06-10T02:44:00.383Z", "content": "Wow, this small gap is quite impressive. Mine are alway around 0.004 (20 times of yours)
", "rawMarkdown": "Wow, this small gap is quite impressive. Mine are alway around 0.004 (20 times of yours)" } ] }, { "id": 342452, "postDate": "2018-06-13T14:39:15.370Z", "content": "Silly question.
\n\nWhat do you guys exacly mean when you say, for example, 10-fold LGB?\nYes, you can use CV to better tune hyperparameters. Particularly, n_estimators. Is that what you mean?
\n\nAnd when you find optimal number of estimators do you retrain on whole dataset?
", "rawMarkdown": "Silly question.\n\nWhat do you guys exacly mean when you say, for example, 10-fold LGB?\nYes, you can use CV to better tune hyperparameters. Particularly, n_estimators. Is that what you mean? \n\nAnd when you find optimal number of estimators do you retrain on whole dataset? ", "votes": 1, "replies": [ { "id": 342463, "postDate": "2018-06-13T15:01:05.843Z", "content": "I guess people mean average the prediction of 10 LGB models in CV instead of retrain and get one model. The advantage of avg is obvious :)
", "rawMarkdown": "I guess people mean average the prediction of 10 LGB models in CV instead of retrain and get one model. The advantage of avg is obvious :) ", "votes": 1 }, { "id": 342521, "postDate": "2018-06-13T16:28:38.163Z", "content": "Oh I see.\nYou can also achieve that with different random seeds. \nBut subsets of training data probably make models more different than random seeds. And that is good.
", "rawMarkdown": "Oh I see.\nYou can also achieve that with different random seeds. \nBut subsets of training data probably make models more different than random seeds. And that is good. " }, { "id": 342532, "postDate": "2018-06-13T16:49:00.773Z", "content": "But that’s not cv tho.
", "rawMarkdown": "But that’s not cv tho. " }, { "id": 342578, "postDate": "2018-06-13T18:29:51.537Z", "content": "10-fold is not used for hyper parameters tuning but just as CV technic. When running 10 fold, it's similar to running 10 times your LGB with 10 different seed and then doing an average. The advantage of using 10 fold, is that you can be sure that all data were used for train and for validation. (which might not be the case if you take random seeds)
", "rawMarkdown": "10-fold is not used for hyper parameters tuning but just as CV technic. When running 10 fold, it's similar to running 10 times your LGB with 10 different seed and then doing an average. The advantage of using 10 fold, is that you can be sure that all data were used for train and for validation. (which might not be the case if you take random seeds)", "votes": 1 }, { "id": 342610, "postDate": "2018-06-13T19:50:04.500Z", "content": "I hope to clear up some confusion here - it's not a silly question at all. It's common conflation of terminology on kaggle to refer to partitioned fold averaging as \"CV\", when it is a distinct method from CV that shares the partitioning structure in common.
\n\nHere are the two methods:
\n\nCross-validation - partition the data into k folds for train/validation splitting to exhaustively use all of the data for validation, with the purpose of evaluating model and hyperparameter performance. You use the evaluation results to guide your model/hyperparameter selection.
K-fold averaging - partition the data into k folds (using the same partitioning as in CV), train the same model on each fold, and average the results to help reduce the variance of a single model. You could add out of fold validation to this process without using it directly for selection, or you can just run with the result of the averaging if you feel good about the model/hyperparameter selection.
The reason the terminology gets confused is that many people take a shortcut and do both things at the same time. They do one k-fold split, use out of fold sets to select something like n_estimators for gradient boosting models with early stopping, then average the resulting models across the k train/validation runs. In theory, this is poor practice. Early stopping overfits to the particular choice of validation fold, so models averaged in this manner within CV inevitably overfit. In practice, the averaging helps mitigate some of the overfitting and the overall harm may be small, but it's something worth keeping in mind. It certainly causes your local score estimates to be at least slightly optimistic.
\n\nThis is a common pitfall in use of CV more generally. People don't realize that by using certain validation sets to guide their model selection, they've reduced the legitimacy of using those same validation sets for estimating generalization error. The best estimates of generalization error are given by sets that have had no influence on model selection. This is why doing CV and setting aside a true hold-out set that only gets used once for evaluation is a common and sound practice.
", "rawMarkdown": "I hope to clear up some confusion here - it's not a silly question at all. It's common conflation of terminology on kaggle to refer to partitioned fold averaging as \"CV\", when it is a distinct method from CV that shares the partitioning structure in common. \n\nHere are the two methods:\n\n1. Cross-validation - partition the data into k folds for train/validation splitting to exhaustively use all of the data for validation, with the purpose of evaluating model and hyperparameter performance. You use the evaluation results to guide your model/hyperparameter selection.\n\n2. K-fold averaging - partition the data into k folds (using the same partitioning as in CV), train the same model on each fold, and average the results to help reduce the variance of a single model. You could add out of fold validation to this process without using it directly for selection, or you can just run with the result of the averaging if you feel good about the model/hyperparameter selection.\n\nThe reason the terminology gets confused is that many people take a shortcut and do both things at the same time. They do one k-fold split, use out of fold sets to select something like n_estimators for gradient boosting models with early stopping, then average the resulting models across the k train/validation runs. In theory, this is poor practice. Early stopping overfits to the particular choice of validation fold, so models averaged in this manner within CV inevitably overfit. In practice, the averaging helps mitigate some of the overfitting and the overall harm may be small, but it's something worth keeping in mind. It certainly causes your local score estimates to be at least slightly optimistic. \n\nThis is a common pitfall in use of CV more generally. People don't realize that by using certain validation sets to guide their model selection, they've reduced the legitimacy of using those same validation sets for estimating generalization error. The best estimates of generalization error are given by sets that have had no influence on model selection. This is why doing CV **and** setting aside a true hold-out set that only gets used once for evaluation is a common and sound practice. ", "votes": 26 }, { "id": 342652, "postDate": "2018-06-13T20:59:59.003Z", "content": "One practical reason why k-fold cross validation is famous around kagglers, is that you can do this approach for several layers, without needing a new hold-out set for each level. This is especially helpful when working together in a team or stacking different models.
", "rawMarkdown": "One practical reason why k-fold cross validation is famous around kagglers, is that you can do this approach for several layers, without needing a new hold-out set for each level. This is especially helpful when working together in a team or stacking different models.", "votes": 4 }, { "id": 342692, "postDate": "2018-06-13T22:43:44.870Z", "content": "@Joe Eddy: nice overview!
\n\n\n\n\nThis is why doing CV and setting aside a true hold-out set that only\n gets used once for evaluation is a common and sound practice.
\n
In this competition at least, I consider the public leaderboard to be my true hold-out set. I prefer to use K-Fold CV to generate my predictions, with K models each built on K-1 folds having their predictions averaged together. I see this as form of de facto model averaging.
\n\nI use early stopping. I agree that it overfits. But it also seems to me that each K-1 folds are best fit by a different # of epochs or trees, so guessing this # without early stopping is also misfitting (either underfitting or overfitting). So the misfitting washes out.
\n\nThe big bonus for me is using the out-of-fold predictions for model selection and to optimize model stacking.
", "rawMarkdown": "@Joe Eddy: nice overview!\n\n> This is why doing CV and setting aside a true hold-out set that only\n> gets used once for evaluation is a common and sound practice.\n\nIn this competition at least, I consider the public leaderboard to be my true hold-out set. I prefer to use K-Fold CV to generate my predictions, with K models each built on K-1 folds having their predictions averaged together. I see this as form of de facto model averaging. \n\nI use early stopping. I agree that it overfits. But it also seems to me that each K-1 folds are best fit by a different # of epochs or trees, so guessing this # without early stopping is also misfitting (either underfitting or overfitting). So the misfitting washes out.\n\nThe big bonus for me is using the out-of-fold predictions for model selection and to optimize model stacking.\n", "votes": 6 }, { "id": 342713, "postDate": "2018-06-14T00:00:36.640Z", "content": "I agree with @Harlan Seymour. The reason you stated is why I am using CV this way and I believe most people in this competition are as well.
", "rawMarkdown": "I agree with @Harlan Seymour. The reason you stated is why I am using CV this way and I believe most people in this competition are as well.\n", "votes": 2 }, { "id": 342722, "postDate": "2018-06-14T00:31:55.450Z", "content": "@Harlan and YaGana, I definitely agree it's very convenient and that overfitting is mostly mitigated. It works well enough to produce good submissions and stacking results. But it's also definitely not optimal. When stacking, generating out of fold predictions on the same folds you early stop on introduces an optimistic bias that your meta learner may overfit to (these predictions are no longer out of fold in a true sense of the model not having seen the data, it now has glimpsed at that data). If that optimistic bias exists across all the models you stack it likely washes out, but if some models were trained differently (e.g. a non-boosting model that doesn't have the same early stopping concept), it might be problematic for the meta learner.
\n\nThere's no reason you can't get the best of both worlds, at the cost of extra computation time. You can run a 5-fold CV for model selection, then run another 5-fold process on a different seed to generate true OOF predictions to stack with. You can take n_estimators as the mean of the 5-fold CV early-stopped n_estimators, or use a similar heuristic. I can't find it right now, but I believe there's a post somewhere where Laurae (LGBM contributor) recommends the heuristic 110% of mean early-stopped n_estimators.
", "rawMarkdown": "@Harlan and YaGana, I definitely agree it's very convenient and that overfitting is mostly mitigated. It works well enough to produce good submissions and stacking results. But it's also *definitely not optimal*. When stacking, generating out of fold predictions on the same folds you early stop on introduces an optimistic bias that your meta learner may overfit to (these predictions are no longer out of fold in a true sense of the model not having seen the data, it now has glimpsed at that data). If that optimistic bias exists across all the models you stack it likely washes out, but if some models were trained differently (e.g. a non-boosting model that doesn't have the same early stopping concept), it might be problematic for the meta learner.\n\nThere's no reason you can't get the best of both worlds, at the cost of extra computation time. You can run a 5-fold CV for model selection, then run another 5-fold process on a different seed to generate true OOF predictions to stack with. You can take n_estimators as the mean of the 5-fold CV early-stopped n_estimators, or use a similar heuristic. I can't find it right now, but I believe there's a post somewhere where Laurae (LGBM contributor) recommends the heuristic 110% of mean early-stopped n_estimators. ", "votes": 2 }, { "id": 342758, "postDate": "2018-06-14T03:16:38.033Z", "content": "@ Joe, you've definitely given me food for thought. Take for example XGBoost. What do you think of this compromise to save compute time?
\n\nNote that I might prefer something like 95% x M to 110% x M. Val score changes very little for the final trees before early stopping while train score goes down significantly i.e. lots of overfitting at the end before early stopping.
\n\nDoesn't this almost entirely mitigate your overfitting worry without having to do a double run? Note that XGBoost takes a long for me in this competition.
\n\n@Joe, I have another question it would be interesting to get your input on. Do you create a model on the entire training set, using in XGBoost, say, 110% x M x (K / (K-1)) for ntree_limit? This model would be used just for test predictions. Maybe average its test predictions with average of K fold test predictions?
", "rawMarkdown": "@ Joe, you've definitely given me food for thought. Take for example XGBoost. What do you think of this compromise to save compute time? \n \n1. Save models and record ntree_limit used in early stopping for all K folds\n2. Calculate mean of ntree_limit used in early stopping over the K folds (call it M)\n3. Reload models with < 110% x M trees and train them up to 110% x M\n4. Call predict() for each model to generate test and out-of-fold predictions, passing 110% x M for ntree_limit for each model.\n\nNote that I might prefer something like 95% x M to 110% x M. Val score changes very little for the final trees before early stopping while train score goes down significantly i.e. lots of overfitting at the end before early stopping.\n\nDoesn't this almost entirely mitigate your overfitting worry without having to do a double run? Note that XGBoost takes a long for me in this competition. \n\n@Joe, I have another question it would be interesting to get your input on. Do you create a model on the entire training set, using in XGBoost, say, 110% x M x (K / (K-1)) for ntree_limit? This model would be used just for test predictions. Maybe average its test predictions with average of K fold test predictions?", "votes": 1 }, { "id": 342763, "postDate": "2018-06-14T03:57:22.260Z", "content": "My personal preference is to not use early stopping. Use trial and error or some initial early stopping to get a certain number of rounds, and then use that number of rounds consistently with all the folds. Also remember to pay attention to the relationship between choice of learning rate and number of rounds needed.
", "rawMarkdown": "My personal preference is to not use early stopping. Use trial and error or some initial early stopping to get a certain number of rounds, and then use that number of rounds consistently with all the folds. Also remember to pay attention to the relationship between choice of learning rate and number of rounds needed.", "votes": 2 }, { "id": 343080, "postDate": "2018-06-14T17:06:36.897Z", "content": "@Joe Eddy
\n\nI agree with a lot of this but if we are talking about optimal then using a heuristic such as 110% of the mean of the early stopped n_estimators certainly isn't optimal.
\n\nP.S. Does anyone know why when I reply to, say, Joe Eddy, the forum thinks I am replying to the person he replied to?
\n\nP.P.S Some of this is covered here which is useful reading.
", "rawMarkdown": "@Joe Eddy\n\nI agree with a lot of this but if we are talking about *optimal* then using a heuristic such as 110% of the mean of the early stopped n_estimators certainly isn't optimal.\n\nP.S. Does anyone know why when I reply to, say, Joe Eddy, the forum thinks I am replying to the person he replied to?\n\nP.P.S Some of this is covered [here][1] which is useful reading.\n\n\n [1]: http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice/", "votes": 1 }, { "id": 343102, "postDate": "2018-06-14T17:37:55.190Z", "content": "@maw501 that's fair, there's no such this as optimality in statistical modeling of distributions that we can't have perfect knowledge of, everything is relative :)
\n\nBut on a practical level, I don't know what you would expect (both in the normal sense of expect, and on expectation in the statistical sense) to work better for generalization performance of n_estimators than something close to the mean of early stopped rounds. Here's a nice experiment in favor of using the mean to estimate the optimal number of rounds from olivier. Do you have in mind something that works better / what would your choice be?
\n\n@Harlan I like that idea a lot, it's not something I've thought of before. I can't think of a reason why it wouldn't mitigate the overfitting issue, sounds like a great time saver and worth a shot to me.
\n\nRe: your second question, I think you could definitely do that, but the gains might be pretty small if you're already using all of the data for training by averaging test predictions across k-folds. I tend to skip that to save time, but using all the training data in one model might be worth it. It aligns with the more traditional way of stacking - generate oof predictions in k-fold, retrain model on entire dataset to generate test predictions (without averaging of models). With relatively smaller datasets like this one my instinct is that averaging on k-folds gives a notable improvement via variance reduction, so that's what I go with.
", "rawMarkdown": "@maw501 that's fair, there's no such this as optimality in statistical modeling of distributions that we can't have perfect knowledge of, everything is relative :) \n\nBut on a practical level, I don't know what you would expect (both in the normal sense of expect, and *on expectation* in the statistical sense) to work better for generalization performance of n_estimators than something close to the mean of early stopped rounds. Here's a [nice experiment][1] in favor of using the mean to estimate the optimal number of rounds from olivier. Do you have in mind something that works better / what would your choice be?\n\n @Harlan I like that idea a lot, it's not something I've thought of before. I can't think of a reason why it wouldn't mitigate the overfitting issue, sounds like a great time saver and worth a shot to me. \n\nRe: your second question, I think you could definitely do that, but the gains might be pretty small if you're already using all of the data for training by averaging test predictions across k-folds. I tend to skip that to save time, but using all the training data in one model might be worth it. It aligns with the more traditional way of stacking - generate oof predictions in k-fold, retrain model on entire dataset to generate test predictions (without averaging of models). With relatively smaller datasets like this one my instinct is that averaging on k-folds gives a notable improvement via variance reduction, so that's what I go with.\n\n\n\n [1]: https://www.kaggle.com/ogrellier/can-early-stopping-generalize/notebook", "votes": 4 }, { "id": 343163, "postDate": "2018-06-14T20:01:47.250Z", "content": "+1 for variance reduction.
", "rawMarkdown": "+1 for variance reduction. " }, { "id": 343188, "postDate": "2018-06-14T21:07:04.853Z", "content": "I have a question about stacking here in Kaggle and for that competition:\nMy current way of doing it is:\n-For 5 folds, train 5 models then predict on the OOF to construct train meta data for that model.\n-Train a full model and create a test meta data.\n(then later on train a meta model etc.)
\n\nWhat I'm wondering is if it makes sense to instead use the average of the 5 models predictions on test data. The same way I (and others) currently test my single models on the LB.\nDoes it make sense to think that the test meta is more stable that way and closer to the train meta (on average) compared to have a different model that created test meta ?
", "rawMarkdown": "I have a question about stacking here in Kaggle and for that competition:\nMy current way of doing it is:\n-For 5 folds, train 5 models then predict on the OOF to construct train meta data for that model.\n-Train a full model and create a test meta data.\n(then later on train a meta model etc.)\n\nWhat I'm wondering is if it makes sense to instead use the average of the 5 models predictions on test data. The same way I (and others) currently test my single models on the LB.\nDoes it make sense to think that the test meta is more stable that way and closer to the train meta (on average) compared to have a different model that created test meta ?", "votes": 1 }, { "id": 343216, "postDate": "2018-06-14T22:30:54.153Z", "content": "Hi Joe,
\n\nAgain, thanks for a thoughtful reply. I definitely agree with your first comment re. optimality and this alongside scepticism is a reasonable grounding for most problems. I also think you somewhat answer your own point when asking for what I would expect to perform better when you point out the elusiveness of optimality. I don't expect any method to be consistently better than any other (for a reasonable choice of methods).
\n\nThis is more complicated clearly and I think a simple (self-evident) answer is that it's entirely problem dependent. I've yet to see any ML technique/approach (including state of the art) that one feels entirely comfortable applying in all contexts. My last two competitions I worried far too much about whether things were theoretically justifiable/sound and paid the price. Worrying about reliably estimating generalization error (which we care about in the real world) vs. performing well on a Kaggle competition vs. having robust statistical backing for one's ideas are all different things.
\n\nMy choice is usually to try most of the approaches listed with a (more recent) favouring to average across the predictions from the folds for the base models. The biggest problem with using a simple heuristic like 110% of the mean of the early stopped n_estimators is that given the extra data it feels a little too much like guesswork for me. Yes under some assumptions I'm sure it's fine, but what are they? Can we even state them?
\n\nFurther if you change from 5-fold to 10-fold to any k-fold - what then? Add in that one should perhaps favour higher entropy validation methods then you should accept some folds will be harder than others for your model and early stopping at least deals with this (of course, at some risk of overfitting) in a consistent manner - additionally you have the benefit of averaging too. If you have a diverse set of base models I don't particularly worry about any single model overfitting too much.
\n\nI do like the idea of using different random seeds for model iteration and oof predictions and will certainly try this.
\n\nEdit extra: Just wondering if we can resolve via a thought experiment. For instance, take an extreme 2-fold example with one really easy fold needing 500 rounds and the other really difficult needing 1500 - using the heuristic we would train on the whole data now with 1, 100 rounds. I'm not sure I'm happy with that vs. averaging the predictions from both which at least predicts at the point when it thinks it has discovered all the information from the data. I'm not saying the difference will be big, or one is consistently better but I like getting variance reduction for free and some comfort in knowing the number of rounds. I guess to some extent it's a function of the dissimilarity in the folds/strength of signal in each but I'm not sure how one would formalise this - so back to my point about not expecting any method to do better on average.
", "rawMarkdown": "Hi Joe,\n\nAgain, thanks for a thoughtful reply. I definitely agree with your first comment re. optimality and this alongside scepticism is a reasonable grounding for most problems. I also think you somewhat answer your own point when asking for what I would *expect* to perform better when you point out the elusiveness of optimality. I don't expect any method to be consistently better than any other (for a reasonable choice of methods).\n\nThis is more complicated clearly and I think a simple (self-evident) answer is that it's entirely problem dependent. I've yet to see any ML technique/approach (including state of the art) that one feels entirely comfortable applying in all contexts. My last two competitions I worried far too much about whether things were theoretically justifiable/sound and paid the price. Worrying about reliably estimating generalization error (which we care about in the real world) vs. performing well on a Kaggle competition vs. having robust statistical backing for one's ideas are all different things.\n\nMy choice is usually to try most of the approaches listed with a (more recent) favouring to average across the predictions from the folds for the base models. The biggest problem with using a simple heuristic like 110% of the mean of the early stopped n_estimators is that given the extra data it feels a little too much like guesswork for me. Yes under some assumptions I'm sure it's fine, but what are they? Can we even state them?\n\nFurther if you change from 5-fold to 10-fold to any **k**-fold - what then? Add in that one should perhaps favour higher entropy validation methods then you should accept some folds will be harder than others for your model and early stopping at least deals with this (of course, at some risk of overfitting) in a consistent manner - additionally you have the benefit of averaging too. If you have a diverse set of base models I don't particularly worry about any single model overfitting too much.\n\nI do like the idea of using different random seeds for model iteration and oof predictions and will certainly try this.\n\nEdit extra: Just wondering if we can resolve via a thought experiment. For instance, take an extreme 2-fold example with one really easy fold needing 500 rounds and the other really difficult needing 1500 - using the heuristic we would train on the whole data now with 1, 100 rounds. I'm not sure I'm happy with that vs. averaging the predictions from both which at least predicts at the point when it thinks it has discovered all the information from the data. I'm not saying the difference will be big, or one is consistently better but I like getting variance reduction for free and some comfort in knowing the number of rounds. I guess to some extent it's a function of the dissimilarity in the folds/strength of signal in each but I'm not sure how one would formalise this - so back to my point about not expecting any method to do better on average.", "votes": 1 }, { "id": 343670, "postDate": "2018-06-15T19:47:05.077Z", "content": "@Joe: As an experiment I applied the procedure I outlined to my latest XGB per-fold models, syncing them to all to be 100% of the mean of the early stopping ntree_limits. My local CV score went up minutely by +0.00001. The newly submitted test predictions had the same RMSE score on the public leaderboard, but ordering \"My Submissions\" by score showed it was not as good as the original.
", "rawMarkdown": "@Joe: As an experiment I applied the procedure I outlined to my latest XGB per-fold models, syncing them to all to be 100% of the mean of the early stopping ntree_limits. My local CV score went up minutely by +0.00001. The newly submitted test predictions had the same RMSE score on the public leaderboard, but ordering \"My Submissions\" by score showed it was not as good as the original.", "votes": 1 }, { "id": 343696, "postDate": "2018-06-15T20:54:37.490Z", "content": "Thanks @Joe Eddy for sparking this useful discussion. And thanks @Harlan Seymour for the update on your experiment.
\n\n@maw501, I would like to know the result of the thought experiment on testing on 2 extreme cases you mentioned. If you ever do that, please report back if you can.
\n\nAveraging is what I am comfortable with at the moment to reduce variance.
", "rawMarkdown": "Thanks @Joe Eddy for sparking this useful discussion. And thanks @Harlan Seymour for the update on your experiment. \n\n@maw501, I would like to know the result of the thought experiment on testing on 2 extreme cases you mentioned. If you ever do that, please report back if you can.\n\nAveraging is what I am comfortable with at the moment to reduce variance. ", "votes": 1 } ] }, { "id": 336590, "postDate": "2018-06-01T01:59:22.720Z", "content": "LGB 0.2223 5-fold average
\n\nupdate:\nnew feature add, 0.2217 without 5 folder
\n\n0.2210 with 5 folder
\n\nupdate 6-15:
\n\n0.2208 with 5 folder
\n\nupdate 6-20:
\n\n0.2205 with 5 folder
\n\nfinal:
\n\n0.2204 with 10 folder
", "rawMarkdown": "LGB 0.2223 5-fold average\n\n**update**:\nnew feature add, 0.2217 without 5 folder\n\n\n0.2210 with 5 folder\n\n\n**update** 6-15:\n\n0.2208 with 5 folder\n\n**update** 6-20:\n\n0.2205 with 5 folder\n\nfinal:\n\n0.2204 with 10 folder", "votes": 1, "replies": [ { "id": 336595, "postDate": "2018-06-01T02:17:59.517Z", "content": "劉sir, 您是用Stack跳到0.2210的麼?
", "rawMarkdown": "劉sir, 您是用Stack跳到0.2210的麼?", "votes": -2 }, { "id": 337169, "postDate": "2018-06-02T06:24:52.117Z", "content": "还没stacking，简单的blend几个public kernel
", "rawMarkdown": "还没stacking，简单的blend几个public kernel", "votes": -2 }, { "id": 337275, "postDate": "2018-06-02T12:38:43.800Z", "content": "Translation: \nLiu sir , are you jumping to 0.2210 with Stack?\n - No stacking, simple blend several public kernel
", "rawMarkdown": "Translation: \nLiu sir , are you jumping to 0.2210 with Stack?\n - No stacking, simple blend several public kernel", "votes": 9 }, { "id": 338644, "postDate": "2018-06-05T13:44:03.090Z", "content": "有啥技巧么，我blend一个0.2212和0.2216结果也就0.2212
", "rawMarkdown": "有啥技巧么，我blend一个0.2212和0.2216结果也就0.2212", "votes": -5 }, { "id": 338654, "postDate": "2018-06-05T14:17:22.973Z", "content": "I guess there is not any techniques. Just use weight to blend public with private kernel. I used blending, too
", "rawMarkdown": "I guess there is not any techniques. Just use weight to blend public with private kernel. I used blending, too" }, { "id": 339058, "postDate": "2018-06-06T08:01:20.183Z", "content": "you need try to improve your single model,then try to blend or stacking
", "rawMarkdown": "you need try to improve your single model,then try to blend or stacking" } ] }, { "id": 330553, "postDate": "2018-05-19T05:14:52.653Z", "content": "Edit : 0.2249
\n\nEdit: 0.2252 with Image_top_1 feature included
\n\nEdit: 0.2259 with RNN
\n\n0.2276 with RNN, Fast Text Embeddings . - No Tuning done, A lot of Scope for improvement and tuning. \nLink to Public kernel is here
\n\n0.2276 RNN - Good for Blend :)
", "rawMarkdown": "Edit : 0.2249\n\nEdit: 0.2252 with Image_top_1 feature included \n\nEdit: 0.2259 with RNN \n\n0.2276 with RNN, Fast Text Embeddings . - No Tuning done, A lot of Scope for improvement and tuning. \nLink to Public kernel is here \n\n[0.2276 RNN - Good for Blend :)][1]\n\n\n [1]: https://www.kaggle.com/shanth84/avito-fast-text-keras-model/code", "votes": 1 }, { "id": 329707, "postDate": "2018-05-17T02:38:18.170Z", "content": "0.2223 with 5fold LGB model (all features are taken from public kernels).
", "rawMarkdown": "0.2223 with 5fold LGB model (all features are taken from public kernels). ", "votes": 1 }, { "id": 329210, "postDate": "2018-05-16T02:38:27.313Z", "content": "Single lgbm 0.223.
", "rawMarkdown": "Single lgbm 0.223. ", "votes": 1 }, { "id": 329185, "postDate": "2018-05-16T00:01:24.690Z", "content": "Single lgbm 0.2215.\nBut using image features from https://www.kaggle.com/bguberfain/vgg16-train-features , my score got worse.\nI would like to know how to use image feature.
", "rawMarkdown": "Single lgbm 0.2215.\nBut using image features from https://www.kaggle.com/bguberfain/vgg16-train-features , my score got worse.\nI would like to know how to use image feature.", "votes": 1, "replies": [ { "id": 329196, "postDate": "2018-05-16T00:39:06.463Z", "content": "Wow, that is so high for a single lgbm model! Would you mind if share some ideas how to improve from 0.223 level to 0.2215 ?
", "rawMarkdown": "Wow, that is so high for a single lgbm model! Would you mind if share some ideas how to improve from 0.223 level to 0.2215 ?" }, { "id": 329209, "postDate": "2018-05-16T02:34:48.440Z", "content": "Price and item-seq-number is important feature. I refer to https://www.kaggle.com/tapioca/item-seq-number-vs-deal-probability/code , and make some features. Good luck!
", "rawMarkdown": "Price and item-seq-number is important feature. I refer to https://www.kaggle.com/tapioca/item-seq-number-vs-deal-probability/code , and make some features. Good luck!", "votes": 2 }, { "id": 329213, "postDate": "2018-05-16T02:43:55.210Z", "content": "Thanks! For the image feature, my thought is: the features from the pre-trained Imagenet model are designed for classification. For example, we want our VGG16 distinguish bicycle from car. However, we already have the features like item_id and category, so directly use imgenet features may not helps. Maybe we can try fine tune those features with the target for the deal probabilities. This is just my naive thought and I haven't try any image features
", "rawMarkdown": "Thanks! For the image feature, my thought is: the features from the pre-trained Imagenet model are designed for classification. For example, we want our VGG16 distinguish bicycle from car. However, we already have the features like item_id and category, so directly use imgenet features may not helps. Maybe we can try fine tune those features with the target for the deal probabilities. This is just my naive thought and I haven't try any image features", "votes": 1 }, { "id": 329214, "postDate": "2018-05-16T02:51:07.357Z", "content": "Wow! Your thought of image feature is very cool! I try to use color base feature because it is easy to use. Thank you!
", "rawMarkdown": "Wow! Your thought of image feature is very cool! I try to use color base feature because it is easy to use. Thank you!" }, { "id": 329451, "postDate": "2018-05-16T13:45:34.857Z", "content": "And also may ask how many models you are using to achieve your LB position now?
", "rawMarkdown": "And also may ask how many models you are using to achieve your LB position now?" }, { "id": 329466, "postDate": "2018-05-16T14:10:34.853Z", "content": "LB 0.2209 by single lgbm. Add some features from active.csv and periods data.
", "rawMarkdown": "LB 0.2209 by single lgbm. Add some features from active.csv and periods data.", "votes": 7 }, { "id": 329646, "postDate": "2018-05-16T21:35:57.487Z", "content": "@takuoko : item-seq-number seems to be exactly the same as doing a count after a group by user no ? I was able to get good features from the price, but not by combining item_seq_number with another feature... Thanks for sharing !
", "rawMarkdown": "@takuoko : item-seq-number seems to be exactly the same as doing a count after a group by user no ? I was able to get good features from the price, but not by combining item_seq_number with another feature... Thanks for sharing !" }, { "id": 329770, "postDate": "2018-05-17T06:39:04.460Z", "content": "Thanks for your comment, Antoine! By using item-seq-number feature, 0.00005 better val score in my 5 folds. So it may be ineffective. But I think item-seq-number is important. So I continue trying to make featrue about that.
", "rawMarkdown": "Thanks for your comment, Antoine! By using item-seq-number feature, 0.00005 better val score in my 5 folds. So it may be ineffective. But I think item-seq-number is important. So I continue trying to make featrue about that. ", "votes": 1 }, { "id": 329895, "postDate": "2018-05-17T13:36:33.833Z", "content": "simple feature from VGG16 is not helpful, because the prediction is not precise nor suitable. I have write a kernel https://www.kaggle.com/liuhdsgoal/about-image-top-1-is-a-classify-label
", "rawMarkdown": "simple feature from VGG16 is not helpful, because the prediction is not precise nor suitable. I have write a kernel https://www.kaggle.com/liuhdsgoal/about-image-top-1-is-a-classify-label" } ] }, { "id": 329122, "postDate": "2018-05-15T19:17:42.043Z", "content": "Single lgbm 0.2229
", "rawMarkdown": "Single lgbm 0.2229", "votes": 1 }, { "id": 329052, "postDate": "2018-05-15T16:02:51.120Z", "content": "LB 0.2249 with only sample train and test dataset via lightgbm
", "rawMarkdown": "LB 0.2249 with only sample train and test dataset via lightgbm", "votes": 1 }, { "id": 328924, "postDate": "2018-05-15T11:41:07.980Z", "content": "My best single NN is 0.2233 without image nor active and period files.
", "rawMarkdown": "My best single NN is 0.2233 without image nor active and period files.", "votes": 1, "replies": [ { "id": 329641, "postDate": "2018-05-16T21:24:57.023Z", "content": "Update:
\n0.2226 for NN
Could you give a hint about architecture of your NN?
", "rawMarkdown": "Could you give a hint about architecture of your NN?" }, { "id": 329745, "postDate": "2018-05-17T05:15:39.803Z", "content": "As @Strideradu commented above,
\n\n> So the NN is for the NLP of description or more like for the classifier for different features just like LGBM
\n\nI use both.
\n\nCategory + Numeric part
\nYou can refer from fastai or some solution in this competition or similar one: \nhttps://www.kaggle.com/c/donorschoose-application-screening
NLP
\nYou can refer from previous competitions like:
\nhttps://www.kaggle.com/c/donorschoose-application-screening
\nhttps://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge\n...\nto know how they dealed with NLP.
Thank you!
", "rawMarkdown": "Thank you!" } ] }, { "id": 336777, "postDate": "2018-06-01T09:02:02.143Z", "content": "NN:\n0.2213 (Categorical, Numerical, W2V)\n10-fold
", "rawMarkdown": "NN:\n0.2213 (Categorical, Numerical, W2V)\n10-fold", "votes": 2 }, { "id": 335073, "postDate": "2018-05-29T05:48:58.033Z", "content": "one LightGBM model 5kfold - 0,2211 (UPD)
", "rawMarkdown": "one LightGBM model 5kfold - 0,2211 (UPD)", "votes": 2, "replies": [ { "id": 335075, "postDate": "2018-05-29T05:55:18.377Z", "content": "incredible
", "rawMarkdown": "incredible" }, { "id": 335214, "postDate": "2018-05-29T11:41:39.043Z", "content": "Almost all ideas i take from public kernels :)
", "rawMarkdown": "Almost all ideas i take from public kernels :)", "votes": 1 }, { "id": 338415, "postDate": "2018-06-05T02:30:16.977Z", "content": "@AlexTru, did you use image features? If so which ones if you can share? They seem to worsen both my CV and LB.
", "rawMarkdown": "@AlexTru, did you use image features? If so which ones if you can share? They seem to worsen both my CV and LB." }, { "id": 338447, "postDate": "2018-06-05T04:28:42.777Z", "content": "Yes, i use features from this two kernels. this and this
", "rawMarkdown": "Yes, i use features from this two kernels. [this][1] and [this][2]\n\n\n [1]: https://www.kaggle.com/wesamelshamy/high-correlation-feature-image-classification-conf\n [2]: https://www.kaggle.com/peterhurford/image-feature-engineering" }, { "id": 339187, "postDate": "2018-06-06T13:30:03.850Z", "content": "UPD 22.07
", "rawMarkdown": "UPD 22.07", "votes": 1 } ] }, { "id": 331503, "postDate": "2018-05-21T12:22:52.577Z", "content": "Update: 0.2220 with variation of the 'ridge trick', 5fold lgbm.
", "rawMarkdown": "Update: 0.2220 with variation of the 'ridge trick', 5fold lgbm.", "votes": 2, "replies": [ { "id": 331505, "postDate": "2018-05-21T12:28:21.953Z", "content": "What's a \"ridge trick\"?
", "rawMarkdown": "What's a \"ridge trick\"?" }, { "id": 331511, "postDate": "2018-05-21T12:37:26.997Z", "content": "Creating an oof prediction from a ridge model and using it as a feature - along the lines of https://www.kaggle.com/him4318/avito-lightgbm-with-ridge-feature-v-2-0
", "rawMarkdown": "Creating an oof prediction from a ridge model and using it as a feature - along the lines of https://www.kaggle.com/him4318/avito-lightgbm-with-ridge-feature-v-2-0", "votes": 1 }, { "id": 331531, "postDate": "2018-05-21T13:05:26.643Z", "content": "Cool. But does that really count as a single model? Though I'd concede at this point I'm just being pedantic. \"Single model\" is a bit of a vanity metric because of things like this, and it's really just about finding models that are individually good but blend together.
\n\nMy LGB has Ridges in it so I wasn't posting about it here. But I'm currently at 0.21611 CV / 0.2200 LB with a single LGB with multiple Ridge models inside. The best individual Ridge is at 0.22674 CV.
", "rawMarkdown": "Cool. But does that really count as a single model? Though I'd concede at this point I'm just being pedantic. \"Single model\" is a bit of a vanity metric because of things like this, and it's really just about finding models that are individually good but blend together.\n\nMy LGB has Ridges in it so I wasn't posting about it here. But I'm currently at 0.21611 CV / 0.2200 LB with a single LGB with multiple Ridge models inside. The best individual Ridge is at 0.22674 CV.", "votes": 5 }, { "id": 331551, "postDate": "2018-05-21T13:30:21.487Z", "content": "Nice one - your gap cv - lb is a bit smaller than mine (~ 0.006). If you don't mind sharing, have you tuned the lgb a lot? I've been rolling mostly with parameters close to the ones from the public kernels so far.
", "rawMarkdown": "Nice one - your gap cv - lb is a bit smaller than mine (~ 0.006). If you don't mind sharing, have you tuned the lgb a lot? I've been rolling mostly with parameters close to the ones from the public kernels so far." }, { "id": 331556, "postDate": "2018-05-21T13:39:06.703Z", "content": "...I sure hope my smaller CV gap does not mean that my LB position will come crashing down at the end. :/
\n\nI have spent some amount of time tuning the LGB by hand to the point where I can no longer easily improve it. I still think I have some room to non-easily improve it, perhaps with Hyperopt or something.
", "rawMarkdown": "...I sure hope my smaller CV gap does not mean that my LB position will come crashing down at the end. :/\n\nI have spent some amount of time tuning the LGB by hand to the point where I can no longer easily improve it. I still think I have some room to non-easily improve it, perhaps with Hyperopt or something." }, { "id": 331561, "postDate": "2018-05-21T13:43:35.087Z", "content": "@Konrad: Are you seeing 0.216XX CV as well? Do you have a lot of other features beyond those in the kernels?
", "rawMarkdown": "@Konrad: Are you seeing 0.216XX CV as well? Do you have a lot of other features beyond those in the kernels?" }, { "id": 331596, "postDate": "2018-05-21T14:48:27.193Z", "content": "Yes and yes - with the latter, i went to town with target encoding. I guess tuning is my next step, as i have barely touched it.
", "rawMarkdown": "Yes and yes - with the latter, i went to town with target encoding. I guess tuning is my next step, as i have barely touched it.", "votes": 1 }, { "id": 331642, "postDate": "2018-05-21T16:14:54.077Z", "content": "Hey I have tried target encoding but it was not helping much as passing the categorical values into LGBM is doing. I think keeping the number of features high is helping the tree to build better.\nwhat do you think?
", "rawMarkdown": "Hey I have tried target encoding but it was not helping much as passing the categorical values into LGBM is doing. I think keeping the number of features high is helping the tree to build better.\nwhat do you think?", "votes": 1 }, { "id": 331665, "postDate": "2018-05-21T17:04:40.073Z", "content": "Interesting - I have exact opposite experience to yours: once I mean-encoded the categoricals (created new numerical cols) and then dropped the originals, my score improved significantly.
", "rawMarkdown": "Interesting - I have exact opposite experience to yours: once I mean-encoded the categoricals (created new numerical cols) and then dropped the originals, my score improved significantly. ", "votes": 3 }, { "id": 331669, "postDate": "2018-05-21T17:09:44.993Z", "content": "maybe I am doing somewhere wrong then. I tried using this
\nhttps://www.kaggle.com/tnarik/likelihood-encoding-of-categorical-features. I am encoding all the categorical values .
@konrad How do you mean encode? You may be at risk of overfitting your CV. When I did mean encoding I ended up in overfit hell for a few days and it took ten submits to get to the bottom of it. So I'm on team LGB native encoding. I might try doing a second LGB with mean encoding though and average the two approaches and see if that helps... I'll calculate it within the CV folds to monitor overfitting better.
", "rawMarkdown": "@konrad How do you mean encode? You may be at risk of overfitting your CV. When I did mean encoding I ended up in overfit hell for a few days and it took ten submits to get to the bottom of it. So I'm on team LGB native encoding. I might try doing a second LGB with mean encoding though and average the two approaches and see if that helps... I'll calculate it within the CV folds to monitor overfitting better.", "votes": 2 }, { "id": 331687, "postDate": "2018-05-21T17:35:05.907Z", "content": "yeah, my CV score was good enough but LB was worst.
", "rawMarkdown": "yeah, my CV score was good enough but LB was worst.", "votes": 2 }, { "id": 331695, "postDate": "2018-05-21T17:45:46.247Z", "content": "@Peter: I did the mean-encoding inside the cv and added heavy regularization, so I think I'm fine (especially that cv - lb is consistently around 0.006, and changes are directionally correct).
", "rawMarkdown": "@Peter: I did the mean-encoding inside the cv and added heavy regularization, so I think I'm fine (especially that cv - lb is consistently around 0.006, and changes are directionally correct).", "votes": 2 }, { "id": 333343, "postDate": "2018-05-25T00:31:58.560Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 340394, "postDate": "2018-06-09T04:46:18.230Z", "content": "10-fold LGB, LB 0.2214.
\n\nTraining a model including some image features. Hopefully it would give me a boost. Finger crossed.
\n\nUpdate: Including image featuers, 10-fold LGB, LB 0.2210
\n\n6-15 Update: manage to improve my LGB to LB 0.2202
\n\n6-19 Update: finally got an LGB with LB 0.2199, XGB with LB 0.2209.
", "rawMarkdown": "10-fold LGB, LB 0.2214.\n\nTraining a model including some image features. Hopefully it would give me a boost. Finger crossed. \n\nUpdate: Including image featuers, 10-fold LGB, LB 0.2210\n\n6-15 Update: manage to improve my LGB to LB 0.2202\n\n6-19 Update: finally got an LGB with LB 0.2199, XGB with LB 0.2209. ", "replies": [ { "id": 343000, "postDate": "2018-06-14T14:21:53.030Z", "content": "10-fold LGB avg or min ？
", "rawMarkdown": "10-fold LGB avg or min ？" }, { "id": 343002, "postDate": "2018-06-14T14:24:21.427Z", "content": "10-fold LGB avg.
", "rawMarkdown": "10-fold LGB avg." } ] }, { "id": 334596, "postDate": "2018-05-28T00:37:22.033Z", "content": "Right now I have 3 models in parallel. A catboost at 0.2237, a LGB at 0.2228 and a Neural network at 0.2240. I'm really impressed by people achieving 0.222X with neural network on that data set. Maybe I'm missing something fundamental or some kind of clever encoding of categoricals.
\n\nNot yet used additional FE from satellite data sets.
", "rawMarkdown": "Right now I have 3 models in parallel. A catboost at 0.2237, a LGB at 0.2228 and a Neural network at 0.2240. I'm really impressed by people achieving 0.222X with neural network on that data set. Maybe I'm missing something fundamental or some kind of clever encoding of categoricals.\n\nNot yet used additional FE from satellite data sets.", "replies": [ { "id": 334699, "postDate": "2018-05-28T08:32:27.917Z", "content": "As for me , I am just using simple label encoding for categorical data .
\n\nI think the NN architecture plays a key role and clever archtitecture can bring huge improvement.
\n\nI tend to rely on deep models just to avoid lot of features engeneering.
", "rawMarkdown": "As for me , I am just using simple label encoding for categorical data . \n\nI think the NN architecture plays a key role and clever archtitecture can bring huge improvement. \n\nI tend to rely on deep models just to avoid lot of features engeneering. ", "votes": 4 }, { "id": 334725, "postDate": "2018-05-28T09:30:27.170Z", "content": "Interesting.
\nIt is a surprise for me to know the simple label encoding for categorical data works well. I will give it a try.
\nThank for your comment
Thanks. I'll play around with some extra ideas. My architecture is fairly straightforward so far, ConvNN for texts, embeddings for categoricals then concat everything and pass it through dense layers.
\n\nMaybe I can try going deeper with the text part.
", "rawMarkdown": "Thanks. I'll play around with some extra ideas. My architecture is fairly straightforward so far, ConvNN for texts, embeddings for categoricals then concat everything and pass it through dense layers.\n\nMaybe I can try going deeper with the text part." }, { "id": 335870, "postDate": "2018-05-30T14:50:52.780Z", "content": "Hi @Serigne!\nYou really just encode cat features to numbers? or use one-hot encoding?
", "rawMarkdown": "Hi @Serigne!\nYou really just encode cat features to numbers? or use one-hot encoding? ", "votes": 1 }, { "id": 335874, "postDate": "2018-05-30T15:05:14.377Z", "content": "Hi Alex... Yes just simple encoding to numbers..Then embed--> concat--> dense layers just like Arroqc..
\n\nBut that's just for categorical features.
\n\nFor text features I try to follow these 4 principles : Embed, Encode, Attend and Predict
", "rawMarkdown": "Hi Alex... Yes just simple encoding to numbers..Then embed--> concat--> dense layers just like Arroqc..\n\nBut that's just for categorical features. \n\nFor text features I try to follow these 4 principles : [Embed, Encode, Attend and Predict][1]\n\n\n [1]: https://explosion.ai/blog/deep-learning-formula-nlp", "votes": 7 }, { "id": 335972, "postDate": "2018-05-30T19:51:50.463Z", "content": "Interesting article Serigne. Thanks. Will try to see if attention helps.
", "rawMarkdown": "Interesting article Serigne. Thanks. Will try to see if attention helps.", "votes": 1 } ] }, { "id": 334448, "postDate": "2018-05-27T12:26:31.887Z", "content": "I got 0.223 using just train.csv and test.csv,How to using train_active.csv which has no labels
", "rawMarkdown": "I got 0.223 using just train.csv and test.csv,How to using train_active.csv which has no labels", "replies": [ { "id": 334827, "postDate": "2018-05-28T14:17:12.837Z", "content": "@ unexceptednull
\n\nYou perform feature engineering with user_id, item_id and merge it to train using 'user_id'\nIt is available in Benjamin's awesome kernel here BENJAMIN FEATURE ENGINEERING WITH TRAIN ACTIVE
\n\nYou can also refer to this kernel for building RNN on these additional features RNN - 0.2246
\n\nHope this helps \nCheers\nShanth
", "rawMarkdown": "@ unexceptednull \n\nYou perform feature engineering with user_id, item_id and merge it to train using 'user_id'\nIt is available in Benjamin's awesome kernel here [BENJAMIN FEATURE ENGINEERING WITH TRAIN ACTIVE ][1]\n\nYou can also refer to this kernel for building RNN on these additional features [RNN - 0.2246][2]\n\nHope this helps \nCheers\nShanth\n\n [1]: https://www.kaggle.com/bminixhofer/aggregated-features-lightgbm/output\n [2]: https://www.kaggle.com/shanth84/rnn-detailed-explanation-0-2246" } ] }, { "id": 330469, "postDate": "2018-05-18T21:18:22.597Z", "content": "I've started with catboost and currently sits at 0.2246 with just train.csv. Edit: (Improved it to 0.2237)
\n\nI'm currently hoping to get further with LGB because of the ability to use sparse matrix.
", "rawMarkdown": "I've started with catboost and currently sits at 0.2246 with just train.csv. Edit: (Improved it to 0.2237)\n\nI'm currently hoping to get further with LGB because of the ability to use sparse matrix.\n\n" }, { "id": 329704, "postDate": "2018-05-17T02:28:57.377Z", "content": "0.2222 with one lgb model
", "rawMarkdown": "0.2222 with one lgb model" }, { "id": 329226, "postDate": "2018-05-16T03:34:41.703Z", "content": "0.2228 with one lgb
", "rawMarkdown": "0.2228 with one lgb" }, { "id": 329137, "postDate": "2018-05-15T20:13:42.170Z", "content": "LGBM with 5-fold CV 0.2231
", "rawMarkdown": "LGBM with 5-fold CV 0.2231", "replies": [ { "id": 329166, "postDate": "2018-05-15T22:25:15.630Z", "content": "Can I know which score for one fold? Is 5CV better than one fold only?
", "rawMarkdown": "Can I know which score for one fold? Is 5CV better than one fold only?" }, { "id": 329168, "postDate": "2018-05-15T22:30:51.207Z", "content": "In my experiment, it boosts about 0.001 level I believed, but I did this when my LB score is around 0.2250
", "rawMarkdown": "In my experiment, it boosts about 0.001 level I believed, but I did this when my LB score is around 0.2250", "votes": 1 }, { "id": 332664, "postDate": "2018-05-23T14:41:55.867Z", "content": "@Strideradu, I was not able to run 5fold CV with Lightgbm, I'm having an error :\nValueError: Supported target types are: ('binary', 'multiclass'). Got 'continuous' instead.
\n\nI'm using the following code :
\n\nlgb_clf = lgb.cv(\n lgbm_params, \n training_data, \n num_boost_round=1000, \n nfold=5,\n verbose_eval=100,\n early_stopping_rounds=50)
\n\nDid it work for you ?\nThanks!
", "rawMarkdown": "@Strideradu, I was not able to run 5fold CV with Lightgbm, I'm having an error :\nValueError: Supported target types are: ('binary', 'multiclass'). Got 'continuous' instead.\n\nI'm using the following code :\n\nlgb_clf = lgb.cv(\n lgbm_params, \n training_data, \n num_boost_round=1000, \n nfold=5,\n verbose_eval=100,\n early_stopping_rounds=50)\n\nDid it work for you ?\nThanks!" }, { "id": 332668, "postDate": "2018-05-23T14:49:33.353Z", "content": "I didn't use CV from lightgbm I use the KFold from sklearn to generate train and validation for each fold then just run fit for each fold data
", "rawMarkdown": "I didn't use CV from lightgbm I use the KFold from sklearn to generate train and validation for each fold then just run fit for each fold data", "votes": 2 }, { "id": 332671, "postDate": "2018-05-23T14:53:20.553Z", "content": "Ok thanks a lot !
", "rawMarkdown": "Ok thanks a lot !" }, { "id": 332778, "postDate": "2018-05-23T18:25:35.923Z", "content": "I got the same error when I used Stratified Kfold by mistake. A regular KFold did the trick
", "rawMarkdown": "I got the same error when I used Stratified Kfold by mistake. A regular KFold did the trick" }, { "id": 332782, "postDate": "2018-05-23T18:30:05.807Z", "content": "@ Strideradu
\n\nI used a 3 Fold in my RNN and got only a 0.0002 boost. Here's what I do
\n\nIs there a better way to do it? Any improvements you can suggest?
", "rawMarkdown": "@ Strideradu \n\nI used a 3 Fold in my RNN and got only a 0.0002 boost. Here's what I do \n\n - For each Kfold initialize a new RNN model and train it. \n - Make predictions for each fold using this model. Delete the model and move to next Fold \n - A simple average of all predictions \n\nIs there a better way to do it? Any improvements you can suggest?" }, { "id": 332787, "postDate": "2018-05-23T18:39:50.697Z", "content": "@Antoine - \nWhat are your lgb_params?
", "rawMarkdown": "@Antoine - \nWhat are your lgb_params?" }, { "id": 332791, "postDate": "2018-05-23T18:52:05.387Z", "content": "@AmirH, I use 0.2 learning rate, 250 num_leaves. Increasing the number of leaves clearly helped a lot...
\n\n@Shanth, I did use a normal Kfold... with the code above and it failed:\nkf = KFold(X.shape[0], n_folds=5, random_state=None, shuffle=False)
\n\nI'm trying to implement it as @strideradu advised, after doing .tocsr() on my sparse Matrix.
\n\nThanks for the help
", "rawMarkdown": "@AmirH, I use 0.2 learning rate, 250 num_leaves. Increasing the number of leaves clearly helped a lot...\n\n@Shanth, I did use a normal Kfold... with the code above and it failed:\nkf = KFold(X.shape[0], n_folds=5, random_state=None, shuffle=False)\n\nI'm trying to implement it as @strideradu advised, after doing .tocsr() on my sparse Matrix.\n\nThanks for the help" }, { "id": 332807, "postDate": "2018-05-23T19:37:05.640Z", "content": "If you want to use stratified Kfold then you need to convert the probabilities to the binned result, some people just round all probability to 0 and 1, and here you can find code for binning https://www.kaggle.com/c/avito-demand-prediction/discussion/56346#330580
", "rawMarkdown": "If you want to use stratified Kfold then you need to convert the probabilities to the binned result, some people just round all probability to 0 and 1, and here you can find code for binning https://www.kaggle.com/c/avito-demand-prediction/discussion/56346#330580", "votes": 1 }, { "id": 332820, "postDate": "2018-05-23T20:32:55.627Z", "content": "@Antoine - make sure your lgb_params have : objective=\"regression\" or something of that kind, and metric = \"rmse\".\nI successfully run lgb.cv with these parameters
", "rawMarkdown": "@Antoine - make sure your lgb_params have : objective=\"regression\" or something of that kind, and metric = \"rmse\".\nI successfully run lgb.cv with these parameters" }, { "id": 332827, "postDate": "2018-05-23T20:57:28.390Z", "content": "Yes that's what I'm using too
\n\nlgbm_params = {\n 'boosting_type': 'gbdt',\n 'objective': 'regression',\n 'metric': 'rmse',
\n\nNot sure what I did wrong...
", "rawMarkdown": "Yes that's what I'm using too\n\nlgbm_params = {\n 'boosting_type': 'gbdt',\n 'objective': 'regression',\n 'metric': 'rmse',\n\nNot sure what I did wrong..." } ] }, { "id": 334810, "postDate": "2018-05-28T13:27:01.550Z", "rawMarkdown": "", "isDeleted": true }, { "id": 330081, "postDate": "2018-05-18T02:21:52.027Z", "rawMarkdown": "", "isDeleted": true }, { "id": 333907, "postDate": "2018-05-26T02:34:02.520Z", "content": "thank you
", "rawMarkdown": "thank you" } ], "comments": [ { "id": 340360, "author_name": "Liu Jilong", "author_url": "", "post_date": "2018-06-09T02:28:20.813000", "content": "I got 0.2181 with a NN network with 10 folds (UPDATED)
", "votes": 17, "replies": [ { "id": 340367, "author_name": "takuoko", "author_url": "", "post_date": "2018-06-09T02:45:00.490000", "content": "Wow, that's great!\nIs the feature amount effective? Or is the network special?
", "votes": 0, "replies": [] }, { "id": 340372, "author_name": "Johnny Liu", "author_url": "", "post_date": "2018-06-09T03:01:40.660000", "content": "Wow,can I ask that using MLP or RNN or CNN?
", "votes": 0, "replies": [] }, { "id": 340378, "author_name": "Liu Jilong", "author_url": "", "post_date": "2018-06-09T03:23:17.113000", "content": "Nothing special. I use four kind features: continous (contains ImageNet score), category, text with embedding, raw image pixel. Four submodels for four kinds feature, concatenated to one tensor, serval Dense Layers. Heavily use BatchNormalization and Dropout. I believe there still have some room to improve because I rarely did feature engineering.
", "votes": 9, "replies": [] }, { "id": 340381, "author_name": "takuoko", "author_url": "", "post_date": "2018-06-09T03:32:07.747000", "content": "Thank you for sharing!\nMy nn is still 0.2222(TT) I will do my best from here!
", "votes": 0, "replies": [] }, { "id": 340382, "author_name": "Liu Jilong", "author_url": "", "post_date": "2018-06-09T03:42:58.173000", "content": "My LGB is still 0.2205 and I have no idea how to improve it. Poor feature engineer skills.
", "votes": 3, "replies": [] }, { "id": 340388, "author_name": "SubikashPal", "author_url": "", "post_date": "2018-06-09T04:12:33.540000", "content": "Wah!!! Great.. R u using self-embedding ? \"Raw image pixel\" - is it like using the raw pixel value of the image..How r u using it in the NN model?
", "votes": 0, "replies": [] }, { "id": 340389, "author_name": "Totoro (ごめん)", "author_url": "", "post_date": "2018-06-09T04:13:28.803000", "content": "Congrats @Liu Jilong,
\n\n\n\n\nraw image pixel
\n
Do you mean the raw image without through any pretrained model?
\nIf yes, that is impressive. How long does it take to train?
Thanks for sharing, I have confused that you train four model for for four kinds feature,then a final model based on the four predictions?
", "votes": 0, "replies": [] }, { "id": 340400, "author_name": "Liu Jilong", "author_url": "", "post_date": "2018-06-09T04:58:48.003000", "content": "Just like any other image network, serval Conv2D layers.
", "votes": 0, "replies": [] }, { "id": 340401, "author_name": "Liu Jilong", "author_url": "", "post_date": "2018-06-09T04:59:20.403000", "content": "UPDATED: about 15min per epoch, 2.5h per fold.
", "votes": 1, "replies": [] }, { "id": 340403, "author_name": "SubikashPal", "author_url": "", "post_date": "2018-06-09T05:08:52.107000", "content": "Thanks Liu... Any insight on text pre-processing / embedding?
", "votes": 0, "replies": [] }, { "id": 340405, "author_name": "Liu Jilong", "author_url": "", "post_date": "2018-06-09T05:19:17.890000", "content": "Just as many kagglers shared, I use self-trained word2vec embedding. I have no idea about russian, so no text pre-processing is used.
", "votes": 0, "replies": [] }, { "id": 340407, "author_name": "Hikkiiiiiiiii", "author_url": "", "post_date": "2018-06-09T05:26:39.460000", "content": "Thank you for sharing.
\n\nMy situation is the opposite of you. I just started trying NN today, and it didn't look good.
\n\nVery valuable experience, thanks :).
", "votes": 0, "replies": [] }, { "id": 340408, "author_name": "Liu Jilong", "author_url": "", "post_date": "2018-06-09T05:31:06.123000", "content": "Many tricks to make NN work well. Good luck.
", "votes": 1, "replies": [] }, { "id": 340410, "author_name": "SubikashPal", "author_url": "", "post_date": "2018-06-09T05:32:25.727000", "content": "Thanks. Appreciate your help...
", "votes": 0, "replies": [] }, { "id": 340430, "author_name": "Johnny Liu", "author_url": "", "post_date": "2018-06-09T06:21:50.623000", "content": "Thanks. wish to see experience sharing for NN model after this competition.I have no idea to train a good NN model.My NN model is poor score.
", "votes": 0, "replies": [] }, { "id": 340688, "author_name": "Webber", "author_url": "", "post_date": "2018-06-10T02:41:29.237000", "content": "Thanks for your sharing. I have a memory problem of training a CNN on images. Since I don't have enough memory to load all images, currently I use a generator to read images batch by batch from disk and feed them to the CNN, which makes the training process very slow (1 hour per epoch). Is there any better way to make it faster?
", "votes": 0, "replies": [] }, { "id": 340691, "author_name": "Liu Jilong", "author_url": "", "post_date": "2018-06-10T03:20:04.987000", "content": "I also tried fit_generator, but fit_generator's result is worse than fit, which is strange and I did not find the reason. As for time-consuming problem, you can try fit_generator's multi-processing parameter.
", "votes": 0, "replies": [] }, { "id": 340699, "author_name": "Strideradu", "author_url": "", "post_date": "2018-06-10T03:47:12.887000", "content": "Wondering 15min/epoch for what kind of GPU?
", "votes": 0, "replies": [] }, { "id": 340702, "author_name": "Liu Jilong", "author_url": "", "post_date": "2018-06-10T04:02:07.473000", "content": "gtx1070
", "votes": 0, "replies": [] }, { "id": 341040, "author_name": "YoungLamb", "author_url": "", "post_date": "2018-06-10T21:38:04.990000", "content": "Hi Liu,\nThanks for your sharing, it's very encouraging. I wonder if you did not use fit_generator, how can you load 1300k images to memory? Thanks!
", "votes": 0, "replies": [] }, { "id": 341651, "author_name": "Endi Niu", "author_url": "", "post_date": "2018-06-12T03:03:38.347000", "content": "Hi Jilong, is your NN model end2end training? Did you use keras concatenate four kinds feature layers? \nThx for your sharing.
", "votes": 0, "replies": [] }, { "id": 342079, "author_name": "To Train Them Is My Cause", "author_url": "", "post_date": "2018-06-12T19:46:52.283000", "content": "\n\n\nI also tried fit_generator, but fit_generator's result is worse than fit, which is strange and I did not find the reason. As for time-consuming problem, you can try fit_generator's multi-processing parameter.
\n
Fit automatically does a few things that are critical for NNs ( such as randomizing input per epoch) that you have to explicitly do with your own fit_generator. Fit_generator automatically should load info on a new thread- so unless you have something crazy expensive in your generator (like a garbage collecting call) it shouldn't be a bottleneck at all-loading is faster than training.
", "votes": 3, "replies": [] }, { "id": 343689, "author_name": "ryches", "author_url": "", "post_date": "2018-06-15T20:30:19.343000", "content": "I've had several issues with using the image data. I think one of the most major ones is that it turns out keras cannot do multiprocessing on windows so it isnt actually loading in images properly beforehand. Second hand is I am trying to do the image resize and padding in line. This ends up costing roughly .05 per image. I worked a little around this using concurrent. Futures, but a full epoch still takes much much more than 15 minutes. Closer to 2-3 hours.. One avenue I spent some time researching is rewriting the images as a hdf5 dataset, but I'm unsure how to load it in alongside my other features, but that has various issues as well regarding shuffling and getting everything aligned
", "votes": 0, "replies": [] }, { "id": 344203, "author_name": "Liu Jilong", "author_url": "", "post_date": "2018-06-17T07:59:08.747000", "content": "Train with image pixels end to end is indeed tough, considering the million count. I did some trick to make it works, also i encountered some strange questions. I will share my workflow after competition ends and hoping for answers.
", "votes": 1, "replies": [] }, { "id": 344290, "author_name": "jetou Xu", "author_url": "", "post_date": "2018-06-17T14:33:53.243000", "content": "thank you! I am looking forward to your model
", "votes": 0, "replies": [] }, { "id": 344749, "author_name": "To Train Them Is My Cause", "author_url": "", "post_date": "2018-06-18T16:10:00.313000", "content": "\n\n\nI think one of the most major ones is that it turns out Keras cannot do multiprocessing on Windows
\n
This is interesting. Can you do your own threading process? maybe something with threading and queue like:
\n\n def data_loader(q,):\n for start in range(0, len(train_imgs), batch_size):\n #do your batch creation code here \n q.put([feat_batch, y_batch])\n q = queue.Queue(maxsize=q_size)\n t1 = threading.Thread(target=data_loader, name='DataLoader', args=(q,))\n t1.start()\n t1.join()\nAnd then just read the generator batches from the queue? I can't really check ( I'm running Linux atm).\nIf that's not possible, I recommend Linux...
", "votes": 2, "replies": [] }, { "id": 344921, "author_name": "ryches", "author_url": "", "post_date": "2018-06-18T23:26:44.383000", "content": "I really should move over to linux at some point, but I am eventually planning on dedicating one system to work like this and the other for games and less demanding functions so dont want to dual boot right away. That being said, I have found some way to kind of work around this using the concurrent.futures builtin python to load in the images with threading and I also preprocessed the images with padding and downscaling so I dont have to do that function within the generator.
", "votes": 1, "replies": [] }, { "id": 345074, "author_name": "Ee Kin Chin", "author_url": "", "post_date": "2018-06-19T06:41:18.443000", "content": "Hi Jilong, In my efforts to include text + image + tabular features, I find that it heavily overfits even before it finishes one epoch, does heavy use of batch normalization help to tackle overfitting? Thanks
", "votes": 0, "replies": [] }, { "id": 345152, "author_name": "Liu Jilong", "author_url": "", "post_date": "2018-06-19T09:28:40.913000", "content": "Yes, i heavily use BN and dropout, you may try add a BN before every Dense Layer.
", "votes": 1, "replies": [] }, { "id": 345417, "author_name": "Rajesh Shreedhar", "author_url": "", "post_date": "2018-06-19T20:57:11.850000", "content": "@Liu Jilong, there are few rows in test set where images are missing, how are you handling those when you say you are using imagent features?
", "votes": 0, "replies": [] }, { "id": 345512, "author_name": "Liu Jilong", "author_url": "", "post_date": "2018-06-20T01:14:15.283000", "content": "I just give them an all-black image for row image pixels and zero score for image net scores. It may have some approaches to improve, but i do not have enough time to handle this.
", "votes": 1, "replies": [] }, { "id": 346569, "author_name": "ryches", "author_url": "", "post_date": "2018-06-22T01:17:55.993000", "content": "Have you tried training a model purely on images? Do you know roughly what training or validation loss that can get down to? I have been working on this for a while and cant seem to get much out of the images.
", "votes": 0, "replies": [] }, { "id": 346622, "author_name": "AlexTru", "author_url": "", "post_date": "2018-06-22T04:09:39.683000", "content": "I tried several CNN models to learn only on images to predict deal probably. I used relu on last several dense layer, and I found out last dense layer (before Dense (1)) predict all zeros, which give me 0.133 rmse loss...
", "votes": 1, "replies": [] }, { "id": 346640, "author_name": "ryches", "author_url": "", "post_date": "2018-06-22T04:56:09.560000", "content": "Mine are consistently getting to .066ish MSE but dont seem to contribute anything to my combined model. Havent inspected what the outputs or the weights areFor a while I thought there was something wrong with my generator but it seems to work fine without the images just fine so dont think that is the issues
", "votes": 0, "replies": [] } ] }, { "id": 338921, "author_name": "Joe Eddy", "author_url": "", "post_date": "2018-06-06T01:33:30.123000", "content": "I'm hitting .2190 with a 5-fold averaged LGBM
", "votes": 9, "replies": [ { "id": 338925, "author_name": "Liu Jilong", "author_url": "", "post_date": "2018-06-06T01:49:48.870000", "content": "That's impressive, Congratulation!
", "votes": 1, "replies": [] }, { "id": 338928, "author_name": "Johnny Liu", "author_url": "", "post_date": "2018-06-06T01:55:08.110000", "content": "Wow,you must find some amazing features~
", "votes": 1, "replies": [] }, { "id": 338929, "author_name": "Marcus Lin", "author_url": "", "post_date": "2018-06-06T01:57:05.233000", "content": "Seems like there is a magic feature for the big jump.
", "votes": 1, "replies": [] }, { "id": 338930, "author_name": "Joe Eddy", "author_url": "", "post_date": "2018-06-06T02:09:09.757000", "content": "Thanks! :)
\n\nI'm using 100+ engineered tabular features, but there are definitely some that have a particularly substantial contribution.
", "votes": 1, "replies": [] }, { "id": 338939, "author_name": "senkin13", "author_url": "", "post_date": "2018-06-06T02:34:12.020000", "content": "good job,did you use target encoding?
", "votes": 1, "replies": [] }, { "id": 339017, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2018-06-06T06:43:56.893000", "content": "Wow, do you mean you didn't use text features at all?
", "votes": 0, "replies": [] }, { "id": 339162, "author_name": "Joe Eddy", "author_url": "", "post_date": "2018-06-06T12:37:55.033000", "content": "Sorry, I do use text features in addition to the tabular ones - could have been more clear
", "votes": 3, "replies": [] }, { "id": 339181, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2018-06-06T13:13:56.817000", "content": "Thank you a lot for a clarification!
", "votes": 2, "replies": [] }, { "id": 340151, "author_name": "gufra", "author_url": "", "post_date": "2018-06-08T13:24:03.263000", "content": "great work! did you use the train_active csv ?
", "votes": 0, "replies": [] } ] }, { "id": 333988, "author_name": "MPWARE", "author_url": "", "post_date": "2018-05-26T08:13:34.680000", "content": "LGB (Categorical, Numerical, TFIDF, Images meta-data + ImageNet + NIMA scoring): 0.2200 (updated)
\n\nXGB (Categorical, Numerical, TFIDF, Images meta-data + ImageNet + NIMA scoring): 0.2216 (updated)
\n\nNN (Categorical, Numerical, W2V, Images meta-data + ImageNet scoring): 0.2198 (updated)
\n\nAll with CV5.
", "votes": 8, "replies": [ { "id": 334098, "author_name": "Strideradu", "author_url": "", "post_date": "2018-05-26T14:27:42.823000", "content": "may i ask what is image meta?
", "votes": 0, "replies": [] }, { "id": 334132, "author_name": "Sohaib Omar", "author_url": "", "post_date": "2018-05-26T16:01:46.743000", "content": "What numerical features are working for you?
", "votes": 0, "replies": [] }, { "id": 334182, "author_name": "MPWARE", "author_url": "", "post_date": "2018-05-26T17:42:30.647000", "content": "Images meta-data is width, height. I've also included the average of VGG16, VGG19, InceptionV3, Xception, ResNet50 ImageNet scoring for each image but it does not really help. It only improves by 0.0002. And for numerical, only the basic ones, log1p(item_seq_number) and log1p(price). I'm trying different target-encoding but currently, as already reported below, it overfits too much.
", "votes": 8, "replies": [] }, { "id": 336600, "author_name": "Daniel S.", "author_url": "", "post_date": "2018-06-01T02:28:34.793000", "content": "does NIMA stand for Neural Image Assessment?
", "votes": 0, "replies": [] }, { "id": 336711, "author_name": "MPWARE", "author_url": "", "post_date": "2018-06-01T06:40:25.510000", "content": "Yes, I used this implementation:\nhttps://github.com/titu1994/neural-image-assessment
\n\nIt improves LGB LB by 0.0002 and XGB by 0.0003 (only). I included mean/std as features from MobileNet and InceptionResNetv2. It does not improve NN (which should mean the NLP part is better).
", "votes": 0, "replies": [] } ] }, { "id": 344107, "author_name": "senkin13", "author_url": "", "post_date": "2018-06-17T02:10:41.900000", "content": "LGB 10-fold 0.2179
", "votes": 5, "replies": [ { "id": 344108, "author_name": "Siyuan Dang", "author_url": "", "post_date": "2018-06-17T02:14:32.873000", "content": "Excellent achieve！Can you show some black magic you used？
", "votes": 0, "replies": [] }, { "id": 344128, "author_name": "senkin13", "author_url": "", "post_date": "2018-06-17T03:33:02.557000", "content": "no magic,features are almost from kernels like aggregations,images,tfidf,tsvd,impute...just try to use more features.
", "votes": 1, "replies": [] }, { "id": 344129, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-06-17T03:36:12.897000", "content": "@senkin13: Is that a CV score or a LB score? Does your model contain a Ridge?
", "votes": 0, "replies": [] }, { "id": 344130, "author_name": "senkin13", "author_url": "", "post_date": "2018-06-17T03:39:13.767000", "content": "LB score without ridge,I prefer to use ridge for stacking instead of lgb features.
", "votes": 1, "replies": [] }, { "id": 344132, "author_name": "Liu Jilong", "author_url": "", "post_date": "2018-06-17T03:52:27.573000", "content": "What do you mean by tsvd and impute? Any reference kernels or information?
", "votes": 0, "replies": [] }, { "id": 344134, "author_name": "senkin13", "author_url": "", "post_date": "2018-06-17T04:07:04.590000", "content": "here: https://www.kaggle.com/krithi07/baseline-model-with-new-features
", "votes": 4, "replies": [] }, { "id": 344146, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-06-17T04:52:48.563000", "content": "Congrats @Senkin13, which image features did you use. The ones I've got makes my scores worse... both CV and LB.
", "votes": 0, "replies": [] }, { "id": 344172, "author_name": "senkin13", "author_url": "", "post_date": "2018-06-17T06:21:42.553000", "content": "A few simple image meta features,not too much improvement compared to Peter Hurford(0.002 improvment)
", "votes": 0, "replies": [] }, { "id": 344178, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-06-17T06:47:15.090000", "content": "I know you called my improvement from images impressive, but I'm not coming anywhere close to 0.2179 single model LGB... Hey, I wonder what would happen if we teamed up and pooled our feature engineering?
", "votes": 0, "replies": [] }, { "id": 344179, "author_name": "Johnny Liu", "author_url": "", "post_date": "2018-06-17T06:52:11.893000", "content": "Hello, I have tried tsvd to tfidf of description and title,but my score became to worse,I choose 32 component tsvd.
", "votes": 1, "replies": [] }, { "id": 344185, "author_name": "Johnny Liu", "author_url": "", "post_date": "2018-06-17T07:13:08.823000", "content": "Maybe I dont choose the right n_ component
", "votes": 0, "replies": [] }, { "id": 344187, "author_name": "senkin13", "author_url": "", "post_date": "2018-06-17T07:14:05.520000", "content": "@Peter Hurford Your proposal is attractive ,there still have 4 days ,let's wait and see.
", "votes": 0, "replies": [] }, { "id": 344188, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2018-06-17T07:15:09.567000", "content": "@Johnny You can always check sum of explained_variance_ratio_. If it is very low, it means you not only lose noise but also information
", "votes": 1, "replies": [] }, { "id": 344189, "author_name": "senkin13", "author_url": "", "post_date": "2018-06-17T07:16:47.913000", "content": "@Johnny Liu I only get a little improvement from tsvd,yes should try different n_component and tfidf parameters.
", "votes": 1, "replies": [] }, { "id": 344191, "author_name": "Cheng", "author_url": "", "post_date": "2018-06-17T07:18:52.707000", "content": "very solid work!
", "votes": 0, "replies": [] }, { "id": 344201, "author_name": "Johnny Liu", "author_url": "", "post_date": "2018-06-17T07:47:13.687000", "content": "thanks @AhmetErdem @senkin13, now I sum explained_variance_ratio_ to find the best component
", "votes": 0, "replies": [] } ] }, { "id": 338049, "author_name": "Andrii Gogniat", "author_url": "", "post_date": "2018-06-04T10:21:58.890000", "content": "LGBM 10-fold avg. - 0.2203 (categorical, numerical, target encoding, text and features predicted using all csv data)
", "votes": 5, "replies": [ { "id": 338078, "author_name": "Johnny Liu", "author_url": "", "post_date": "2018-06-04T11:43:39.543000", "content": "Hello, does target encoding contribute your score a lot?
", "votes": 0, "replies": [] }, { "id": 338083, "author_name": "Larry Freeman", "author_url": "", "post_date": "2018-06-04T12:00:17.947000", "content": "When you say \"features predicted using all csv data\", does that mean that you found some interesting features leveraging train_active/test_active/periods_train/periods_test beyond the ones identified by Benjamin Minixhofer?
", "votes": 1, "replies": [] }, { "id": 338100, "author_name": "Andrii Gogniat", "author_url": "", "post_date": "2018-06-04T12:28:06.580000", "content": "Previous lgbm was at 0.2215 but TE was not the only features added so it is hard to say how much TE contributed to score. Still half of features in top-100 are TEs.
", "votes": 2, "replies": [] }, { "id": 338104, "author_name": "Andrii Gogniat", "author_url": "", "post_date": "2018-06-04T12:32:30.757000", "content": "@Larry Currently I am using only 'times up' and 'days up' predictions, but I think it is worth trying to fill other missing values by predicting them from all csv data.
", "votes": 0, "replies": [] }, { "id": 338111, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2018-06-04T12:48:16.607000", "content": "@Andrii Do you mean you trained a model to estimate 'times up' and 'days up' from the other features? Or you just used the features described in Benjamin's kernel?
", "votes": 0, "replies": [] }, { "id": 338116, "author_name": "Andrii Gogniat", "author_url": "", "post_date": "2018-06-04T12:59:20.747000", "content": "@Dmitriy I am using both - averages from kernel and model predictions.
", "votes": 2, "replies": [] }, { "id": 338427, "author_name": "Johnny Liu", "author_url": "", "post_date": "2018-06-05T03:24:17.920000", "content": "wow, thanks for your replying.I have not try to using target encoding now
", "votes": 0, "replies": [] } ] }, { "id": 333337, "author_name": "cab", "author_url": "", "post_date": "2018-05-24T23:45:27.317000", "content": "NN:
\n0.2218 Fasttext
\n0.2215 Self-training embedding
Thanks @Dieter
", "votes": 5, "replies": [ { "id": 333362, "author_name": "Strideradu", "author_url": "", "post_date": "2018-05-25T02:50:23.843000", "content": "Wow your fasttext result is great. Are u using the crawl one or wiki one? Did you play a lot with different NN structures?
", "votes": 0, "replies": [] }, { "id": 333366, "author_name": "cab", "author_url": "", "post_date": "2018-05-25T03:09:20.937000", "content": "I use wiki 300D.
\nI tried some NLP parts such as: RNN, Bi-GRU, Bi-LSTM, CNN, ... (with Attention and without Attention).
@Ghost can you tell us how much do you think contributes the NLP part to your score? Let's say you take a naive approach with a single gru layer, how much would that worsen your result?
", "votes": 0, "replies": [] }, { "id": 333372, "author_name": "cab", "author_url": "", "post_date": "2018-05-25T03:35:58.927000", "content": "@Dieter
\nI am not sure that I understand your question correctly. Please feel free to ask.
\nWithout NLP, my model is really bad. It is arround 0.2240. Then I tried to add NLP.
I have not ran all of the NLP parts above, so I don't have results of each part.
\nTo determine which NLP part is good or not. I have a training plan as follows:
You will have list of cross-validation of model in different NLP architectures (GRU, CNN, LSTM,...).\nSelect the best one and go fully training.
\n\nMy top 3 best NLP parts are:\nCapsule --> CNN --> Bi-GRU --> ...
", "votes": 3, "replies": [] }, { "id": 333377, "author_name": "Dieter", "author_url": "", "post_date": "2018-05-25T03:52:36.307000", "content": "\n\n\nWithout NLP
\n
you mean skipping text completely?
", "votes": 0, "replies": [] }, { "id": 333379, "author_name": "cab", "author_url": "", "post_date": "2018-05-25T03:56:39.793000", "content": "Yeap.
\nWithout NLP, I only have numeric and categorical features.
Thank you, that helps :)
", "votes": 0, "replies": [] }, { "id": 333499, "author_name": "Liu Jilong", "author_url": "", "post_date": "2018-05-25T09:17:35.427000", "content": "0.2218 is this a single model or average of 5 fold?
", "votes": 0, "replies": [] }, { "id": 334030, "author_name": "Dieter", "author_url": "", "post_date": "2018-05-26T10:47:38.253000", "content": "@Ghost you must be very talented. You joined kaggle 2 days ago, and already have models that score top 50 in LB. I can only salute.
", "votes": 1, "replies": [] }, { "id": 334099, "author_name": "Shanth", "author_url": "", "post_date": "2018-05-26T14:32:58.730000", "content": "0.2240 without Text is a strong model I think. \nDid you use image meta features in this?
", "votes": 0, "replies": [] }, { "id": 334592, "author_name": "cab", "author_url": "", "post_date": "2018-05-27T23:48:41.247000", "content": "Sorry for the late reply. I have not tried with image meta-features. The result is after CV.
", "votes": 0, "replies": [] } ] }, { "id": 329320, "author_name": "Serigne ", "author_url": "", "post_date": "2018-05-16T08:25:08.463000", "content": "I am working exclusively with NN for now..
\n\nSo far, best single NN model : LB = 0.2246 ( 5 folds CV RMSE : 0.2215xxx ), no image features either
\n\nThe model was scoring 0.244xxx on LB when I started it , 5 days ago...(Some ideas from kernels and discussion topics helped me in the improvement)
\n\nEdit : The NN model is now scoring 0.2223 on LB
", "votes": 6, "replies": [ { "id": 330506, "author_name": "Totoro (ごめん)", "author_url": "", "post_date": "2018-05-19T01:47:12.347000", "content": "Nice.
\nMy model now is 0.2222 LB. Can I know which pretrained embedding are you using?
Thanks :)
\n\nI am using self-trained embeddings (following the idea suggested by Dieter )
", "votes": 0, "replies": [] }, { "id": 330561, "author_name": "Totoro (ごめん)", "author_url": "", "post_date": "2018-05-19T05:32:38.377000", "content": "Amazing guy.
\nI could not get the high result with this embedding :D
@ Totoro - Did you try both options ( trainable = True and Trainable = False )??
", "votes": 0, "replies": [] }, { "id": 330567, "author_name": "Totoro (ごめん)", "author_url": "", "post_date": "2018-05-19T05:51:58.893000", "content": "I use pretrained embedding, so trainble = False. I tried trainable = True, but it is worse for me
", "votes": 0, "replies": [] }, { "id": 330568, "author_name": "Serigne ", "author_url": "", "post_date": "2018-05-19T05:54:44.660000", "content": "If it's any consolation...the biggest improvements in my model are not from the embedding itself :)
", "votes": 0, "replies": [] }, { "id": 330571, "author_name": "Totoro (ごめん)", "author_url": "", "post_date": "2018-05-19T05:59:16.977000", "content": "Interesting
\nHope to see your model when the competition ends :D
Thank you
", "votes": 0, "replies": [] } ] }, { "id": 342176, "author_name": "pocket", "author_url": "", "post_date": "2018-06-13T03:26:59.310000", "content": "Single model LightGBM 0.2199, with 5seed-avg 0.2195
", "votes": 3, "replies": [ { "id": 343951, "author_name": "Siyuan Dang", "author_url": "", "post_date": "2018-06-16T15:11:08.827000", "content": "could you show some hints about how you achieve this?
", "votes": 0, "replies": [] }, { "id": 345827, "author_name": "pocket", "author_url": "", "post_date": "2018-06-20T14:44:08.317000", "content": "Sure, it will be in my write up if I get gold :)
", "votes": 3, "replies": [] } ] }, { "id": 339507, "author_name": "Nooh", "author_url": "", "post_date": "2018-06-07T04:21:41.537000", "content": "Finally! Able to achieve 2210-XGB and 2217-LGB. No image features though.\nI was stuck earlier and started from scratch, seems the right decision. Best of luck guys.
", "votes": 3, "replies": [ { "id": 339528, "author_name": "Johnny Liu", "author_url": "", "post_date": "2018-06-07T04:58:37.717000", "content": "Wow,Congratulation! Please sharing your solution after competition,I'm every interested how to achieve the score.
", "votes": 3, "replies": [] }, { "id": 339547, "author_name": "Larry Freeman", "author_url": "", "post_date": "2018-06-07T05:56:38.403000", "content": "Is your XGB model using the same features as LGBM? I am finding that my LGBM models are both faster and perform better. I am able to get to 2217-LGB but not even 2217-XGB.
", "votes": 2, "replies": [] }, { "id": 339568, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-06-07T06:27:00.560000", "content": "Congratulations @Nooh on getting those scores. Any hints you can share?
\n\n@Larry Freeman, I have the same issue with you. My best LGBM is at 0.2215 but xgb is 0.2232. Both with single runs. The speed difference is not surprising as LGBM is always faster than xgb but in my experience their performance is comparable. Hence my surprise that I am not able to get xgb to match lgb performance here.
", "votes": 2, "replies": [] }, { "id": 339648, "author_name": "Nooh", "author_url": "", "post_date": "2018-06-07T09:18:25.063000", "content": "Thanks Liu, though I'm really bad in documentation and my code is a mess right now but will surely post a brief if I don't end up over-fitting.
", "votes": 1, "replies": [] }, { "id": 339649, "author_name": "Nooh", "author_url": "", "post_date": "2018-06-07T09:20:37.430000", "content": "Yes Larry, I can say XGB has 98% same features as of my LGB. \nMy XGB took around 15-hours to train. :/
", "votes": 2, "replies": [] }, { "id": 339651, "author_name": "Nooh", "author_url": "", "post_date": "2018-06-07T09:28:44.263000", "content": "Thanks a lot YaGana Sheriff-Hussaini!
\n\nNothing special in features though. I'm using nearly all the features used by this kernel. Played with some statistical features around deal_prob and Price. And another tip, try to thoroughly analyze image_top_1 and item_seq_no; I handcrafted something using them. After all that I tuned the hyperparams.
\n\nBest of luck buddy!
", "votes": 7, "replies": [] }, { "id": 339863, "author_name": "Nooh", "author_url": "", "post_date": "2018-06-07T19:44:29.180000", "content": "Added OOF ridge in LGB and now hitting 2211. :)
", "votes": 3, "replies": [] }, { "id": 340353, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-06-09T01:29:08.370000", "content": "Thanks @Nooh and best of luck to you too.
\n\nI just managed to get my lgb to LB=0.2212 but still have some optimization to do. You are right the xgb takes a long time for me as well, hence I am trying to optimize my features with LGB before running the xgb model again. Like you, I use the same features for both.
", "votes": 1, "replies": [] }, { "id": 340584, "author_name": "Nooh", "author_url": "", "post_date": "2018-06-09T17:43:15.437000", "content": "Okay, so I'm convinced now that text features are bringing in some improvements in my models. Text normalization seems promising in my case. I just did some very basic normalization on title+desc and it is progressive. I should quote the original author here i.e; ololo (who stood 5th in the previous avito competition). I just stumbled upon his code and there is a lot to scavenge from his repo here
\n\nI used this script to get some text features. Hope this helps.
", "votes": 7, "replies": [] }, { "id": 340759, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-06-10T08:21:03.113000", "content": "Thanks @Nooh. I have a simple text normalization but after reading your comment, I tried ololo's i.e. \"def normalize(text):\" and I get the following error:
\n\n\n", "votes": 2, "replies": [] }, { "id": 340775, "author_name": "Nooh", "author_url": "", "post_date": "2018-06-10T09:24:03.940000", "content": "File \"../src/script.py\", line 98\n text = re.sub(ur'(?<=[^а-я])(' + shortenings + ')[.]', ur'\\1 ', text)\n ^\n SyntaxError: invalid syntax
\n
Hi YaGana Sheriff-Hussaini. His code is for Python 2.x\nAre you on Python3x? If so, make sure you port it accordingly.\nI didn't used it as is actually instead get ideas from his code and made my own normalizations.
", "votes": 2, "replies": [] }, { "id": 341220, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-06-11T08:22:41.477000", "content": "@Nooh you are right Ololo's code is for python2.x. I am using python3 and wrote a better text normalization than the basic one I had. This one improved both my CV and LB scores.
", "votes": 1, "replies": [] }, { "id": 341293, "author_name": "Nooh", "author_url": "", "post_date": "2018-06-11T11:25:21.387000", "content": "Great bro!, happy to hear that. I'm making up some image meta features' code. I hope that will bring in some improvement. If so, I'll share the code+data. \nFingers crossed. :)
", "votes": 1, "replies": [] }, { "id": 341351, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-06-11T13:25:45.630000", "content": "I hope you get a better results with image features than I do. I have a weak machine hence using a different LGBM model with reduced features that includes image ones but only scores 0.21777 CV and 0.2233 LB.
", "votes": 0, "replies": [] }, { "id": 341522, "author_name": "Nooh", "author_url": "", "post_date": "2018-06-11T18:12:12.117000", "content": "Hopefully yeah. I'm done with test set images and training set is running for 1 day now. I think it will take a couple of more hours.\nHow many models have you ensembled to get the current lb score you have? I have 4 of them, 3 lgbms and 1 xgb.\nTry training different lgbms with different features and different hyperparams. You can get all your features in that way i guess.
", "votes": 0, "replies": [] }, { "id": 341598, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-06-11T21:21:31.863000", "content": "Six.
", "votes": 0, "replies": [] }, { "id": 344448, "author_name": "high&mean", "author_url": "", "post_date": "2018-06-18T03:05:43.033000", "content": "which image meta features will you care for？
", "votes": 0, "replies": [] } ] }, { "id": 337277, "author_name": "Dieter", "author_url": "", "post_date": "2018-06-02T12:40:10.413000", "content": "We are at
\n\nImpressive @Dieter. Are your results for single or N-Folds. If so is it 3 or 5 folds?
\n\nMy best single model is LGBM with LB = 0.2217 single run. I have not run it for N-Folds yet.
", "votes": 1, "replies": [] }, { "id": 337916, "author_name": "Dieter", "author_url": "", "post_date": "2018-06-04T03:47:50.800000", "content": "5 fold
", "votes": 2, "replies": [] }, { "id": 338044, "author_name": "Larry Freeman", "author_url": "", "post_date": "2018-06-04T10:04:21.810000", "content": "What features are you using for LGB .2206? That is very impressive.
", "votes": 0, "replies": [] }, { "id": 338101, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-06-04T12:29:12.633000", "content": "@Dieter, if you can share , what is the difference in score between a single run and 5-Fold?
", "votes": 0, "replies": [] }, { "id": 338426, "author_name": "Dieter", "author_url": "", "post_date": "2018-06-05T03:15:44.950000", "content": "@Larry, I guess some part of the good score are image features
\n\n@YaGana Sheriff-Hussaini difference is about 0.001
", "votes": 2, "replies": [] }, { "id": 342435, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-06-13T14:10:32.643000", "content": "@Dieter, the difference for me is 0.0003
", "votes": 0, "replies": [] } ] }, { "id": 335359, "author_name": "Larry Freeman", "author_url": "", "post_date": "2018-05-29T16:31:37.513000", "content": "My best so far is an LGB model with .2216 (recently improved) after a 10-fold average.
", "votes": 3, "replies": [ { "id": 335485, "author_name": "Antoine", "author_url": "", "post_date": "2018-05-29T20:05:05.627000", "content": "Good job! Your public score is 0.2195, how did you earn those 0.002 on your score ? with some blending ?\nSame thing @AlexTru there is a significant gap between your LB and this LGB model. I'd understand if you don't want to share...
", "votes": 0, "replies": [] }, { "id": 335487, "author_name": "Larry Freeman", "author_url": "", "post_date": "2018-05-29T20:09:31.143000", "content": "I used lessons learned from here:\nhttps://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/52557
", "votes": 2, "replies": [] }, { "id": 335510, "author_name": "Antoine", "author_url": "", "post_date": "2018-05-29T20:50:30.400000", "content": "Thanks for sharing this link !
", "votes": 0, "replies": [] }, { "id": 335518, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-05-29T21:21:05.710000", "content": "@larryfreeman Are you using train/test-time augments? I assume you're not doing psuedo-labeling?
", "votes": 0, "replies": [] }, { "id": 335542, "author_name": "Larry Freeman", "author_url": "", "post_date": "2018-05-29T22:17:33.987000", "content": "I am making attempts to use multiple methods (train/test augmentation, pseudo-labeling, and (less robust) cv + stacking framework). I think its mostly stacking that's contributing to score.
\n\n@PeterHurford, congrats to you on your excellent score. Any reason why you assume that I'm not doing pseudo-labeling? Is this something that you have tried? I have not gotten too much mileage from it yet but I am working on it. :-)
", "votes": 0, "replies": [] }, { "id": 335546, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-05-29T22:23:42.073000", "content": "@larryfreeman Thanks for the congrats - I hope it holds! :)
\n\n...My assumption is that pseudo-labeling is only worthwhile when you have really accurate models. We were lucky to be in that case in the toxic competition, though I stupidly rejected the idea and didn't try it there. With RMSE in the >0.2 range, I don't think our models are accurate enough, so I'm rejecting the idea again. I have not tried it. I hope it's the right call this time. ;)
", "votes": 3, "replies": [] }, { "id": 335549, "author_name": "Dieter", "author_url": "", "post_date": "2018-05-29T22:27:27.060000", "content": "additionally train and test distributions are quite close, which reduces the benefit from pseudo labeling
", "votes": 6, "replies": [] }, { "id": 335553, "author_name": "Larry Freeman", "author_url": "", "post_date": "2018-05-29T22:30:46.693000", "content": "Thanks, @Peter and @Dieter. I am still developing my intuition about pseudo labeling. Cheers.
", "votes": 0, "replies": [] }, { "id": 335555, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-05-29T22:31:39.537000", "content": "@larryfreeman I'd also be really curious to hear if augmentation is helpful. My assumption is that with 10x the data here compared to toxic, plus the impact of categoricals/images in addition to text, the benefits of adding translated data would be much lower now. ...But I could also be wrong about that.
", "votes": 1, "replies": [] }, { "id": 335556, "author_name": "Larry Freeman", "author_url": "", "post_date": "2018-05-29T22:35:47.767000", "content": "I share your assumption on data augmentation. At this point, I'm trying everything that I can think of just to see what happens and to see if it gives me any other ideas to try.
", "votes": 1, "replies": [] }, { "id": 335559, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-05-29T22:38:52.247000", "content": "@larryfreeman It's awesome to try stuff. This competition is overwhelming in the amount of things there are to try. I hope you get lucky and one of these ideas pans out for you. :)
", "votes": 1, "replies": [] }, { "id": 335584, "author_name": "Larry Freeman", "author_url": "", "post_date": "2018-05-30T00:50:49.670000", "content": "Thanks, @Peter. Best of luck to you too. :-)
", "votes": 0, "replies": [] } ] }, { "id": 328983, "author_name": "ms", "author_url": "", "post_date": "2018-05-15T13:29:53.330000", "content": "Got 0.2222 with NN just train and test
", "votes": 3, "replies": [ { "id": 329135, "author_name": "Strideradu", "author_url": "", "post_date": "2018-05-15T20:13:18.727000", "content": "So the NN is for the NLP of description or more like for the classifier for different features just like LGBM
", "votes": 0, "replies": [] }, { "id": 329244, "author_name": "Dieter", "author_url": "", "post_date": "2018-05-16T04:40:09.050000", "content": "I guess both. At least I use it for both. It makes it much easier for me to connect different types of information.
", "votes": 0, "replies": [] } ] }, { "id": 341626, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2018-06-12T00:36:43.723000", "content": "After a lot of hard work I got my single LGB (with 5 fold) down to 0.2200 on LB. Yay !
", "votes": 4, "replies": [ { "id": 341667, "author_name": "Nooh", "author_url": "", "post_date": "2018-06-12T04:12:58.910000", "content": "Good going buddy!
", "votes": 0, "replies": [] }, { "id": 342197, "author_name": "Johnny Liu", "author_url": "", "post_date": "2018-06-13T04:31:23.570000", "content": "Good work, buddy!
", "votes": 0, "replies": [] } ] }, { "id": 340365, "author_name": "Hikkiiiiiiiii", "author_url": "", "post_date": "2018-06-09T02:38:39.960000", "content": "0.2190 10-fold LGB
", "votes": 4, "replies": [ { "id": 340375, "author_name": "Liu Jilong", "author_url": "", "post_date": "2018-06-09T03:15:45.817000", "content": "Do you known what is lb difference between 5-fold and 10-fold?
", "votes": 1, "replies": [] }, { "id": 340379, "author_name": "KALE", "author_url": "", "post_date": "2018-06-09T03:23:30.757000", "content": "Approximately 0.0001-0.0002
", "votes": 1, "replies": [] }, { "id": 340406, "author_name": "Hikkiiiiiiiii", "author_url": "", "post_date": "2018-06-09T05:22:56.690000", "content": "@Liu Jilong, I've only tried 1fold and 10fold.
\n\nBut I think it should be around 0.0002
Congratulation!! you guys are so cool. upvote for you guys.
", "votes": 0, "replies": [] }, { "id": 340689, "author_name": "Webber", "author_url": "", "post_date": "2018-06-10T02:44:00.383000", "content": "Wow, this small gap is quite impressive. Mine are alway around 0.004 (20 times of yours)
", "votes": 0, "replies": [] } ] }, { "id": 342452, "author_name": "Stanislav K.", "author_url": "", "post_date": "2018-06-13T14:39:15.370000", "content": "Silly question.
\n\nWhat do you guys exacly mean when you say, for example, 10-fold LGB?\nYes, you can use CV to better tune hyperparameters. Particularly, n_estimators. Is that what you mean?
\n\nAnd when you find optimal number of estimators do you retrain on whole dataset?
", "votes": 1, "replies": [ { "id": 342463, "author_name": "yimacs", "author_url": "", "post_date": "2018-06-13T15:01:05.843000", "content": "I guess people mean average the prediction of 10 LGB models in CV instead of retrain and get one model. The advantage of avg is obvious :)
", "votes": 1, "replies": [] }, { "id": 342521, "author_name": "Stanislav K.", "author_url": "", "post_date": "2018-06-13T16:28:38.163000", "content": "Oh I see.\nYou can also achieve that with different random seeds. \nBut subsets of training data probably make models more different than random seeds. And that is good.
", "votes": 0, "replies": [] }, { "id": 342532, "author_name": "yimacs", "author_url": "", "post_date": "2018-06-13T16:49:00.773000", "content": "But that’s not cv tho.
", "votes": 0, "replies": [] }, { "id": 342578, "author_name": "Antoine", "author_url": "", "post_date": "2018-06-13T18:29:51.537000", "content": "10-fold is not used for hyper parameters tuning but just as CV technic. When running 10 fold, it's similar to running 10 times your LGB with 10 different seed and then doing an average. The advantage of using 10 fold, is that you can be sure that all data were used for train and for validation. (which might not be the case if you take random seeds)
", "votes": 1, "replies": [] }, { "id": 342610, "author_name": "Joe Eddy", "author_url": "", "post_date": "2018-06-13T19:50:04.500000", "content": "I hope to clear up some confusion here - it's not a silly question at all. It's common conflation of terminology on kaggle to refer to partitioned fold averaging as \"CV\", when it is a distinct method from CV that shares the partitioning structure in common.
\n\nHere are the two methods:
\n\nCross-validation - partition the data into k folds for train/validation splitting to exhaustively use all of the data for validation, with the purpose of evaluating model and hyperparameter performance. You use the evaluation results to guide your model/hyperparameter selection.
K-fold averaging - partition the data into k folds (using the same partitioning as in CV), train the same model on each fold, and average the results to help reduce the variance of a single model. You could add out of fold validation to this process without using it directly for selection, or you can just run with the result of the averaging if you feel good about the model/hyperparameter selection.
The reason the terminology gets confused is that many people take a shortcut and do both things at the same time. They do one k-fold split, use out of fold sets to select something like n_estimators for gradient boosting models with early stopping, then average the resulting models across the k train/validation runs. In theory, this is poor practice. Early stopping overfits to the particular choice of validation fold, so models averaged in this manner within CV inevitably overfit. In practice, the averaging helps mitigate some of the overfitting and the overall harm may be small, but it's something worth keeping in mind. It certainly causes your local score estimates to be at least slightly optimistic.
\n\nThis is a common pitfall in use of CV more generally. People don't realize that by using certain validation sets to guide their model selection, they've reduced the legitimacy of using those same validation sets for estimating generalization error. The best estimates of generalization error are given by sets that have had no influence on model selection. This is why doing CV and setting aside a true hold-out set that only gets used once for evaluation is a common and sound practice.
", "votes": 26, "replies": [] }, { "id": 342652, "author_name": "Dieter", "author_url": "", "post_date": "2018-06-13T20:59:59.003000", "content": "One practical reason why k-fold cross validation is famous around kagglers, is that you can do this approach for several layers, without needing a new hold-out set for each level. This is especially helpful when working together in a team or stacking different models.
", "votes": 4, "replies": [] }, { "id": 342692, "author_name": "Harlan Seymour", "author_url": "", "post_date": "2018-06-13T22:43:44.870000", "content": "@Joe Eddy: nice overview!
\n\n\n\n\nThis is why doing CV and setting aside a true hold-out set that only\n gets used once for evaluation is a common and sound practice.
\n
In this competition at least, I consider the public leaderboard to be my true hold-out set. I prefer to use K-Fold CV to generate my predictions, with K models each built on K-1 folds having their predictions averaged together. I see this as form of de facto model averaging.
\n\nI use early stopping. I agree that it overfits. But it also seems to me that each K-1 folds are best fit by a different # of epochs or trees, so guessing this # without early stopping is also misfitting (either underfitting or overfitting). So the misfitting washes out.
\n\nThe big bonus for me is using the out-of-fold predictions for model selection and to optimize model stacking.
", "votes": 6, "replies": [] }, { "id": 342713, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-06-14T00:00:36.640000", "content": "I agree with @Harlan Seymour. The reason you stated is why I am using CV this way and I believe most people in this competition are as well.
", "votes": 2, "replies": [] }, { "id": 342722, "author_name": "Joe Eddy", "author_url": "", "post_date": "2018-06-14T00:31:55.450000", "content": "@Harlan and YaGana, I definitely agree it's very convenient and that overfitting is mostly mitigated. It works well enough to produce good submissions and stacking results. But it's also definitely not optimal. When stacking, generating out of fold predictions on the same folds you early stop on introduces an optimistic bias that your meta learner may overfit to (these predictions are no longer out of fold in a true sense of the model not having seen the data, it now has glimpsed at that data). If that optimistic bias exists across all the models you stack it likely washes out, but if some models were trained differently (e.g. a non-boosting model that doesn't have the same early stopping concept), it might be problematic for the meta learner.
\n\nThere's no reason you can't get the best of both worlds, at the cost of extra computation time. You can run a 5-fold CV for model selection, then run another 5-fold process on a different seed to generate true OOF predictions to stack with. You can take n_estimators as the mean of the 5-fold CV early-stopped n_estimators, or use a similar heuristic. I can't find it right now, but I believe there's a post somewhere where Laurae (LGBM contributor) recommends the heuristic 110% of mean early-stopped n_estimators.
", "votes": 2, "replies": [] }, { "id": 342758, "author_name": "Harlan Seymour", "author_url": "", "post_date": "2018-06-14T03:16:38.033000", "content": "@ Joe, you've definitely given me food for thought. Take for example XGBoost. What do you think of this compromise to save compute time?
\n\nNote that I might prefer something like 95% x M to 110% x M. Val score changes very little for the final trees before early stopping while train score goes down significantly i.e. lots of overfitting at the end before early stopping.
\n\nDoesn't this almost entirely mitigate your overfitting worry without having to do a double run? Note that XGBoost takes a long for me in this competition.
\n\n@Joe, I have another question it would be interesting to get your input on. Do you create a model on the entire training set, using in XGBoost, say, 110% x M x (K / (K-1)) for ntree_limit? This model would be used just for test predictions. Maybe average its test predictions with average of K fold test predictions?
", "votes": 1, "replies": [] }, { "id": 342763, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-06-14T03:57:22.260000", "content": "My personal preference is to not use early stopping. Use trial and error or some initial early stopping to get a certain number of rounds, and then use that number of rounds consistently with all the folds. Also remember to pay attention to the relationship between choice of learning rate and number of rounds needed.
", "votes": 2, "replies": [] }, { "id": 343080, "author_name": "Mark Worrall", "author_url": "", "post_date": "2018-06-14T17:06:36.897000", "content": "@Joe Eddy
\n\nI agree with a lot of this but if we are talking about optimal then using a heuristic such as 110% of the mean of the early stopped n_estimators certainly isn't optimal.
\n\nP.S. Does anyone know why when I reply to, say, Joe Eddy, the forum thinks I am replying to the person he replied to?
\n\nP.P.S Some of this is covered here which is useful reading.
", "votes": 1, "replies": [] }, { "id": 343102, "author_name": "Joe Eddy", "author_url": "", "post_date": "2018-06-14T17:37:55.190000", "content": "@maw501 that's fair, there's no such this as optimality in statistical modeling of distributions that we can't have perfect knowledge of, everything is relative :)
\n\nBut on a practical level, I don't know what you would expect (both in the normal sense of expect, and on expectation in the statistical sense) to work better for generalization performance of n_estimators than something close to the mean of early stopped rounds. Here's a nice experiment in favor of using the mean to estimate the optimal number of rounds from olivier. Do you have in mind something that works better / what would your choice be?
\n\n@Harlan I like that idea a lot, it's not something I've thought of before. I can't think of a reason why it wouldn't mitigate the overfitting issue, sounds like a great time saver and worth a shot to me.
\n\nRe: your second question, I think you could definitely do that, but the gains might be pretty small if you're already using all of the data for training by averaging test predictions across k-folds. I tend to skip that to save time, but using all the training data in one model might be worth it. It aligns with the more traditional way of stacking - generate oof predictions in k-fold, retrain model on entire dataset to generate test predictions (without averaging of models). With relatively smaller datasets like this one my instinct is that averaging on k-folds gives a notable improvement via variance reduction, so that's what I go with.
", "votes": 4, "replies": [] }, { "id": 343163, "author_name": "yimacs", "author_url": "", "post_date": "2018-06-14T20:01:47.250000", "content": "+1 for variance reduction.
", "votes": 0, "replies": [] }, { "id": 343188, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2018-06-14T21:07:04.853000", "content": "I have a question about stacking here in Kaggle and for that competition:\nMy current way of doing it is:\n-For 5 folds, train 5 models then predict on the OOF to construct train meta data for that model.\n-Train a full model and create a test meta data.\n(then later on train a meta model etc.)
\n\nWhat I'm wondering is if it makes sense to instead use the average of the 5 models predictions on test data. The same way I (and others) currently test my single models on the LB.\nDoes it make sense to think that the test meta is more stable that way and closer to the train meta (on average) compared to have a different model that created test meta ?
", "votes": 1, "replies": [] }, { "id": 343216, "author_name": "Mark Worrall", "author_url": "", "post_date": "2018-06-14T22:30:54.153000", "content": "Hi Joe,
\n\nAgain, thanks for a thoughtful reply. I definitely agree with your first comment re. optimality and this alongside scepticism is a reasonable grounding for most problems. I also think you somewhat answer your own point when asking for what I would expect to perform better when you point out the elusiveness of optimality. I don't expect any method to be consistently better than any other (for a reasonable choice of methods).
\n\nThis is more complicated clearly and I think a simple (self-evident) answer is that it's entirely problem dependent. I've yet to see any ML technique/approach (including state of the art) that one feels entirely comfortable applying in all contexts. My last two competitions I worried far too much about whether things were theoretically justifiable/sound and paid the price. Worrying about reliably estimating generalization error (which we care about in the real world) vs. performing well on a Kaggle competition vs. having robust statistical backing for one's ideas are all different things.
\n\nMy choice is usually to try most of the approaches listed with a (more recent) favouring to average across the predictions from the folds for the base models. The biggest problem with using a simple heuristic like 110% of the mean of the early stopped n_estimators is that given the extra data it feels a little too much like guesswork for me. Yes under some assumptions I'm sure it's fine, but what are they? Can we even state them?
\n\nFurther if you change from 5-fold to 10-fold to any k-fold - what then? Add in that one should perhaps favour higher entropy validation methods then you should accept some folds will be harder than others for your model and early stopping at least deals with this (of course, at some risk of overfitting) in a consistent manner - additionally you have the benefit of averaging too. If you have a diverse set of base models I don't particularly worry about any single model overfitting too much.
\n\nI do like the idea of using different random seeds for model iteration and oof predictions and will certainly try this.
\n\nEdit extra: Just wondering if we can resolve via a thought experiment. For instance, take an extreme 2-fold example with one really easy fold needing 500 rounds and the other really difficult needing 1500 - using the heuristic we would train on the whole data now with 1, 100 rounds. I'm not sure I'm happy with that vs. averaging the predictions from both which at least predicts at the point when it thinks it has discovered all the information from the data. I'm not saying the difference will be big, or one is consistently better but I like getting variance reduction for free and some comfort in knowing the number of rounds. I guess to some extent it's a function of the dissimilarity in the folds/strength of signal in each but I'm not sure how one would formalise this - so back to my point about not expecting any method to do better on average.
", "votes": 1, "replies": [] }, { "id": 343670, "author_name": "Harlan Seymour", "author_url": "", "post_date": "2018-06-15T19:47:05.077000", "content": "@Joe: As an experiment I applied the procedure I outlined to my latest XGB per-fold models, syncing them to all to be 100% of the mean of the early stopping ntree_limits. My local CV score went up minutely by +0.00001. The newly submitted test predictions had the same RMSE score on the public leaderboard, but ordering \"My Submissions\" by score showed it was not as good as the original.
", "votes": 1, "replies": [] }, { "id": 343696, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-06-15T20:54:37.490000", "content": "Thanks @Joe Eddy for sparking this useful discussion. And thanks @Harlan Seymour for the update on your experiment.
\n\n@maw501, I would like to know the result of the thought experiment on testing on 2 extreme cases you mentioned. If you ever do that, please report back if you can.
\n\nAveraging is what I am comfortable with at the moment to reduce variance.
", "votes": 1, "replies": [] } ] }, { "id": 336590, "author_name": "Johnny Liu", "author_url": "", "post_date": "2018-06-01T01:59:22.720000", "content": "LGB 0.2223 5-fold average
\n\nupdate:\nnew feature add, 0.2217 without 5 folder
\n\n0.2210 with 5 folder
\n\nupdate 6-15:
\n\n0.2208 with 5 folder
\n\nupdate 6-20:
\n\n0.2205 with 5 folder
\n\nfinal:
\n\n0.2204 with 10 folder
", "votes": 1, "replies": [ { "id": 336595, "author_name": "Marcus Lin", "author_url": "", "post_date": "2018-06-01T02:17:59.517000", "content": "劉sir, 您是用Stack跳到0.2210的麼?
", "votes": -2, "replies": [] }, { "id": 337169, "author_name": "Johnny Liu", "author_url": "", "post_date": "2018-06-02T06:24:52.117000", "content": "还没stacking，简单的blend几个public kernel
", "votes": -2, "replies": [] }, { "id": 337275, "author_name": "Dieter", "author_url": "", "post_date": "2018-06-02T12:38:43.800000", "content": "Translation: \nLiu sir , are you jumping to 0.2210 with Stack?\n - No stacking, simple blend several public kernel
", "votes": 9, "replies": [] }, { "id": 338644, "author_name": "ea", "author_url": "", "post_date": "2018-06-05T13:44:03.090000", "content": "有啥技巧么，我blend一个0.2212和0.2216结果也就0.2212
", "votes": -5, "replies": [] }, { "id": 338654, "author_name": "Diangu Xie", "author_url": "", "post_date": "2018-06-05T14:17:22.973000", "content": "I guess there is not any techniques. Just use weight to blend public with private kernel. I used blending, too
", "votes": 0, "replies": [] }, { "id": 339058, "author_name": "Johnny Liu", "author_url": "", "post_date": "2018-06-06T08:01:20.183000", "content": "you need try to improve your single model,then try to blend or stacking
", "votes": 0, "replies": [] } ] }, { "id": 330553, "author_name": "Shanth", "author_url": "", "post_date": "2018-05-19T05:14:52.653000", "content": "Edit : 0.2249
\n\nEdit: 0.2252 with Image_top_1 feature included
\n\nEdit: 0.2259 with RNN
\n\n0.2276 with RNN, Fast Text Embeddings . - No Tuning done, A lot of Scope for improvement and tuning. \nLink to Public kernel is here
\n\n0.2276 RNN - Good for Blend :)
", "votes": 1, "replies": [] }, { "id": 329707, "author_name": "Phat Hoang", "author_url": "", "post_date": "2018-05-17T02:38:18.170000", "content": "0.2223 with 5fold LGB model (all features are taken from public kernels).
", "votes": 1, "replies": [] }, { "id": 329210, "author_name": "Chau Huynh", "author_url": "", "post_date": "2018-05-16T02:38:27.313000", "content": "Single lgbm 0.223.
", "votes": 1, "replies": [] }, { "id": 329185, "author_name": "takuoko", "author_url": "", "post_date": "2018-05-16T00:01:24.690000", "content": "Single lgbm 0.2215.\nBut using image features from https://www.kaggle.com/bguberfain/vgg16-train-features , my score got worse.\nI would like to know how to use image feature.
", "votes": 1, "replies": [ { "id": 329196, "author_name": "Strideradu", "author_url": "", "post_date": "2018-05-16T00:39:06.463000", "content": "Wow, that is so high for a single lgbm model! Would you mind if share some ideas how to improve from 0.223 level to 0.2215 ?
", "votes": 0, "replies": [] }, { "id": 329209, "author_name": "takuoko", "author_url": "", "post_date": "2018-05-16T02:34:48.440000", "content": "Price and item-seq-number is important feature. I refer to https://www.kaggle.com/tapioca/item-seq-number-vs-deal-probability/code , and make some features. Good luck!
", "votes": 2, "replies": [] }, { "id": 329213, "author_name": "Strideradu", "author_url": "", "post_date": "2018-05-16T02:43:55.210000", "content": "Thanks! For the image feature, my thought is: the features from the pre-trained Imagenet model are designed for classification. For example, we want our VGG16 distinguish bicycle from car. However, we already have the features like item_id and category, so directly use imgenet features may not helps. Maybe we can try fine tune those features with the target for the deal probabilities. This is just my naive thought and I haven't try any image features
", "votes": 1, "replies": [] }, { "id": 329214, "author_name": "takuoko", "author_url": "", "post_date": "2018-05-16T02:51:07.357000", "content": "Wow! Your thought of image feature is very cool! I try to use color base feature because it is easy to use. Thank you!
", "votes": 0, "replies": [] }, { "id": 329451, "author_name": "Strideradu", "author_url": "", "post_date": "2018-05-16T13:45:34.857000", "content": "And also may ask how many models you are using to achieve your LB position now?
", "votes": 0, "replies": [] }, { "id": 329466, "author_name": "takuoko", "author_url": "", "post_date": "2018-05-16T14:10:34.853000", "content": "LB 0.2209 by single lgbm. Add some features from active.csv and periods data.
", "votes": 7, "replies": [] }, { "id": 329646, "author_name": "Antoine", "author_url": "", "post_date": "2018-05-16T21:35:57.487000", "content": "@takuoko : item-seq-number seems to be exactly the same as doing a count after a group by user no ? I was able to get good features from the price, but not by combining item_seq_number with another feature... Thanks for sharing !
", "votes": 0, "replies": [] }, { "id": 329770, "author_name": "takuoko", "author_url": "", "post_date": "2018-05-17T06:39:04.460000", "content": "Thanks for your comment, Antoine! By using item-seq-number feature, 0.00005 better val score in my 5 folds. So it may be ineffective. But I think item-seq-number is important. So I continue trying to make featrue about that.
", "votes": 1, "replies": [] }, { "id": 329895, "author_name": "Johnny Liu", "author_url": "", "post_date": "2018-05-17T13:36:33.833000", "content": "simple feature from VGG16 is not helpful, because the prediction is not precise nor suitable. I have write a kernel https://www.kaggle.com/liuhdsgoal/about-image-top-1-is-a-classify-label
", "votes": 0, "replies": [] } ] }, { "id": 329122, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2018-05-15T19:17:42.043000", "content": "Single lgbm 0.2229
", "votes": 1, "replies": [] }, { "id": 329052, "author_name": "Victor An", "author_url": "", "post_date": "2018-05-15T16:02:51.120000", "content": "LB 0.2249 with only sample train and test dataset via lightgbm
", "votes": 1, "replies": [] }, { "id": 328924, "author_name": "Totoro (ごめん)", "author_url": "", "post_date": "2018-05-15T11:41:07.980000", "content": "My best single NN is 0.2233 without image nor active and period files.
", "votes": 1, "replies": [ { "id": 329641, "author_name": "Totoro (ごめん)", "author_url": "", "post_date": "2018-05-16T21:24:57.023000", "content": "Update:
\n0.2226 for NN
Could you give a hint about architecture of your NN?
", "votes": 0, "replies": [] }, { "id": 329745, "author_name": "Totoro (ごめん)", "author_url": "", "post_date": "2018-05-17T05:15:39.803000", "content": "As @Strideradu commented above,
\n\n> So the NN is for the NLP of description or more like for the classifier for different features just like LGBM
\n\nI use both.
\n\nCategory + Numeric part
\nYou can refer from fastai or some solution in this competition or similar one: \nhttps://www.kaggle.com/c/donorschoose-application-screening
NLP
\nYou can refer from previous competitions like:
\nhttps://www.kaggle.com/c/donorschoose-application-screening
\nhttps://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge\n...\nto know how they dealed with NLP.
Thank you!
", "votes": 0, "replies": [] } ] }, { "id": 336777, "author_name": "jetou Xu", "author_url": "", "post_date": "2018-06-01T09:02:02.143000", "content": "NN:\n0.2213 (Categorical, Numerical, W2V)\n10-fold
", "votes": 2, "replies": [] }, { "id": 335073, "author_name": "AlexTru", "author_url": "", "post_date": "2018-05-29T05:48:58.033000", "content": "one LightGBM model 5kfold - 0,2211 (UPD)
", "votes": 2, "replies": [ { "id": 335075, "author_name": "Marcus Lin", "author_url": "", "post_date": "2018-05-29T05:55:18.377000", "content": "incredible
", "votes": 0, "replies": [] }, { "id": 335214, "author_name": "AlexTru", "author_url": "", "post_date": "2018-05-29T11:41:39.043000", "content": "Almost all ideas i take from public kernels :)
", "votes": 1, "replies": [] }, { "id": 338415, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-06-05T02:30:16.977000", "content": "@AlexTru, did you use image features? If so which ones if you can share? They seem to worsen both my CV and LB.
", "votes": 0, "replies": [] }, { "id": 338447, "author_name": "AlexTru", "author_url": "", "post_date": "2018-06-05T04:28:42.777000", "content": "Yes, i use features from this two kernels. this and this
", "votes": 0, "replies": [] }, { "id": 339187, "author_name": "AlexTru", "author_url": "", "post_date": "2018-06-06T13:30:03.850000", "content": "UPD 22.07
", "votes": 1, "replies": [] } ] }, { "id": 331503, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2018-05-21T12:22:52.577000", "content": "Update: 0.2220 with variation of the 'ridge trick', 5fold lgbm.
", "votes": 2, "replies": [ { "id": 331505, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-05-21T12:28:21.953000", "content": "What's a \"ridge trick\"?
", "votes": 0, "replies": [] }, { "id": 331511, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2018-05-21T12:37:26.997000", "content": "Creating an oof prediction from a ridge model and using it as a feature - along the lines of https://www.kaggle.com/him4318/avito-lightgbm-with-ridge-feature-v-2-0
", "votes": 1, "replies": [] }, { "id": 331531, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-05-21T13:05:26.643000", "content": "Cool. But does that really count as a single model? Though I'd concede at this point I'm just being pedantic. \"Single model\" is a bit of a vanity metric because of things like this, and it's really just about finding models that are individually good but blend together.
\n\nMy LGB has Ridges in it so I wasn't posting about it here. But I'm currently at 0.21611 CV / 0.2200 LB with a single LGB with multiple Ridge models inside. The best individual Ridge is at 0.22674 CV.
", "votes": 5, "replies": [] }, { "id": 331551, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2018-05-21T13:30:21.487000", "content": "Nice one - your gap cv - lb is a bit smaller than mine (~ 0.006). If you don't mind sharing, have you tuned the lgb a lot? I've been rolling mostly with parameters close to the ones from the public kernels so far.
", "votes": 0, "replies": [] }, { "id": 331556, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-05-21T13:39:06.703000", "content": "...I sure hope my smaller CV gap does not mean that my LB position will come crashing down at the end. :/
\n\nI have spent some amount of time tuning the LGB by hand to the point where I can no longer easily improve it. I still think I have some room to non-easily improve it, perhaps with Hyperopt or something.
", "votes": 0, "replies": [] }, { "id": 331561, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-05-21T13:43:35.087000", "content": "@Konrad: Are you seeing 0.216XX CV as well? Do you have a lot of other features beyond those in the kernels?
", "votes": 0, "replies": [] }, { "id": 331596, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2018-05-21T14:48:27.193000", "content": "Yes and yes - with the latter, i went to town with target encoding. I guess tuning is my next step, as i have barely touched it.
", "votes": 1, "replies": [] }, { "id": 331642, "author_name": "Himanshu Chaudhary", "author_url": "", "post_date": "2018-05-21T16:14:54.077000", "content": "Hey I have tried target encoding but it was not helping much as passing the categorical values into LGBM is doing. I think keeping the number of features high is helping the tree to build better.\nwhat do you think?
", "votes": 1, "replies": [] }, { "id": 331665, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2018-05-21T17:04:40.073000", "content": "Interesting - I have exact opposite experience to yours: once I mean-encoded the categoricals (created new numerical cols) and then dropped the originals, my score improved significantly.
", "votes": 3, "replies": [] }, { "id": 331669, "author_name": "Himanshu Chaudhary", "author_url": "", "post_date": "2018-05-21T17:09:44.993000", "content": "maybe I am doing somewhere wrong then. I tried using this
\nhttps://www.kaggle.com/tnarik/likelihood-encoding-of-categorical-features. I am encoding all the categorical values .
@konrad How do you mean encode? You may be at risk of overfitting your CV. When I did mean encoding I ended up in overfit hell for a few days and it took ten submits to get to the bottom of it. So I'm on team LGB native encoding. I might try doing a second LGB with mean encoding though and average the two approaches and see if that helps... I'll calculate it within the CV folds to monitor overfitting better.
", "votes": 2, "replies": [] }, { "id": 331687, "author_name": "Himanshu Chaudhary", "author_url": "", "post_date": "2018-05-21T17:35:05.907000", "content": "yeah, my CV score was good enough but LB was worst.
", "votes": 2, "replies": [] }, { "id": 331695, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2018-05-21T17:45:46.247000", "content": "@Peter: I did the mean-encoding inside the cv and added heavy regularization, so I think I'm fine (especially that cv - lb is consistently around 0.006, and changes are directionally correct).
", "votes": 2, "replies": [] }, { "id": 333343, "author_name": "", "author_url": "", "post_date": "2018-05-25T00:31:58.560000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 340394, "author_name": "Xuan Cao", "author_url": "", "post_date": "2018-06-09T04:46:18.230000", "content": "10-fold LGB, LB 0.2214.
\n\nTraining a model including some image features. Hopefully it would give me a boost. Finger crossed.
\n\nUpdate: Including image featuers, 10-fold LGB, LB 0.2210
\n\n6-15 Update: manage to improve my LGB to LB 0.2202
\n\n6-19 Update: finally got an LGB with LB 0.2199, XGB with LB 0.2209.
", "votes": 0, "replies": [ { "id": 343000, "author_name": "high&mean", "author_url": "", "post_date": "2018-06-14T14:21:53.030000", "content": "10-fold LGB avg or min ？
", "votes": 0, "replies": [] }, { "id": 343002, "author_name": "Xuan Cao", "author_url": "", "post_date": "2018-06-14T14:24:21.427000", "content": "10-fold LGB avg.
", "votes": 0, "replies": [] } ] }, { "id": 334596, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2018-05-28T00:37:22.033000", "content": "Right now I have 3 models in parallel. A catboost at 0.2237, a LGB at 0.2228 and a Neural network at 0.2240. I'm really impressed by people achieving 0.222X with neural network on that data set. Maybe I'm missing something fundamental or some kind of clever encoding of categoricals.
\n\nNot yet used additional FE from satellite data sets.
", "votes": 0, "replies": [ { "id": 334699, "author_name": "Serigne ", "author_url": "", "post_date": "2018-05-28T08:32:27.917000", "content": "As for me , I am just using simple label encoding for categorical data .
\n\nI think the NN architecture plays a key role and clever archtitecture can bring huge improvement.
\n\nI tend to rely on deep models just to avoid lot of features engeneering.
", "votes": 4, "replies": [] }, { "id": 334725, "author_name": "cab", "author_url": "", "post_date": "2018-05-28T09:30:27.170000", "content": "Interesting.
\nIt is a surprise for me to know the simple label encoding for categorical data works well. I will give it a try.
\nThank for your comment
Thanks. I'll play around with some extra ideas. My architecture is fairly straightforward so far, ConvNN for texts, embeddings for categoricals then concat everything and pass it through dense layers.
\n\nMaybe I can try going deeper with the text part.
", "votes": 0, "replies": [] }, { "id": 335870, "author_name": "AlexTru", "author_url": "", "post_date": "2018-05-30T14:50:52.780000", "content": "Hi @Serigne!\nYou really just encode cat features to numbers? or use one-hot encoding?
", "votes": 1, "replies": [] }, { "id": 335874, "author_name": "Serigne ", "author_url": "", "post_date": "2018-05-30T15:05:14.377000", "content": "Hi Alex... Yes just simple encoding to numbers..Then embed--> concat--> dense layers just like Arroqc..
\n\nBut that's just for categorical features.
\n\nFor text features I try to follow these 4 principles : Embed, Encode, Attend and Predict
", "votes": 7, "replies": [] }, { "id": 335972, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2018-05-30T19:51:50.463000", "content": "Interesting article Serigne. Thanks. Will try to see if attention helps.
", "votes": 1, "replies": [] } ] }, { "id": 334448, "author_name": "ea", "author_url": "", "post_date": "2018-05-27T12:26:31.887000", "content": "I got 0.223 using just train.csv and test.csv,How to using train_active.csv which has no labels
", "votes": 0, "replies": [ { "id": 334827, "author_name": "Shanth", "author_url": "", "post_date": "2018-05-28T14:17:12.837000", "content": "@ unexceptednull
\n\nYou perform feature engineering with user_id, item_id and merge it to train using 'user_id'\nIt is available in Benjamin's awesome kernel here BENJAMIN FEATURE ENGINEERING WITH TRAIN ACTIVE
\n\nYou can also refer to this kernel for building RNN on these additional features RNN - 0.2246
\n\nHope this helps \nCheers\nShanth
", "votes": 0, "replies": [] } ] }, { "id": 330469, "author_name": "", "author_url": "", "post_date": "2018-05-18T21:18:22.597000", "content": "", "votes": 0, "replies": [] }, { "id": 329704, "author_name": "", "author_url": "", "post_date": "2018-05-17T02:28:57.377000", "content": "", "votes": 0, "replies": [] }, { "id": 329226, "author_name": "", "author_url": "", "post_date": "2018-05-16T03:34:41.703000", "content": "", "votes": 0, "replies": [] }, { "id": 329137, "author_name": "", "author_url": "", "post_date": "2018-05-15T20:13:42.170000", "content": "", "votes": 0, "replies": [ { "id": 329166, "author_name": "", "author_url": "", "post_date": "2018-05-15T22:25:15.630000", "content": "", "votes": 0, "replies": [] }, { "id": 329168, "author_name": "", "author_url": "", "post_date": "2018-05-15T22:30:51.207000", "content": "", "votes": 1, "replies": [] }, { "id": 332664, "author_name": "", "author_url": "", "post_date": "2018-05-23T14:41:55.867000", "content": "", "votes": 0, "replies": [] }, { "id": 332668, "author_name": "", "author_url": "", "post_date": "2018-05-23T14:49:33.353000", "content": "", "votes": 2, "replies": [] }, { "id": 332671, "author_name": "", "author_url": "", "post_date": "2018-05-23T14:53:20.553000", "content": "", "votes": 0, "replies": [] }, { "id": 332778, "author_name": "", "author_url": "", "post_date": "2018-05-23T18:25:35.923000", "content": "", "votes": 0, "replies": [] }, { "id": 332782, "author_name": "", "author_url": "", "post_date": "2018-05-23T18:30:05.807000", "content": "", "votes": 0, "replies": [] }, { "id": 332787, "author_name": "", "author_url": "", "post_date": "2018-05-23T18:39:50.697000", "content": "", "votes": 0, "replies": [] }, { "id": 332791, "author_name": "", "author_url": "", "post_date": "2018-05-23T18:52:05.387000", "content": "", "votes": 0, "replies": [] }, { "id": 332807, "author_name": "", "author_url": "", "post_date": "2018-05-23T19:37:05.640000", "content": "", "votes": 1, "replies": [] }, { "id": 332820, "author_name": "", "author_url": "", "post_date": "2018-05-23T20:32:55.627000", "content": "", "votes": 0, "replies": [] }, { "id": 332827, "author_name": "", "author_url": "", "post_date": "2018-05-23T20:57:28.390000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 334810, "author_name": "", "author_url": "", "post_date": "2018-05-28T13:27:01.550000", "content": "", "votes": 0, "replies": [] }, { "id": 330081, "author_name": "", "author_url": "", "post_date": "2018-05-18T02:21:52.027000", "content": "", "votes": 0, "replies": [] }, { "id": 333907, "author_name": "", "author_url": "", "post_date": "2018-05-26T02:34:02.520000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "328917": "I wonder what the best single model score is. I got my NN to 0.2240, but havn't included any image features nor the active and period files yet. How much is the improvement with image and co.?", "340360": "I got 0.2181 with a NN network with 10 folds (UPDATED)\n", "338921": "I'm hitting .2190 with a 5-fold averaged LGBM", "333988": "LGB (Categorical, Numerical, TFIDF, Images meta-data + ImageNet + NIMA scoring): 0.2200 (updated)\n\nXGB (Categorical, Numerical, TFIDF, Images meta-data + ImageNet + NIMA scoring): 0.2216 (updated)\n\nNN (Categorical, Numerical, W2V, Images meta-data + ImageNet scoring): 0.2198 (updated)\n\nAll with CV5.", "344107": "LGB 10-fold 0.2179", "338049": "LGBM 10-fold avg. - 0.2203 (categorical, numerical, target encoding, text and features predicted using all csv data)", "333337": "NN: \n0.2218 Fasttext \n0.2215 Self-training embedding \n\nThanks @Dieter", "329320": "I am working exclusively with NN for now..\n\nSo far, best single NN model : LB = 0.2246 ( 5 folds CV RMSE : 0.2215xxx ), no image features either\n\nThe model was scoring 0.244xxx on LB when I started it , 5 days ago...(Some ideas from kernels and discussion topics helped me in the improvement) \n\n**Edit :** The NN model is now scoring 0.2223 on LB ", "342176": "Single model LightGBM 0.2199, with 5seed-avg 0.2195", "339507": "Finally! Able to achieve 2210-XGB and 2217-LGB. No image features though.\nI was stuck earlier and started from scratch, seems the right decision. Best of luck guys.", "337277": "We are at\n\n - LGB 0.2200 \n - XGB 0.2213 \n - Catboost 0.2208\n - NN 0.2194", "335359": "My best so far is an LGB model with .2216 (recently improved) after a 10-fold average.", "328983": "Got 0.2222 with NN just train and test", "341626": "After a lot of hard work I got my single LGB (with 5 fold) down to 0.2200 on LB. Yay !", "340365": "0.2190 10-fold LGB", "342452": "Silly question.\n\nWhat do you guys exacly mean when you say, for example, 10-fold LGB?\nYes, you can use CV to better tune hyperparameters. Particularly, n_estimators. Is that what you mean? \n\nAnd when you find optimal number of estimators do you retrain on whole dataset? ", "336590": "LGB 0.2223 5-fold average\n\n**update**:\nnew feature add, 0.2217 without 5 folder\n\n\n0.2210 with 5 folder\n\n\n**update** 6-15:\n\n0.2208 with 5 folder\n\n**update** 6-20:\n\n0.2205 with 5 folder\n\nfinal:\n\n0.2204 with 10 folder", "330553": "Edit : 0.2249\n\nEdit: 0.2252 with Image_top_1 feature included \n\nEdit: 0.2259 with RNN \n\n0.2276 with RNN, Fast Text Embeddings . - No Tuning done, A lot of Scope for improvement and tuning. \nLink to Public kernel is here \n\n[0.2276 RNN - Good for Blend :)][1]\n\n\n [1]: https://www.kaggle.com/shanth84/avito-fast-text-keras-model/code", "329707": "0.2223 with 5fold LGB model (all features are taken from public kernels). ", "329210": "Single lgbm 0.223. ", "329185": "Single lgbm 0.2215.\nBut using image features from https://www.kaggle.com/bguberfain/vgg16-train-features , my score got worse.\nI would like to know how to use image feature.", "329122": "Single lgbm 0.2229", "329052": "LB 0.2249 with only sample train and test dataset via lightgbm", "328924": "My best single NN is 0.2233 without image nor active and period files.", "336777": "NN:\n0.2213 (Categorical, Numerical, W2V)\n10-fold", "335073": "one LightGBM model 5kfold - 0,2211 (UPD)", "331503": "Update: 0.2220 with variation of the 'ridge trick', 5fold lgbm.", "340394": "10-fold LGB, LB 0.2214.\n\nTraining a model including some image features. Hopefully it would give me a boost. Finger crossed. \n\nUpdate: Including image featuers, 10-fold LGB, LB 0.2210\n\n6-15 Update: manage to improve my LGB to LB 0.2202\n\n6-19 Update: finally got an LGB with LB 0.2199, XGB with LB 0.2209. ", "334596": "Right now I have 3 models in parallel. A catboost at 0.2237, a LGB at 0.2228 and a Neural network at 0.2240. I'm really impressed by people achieving 0.222X with neural network on that data set. Maybe I'm missing something fundamental or some kind of clever encoding of categoricals.\n\nNot yet used additional FE from satellite data sets.", "334448": "I got 0.223 using just train.csv and test.csv,How to using train_active.csv which has no labels", "330469": "I've started with catboost and currently sits at 0.2246 with just train.csv. Edit: (Improved it to 0.2237)\n\nI'm currently hoping to get further with LGB because of the ability to use sparse matrix.\n\n", "329704": "0.2222 with one lgb model", "329226": "0.2228 with one lgb", "329137": "LGBM with 5-fold CV 0.2231", "334810": "", "330081": "", "333907": "thank you" } }