First of all I should thank Kaggle for this wonderful competition. As this is my first competition I was more in a learning mode than in contribution mode. Thoughts and Ideas from different people have helped me in understanding the ideology and process of solving a problem.
\n\nI'm more than happy to thank the people who have directly or indirectly helped me to gain a lot of knowledge with this competition.
\n\n@pranav84, @cpmpml, @anttip, @authman, @yuliagm, @nanomathias, @anokas, @bk0000, @aharless, @tunguz, @tetyanayatsenko, @sohaibomar, @tpthegreat, @aquatic, @chengju, @rrqqmm, @hengck23, @wythhh, @mrbeer
\n\nLessons Learnt
\n\nFirst of all I should thank Kaggle for this wonderful competition. As this is my first competition I was more in a learning mode than in contribution mode. Thoughts and Ideas from different people have helped me in understanding the ideology and process of solving a problem.
\n\nI'm more than happy to thank the people who have directly or indirectly helped me to gain a lot of knowledge with this competition.
\n\n@pranav84, @cpmpml, @anttip, @authman, @yuliagm, @nanomathias, @anokas, @bk0000, @aharless, @tunguz, @tetyanayatsenko, @sohaibomar, @tpthegreat, @aquatic, @chengju, @rrqqmm, @hengck23, @wythhh, @mrbeer
\n\nLessons Learnt
\n\nThanks Samrat, usually I'm less active in the board, this time I found something that helped me and wanted to help others as well, as they helped me.
\n\nFor me, this competition taught me:\n - How I should optimize memory
\n\nImportance of an accurate CV. My iterations were slow, but the score improved after each iteration.
Correct workflow order -> CV, feature engineering, algorithm optimisation, ensemble.
Using pandas builtin methods - because of the sheer size of the data, lambda functions were extremely slow. I had to use dt, isin, multiindex groupby, multi index sort
Things I want to learn in the next competition:\n- working in a team\n- improving my deep learning ability.\n- buying more 32GB RAM
", "rawMarkdown": "Thanks Samrat, usually I'm less active in the board, this time I found something that helped me and wanted to help others as well, as they helped me.\n\nFor me, this competition taught me:\n - How I should optimize memory\n\n - Importance of an accurate CV. My iterations were slow, but the score improved after each iteration.\n\n - Correct workflow order -> CV, feature engineering, algorithm optimisation, ensemble.\n\n - Using pandas builtin methods - because of the sheer size of the data, lambda functions were extremely slow. I had to use dt, isin, multiindex groupby, multi index sort\n\nThings I want to learn in the next competition:\n- working in a team\n- improving my deep learning ability.\n- buying more 32GB RAM", "votes": 14, "replies": [ { "id": 323527, "postDate": "2018-05-05T11:17:20.293Z", "content": "\n\n\nCorrect workflow order -> CV, feature engineering, algorithm optimisation, ensemble.
\n
Right order indeed. I'm at 'algorithm optimisaiton', if this is how you call algorithm parameter tuning (aka hyper parameter optimization) ;) Hope to have time to do some ensembling, but time is running short.
", "rawMarkdown": "> Correct workflow order -> CV, feature engineering, algorithm optimisation, ensemble.\n\nRight order indeed. I'm at 'algorithm optimisaiton', if this is how you call algorithm parameter tuning (aka hyper parameter optimization) ;) Hope to have time to do some ensembling, but time is running short.\n", "votes": 2 }, { "id": 323536, "postDate": "2018-05-05T12:28:14.623Z", "content": "Very much looking forward to your post-competition CV-building tutorials @cpmpml :-)
", "rawMarkdown": "Very much looking forward to your post-competition CV-building tutorials @cpmpml :-)", "votes": 1 }, { "id": 323543, "postDate": "2018-05-05T12:46:33.883Z", "content": "Buying more 32GB RAM, LOL!!! What I've learnt, becoming rich next time. ToT
", "rawMarkdown": "Buying more 32GB RAM, LOL!!! What I've learnt, becoming rich next time. ToT" }, { "id": 323551, "postDate": "2018-05-05T13:07:41.123Z", "content": "@authman, your suggestion to use 'two_round':True helped me a lot. I can now add way more features that overfit ;)
\n\nMore seriously, this is a great improvement, peak memory use was divided by almost 2. Thanks.
", "rawMarkdown": "@authman, your suggestion to use 'two_round':True helped me a lot. I can now add way more features that overfit ;) \n\nMore seriously, this is a great improvement, peak memory use was divided by almost 2. Thanks.", "votes": 2 }, { "id": 323625, "postDate": "2018-05-05T18:03:44.473Z", "content": "@CPMP I grinded 0.0005 from ensembling :), and I meant for hyper parameter tuning. \nGood luck
", "rawMarkdown": "@CPMP I grinded 0.0005 from ensembling :), and I meant for hyper parameter tuning. \nGood luck", "votes": 1 }, { "id": 323628, "postDate": "2018-05-05T18:11:13.110Z", "content": "@Yair Beer, Good, I have some hope for the next 2 days then ;) So far I was just averaging few runs, and get a 0.0001 uplift at most.
", "rawMarkdown": "@Yair Beer, Good, I have some hope for the next 2 days then ;) So far I was just averaging few runs, and get a 0.0001 uplift at most.", "votes": 2 }, { "id": 323780, "postDate": "2018-05-06T07:39:15.670Z", "content": "@Yair Beer\nCould you please share what you mean by \"CV\" as a first thing to do?\nYou mean that before feature engineering you first find a good way to split the data into Train and CV?
\n\nIsn't this something you do after you engineer some features?
", "rawMarkdown": "@Yair Beer\nCould you please share what you mean by \"CV\" as a first thing to do?\nYou mean that before feature engineering you first find a good way to split the data into Train and CV?\n\nIsn't this something you do after you engineer some features?" }, { "id": 324111, "postDate": "2018-05-07T07:11:08.440Z", "content": "Hi Amir,\nYou need to have a proper CV to understand which features would actually improve the test rather than overfit. A good example for me was the \"ip\" feature.\nI used the test hours inside the 9th day for evaluation and trained using (most of the time) only day 8.
\n\nWhen I wanted to speed things up for the hyper parameter optimization I used only third of the rows
\n\ntrain = train.iloc[::3,:]\nUnfortunately it has to me done after the feature creation stage because the feature use memory (such as delta time) which I wanted to preserve.
", "rawMarkdown": "Hi Amir,\nYou need to have a proper CV to understand which features would actually improve the test rather than overfit. A good example for me was the \"ip\" feature.\nI used the test hours inside the 9th day for evaluation and trained using (most of the time) only day 8.\n\nWhen I wanted to speed things up for the hyper parameter optimization I used only third of the rows \n\n train = train.iloc[::3,:]\n\nUnfortunately it has to me done after the feature creation stage because the feature use memory (such as delta time) which I wanted to preserve.", "votes": 1 }, { "id": 324131, "postDate": "2018-05-07T08:05:33.550Z", "content": "@Yair Beer\nThanks for our response!\nI understand the importance of CV of course.\n2 things:\n1 - give that your CV is good, How do you know if a feature overfits? will it simply be of low importance? or the AUC would be lower with it rather than without it? or maybe the AUC goes up but the TRAINING vs CV AUC difference goes up as well?
\n\n2 - I considered optimizing hyper parameters with less data but then i decided to do it with full data, although slower, i thought that many hyperparameters like \"min_child_weight\" or \"min_data_in_leaf\" or stuff like that, are really dependent on the size of the actual data you are training on.\nWhat is your take on that?
\n\nThanks again!
", "rawMarkdown": "@Yair Beer\nThanks for our response!\nI understand the importance of CV of course.\n2 things:\n1 - give that your CV is good, How do you know if a feature overfits? will it simply be of low importance? or the AUC would be lower with it rather than without it? or maybe the AUC goes up but the TRAINING vs CV AUC difference goes up as well?\n\n2 - I considered optimizing hyper parameters with less data but then i decided to do it with full data, although slower, i thought that many hyperparameters like \"min_child_weight\" or \"min_data_in_leaf\" or stuff like that, are really dependent on the size of the actual data you are training on.\nWhat is your take on that?\n\nThanks again!" }, { "id": 324136, "postDate": "2018-05-07T08:22:03.053Z", "content": "for lgm, the early stopping would kick in sooner. For algorithms without early stopping the gap between the train and test would be larger.
You are right. But it works for most of them and you can also create the parameter value as an percent oc the total number of samples.
@Yair Beer
\n\n1 - Let's take for example a model that gives Training AUC 0.983 and CV AUC of 0.981,\nAdding an overfitting feature might take us to maybe Training AUC 0.99 and CV AUC of 0.982.\nIn terms of overfitting, we do see the gap between the 2 as very large, but the overall CV AUC still went up a notch.\nWould you use such a feature?
\n\n@amirh in this competition I would because I found the CV to be very consistent. There could be competitions where I wouldn't dare.
", "rawMarkdown": "@amirh in this competition I would because I found the CV to be very consistent. There could be competitions where I wouldn't dare." }, { "id": 324182, "postDate": "2018-05-07T10:25:27.820Z", "content": "@AmirH, @Yair, very interesting discussion. I often ask myself similar questions. Here is one I think I can answer because I had the case in this competition.
\n\n\n\n\n1 - Let's take for example a model that gives Training AUC 0.983 and CV AUC of 0.981, Adding an overfitting feature might take us to maybe Training AUC 0.99 and CV AUC of 0.982. In terms of overfitting, we do see the gap between the 2 as very large, but the overall CV AUC still went up a notch. Would you use such a feature?
\n
I would much prefer the former, as it shows a much better generalization power. In my case, the public LB score of the latter was way lower. Here are the actual values:
\n\nTrain score: CV score: LB score\n\n0.99004 0.98040 0.9675\n\n0.98313 0.97937 0.9694\nIn order to detect this form of overfiting I often, if not always, look at the gap between train and validation. In this competition computing the train metrics is really time consuming, hence I skipped it in the last week or so.
", "rawMarkdown": "@AmirH, @Yair, very interesting discussion. I often ask myself similar questions. Here is one I think I can answer because I had the case in this competition. \n\n> 1 - Let's take for example a model that gives Training AUC 0.983 and CV AUC of 0.981, Adding an overfitting feature might take us to maybe Training AUC 0.99 and CV AUC of 0.982. In terms of overfitting, we do see the gap between the 2 as very large, but the overall CV AUC still went up a notch. Would you use such a feature?\n\nI would much prefer the former, as it shows a much better generalization power. In my case, the public LB score of the latter was way lower. Here are the actual values:\n\n Train score:\t\tCV score:\t\tLB score\n \n 0.99004\t\t\t0.98040\t\t\t0.9675\n \n 0.98313\t\t\t0.97937 \t\t0.9694\n\nIn order to detect this form of overfiting I often, if not always, look at the gap between train and validation. In this competition computing the train metrics is really time consuming, hence I skipped it in the last week or so." } ] }, { "id": 323421, "postDate": "2018-05-05T05:26:31.263Z", "content": "Thank you too. Very pleased to be on your list of people. Your action list is great but it lacks an important one I discussed in Advice to newbies: test your ideas yourself instead of asking if they can work.
", "rawMarkdown": "Thank you too. Very pleased to be on your list of people. Your action list is great but it lacks an important one I discussed in [Advice to newbies][1]: test your ideas yourself instead of asking if they can work.\n\n\n [1]: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/53251", "votes": 7, "replies": [ { "id": 323723, "postDate": "2018-05-06T02:12:52.500Z", "content": "Thanks for @CPMP,your advice is helpful
", "rawMarkdown": "Thanks for @CPMP,your advice is helpful", "votes": 2 } ] }, { "id": 323404, "postDate": "2018-05-05T03:33:18.930Z", "content": "Thank you Samrat, for this post and for your own contributions. It never ceases to amaze me how many good, talented Data Scientists are out there, and in every competition I manage to learn so much from “novices”. Your “Lessons Learned” is a valuable list for Kagglers of all levels of experience. Good luck with the rest of this competition, and hope to see you in many more in the future.
", "rawMarkdown": "Thank you Samrat, for this post and for your own contributions. It never ceases to amaze me how many good, talented Data Scientists are out there, and in every competition I manage to learn so much from “novices”. Your “Lessons Learned” is a valuable list for Kagglers of all levels of experience. Good luck with the rest of this competition, and hope to see you in many more in the future.", "votes": 6 }, { "id": 323497, "postDate": "2018-05-05T09:48:08.630Z", "content": "Thanks for the mention Samrat :) I'm very glad to hear that. Personally, I learn a lot from community in every Kaggle competition and wanted to say a huge thanks to fellow Kagglers too for sharing their innovative ideas/ approaches, feedback and suggestions.
", "rawMarkdown": "Thanks for the mention Samrat :) I'm very glad to hear that. Personally, I learn a lot from community in every Kaggle competition and wanted to say a huge thanks to fellow Kagglers too for sharing their innovative ideas/ approaches, feedback and suggestions. ", "votes": 5 }, { "id": 323540, "postDate": "2018-05-05T12:35:06.943Z", "content": "TY For the mention :-). My favorites this competition are:
\n\n@aharless - Andy Harless: Currently ranked Top 5 in both Discussions and Kernels on Kaggle, he's no doubt a hard worker. For this competition alone, he's put out 82 public kernels (wtx?)! Really democratizing his knowledge.
\n\nAnd, @nanomathias - Nano Mathias: Creator of the famous Feature Engineering & Importance Testing kernel. This kernel experimented with many features and feature groupings that people then expanded upon in their own kernels, and served as a good spring board for FE.
", "rawMarkdown": "TY For the mention :-). My favorites this competition are:\n\n@aharless - Andy Harless: Currently ranked Top 5 in both Discussions and Kernels on Kaggle, he's no doubt a hard worker. For this competition alone, he's put out **82 public kernels** (wtx?)! Really democratizing his knowledge.\n\nAnd, @nanomathias - Nano Mathias: Creator of the famous Feature Engineering & Importance Testing kernel. This kernel experimented with many features and feature groupings that people then expanded upon in their own kernels, and served as a good spring board for FE.\n", "votes": 6 }, { "id": 323781, "postDate": "2018-05-06T07:41:04.183Z", "content": "Thanks a lot @Samrat,\nIt's my first competition also, and I have learned A LOT (I think...)\nThanks for sharing your thoughts!
", "rawMarkdown": "Thanks a lot @Samrat,\nIt's my first competition also, and I have learned A LOT (I think...)\nThanks for sharing your thoughts!", "votes": 3 }, { "id": 323558, "postDate": "2018-05-05T13:36:47.380Z", "content": "Thank you Samrat :)
\n\nYour lessons are all correct and good things for us all to follow. Good luck!
", "rawMarkdown": "Thank you Samrat :)\n\nYour lessons are all correct and good things for us all to follow. Good luck!", "votes": 4 }, { "id": 323482, "postDate": "2018-05-05T08:31:21.687Z", "content": "Thank you too. :) And Good Luck for private LB.
", "rawMarkdown": "Thank you too. :) And Good Luck for private LB.", "votes": 4 }, { "id": 323648, "postDate": "2018-05-05T19:01:58.217Z", "content": "And thank you too, your discussion give me the motivation to tune the single model.
", "rawMarkdown": "And thank you too, your discussion give me the motivation to tune the single model.", "votes": 1 }, { "id": 323552, "postDate": "2018-05-05T13:09:19.970Z", "content": "Great list ! Thanks for the share ! The Kaggle community is amazing to learn in this field
", "rawMarkdown": "Great list ! Thanks for the share ! The Kaggle community is amazing to learn in this field ", "votes": 1 }, { "id": 324212, "postDate": "2018-05-07T11:59:18.457Z", "content": "Another lesson learnt. don't leave the competition for a few days , it is better to give some time daily, i left for a few days, lost considerable ground, and now with a few hours left, struggling to improve my model!
", "rawMarkdown": "Another lesson learnt. don't leave the competition for a few days , it is better to give some time daily, i left for a few days, lost considerable ground, and now with a few hours left, struggling to improve my model!", "votes": 2, "replies": [ { "id": 324237, "postDate": "2018-05-07T13:16:13.487Z", "content": "I feel you on this one. I left for a week, lost a potential team-up, lost ~300 spots on LB, and never fully recovered.
", "rawMarkdown": "I feel you on this one. I left for a week, lost a potential team-up, lost ~300 spots on LB, and never fully recovered.", "votes": 1 } ] }, { "id": 323722, "postDate": "2018-05-06T02:11:41.180Z", "content": "People in kaggle are so nice, and willing to share knowledge
", "rawMarkdown": "People in kaggle are so nice, and willing to share knowledge", "votes": 2 }, { "id": 323661, "postDate": "2018-05-05T19:54:00.377Z", "content": "I am new in Kaggle and learned a lot from it, the community here are the most open to share knowledge and reply to questions and the list you shared are awsome.
\n\nIn this completion I learned: \n - How to use Light GBM in depth.\n - How to work with big data and utilize the RAM to maximum.\n - Learned new models based on wordbatch.\n - New ideas for feature engineering and how one good feature can enhance the score.
", "rawMarkdown": "I am new in Kaggle and learned a lot from it, the community here are the most open to share knowledge and reply to questions and the list you shared are awsome.\n\nIn this completion I learned: \n - How to use Light GBM in depth.\n - How to work with big data and utilize the RAM to maximum.\n - Learned new models based on wordbatch.\n - New ideas for feature engineering and how one good feature can enhance the score.", "votes": 2 }, { "id": 323502, "postDate": "2018-05-05T10:02:17.777Z", "content": "Well done Samrat! I guess you just have to take the jump, and not wait too long. I am also increasingly keeping an eye on new comps......and am expecting to get stuck many times ;-)
", "rawMarkdown": "Well done Samrat! I guess you just have to take the jump, and not wait too long. I am also increasingly keeping an eye on new comps......and am expecting to get stuck many times ;-)", "votes": 2 }, { "id": 324128, "postDate": "2018-05-07T07:58:17.203Z", "content": "Good Luck Samrat
", "rawMarkdown": "Good Luck Samrat" }, { "id": 324068, "postDate": "2018-05-07T04:36:07.757Z", "content": "Started late, My first competiton as well thanks for good list of thumb rules, still a lot to learn
", "rawMarkdown": "Started late, My first competiton as well thanks for good list of thumb rules, still a lot to learn" }, { "id": 324062, "postDate": "2018-05-07T03:43:57.243Z", "content": "good~
", "rawMarkdown": "good~" }, { "id": 323783, "postDate": "2018-05-06T07:43:48.983Z", "content": "Super work Samrat...
", "rawMarkdown": "Super work Samrat..." }, { "id": 323773, "postDate": "2018-05-06T07:01:01.440Z", "content": "I am new too in Kaggle competition, and learned many things from many people in this forum. Thanks for everyone.
", "rawMarkdown": "I am new too in Kaggle competition, and learned many things from many people in this forum. Thanks for everyone." }, { "id": 324006, "postDate": "2018-05-06T22:27:21.627Z", "rawMarkdown": "", "votes": -8, "isDeleted": true }, { "id": 323933, "postDate": "2018-05-06T17:53:44.807Z", "content": "Thanks for that list on lessons!
", "rawMarkdown": "Thanks for that list on lessons!\n" }, { "id": 323891, "postDate": "2018-05-06T15:03:06.493Z", "content": "Thanks.
", "rawMarkdown": "Thanks." } ], "comments": [ { "id": 323450, "author_name": "Yair Beer", "author_url": "", "post_date": "2018-05-05T07:15:41.127000", "content": "Thanks Samrat, usually I'm less active in the board, this time I found something that helped me and wanted to help others as well, as they helped me.
\n\nFor me, this competition taught me:\n - How I should optimize memory
\n\nImportance of an accurate CV. My iterations were slow, but the score improved after each iteration.
Correct workflow order -> CV, feature engineering, algorithm optimisation, ensemble.
Using pandas builtin methods - because of the sheer size of the data, lambda functions were extremely slow. I had to use dt, isin, multiindex groupby, multi index sort
Things I want to learn in the next competition:\n- working in a team\n- improving my deep learning ability.\n- buying more 32GB RAM
", "votes": 14, "replies": [ { "id": 323527, "author_name": "CPMP", "author_url": "", "post_date": "2018-05-05T11:17:20.293000", "content": "\n\n\nCorrect workflow order -> CV, feature engineering, algorithm optimisation, ensemble.
\n
Right order indeed. I'm at 'algorithm optimisaiton', if this is how you call algorithm parameter tuning (aka hyper parameter optimization) ;) Hope to have time to do some ensembling, but time is running short.
", "votes": 2, "replies": [] }, { "id": 323536, "author_name": "عثمان", "author_url": "", "post_date": "2018-05-05T12:28:14.623000", "content": "Very much looking forward to your post-competition CV-building tutorials @cpmpml :-)
", "votes": 1, "replies": [] }, { "id": 323543, "author_name": "Laevatein", "author_url": "", "post_date": "2018-05-05T12:46:33.883000", "content": "Buying more 32GB RAM, LOL!!! What I've learnt, becoming rich next time. ToT
", "votes": 0, "replies": [] }, { "id": 323551, "author_name": "CPMP", "author_url": "", "post_date": "2018-05-05T13:07:41.123000", "content": "@authman, your suggestion to use 'two_round':True helped me a lot. I can now add way more features that overfit ;)
\n\nMore seriously, this is a great improvement, peak memory use was divided by almost 2. Thanks.
", "votes": 2, "replies": [] }, { "id": 323625, "author_name": "Yair Beer", "author_url": "", "post_date": "2018-05-05T18:03:44.473000", "content": "@CPMP I grinded 0.0005 from ensembling :), and I meant for hyper parameter tuning. \nGood luck
", "votes": 1, "replies": [] }, { "id": 323628, "author_name": "CPMP", "author_url": "", "post_date": "2018-05-05T18:11:13.110000", "content": "@Yair Beer, Good, I have some hope for the next 2 days then ;) So far I was just averaging few runs, and get a 0.0001 uplift at most.
", "votes": 2, "replies": [] }, { "id": 323780, "author_name": "AmirH", "author_url": "", "post_date": "2018-05-06T07:39:15.670000", "content": "@Yair Beer\nCould you please share what you mean by \"CV\" as a first thing to do?\nYou mean that before feature engineering you first find a good way to split the data into Train and CV?
\n\nIsn't this something you do after you engineer some features?
", "votes": 0, "replies": [] }, { "id": 324111, "author_name": "Yair Beer", "author_url": "", "post_date": "2018-05-07T07:11:08.440000", "content": "Hi Amir,\nYou need to have a proper CV to understand which features would actually improve the test rather than overfit. A good example for me was the \"ip\" feature.\nI used the test hours inside the 9th day for evaluation and trained using (most of the time) only day 8.
\n\nWhen I wanted to speed things up for the hyper parameter optimization I used only third of the rows
\n\ntrain = train.iloc[::3,:]\nUnfortunately it has to me done after the feature creation stage because the feature use memory (such as delta time) which I wanted to preserve.
", "votes": 1, "replies": [] }, { "id": 324131, "author_name": "AmirH", "author_url": "", "post_date": "2018-05-07T08:05:33.550000", "content": "@Yair Beer\nThanks for our response!\nI understand the importance of CV of course.\n2 things:\n1 - give that your CV is good, How do you know if a feature overfits? will it simply be of low importance? or the AUC would be lower with it rather than without it? or maybe the AUC goes up but the TRAINING vs CV AUC difference goes up as well?
\n\n2 - I considered optimizing hyper parameters with less data but then i decided to do it with full data, although slower, i thought that many hyperparameters like \"min_child_weight\" or \"min_data_in_leaf\" or stuff like that, are really dependent on the size of the actual data you are training on.\nWhat is your take on that?
\n\nThanks again!
", "votes": 0, "replies": [] }, { "id": 324136, "author_name": "Yair Beer", "author_url": "", "post_date": "2018-05-07T08:22:03.053000", "content": "for lgm, the early stopping would kick in sooner. For algorithms without early stopping the gap between the train and test would be larger.
You are right. But it works for most of them and you can also create the parameter value as an percent oc the total number of samples.
@Yair Beer
\n\n1 - Let's take for example a model that gives Training AUC 0.983 and CV AUC of 0.981,\nAdding an overfitting feature might take us to maybe Training AUC 0.99 and CV AUC of 0.982.\nIn terms of overfitting, we do see the gap between the 2 as very large, but the overall CV AUC still went up a notch.\nWould you use such a feature?
\n\n@amirh in this competition I would because I found the CV to be very consistent. There could be competitions where I wouldn't dare.
", "votes": 0, "replies": [] }, { "id": 324182, "author_name": "CPMP", "author_url": "", "post_date": "2018-05-07T10:25:27.820000", "content": "@AmirH, @Yair, very interesting discussion. I often ask myself similar questions. Here is one I think I can answer because I had the case in this competition.
\n\n\n\n\n1 - Let's take for example a model that gives Training AUC 0.983 and CV AUC of 0.981, Adding an overfitting feature might take us to maybe Training AUC 0.99 and CV AUC of 0.982. In terms of overfitting, we do see the gap between the 2 as very large, but the overall CV AUC still went up a notch. Would you use such a feature?
\n
I would much prefer the former, as it shows a much better generalization power. In my case, the public LB score of the latter was way lower. Here are the actual values:
\n\nTrain score: CV score: LB score\n\n0.99004 0.98040 0.9675\n\n0.98313 0.97937 0.9694\nIn order to detect this form of overfiting I often, if not always, look at the gap between train and validation. In this competition computing the train metrics is really time consuming, hence I skipped it in the last week or so.
", "votes": 0, "replies": [] } ] }, { "id": 323421, "author_name": "CPMP", "author_url": "", "post_date": "2018-05-05T05:26:31.263000", "content": "Thank you too. Very pleased to be on your list of people. Your action list is great but it lacks an important one I discussed in Advice to newbies: test your ideas yourself instead of asking if they can work.
", "votes": 7, "replies": [ { "id": 323723, "author_name": "Johnny Liu", "author_url": "", "post_date": "2018-05-06T02:12:52.500000", "content": "Thanks for @CPMP,your advice is helpful
", "votes": 2, "replies": [] } ] }, { "id": 323404, "author_name": "Bojan Tunguz", "author_url": "", "post_date": "2018-05-05T03:33:18.930000", "content": "Thank you Samrat, for this post and for your own contributions. It never ceases to amaze me how many good, talented Data Scientists are out there, and in every competition I manage to learn so much from “novices”. Your “Lessons Learned” is a valuable list for Kagglers of all levels of experience. Good luck with the rest of this competition, and hope to see you in many more in the future.
", "votes": 6, "replies": [] }, { "id": 323497, "author_name": "Pranav Pandya", "author_url": "", "post_date": "2018-05-05T09:48:08.630000", "content": "Thanks for the mention Samrat :) I'm very glad to hear that. Personally, I learn a lot from community in every Kaggle competition and wanted to say a huge thanks to fellow Kagglers too for sharing their innovative ideas/ approaches, feedback and suggestions.
", "votes": 5, "replies": [] }, { "id": 323540, "author_name": "عثمان", "author_url": "", "post_date": "2018-05-05T12:35:06.943000", "content": "TY For the mention :-). My favorites this competition are:
\n\n@aharless - Andy Harless: Currently ranked Top 5 in both Discussions and Kernels on Kaggle, he's no doubt a hard worker. For this competition alone, he's put out 82 public kernels (wtx?)! Really democratizing his knowledge.
\n\nAnd, @nanomathias - Nano Mathias: Creator of the famous Feature Engineering & Importance Testing kernel. This kernel experimented with many features and feature groupings that people then expanded upon in their own kernels, and served as a good spring board for FE.
", "votes": 6, "replies": [] }, { "id": 323781, "author_name": "AmirH", "author_url": "", "post_date": "2018-05-06T07:41:04.183000", "content": "Thanks a lot @Samrat,\nIt's my first competition also, and I have learned A LOT (I think...)\nThanks for sharing your thoughts!
", "votes": 3, "replies": [] }, { "id": 323558, "author_name": "anokas", "author_url": "", "post_date": "2018-05-05T13:36:47.380000", "content": "Thank you Samrat :)
\n\nYour lessons are all correct and good things for us all to follow. Good luck!
", "votes": 4, "replies": [] }, { "id": 323482, "author_name": "Sohaib Omar", "author_url": "", "post_date": "2018-05-05T08:31:21.687000", "content": "Thank you too. :) And Good Luck for private LB.
", "votes": 4, "replies": [] }, { "id": 323648, "author_name": "Toaru", "author_url": "", "post_date": "2018-05-05T19:01:58.217000", "content": "And thank you too, your discussion give me the motivation to tune the single model.
", "votes": 1, "replies": [] }, { "id": 323552, "author_name": "Nathan Lauga", "author_url": "", "post_date": "2018-05-05T13:09:19.970000", "content": "Great list ! Thanks for the share ! The Kaggle community is amazing to learn in this field
", "votes": 1, "replies": [] }, { "id": 324212, "author_name": "nickhillator", "author_url": "", "post_date": "2018-05-07T11:59:18.457000", "content": "Another lesson learnt. don't leave the competition for a few days , it is better to give some time daily, i left for a few days, lost considerable ground, and now with a few hours left, struggling to improve my model!
", "votes": 2, "replies": [ { "id": 324237, "author_name": "عثمان", "author_url": "", "post_date": "2018-05-07T13:16:13.487000", "content": "I feel you on this one. I left for a week, lost a potential team-up, lost ~300 spots on LB, and never fully recovered.
", "votes": 1, "replies": [] } ] }, { "id": 323722, "author_name": "Johnny Liu", "author_url": "", "post_date": "2018-05-06T02:11:41.180000", "content": "People in kaggle are so nice, and willing to share knowledge
", "votes": 2, "replies": [] }, { "id": 323661, "author_name": "A.Barqawi", "author_url": "", "post_date": "2018-05-05T19:54:00.377000", "content": "I am new in Kaggle and learned a lot from it, the community here are the most open to share knowledge and reply to questions and the list you shared are awsome.
\n\nIn this completion I learned: \n - How to use Light GBM in depth.\n - How to work with big data and utilize the RAM to maximum.\n - Learned new models based on wordbatch.\n - New ideas for feature engineering and how one good feature can enhance the score.
", "votes": 2, "replies": [] }, { "id": 323502, "author_name": "Erik Bruin", "author_url": "", "post_date": "2018-05-05T10:02:17.777000", "content": "Well done Samrat! I guess you just have to take the jump, and not wait too long. I am also increasingly keeping an eye on new comps......and am expecting to get stuck many times ;-)
", "votes": 2, "replies": [] }, { "id": 324128, "author_name": "Mahesh Kulkarni", "author_url": "", "post_date": "2018-05-07T07:58:17.203000", "content": "Good Luck Samrat
", "votes": 0, "replies": [] }, { "id": 324068, "author_name": "Manimaran Paneerselvam", "author_url": "", "post_date": "2018-05-07T04:36:07.757000", "content": "Started late, My first competiton as well thanks for good list of thumb rules, still a lot to learn
", "votes": 0, "replies": [] }, { "id": 324062, "author_name": "Agoer", "author_url": "", "post_date": "2018-05-07T03:43:57.243000", "content": "good~
", "votes": 0, "replies": [] }, { "id": 323783, "author_name": "Spidy", "author_url": "", "post_date": "2018-05-06T07:43:48.983000", "content": "Super work Samrat...
", "votes": 0, "replies": [] }, { "id": 323773, "author_name": "sokazaki", "author_url": "", "post_date": "2018-05-06T07:01:01.440000", "content": "I am new too in Kaggle competition, and learned many things from many people in this forum. Thanks for everyone.
", "votes": 0, "replies": [] }, { "id": 324006, "author_name": "", "author_url": "", "post_date": "2018-05-06T22:27:21.627000", "content": "", "votes": -8, "replies": [] }, { "id": 323933, "author_name": "Ramesha C Gowda", "author_url": "", "post_date": "2018-05-06T17:53:44.807000", "content": "Thanks for that list on lessons!
", "votes": 0, "replies": [] }, { "id": 323891, "author_name": "Raajesh Laguduva Rameshbabu", "author_url": "", "post_date": "2018-05-06T15:03:06.493000", "content": "Thanks.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "323398": "First of all I should thank Kaggle for this wonderful competition. As this is my first competition I was more in a learning mode than in contribution mode. Thoughts and Ideas from different people have helped me in understanding the ideology and process of solving a problem.\n\nI'm more than happy to thank the people who have directly or indirectly helped me to gain a lot of knowledge with this competition.\n\n@pranav84, @cpmpml, @anttip, @authman, @yuliagm, @nanomathias, @anokas, @bk0000, @aharless, @tunguz, @tetyanayatsenko, @sohaibomar, @tpthegreat, @aquatic, @chengju, @rrqqmm, @hengck23, @wythhh, @mrbeer\n\n- PS: The above list was in no particular order except the first one :-) Though I never had an interaction with Pranav Pandya, his starter kernels have helped me a lot to kick of with this competition.\n- PPS: If you are wondering if you never interacted with me but your name is on the list then it could be that your contribution to the community helped me or it could be that I have assumed you as a pseudo competitor and trying to beat your score :-D\n\n**Lessons Learnt**\n\n- Don't be in a hurry.\n- Don't chase the LB from Day 1.\n- Do proper EDA. See what best you can do to understand the data.\n- Read the internal working mechanism of the algorithm\n- Follow the Discussions\n- Go through the kernels and even a small point can help you\n- Read the research papers in the related areas.\n- Maintain a proper version control\n- Try optimizing the code. It will save a lot of time and computation.\n- Take advice from experts\n- Try answering the questions from learners\n- Try cloud services to see if they can help with better computation resources\n- Have patience\n- Note down all your mistakes and this will definitely help us in the next competition.\n", "323450": "Thanks Samrat, usually I'm less active in the board, this time I found something that helped me and wanted to help others as well, as they helped me.\n\nFor me, this competition taught me:\n - How I should optimize memory\n\n - Importance of an accurate CV. My iterations were slow, but the score improved after each iteration.\n\n - Correct workflow order -> CV, feature engineering, algorithm optimisation, ensemble.\n\n - Using pandas builtin methods - because of the sheer size of the data, lambda functions were extremely slow. I had to use dt, isin, multiindex groupby, multi index sort\n\nThings I want to learn in the next competition:\n- working in a team\n- improving my deep learning ability.\n- buying more 32GB RAM", "323421": "Thank you too. Very pleased to be on your list of people. Your action list is great but it lacks an important one I discussed in [Advice to newbies][1]: test your ideas yourself instead of asking if they can work.\n\n\n [1]: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/53251", "323404": "Thank you Samrat, for this post and for your own contributions. It never ceases to amaze me how many good, talented Data Scientists are out there, and in every competition I manage to learn so much from “novices”. Your “Lessons Learned” is a valuable list for Kagglers of all levels of experience. Good luck with the rest of this competition, and hope to see you in many more in the future.", "323497": "Thanks for the mention Samrat :) I'm very glad to hear that. Personally, I learn a lot from community in every Kaggle competition and wanted to say a huge thanks to fellow Kagglers too for sharing their innovative ideas/ approaches, feedback and suggestions. ", "323540": "TY For the mention :-). My favorites this competition are:\n\n@aharless - Andy Harless: Currently ranked Top 5 in both Discussions and Kernels on Kaggle, he's no doubt a hard worker. For this competition alone, he's put out **82 public kernels** (wtx?)! Really democratizing his knowledge.\n\nAnd, @nanomathias - Nano Mathias: Creator of the famous Feature Engineering & Importance Testing kernel. This kernel experimented with many features and feature groupings that people then expanded upon in their own kernels, and served as a good spring board for FE.\n", "323781": "Thanks a lot @Samrat,\nIt's my first competition also, and I have learned A LOT (I think...)\nThanks for sharing your thoughts!", "323558": "Thank you Samrat :)\n\nYour lessons are all correct and good things for us all to follow. Good luck!", "323482": "Thank you too. :) And Good Luck for private LB.", "323648": "And thank you too, your discussion give me the motivation to tune the single model.", "323552": "Great list ! Thanks for the share ! The Kaggle community is amazing to learn in this field ", "324212": "Another lesson learnt. don't leave the competition for a few days , it is better to give some time daily, i left for a few days, lost considerable ground, and now with a few hours left, struggling to improve my model!", "323722": "People in kaggle are so nice, and willing to share knowledge", "323661": "I am new in Kaggle and learned a lot from it, the community here are the most open to share knowledge and reply to questions and the list you shared are awsome.\n\nIn this completion I learned: \n - How to use Light GBM in depth.\n - How to work with big data and utilize the RAM to maximum.\n - Learned new models based on wordbatch.\n - New ideas for feature engineering and how one good feature can enhance the score.", "323502": "Well done Samrat! I guess you just have to take the jump, and not wait too long. I am also increasingly keeping an eye on new comps......and am expecting to get stuck many times ;-)", "324128": "Good Luck Samrat", "324068": "Started late, My first competiton as well thanks for good list of thumb rules, still a lot to learn", "324062": "good~", "323783": "Super work Samrat...", "323773": "I am new too in Kaggle competition, and learned many things from many people in this forum. Thanks for everyone.", "324006": "", "323933": "Thanks for that list on lessons!\n", "323891": "Thanks." } }