To me, this competition is not worth spending too much time on extracting better features, or tuning models... A simple trick given the prior knowledge of private test set would win. The data is not statistical enough, and there’s no fun of climbing the LB like some optimization competitions (i.e., the annual Santa competitions by Kaggle). A strong model does not guarantee to win. The train/public/private sets are very much different. \nJust a tricky competition though.
\n\nFeature engineering and model build-up is ONLY effective if everybody is not given that kind of test leak. Otherwise, the leak cancels out all machine learning engineering efforts.
", "messages": [ { "id": "542667", "postDate": "06/04/2019 03:12:05", "content": "To me, this competition is not worth spending too much time on extracting better features, or tuning models... A simple trick given the prior knowledge of private test set would win. The data is not statistical enough, and there’s no fun of climbing the LB like some optimization competitions (i.e., the annual Santa competitions by Kaggle). A strong model does not guarantee to win. The train/public/private sets are very much different. \nJust a tricky competition though.
\n\nFeature engineering and model build-up is ONLY effective if everybody is not given that kind of test leak. Otherwise, the leak cancels out all machine learning engineering efforts.
", "rawMarkdown": "To me, this competition is not worth spending too much time on extracting better features, or tuning models... A simple trick given the prior knowledge of private test set would win. The data is not statistical enough, and there’s no fun of climbing the LB like some optimization competitions (i.e., the annual Santa competitions by Kaggle). A strong model does not guarantee to win. The train/public/private sets are very much different. \nJust a tricky competition though. \n\nFeature engineering and model build-up is ONLY effective if everybody is not given that kind of test leak. Otherwise, the leak cancels out all machine learning engineering efforts.", "votes": null }, { "id": "542674", "postDate": "06/04/2019 03:22:55", "content": "sadly as a novice, i spending much time here as my first competition T.T . but i learn a lot from you, cpmp,and other masters. As per suggestion by you, i agree that if i am not doing much time for extracting features and tuning models i should at least get medals from my earlier submissions.
", "rawMarkdown": "sadly as a novice, i spending much time here as my first competition T.T . but i learn a lot from you, cpmp,and other masters. As per suggestion by you, i agree that if i am not doing much time for extracting features and tuning models i should at least get medals from my earlier submissions.", "votes": null }, { "id": "542684", "postDate": "06/04/2019 03:30:07", "content": "If I know this competition has too many such pitfalls, I would have NOT participated in the first place. It took me too much time spending on extracting useful features and better models. In the end, my best submission scored 1.380 on public LB, which will give 2.37 on private LB. And some other guy discovered that a 1.800 public LB would also earn that equivalent private LB. I regret that I took too much time on this competition. If I just came in in the last month, I would have more joy, rather than sticking at the beginning and now see a lot of guys winning by pure luck or by some simple tricks. Of course there are teams with skills, but it's counted by fingers.
", "rawMarkdown": "If I know this competition has too many such pitfalls, I would have NOT participated in the first place. It took me too much time spending on extracting useful features and better models. In the end, my best submission scored 1.380 on public LB, which will give 2.37 on private LB. And some other guy discovered that a 1.800 public LB would also earn that equivalent private LB. I regret that I took too much time on this competition. If I just came in in the last month, I would have more joy, rather than sticking at the beginning and now see a lot of guys winning by pure luck or by some simple tricks. Of course there are teams with skills, but it's counted by fingers.", "votes": null }, { "id": "542700", "postDate": "06/04/2019 03:39:30", "content": "Yes, i saw the post where the public test score is >1.8 but able to get score <2.4 in private test.
\n\nI also spend many nights, lack of sleep for a month (as i join this competition about 1 month ago) .
\n\nBut i will be very pleased if you could share your approach, how you can manage to avoid a big shake up in the end. I admire everyone who in the top 100 in public test but still can manage to be in top 100 in private test. I think you are a true expert in this field and i eager to learn a lot from the best. Many thanks :)
", "rawMarkdown": "Yes, i saw the post where the public test score is >1.8 but able to get score <2.4 in private test.\n\nI also spend many nights, lack of sleep for a month (as i join this competition about 1 month ago) .\n\nBut i will be very pleased if you could share your approach, how you can manage to avoid a big shake up in the end. I admire everyone who in the top 100 in public test but still can manage to be in top 100 in private test. I think you are a true expert in this field and i eager to learn a lot from the best. Many thanks :)", "votes": null }, { "id": "542701", "postDate": "06/04/2019 03:40:34", "content": "Please don't be discouraged! You did an excellent job!! 23rd place is an amazing feat. And you learned a lot. Next time you can take this knowledge and go even further. Although 23rd is something to be really really proud of.
\n\nI totally understand how you feel though. Earlier this year I poured my heart into the NFL competition I really though I had a chance to win. When I lost I felt crushed and said \"I would have not participated in the first place if I had known\" also. I also can understand it must be frustrating knowing a leak is the reason you missed out. Next time...
", "rawMarkdown": "Please don't be discouraged! You did an excellent job!! 23rd place is an amazing feat. And you learned a lot. Next time you can take this knowledge and go even further. Although 23rd is something to be really really proud of.\n\nI totally understand how you feel though. Earlier this year I poured my heart into the NFL competition I really though I had a chance to win. When I lost I felt crushed and said \"I would have not participated in the first place if I had known\" also. I also can understand it must be frustrating knowing a leak is the reason you missed out. Next time...", "votes": null }, { "id": "542711", "postDate": "06/04/2019 03:52:52", "content": "What saved me is simply using a linear regression stack model on all oof predictions of other 1st level models (LGB, XGB, Catboost, RF, SVM), without any further manipulation. The public 1st level models scored around 1.29 to 1.31 on public LB. That stacked submission has worse validation MAE than any of the 1st level models, but better MSE. The submission file has much bigger mean and median than each of the 1st level model. I doubt that file very much, but in the end I still trust it anyway thanks to my intuition. The public LB is worse by 0.05. I almost did not select that file, and I would have ended up around rank 300+. Luckily that file gave me 23rd place. Not sure what to say more now, but I bet there is no CV that can win without prior knowledge of the private test set.
", "rawMarkdown": "What saved me is simply using a linear regression stack model on all oof predictions of other 1st level models (LGB, XGB, Catboost, RF, SVM), without any further manipulation. The public 1st level models scored around 1.29 to 1.31 on public LB. That stacked submission has worse validation MAE than any of the 1st level models, but better MSE. The submission file has much bigger mean and median than each of the 1st level model. I doubt that file very much, but in the end I still trust it anyway thanks to my intuition. The public LB is worse by 0.05. I almost did not select that file, and I would have ended up around rank 300+. Luckily that file gave me 23rd place. Not sure what to say more now, but I bet there is no CV that can win without prior knowledge of the private test set.", "votes": null }, { "id": "542723", "postDate": "06/04/2019 04:04:32", "content": "Thanks for the share Kha Vo
", "rawMarkdown": "Thanks for the share Kha Vo", "votes": null }, { "id": "542750", "postDate": "06/04/2019 04:29:20", "content": "I think the data was not enough to use all the power of Machine Learning. The chunks were too small and it was too few data for a RNN or CNN to work well. It is required more data (more experiments) and larger chunks to make state of the art predictions.
", "rawMarkdown": "I think the data was not enough to use all the power of Machine Learning. The chunks were too small and it was too few data for a RNN or CNN to work well. It is required more data (more experiments) and larger chunks to make state of the art predictions.", "votes": null }, { "id": "542777", "postDate": "06/04/2019 04:52:17", "content": "Yes. All the gambler wins, including me. I admitted that I managed to survive just by LUCK, by taking the knowledge of private test set. That submission give worse local CV MAE than some of my other traditional CV trained on pure train data only.
", "rawMarkdown": "Yes. All the gambler wins, including me. I admitted that I managed to survive just by LUCK, by taking the knowledge of private test set. That submission give worse local CV MAE than some of my other traditional CV trained on pure train data only.", "votes": null }, { "id": "542778", "postDate": "06/04/2019 04:52:46", "content": "Actually the prior knowledge of the testset is a form of leak. Fortunately the leak was disclosed in the forums
", "rawMarkdown": "Actually the prior knowledge of the testset is a form of leak. Fortunately the leak was disclosed in the [forums](https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/90664)", "votes": null }, { "id": "542779", "postDate": "06/04/2019 04:53:53", "content": "That makes this competition a tricky one. All efforts in feature engineering and model build up are ineffective.
", "rawMarkdown": "That makes this competition a tricky one. All efforts in feature engineering and model build up are ineffective.", "votes": null }, { "id": "543273", "postDate": "06/04/2019 12:34:09", "content": "I also learned that we need to use all the information that we can gather in order to win. Most of the competition contains some kind of \"leak\"(=information about test score) even though its leak type varies...
", "rawMarkdown": "I also learned that we need to use all the information that we can gather in order to win. Most of the competition contains some kind of \"leak\"(=information about test score) even though its leak type varies...", "votes": null }, { "id": "543281", "postDate": "06/04/2019 12:44:57", "content": "The leak was accessible to all, and what made a difference is machine learning.
\n\nI have a general additive model that cannot use sample weights, hence we could not tune it using test estimates. Yet, this model would claim 12th rank here. Just using 16 fold CV and proper feature engineering.
", "rawMarkdown": "The leak was accessible to all, and what made a difference is machine learning.\n\nI have a general additive model that cannot use sample weights, hence we could not tune it using test estimates. Yet, this model would claim 12th rank here. Just using 16 fold CV and proper feature engineering.", "votes": null }, { "id": "543331", "postDate": "06/04/2019 13:31:40", "content": "Except some exceptional models like yours, otherwise test leak overwhelm any power to get a good result.
", "rawMarkdown": "Except some exceptional models like yours, otherwise test leak overwhelm any power to get a good result.", "votes": null }, { "id": "543345", "postDate": "06/04/2019 13:40:16", "content": "I agree, and I had no idea that this model would fare that well. Using test estimates was clearly the way to do.
", "rawMarkdown": "I agree, and I had no idea that this model would fare that well. Using test estimates was clearly the way to do.", "votes": null }, { "id": "543358", "postDate": "06/04/2019 13:46:20", "content": "Exactly @khahuras . People should be complaining about the leakage. That made all the difference here.
", "rawMarkdown": "Exactly @khahuras . People should be complaining about the leakage. That made all the difference here.", "votes": null }, { "id": "543400", "postDate": "06/04/2019 14:09:43", "content": "Congrats everybody and thanks a lot for your shares but...
\n\nTRUST YOUR TEST-SET LEAK !!!
\n\nLook at the file:
\n\n\n\n\nversion 105 = version 104 * 1.1
\n \n\n\nversion 106 = version 104 * 1.2 (GOLD)
\n
I do not want to fine-tune more... Too much punishment!!!
\n\nI will try to do it better in the next competition.
", "rawMarkdown": "Congrats everybody and thanks a lot for your shares but... \n\nTRUST YOUR TEST-SET LEAK !!!\n\nLook at the file: \n\n> version 105 = version 104 * 1.1 \n>> version 106 = version 104 * 1.2 (GOLD)\n\nI do not want to fine-tune more... Too much punishment!!!\n\nI will try to do it better in the next competition.", "votes": null }, { "id": "543401", "postDate": "06/04/2019 14:09:56", "content": "I agree with Kha Vo's comments. I thought these machine learning competitions were supposed to be focused on identifying and developing best case machine learning techniques and not focused on detective work to find data leaks. In the last two competitions I have entered discovery and use of info associated with data leaks has been critical to success. I know I sound like sour grapes but I did not feel I learned much after spending a lot of time trying to define and improve useful features on a very random train data set.
\n\nThe very poor correlation between public and private LB results clearly kept a lot of people from developing better models. The purpose of this competition was supposed to be looking for ways to improve earthquake detection which would benefit everyone and I don't think it succeeded due to the quality and qty of info provided by LANL.
\n\nCPMP's results on the public and private LB were great but they are very much the exception. It looks like the average shift on the private LB for everyone else was very large.
", "rawMarkdown": "I agree with Kha Vo's comments. I thought these machine learning competitions were supposed to be focused on identifying and developing best case machine learning techniques and not focused on detective work to find data leaks. In the last two competitions I have entered discovery and use of info associated with data leaks has been critical to success. I know I sound like sour grapes but I did not feel I learned much after spending a lot of time trying to define and improve useful features on a very random train data set. \n\n The very poor correlation between public and private LB results clearly kept a lot of people from developing better models. The purpose of this competition was supposed to be looking for ways to improve earthquake detection which would benefit everyone and I don't think it succeeded due to the quality and qty of info provided by LANL. \n\n CPMP's results on the public and private LB were great but they are very much the exception. It looks like the average shift on the private LB for everyone else was very large.", "votes": null }, { "id": "543595", "postDate": "06/04/2019 16:08:01", "content": "i remember there was about 200 participants when I start working on this. I personally do not like tricky features, thus focused more on NN based models. I gave up after work on it for 3-4 of my weekends because i felt training set is small and quite different with testing. but there always a way, like looking at testing set and leaking set. I guess the organizers are a way too scientific research thinking but Kagglers have another way to success. anyway, it is not so bad to myself and worth the time i have spent.
", "rawMarkdown": "i remember there was about 200 participants when I start working on this. I personally do not like tricky features, thus focused more on NN based models. I gave up after work on it for 3-4 of my weekends because i felt training set is small and quite different with testing. but there always a way, like looking at testing set and leaking set. I guess the organizers are a way too scientific research thinking but Kagglers have another way to success. anyway, it is not so bad to myself and worth the time i have spent.", "votes": null }, { "id": "543789", "postDate": "06/04/2019 20:26:56", "content": "The focus should be on creating good models, that can extract good features from the data for predictions, not humanly creating 80-100 features.
", "rawMarkdown": "The focus should be on creating good models, that can extract good features from the data for predictions, not humanly creating 80-100 features.", "votes": null } ], "comments": [ { "id": 542674, "author_name": "arisukma", "author_url": "", "post_date": "06/04/2019 03:22:55", "content": "sadly as a novice, i spending much time here as my first competition T.T . but i learn a lot from you, cpmp,and other masters. As per suggestion by you, i agree that if i am not doing much time for extracting features and tuning models i should at least get medals from my earlier submissions.
", "votes": null, "replies": [ { "id": 542684, "author_name": "khahuras", "author_url": "", "post_date": "06/04/2019 03:30:07", "content": "If I know this competition has too many such pitfalls, I would have NOT participated in the first place. It took me too much time spending on extracting useful features and better models. In the end, my best submission scored 1.380 on public LB, which will give 2.37 on private LB. And some other guy discovered that a 1.800 public LB would also earn that equivalent private LB. I regret that I took too much time on this competition. If I just came in in the last month, I would have more joy, rather than sticking at the beginning and now see a lot of guys winning by pure luck or by some simple tricks. Of course there are teams with skills, but it's counted by fingers.
", "votes": null, "replies": [] }, { "id": 542700, "author_name": "arisukma", "author_url": "", "post_date": "06/04/2019 03:39:30", "content": "Yes, i saw the post where the public test score is >1.8 but able to get score <2.4 in private test.
\n\nI also spend many nights, lack of sleep for a month (as i join this competition about 1 month ago) .
\n\nBut i will be very pleased if you could share your approach, how you can manage to avoid a big shake up in the end. I admire everyone who in the top 100 in public test but still can manage to be in top 100 in private test. I think you are a true expert in this field and i eager to learn a lot from the best. Many thanks :)
", "votes": null, "replies": [] }, { "id": 542701, "author_name": "robikscube", "author_url": "", "post_date": "06/04/2019 03:40:34", "content": "Please don't be discouraged! You did an excellent job!! 23rd place is an amazing feat. And you learned a lot. Next time you can take this knowledge and go even further. Although 23rd is something to be really really proud of.
\n\nI totally understand how you feel though. Earlier this year I poured my heart into the NFL competition I really though I had a chance to win. When I lost I felt crushed and said \"I would have not participated in the first place if I had known\" also. I also can understand it must be frustrating knowing a leak is the reason you missed out. Next time...
", "votes": null, "replies": [] }, { "id": 542711, "author_name": "khahuras", "author_url": "", "post_date": "06/04/2019 03:52:52", "content": "What saved me is simply using a linear regression stack model on all oof predictions of other 1st level models (LGB, XGB, Catboost, RF, SVM), without any further manipulation. The public 1st level models scored around 1.29 to 1.31 on public LB. That stacked submission has worse validation MAE than any of the 1st level models, but better MSE. The submission file has much bigger mean and median than each of the 1st level model. I doubt that file very much, but in the end I still trust it anyway thanks to my intuition. The public LB is worse by 0.05. I almost did not select that file, and I would have ended up around rank 300+. Luckily that file gave me 23rd place. Not sure what to say more now, but I bet there is no CV that can win without prior knowledge of the private test set.
", "votes": null, "replies": [] }, { "id": 542723, "author_name": "arisukma", "author_url": "", "post_date": "06/04/2019 04:04:32", "content": "Thanks for the share Kha Vo
", "votes": null, "replies": [] } ] }, { "id": 542750, "author_name": "carlospk", "author_url": "", "post_date": "06/04/2019 04:29:20", "content": "I think the data was not enough to use all the power of Machine Learning. The chunks were too small and it was too few data for a RNN or CNN to work well. It is required more data (more experiments) and larger chunks to make state of the art predictions.
", "votes": null, "replies": [ { "id": 542777, "author_name": "khahuras", "author_url": "", "post_date": "06/04/2019 04:52:17", "content": "Yes. All the gambler wins, including me. I admitted that I managed to survive just by LUCK, by taking the knowledge of private test set. That submission give worse local CV MAE than some of my other traditional CV trained on pure train data only.
", "votes": null, "replies": [] } ] }, { "id": 542778, "author_name": "titericz", "author_url": "", "post_date": "06/04/2019 04:52:46", "content": "Actually the prior knowledge of the testset is a form of leak. Fortunately the leak was disclosed in the forums
", "votes": null, "replies": [ { "id": 542779, "author_name": "khahuras", "author_url": "", "post_date": "06/04/2019 04:53:53", "content": "That makes this competition a tricky one. All efforts in feature engineering and model build up are ineffective.
", "votes": null, "replies": [] }, { "id": 543358, "author_name": "titericz", "author_url": "", "post_date": "06/04/2019 13:46:20", "content": "Exactly @khahuras . People should be complaining about the leakage. That made all the difference here.
", "votes": null, "replies": [] } ] }, { "id": 543273, "author_name": "corochann", "author_url": "", "post_date": "06/04/2019 12:34:09", "content": "I also learned that we need to use all the information that we can gather in order to win. Most of the competition contains some kind of \"leak\"(=information about test score) even though its leak type varies...
", "votes": null, "replies": [] }, { "id": 543281, "author_name": "cpmpml", "author_url": "", "post_date": "06/04/2019 12:44:57", "content": "The leak was accessible to all, and what made a difference is machine learning.
\n\nI have a general additive model that cannot use sample weights, hence we could not tune it using test estimates. Yet, this model would claim 12th rank here. Just using 16 fold CV and proper feature engineering.
", "votes": null, "replies": [ { "id": 543331, "author_name": "khahuras", "author_url": "", "post_date": "06/04/2019 13:31:40", "content": "Except some exceptional models like yours, otherwise test leak overwhelm any power to get a good result.
", "votes": null, "replies": [] }, { "id": 543345, "author_name": "cpmpml", "author_url": "", "post_date": "06/04/2019 13:40:16", "content": "I agree, and I had no idea that this model would fare that well. Using test estimates was clearly the way to do.
", "votes": null, "replies": [] } ] }, { "id": 543400, "author_name": "coreacasa", "author_url": "", "post_date": "06/04/2019 14:09:43", "content": "Congrats everybody and thanks a lot for your shares but...
\n\nTRUST YOUR TEST-SET LEAK !!!
\n\nLook at the file:
\n\n\n\n\nversion 105 = version 104 * 1.1
\n \n\n\nversion 106 = version 104 * 1.2 (GOLD)
\n
I do not want to fine-tune more... Too much punishment!!!
\n\nI will try to do it better in the next competition.
", "votes": null, "replies": [] }, { "id": 543401, "author_name": "bkosar1640", "author_url": "", "post_date": "06/04/2019 14:09:56", "content": "I agree with Kha Vo's comments. I thought these machine learning competitions were supposed to be focused on identifying and developing best case machine learning techniques and not focused on detective work to find data leaks. In the last two competitions I have entered discovery and use of info associated with data leaks has been critical to success. I know I sound like sour grapes but I did not feel I learned much after spending a lot of time trying to define and improve useful features on a very random train data set.
\n\nThe very poor correlation between public and private LB results clearly kept a lot of people from developing better models. The purpose of this competition was supposed to be looking for ways to improve earthquake detection which would benefit everyone and I don't think it succeeded due to the quality and qty of info provided by LANL.
\n\nCPMP's results on the public and private LB were great but they are very much the exception. It looks like the average shift on the private LB for everyone else was very large.
", "votes": null, "replies": [] }, { "id": 543595, "author_name": "asterisk", "author_url": "", "post_date": "06/04/2019 16:08:01", "content": "i remember there was about 200 participants when I start working on this. I personally do not like tricky features, thus focused more on NN based models. I gave up after work on it for 3-4 of my weekends because i felt training set is small and quite different with testing. but there always a way, like looking at testing set and leaking set. I guess the organizers are a way too scientific research thinking but Kagglers have another way to success. anyway, it is not so bad to myself and worth the time i have spent.
", "votes": null, "replies": [] }, { "id": 543789, "author_name": "joejeo1", "author_url": "", "post_date": "06/04/2019 20:26:56", "content": "The focus should be on creating good models, that can extract good features from the data for predictions, not humanly creating 80-100 features.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "542667": "To me, this competition is not worth spending too much time on extracting better features, or tuning models... A simple trick given the prior knowledge of private test set would win. The data is not statistical enough, and there’s no fun of climbing the LB like some optimization competitions (i.e., the annual Santa competitions by Kaggle). A strong model does not guarantee to win. The train/public/private sets are very much different. \nJust a tricky competition though. \n\nFeature engineering and model build-up is ONLY effective if everybody is not given that kind of test leak. Otherwise, the leak cancels out all machine learning engineering efforts.", "542674": "sadly as a novice, i spending much time here as my first competition T.T . but i learn a lot from you, cpmp,and other masters. As per suggestion by you, i agree that if i am not doing much time for extracting features and tuning models i should at least get medals from my earlier submissions.", "542684": "If I know this competition has too many such pitfalls, I would have NOT participated in the first place. It took me too much time spending on extracting useful features and better models. In the end, my best submission scored 1.380 on public LB, which will give 2.37 on private LB. And some other guy discovered that a 1.800 public LB would also earn that equivalent private LB. I regret that I took too much time on this competition. If I just came in in the last month, I would have more joy, rather than sticking at the beginning and now see a lot of guys winning by pure luck or by some simple tricks. Of course there are teams with skills, but it's counted by fingers.", "542700": "Yes, i saw the post where the public test score is >1.8 but able to get score <2.4 in private test.\n\nI also spend many nights, lack of sleep for a month (as i join this competition about 1 month ago) .\n\nBut i will be very pleased if you could share your approach, how you can manage to avoid a big shake up in the end. I admire everyone who in the top 100 in public test but still can manage to be in top 100 in private test. I think you are a true expert in this field and i eager to learn a lot from the best. Many thanks :)", "542701": "Please don't be discouraged! You did an excellent job!! 23rd place is an amazing feat. And you learned a lot. Next time you can take this knowledge and go even further. Although 23rd is something to be really really proud of.\n\nI totally understand how you feel though. Earlier this year I poured my heart into the NFL competition I really though I had a chance to win. When I lost I felt crushed and said \"I would have not participated in the first place if I had known\" also. I also can understand it must be frustrating knowing a leak is the reason you missed out. Next time...", "542711": "What saved me is simply using a linear regression stack model on all oof predictions of other 1st level models (LGB, XGB, Catboost, RF, SVM), without any further manipulation. The public 1st level models scored around 1.29 to 1.31 on public LB. That stacked submission has worse validation MAE than any of the 1st level models, but better MSE. The submission file has much bigger mean and median than each of the 1st level model. I doubt that file very much, but in the end I still trust it anyway thanks to my intuition. The public LB is worse by 0.05. I almost did not select that file, and I would have ended up around rank 300+. Luckily that file gave me 23rd place. Not sure what to say more now, but I bet there is no CV that can win without prior knowledge of the private test set.", "542723": "Thanks for the share Kha Vo", "542750": "I think the data was not enough to use all the power of Machine Learning. The chunks were too small and it was too few data for a RNN or CNN to work well. It is required more data (more experiments) and larger chunks to make state of the art predictions.", "542777": "Yes. All the gambler wins, including me. I admitted that I managed to survive just by LUCK, by taking the knowledge of private test set. That submission give worse local CV MAE than some of my other traditional CV trained on pure train data only.", "542778": "Actually the prior knowledge of the testset is a form of leak. Fortunately the leak was disclosed in the [forums](https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/90664)", "542779": "That makes this competition a tricky one. All efforts in feature engineering and model build up are ineffective.", "543273": "I also learned that we need to use all the information that we can gather in order to win. Most of the competition contains some kind of \"leak\"(=information about test score) even though its leak type varies...", "543281": "The leak was accessible to all, and what made a difference is machine learning.\n\nI have a general additive model that cannot use sample weights, hence we could not tune it using test estimates. Yet, this model would claim 12th rank here. Just using 16 fold CV and proper feature engineering.", "543331": "Except some exceptional models like yours, otherwise test leak overwhelm any power to get a good result.", "543345": "I agree, and I had no idea that this model would fare that well. Using test estimates was clearly the way to do.", "543358": "Exactly @khahuras . People should be complaining about the leakage. That made all the difference here.", "543400": "Congrats everybody and thanks a lot for your shares but... \n\nTRUST YOUR TEST-SET LEAK !!!\n\nLook at the file: \n\n> version 105 = version 104 * 1.1 \n>> version 106 = version 104 * 1.2 (GOLD)\n\nI do not want to fine-tune more... Too much punishment!!!\n\nI will try to do it better in the next competition.", "543401": "I agree with Kha Vo's comments. I thought these machine learning competitions were supposed to be focused on identifying and developing best case machine learning techniques and not focused on detective work to find data leaks. In the last two competitions I have entered discovery and use of info associated with data leaks has been critical to success. I know I sound like sour grapes but I did not feel I learned much after spending a lot of time trying to define and improve useful features on a very random train data set. \n\n The very poor correlation between public and private LB results clearly kept a lot of people from developing better models. The purpose of this competition was supposed to be looking for ways to improve earthquake detection which would benefit everyone and I don't think it succeeded due to the quality and qty of info provided by LANL. \n\n CPMP's results on the public and private LB were great but they are very much the exception. It looks like the average shift on the private LB for everyone else was very large.", "543595": "i remember there was about 200 participants when I start working on this. I personally do not like tricky features, thus focused more on NN based models. I gave up after work on it for 3-4 of my weekends because i felt training set is small and quite different with testing. but there always a way, like looking at testing set and leaking set. I guess the organizers are a way too scientific research thinking but Kagglers have another way to success. anyway, it is not so bad to myself and worth the time i have spent.", "543789": "The focus should be on creating good models, that can extract good features from the data for predictions, not humanly creating 80-100 features." }, "source": "meta" }