![\"\"](\"https://i.imgur.com/TTvkiWn.png\")
\nCredit @ilu000 for confirming with this imageAbout a month ago a thread discussed the discovery that the data is actually from P4677.
\n\n\nCredit @ilu000 for confirming with this image
Recently I haven't seen much discussion about this- but I've always had it in the back of my head that the team(s) that use this information to their advantage will end up winning. Especially if it is confirmed that the test set is in fact the same as the image below.
\n\n![\"\"](\"https://i.imgur.com/t9jOaPf.png\")
If you know, even roughly, what kind of quakes (average TTF, number of miniquakes, etc) are in the public and private test set - you could optimize your model for these types of quakes without it being traceable. Just select parameters that improve your CV for similar types of quakes.
\n\nPlease tell me if I'm misunderstanding something, because it would be a shame for a leak like this to decide the results.
", "messages": [ { "id": 541157, "postDate": "2019-06-01T22:48:55.630Z", "content": "About a month ago a thread discussed the discovery that the data is actually from P4677.
\n\n\nCredit @ilu000 for confirming with this image
Recently I haven't seen much discussion about this- but I've always had it in the back of my head that the team(s) that use this information to their advantage will end up winning. Especially if it is confirmed that the test set is in fact the same as the image below.
\n\n
If you know, even roughly, what kind of quakes (average TTF, number of miniquakes, etc) are in the public and private test set - you could optimize your model for these types of quakes without it being traceable. Just select parameters that improve your CV for similar types of quakes.
\n\nPlease tell me if I'm misunderstanding something, because it would be a shame for a leak like this to decide the results.
", "rawMarkdown": "About a month ago a thread discussed the discovery that the [data is actually from P4677.](https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/90664#latest-535844)\n\n\nCredit @ilu000 for confirming with this image\n\nRecently I haven't seen much discussion about this- but I've always had it in the back of my head that the team(s) that use this information to their advantage will end up winning. Especially if it is confirmed that the test set is in fact the same as the image below.\n\n\n\nIf you know, even roughly, what kind of quakes (average TTF, number of miniquakes, etc) are in the public and private test set - you could optimize your model for these types of quakes without it being traceable. Just select parameters that improve your CV for similar types of quakes.\n\nPlease tell me if I'm misunderstanding something, because it would be a shame for a leak like this to decide the results.", "votes": 18 }, { "id": 541163, "postDate": "2019-06-01T23:33:50.340Z", "content": "Yes, it looks like the same dataset :)
\n\nIf the test set is the same as in the publication, it's a shame. It is demotivating.
", "rawMarkdown": "Yes, it looks like the same dataset :)\n\nIf the test set is the same as in the publication, it's a shame. It is demotivating.\n\n", "votes": 7 }, { "id": 541651, "postDate": "2019-06-02T20:32:38.887Z", "content": "Yes, it does.\nI am almost sure that test set is exactly shuffled version of the one shown in that figure and I can tell you if the top teams pick their highest-LB-score submission they're gonna shake badly. \nThe reason is, the test split shown in that figure has a mean of 5.7 while the public portion of it has a mean of 4.17 (I am not sure about the exact numbers since it's been a while I decided not to compete in this competition anymore).\nTo my understanding the models (at least the ones discussed publicly) fail to capture extreme TTFs. Hence, if your model is good on the public leaderboard then it should be good at capturing lower TTFs so it would fail to capture highest TTFs and you are gonna fail on the private leaderboard.
\n\nDisclaimer: I am sure there are teams with models capable of capturing the whole range of TTFs. Good luck to you all.
", "rawMarkdown": "Yes, it does.\nI am almost sure that test set is exactly shuffled version of the one shown in that figure and I can tell you if the top teams pick their highest-LB-score submission they're gonna shake badly. \nThe reason is, the test split shown in that figure has a mean of 5.7 while the public portion of it has a mean of 4.17 (I am not sure about the exact numbers since it's been a while I decided not to compete in this competition anymore).\nTo my understanding the models (at least the ones discussed publicly) fail to capture extreme TTFs. Hence, if your model is good on the public leaderboard then it should be good at capturing lower TTFs so it would fail to capture highest TTFs and you are gonna fail on the private leaderboard. \n\nDisclaimer: I am sure there are teams with models capable of capturing the whole range of TTFs. Good luck to you all.", "votes": 6, "replies": [ { "id": 541787, "postDate": "2019-06-03T03:15:36.143Z", "content": "So,a submission with higher mean will have higher score at private lb?
", "rawMarkdown": "So,a submission with higher mean will have higher score at private lb?" }, { "id": 541812, "postDate": "2019-06-03T03:53:14.393Z", "content": "What you said is not necessarily true.\n1) Better LB scores on low ttf scores will most probably lead to better private LB score (assuming the model is strong, and the person doing machine learning does not do any manual manipulation which can cause overfitting on public LB). The improvement on private LB is slower. A model which cannot capture low ttf, will more probably cannot capture high ttf as well.\n2) High ttf samples are unpredictable. People doing CV with more focus on high ttf, will probably suffer, because their CV with high ttf is although good, it is overfitted with the specific types of EQs in train.\n3) There is no guarantee a submission with high mean or high median will lead to better LB, except with very near public LB scores. Given a submission with LB 1.350 and mean 5, and the other 1.450 with mean 5.5, I prefer the 1.350.
", "rawMarkdown": "What you said is not necessarily true.\n1) Better LB scores on low ttf scores will most probably lead to better private LB score (assuming the model is strong, and the person doing machine learning does not do any manual manipulation which can cause overfitting on public LB). The improvement on private LB is slower. A model which cannot capture low ttf, will more probably cannot capture high ttf as well.\n2) High ttf samples are unpredictable. People doing CV with more focus on high ttf, will probably suffer, because their CV with high ttf is although good, it is overfitted with the specific types of EQs in train.\n3) There is no guarantee a submission with high mean or high median will lead to better LB, except with very near public LB scores. Given a submission with LB 1.350 and mean 5, and the other 1.450 with mean 5.5, I prefer the 1.350.\n", "votes": 2 }, { "id": 541839, "postDate": "2019-06-03T05:10:35.500Z", "content": "@takeiy not necessarily. It also depends on other factors. \n@khahuras
\n\n> A model which cannot capture low ttf, will more probably cannot capture high ttf as well.
\n\nI am afraid this wasn't the case for any of my models. I think other people observed the same.
\n\n> High ttf samples are unpredictable.
\n\nI remember cpmp had a model which was able to capture higher ttfs. I also have a model that captures high TTFs very well.
\n\n> There is no guarantee a submission with high mean or high median will lead to better LB
\n\nI agree.
\n\nPS. Given that test set as a whole has mean of 5.7 and public test set (which is 13% of the whole test set) has mean of 4.17 we can safely say that private test set (which is 87% of the whole test set) has a mean of 6. These are all our speculations after all and I can be totally wrong. We will find out in next 12 hours.
", "rawMarkdown": "@takeiy not necessarily. It also depends on other factors. \n@khahuras \n\n > A model which cannot capture low ttf, will more probably cannot capture high ttf as well.\n\nI am afraid this wasn't the case for any of my models. [I think other people observed the same](https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/91125#latest-536029).\n\n> High ttf samples are unpredictable.\n\nI remember cpmp had a model which was able to capture higher ttfs. I also have a model that captures high TTFs very well. \n\n> There is no guarantee a submission with high mean or high median will lead to better LB\n\nI agree.\n\nPS. Given that test set as a whole has mean of 5.7 and public test set (which is 13% of the whole test set) has mean of 4.17 we can safely say that private test set (which is 87% of the whole test set) has a mean of 6. These are all our speculations after all and I can be totally wrong. We will find out in next 12 hours." }, { "id": 541843, "postDate": "2019-06-03T05:17:05.970Z", "content": "My model can capture above ttf=12, but I still say that those are just estimations which are averaged by the models from all of the high ttf distributions. What I mean \"unpredictable\" here is that given all samples above 12s of ttf, no feature can discriminate that, such that we cannot predict when exactly the quake will occur after 100 years or 105 years.
", "rawMarkdown": "My model can capture above ttf=12, but I still say that those are just estimations which are averaged by the models from all of the high ttf distributions. What I mean \"unpredictable\" here is that given all samples above 12s of ttf, no feature can discriminate that, such that we cannot predict when exactly the quake will occur after 100 years or 105 years." }, { "id": 541911, "postDate": "2019-06-03T07:27:28.843Z", "content": "@khahuras, I have a few models that can predict ttf>12 and have one that can detect up to ttf=17. The model with the highest ttf has a cv of 2.13 so it is really hard to say that submission with high mean/median is the best answer. We will know tomorrow for sure. Good luck everyone :-)
", "rawMarkdown": "@khahuras, I have a few models that can predict ttf>12 and have one that can detect up to ttf=17. The model with the highest ttf has a cv of 2.13 so it is really hard to say that submission with high mean/median is the best answer. We will know tomorrow for sure. Good luck everyone :-)", "votes": 1 }, { "id": 542707, "postDate": "2019-06-04T03:48:53.590Z", "content": "@sheriytm do you still believe its hard to say submission with high mean is the best? ;)
", "rawMarkdown": "@sheriytm do you still believe its hard to say submission with high mean is the best? ;)" } ] }, { "id": 542665, "postDate": "2019-06-04T03:07:22.087Z", "content": "Wish I read it.
", "rawMarkdown": "Wish I read it.", "votes": 1 }, { "id": 541469, "postDate": "2019-06-02T13:53:23.887Z", "content": "Answering to your question: I think so ;)
", "rawMarkdown": "Answering to your question: I think so ;)", "votes": 1, "replies": [ { "id": 541516, "postDate": "2019-06-02T15:32:19.520Z", "content": "😄 thanks for answering directly. Hopefully the winners will post a write up. I’m looking forward to reading about the creative ways used to exploit this leak.
", "rawMarkdown": "😄 thanks for answering directly. Hopefully the winners will post a write up. I’m looking forward to reading about the creative ways used to exploit this leak.", "votes": 1 } ] }, { "id": 541433, "postDate": "2019-06-02T12:47:27.287Z", "content": "So..... nobody is answering my question directly I'll ask it another way. Is this a leak? Is it okay (morally or against the rules) to- for example - count the pixels in this image to determine the best test set mean TTF and build your model around it? Do we think the winning/top teams will have done that?
\n\nI'm asking because my team is not doing that- but if everyone else is and it's understood to be okay, maybe we should.
", "rawMarkdown": "So..... nobody is answering my question directly I'll ask it another way. Is this a leak? Is it okay (morally or against the rules) to- for example - count the pixels in this image to determine the best test set mean TTF and build your model around it? Do we think the winning/top teams will have done that?\n\nI'm asking because my team is not doing that- but if everyone else is and it's understood to be okay, maybe we should.", "votes": 1, "replies": [ { "id": 541435, "postDate": "2019-06-02T12:51:46.830Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 541453, "postDate": "2019-06-02T13:31:52.333Z", "content": "This paper is listed here: https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/77240#523604 hence is disclosed enough to meet rules requirement.
", "rawMarkdown": "This paper is listed here: https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/77240#523604 hence is disclosed enough to meet rules requirement." }, { "id": 541454, "postDate": "2019-06-02T13:36:31.007Z", "content": "\n\n\nIs this a leak?
\n
If a leak (are we sure test data is the same?) then it is rather indirect as you have no way to map any of the test segment to whatever is shown in the paper.
\n\nAssume you can compute the test mean from the paper picture. How would you use that information?
", "rawMarkdown": "> Is this a leak? \n\nIf a leak (are we sure test data is the same?) then it is rather indirect as you have no way to map any of the test segment to whatever is shown in the paper.\n\nAssume you can compute the test mean from the paper picture. How would you use that information?" }, { "id": 541462, "postDate": "2019-06-02T13:42:11.697Z", "content": "\n\n\nIs it okay (morally or against the rules)
\n
Which rule would be violated here?
\n\nMoral is another topic I prefer not to enter 35 hours before competition end ;)
\n\nHow to deal with leaks (if this is one) is one area where Kaggle and real world differ. If you find a leak in real world then you'd want to modify the data you use for building and testing models in order to remove the leak.
", "rawMarkdown": "> Is it okay (morally or against the rules) \n\nWhich rule would be violated here?\n\nMoral is another topic I prefer not to enter 35 hours before competition end ;)\n\nHow to deal with leaks (if this is one) is one area where Kaggle and real world differ. If you find a leak in real world then you'd want to modify the data you use for building and testing models in order to remove the leak. ", "votes": 2 }, { "id": 541515, "postDate": "2019-06-02T15:28:58.563Z", "content": "> Assume you can compute the test mean from the paper picture. How would you use that information?
\n\nI haven’t spent time to figure out how. But I’m guessing the winning team will have. For one you could compute the distribution of the true target variable and plot it vs the distribution of your predictions. This would serve as extra validation of if features will improve the model on the full test set.
\n\nA huge advantage for teams that did this... assuming private test actually is the one in the image.
", "rawMarkdown": "> Assume you can compute the test mean from the paper picture. How would you use that information?\n\nI haven’t spent time to figure out how. But I’m guessing the winning team will have. For one you could compute the distribution of the true target variable and plot it vs the distribution of your predictions. This would serve as extra validation of if features will improve the model on the full test set. \n\nA huge advantage for teams that did this... assuming private test actually is the one in the image." }, { "id": 541521, "postDate": "2019-06-02T15:49:49.733Z", "content": "What do you think about my previous post above ? Will it work? let say, we just use EQ with TTF peak >10 for training the model.
", "rawMarkdown": "What do you think about my previous post above ? Will it work? let say, we just use EQ with TTF peak >10 for training the model." }, { "id": 541533, "postDate": "2019-06-02T16:08:42.453Z", "content": "That is an asumption at the end. It involves some risk, because it is also possible that the test set is not the one in the picture
", "rawMarkdown": "That is an asumption at the end. It involves some risk, because it is also possible that the test set is not the one in the picture", "votes": 1 }, { "id": 541581, "postDate": "2019-06-02T17:21:33.660Z", "content": "I'm starting to think it would be risker not to at least assume it for one of the final submissions.
", "rawMarkdown": "I'm starting to think it would be risker not to at least assume it for one of the final submissions." }, { "id": 541818, "postDate": "2019-06-03T04:24:29.353Z", "content": "My guess (?):
\n\nNote that \"some\" means \"nearly all\" as overfitting the leaderboard, at least for one entry, is generally done
", "rawMarkdown": "My guess (?):\n\n1. Measure the ttf *distribution* from the figure\n2. Probe the LB to find the best model for the Public LB (as some have done) and hence the public LB distribution - call it X.\n3. Use that to determine the private LB distribution ((1-X) or something)\n4. Cherry-pick the training dataset to obtain a dataset with that measured private LB distribution\n5. Train. I am guesings there will be a bias to the private LB.\n\nNote that \"some\" means \"nearly all\" as overfitting the leaderboard, at least for one entry, is generally done", "votes": 1 }, { "id": 541825, "postDate": "2019-06-03T04:37:06.213Z", "content": "@petewills I have done all those things, and never be successful even with public LB fitting. We all have only 2 submissions with a lot of CV strategies, models, and features to choose from (that does not use this kind of information). And finally, after the last two days agonizing on which should I choose for final judgement, I decided not to gamble with those assumptions, or not to waste submission file for that, as I don't want to regret after all.
", "rawMarkdown": "@petewills I have done all those things, and never be successful even with public LB fitting. We all have only 2 submissions with a lot of CV strategies, models, and features to choose from (that does not use this kind of information). And finally, after the last two days agonizing on which should I choose for final judgement, I decided not to gamble with those assumptions, or not to waste submission file for that, as I don't want to regret after all. ", "votes": 1 }, { "id": 542128, "postDate": "2019-06-03T13:40:21.617Z", "content": "\n\n\nWill it work?
\n
There is only one way to know: try it.
", "rawMarkdown": "> Will it work? \n\nThere is only one way to know: try it.", "votes": 1 } ] }, { "id": 542654, "postDate": "2019-06-04T02:52:13.673Z", "content": "This post helped a lot of people, including me.
", "rawMarkdown": "[This](https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/90664#latest-535844) post helped a lot of people, including me.", "votes": 2, "replies": [ { "id": 542678, "postDate": "2019-06-04T03:26:18.147Z", "content": "I wish I had focused more on it. It did end up helping us in the end- but there wasn't enough time to gain as much from it as I wished.
", "rawMarkdown": "I wish I had focused more on it. It did end up helping us in the end- but there wasn't enough time to gain as much from it as I wished.", "votes": 1 }, { "id": 542730, "postDate": "2019-06-04T04:10:16.893Z", "content": "It didn't helped too much to the ones that knew it! Haha
", "rawMarkdown": "It didn't helped too much to the ones that knew it! Haha " } ] }, { "id": 541978, "postDate": "2019-06-03T09:17:57.190Z", "content": "\n\n\nIs this a leak?
\n
In my view if test is finally confirmed to be there then it is indeed a leak .
\n\nNot the kind of leak that renders a competition worthless for the organizers but easily important enough to condition strongly private leaderboard. If finally confirmed.
\n\nAbout kaggle rules, usually \"non comp. destructive leaks\" are allowed (encouraged?) to exploit. In this case doesn't seem easy but possibly some insights can be gained from a simple glance at the experiment.
", "rawMarkdown": ">Is this a leak?\n\nIn my view if test is finally confirmed to be there then it is indeed a leak . \n\nNot the kind of leak that renders a competition worthless for the organizers but easily important enough to condition strongly private leaderboard. If finally confirmed.\n\nAbout kaggle rules, usually \"non comp. destructive leaks\" are allowed (encouraged?) to exploit. In this case doesn't seem easy but possibly some insights can be gained from a simple glance at the experiment.\n", "votes": 2 }, { "id": 541367, "postDate": "2019-06-02T10:10:15.957Z", "content": "Well, at the cost of $50k you'd want to hope the sponsors get value for their money. Some of the kernels I've seen, geez, the features involved makes it commercially impractical in my very humble opinion. Makes me wonder how many kernels barely hit silver or bronze, yet commercially would have been a better result for the sponsors. Less accurate but 100 lines of neat, clean code that runs in a jiffy.
", "rawMarkdown": "Well, at the cost of $50k you'd want to hope the sponsors get value for their money. Some of the kernels I've seen, geez, the features involved makes it commercially impractical in my *very* humble opinion. Makes me wonder how many kernels barely hit silver or bronze, yet commercially would have been a better result for the sponsors. Less accurate but 100 lines of neat, clean code that runs in a jiffy.", "votes": 2, "replies": [ { "id": 541382, "postDate": "2019-06-02T10:48:51.393Z", "content": "Who joins a Kaggle competition to make value for the sponsor raise her/his hand ;)
", "rawMarkdown": "Who joins a Kaggle competition to make value for the sponsor raise her/his hand ;)", "votes": 2 }, { "id": 541395, "postDate": "2019-06-02T11:24:11.407Z", "content": "crickets chirping...
", "rawMarkdown": "crickets chirping...", "votes": 1 }, { "id": 541409, "postDate": "2019-06-02T11:58:17.143Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 541464, "postDate": "2019-06-02T13:44:33.533Z", "content": "I think they will rather look at winning solutions, which probably will be (very) different from public kernels.
", "rawMarkdown": "I think they will rather look at winning solutions, which probably will be (very) different from public kernels.", "votes": 1 } ] }, { "id": 542625, "postDate": "2019-06-04T02:26:17.680Z", "content": "Did this post end up helping anyone?
", "rawMarkdown": "Did this post end up helping anyone?" }, { "id": 542410, "postDate": "2019-06-03T21:10:50.193Z", "content": "I have a neat NN based solution, but does not work well although I've tried my best to tune it.
", "rawMarkdown": "I have a neat NN based solution, but does not work well although I've tried my best to tune it. " }, { "id": 541450, "postDate": "2019-06-02T13:26:09.460Z", "content": "If this is true, the the remaining of test data/private data will be all the EQ with TTF > 10, as the public test already evaluate the EQ with TTF <10 and +-4 as Mykper said in another discussion. Am i correct ?
", "rawMarkdown": "If this is true, the the remaining of test data/private data will be all the EQ with TTF > 10, as the public test already evaluate the EQ with TTF <10 and +-4 as Mykper said in another discussion. Am i correct ? " }, { "id": 541349, "postDate": "2019-06-02T09:26:35.293Z", "content": "Unique innovations in the field of Data Science, supported by a documented working protoype should win in my opinion
", "rawMarkdown": "Unique innovations in the field of Data Science, supported by a documented working protoype should win in my opinion \n", "replies": [ { "id": 541408, "postDate": "2019-06-02T11:53:58.200Z", "content": "Winning a Kaggle competition is defined by an automated procedure without any human bias. Why do you want to add a human bias?
", "rawMarkdown": "Winning a Kaggle competition is defined by an automated procedure without any human bias. Why do you want to add a human bias?", "votes": 2 }, { "id": 541420, "postDate": "2019-06-02T12:28:44.103Z", "content": "\"without any human bias\": Do you still think it's the case all after some past competitions were won by LB probings and human tricks?
", "rawMarkdown": "\"without any human bias\": Do you still think it's the case all after some past competitions were won by LB probings and human tricks?" }, { "id": 541441, "postDate": "2019-06-02T12:55:34.900Z", "content": "Winning a Kaggle competition is defined by an automated procedure without any human bias.\nWell... maybe using is supposed to be defined instead of is defined would be more meaningful here.
What human bias happens when evaluating a submission?
", "rawMarkdown": "What human bias happens when evaluating a submission?", "votes": 1 }, { "id": 541485, "postDate": "2019-06-02T14:22:21.547Z", "content": "None, now that I think about it the evaluation procedure has nothing to do with human bias and is done pretty much objectively.
", "rawMarkdown": "None, now that I think about it **the evaluation procedure** has nothing to do with human bias and is done pretty much objectively.", "votes": 1 } ] } ], "comments": [ { "id": 541163, "author_name": "Dre@mer", "author_url": "", "post_date": "2019-06-01T23:33:50.340000", "content": "Yes, it looks like the same dataset :)
\n\nIf the test set is the same as in the publication, it's a shame. It is demotivating.
", "votes": 7, "replies": [] }, { "id": 541651, "author_name": "Massoud Hosseinali", "author_url": "", "post_date": "2019-06-02T20:32:38.887000", "content": "Yes, it does.\nI am almost sure that test set is exactly shuffled version of the one shown in that figure and I can tell you if the top teams pick their highest-LB-score submission they're gonna shake badly. \nThe reason is, the test split shown in that figure has a mean of 5.7 while the public portion of it has a mean of 4.17 (I am not sure about the exact numbers since it's been a while I decided not to compete in this competition anymore).\nTo my understanding the models (at least the ones discussed publicly) fail to capture extreme TTFs. Hence, if your model is good on the public leaderboard then it should be good at capturing lower TTFs so it would fail to capture highest TTFs and you are gonna fail on the private leaderboard.
\n\nDisclaimer: I am sure there are teams with models capable of capturing the whole range of TTFs. Good luck to you all.
", "votes": 6, "replies": [ { "id": 541787, "author_name": "ZeroWen", "author_url": "", "post_date": "2019-06-03T03:15:36.143000", "content": "So,a submission with higher mean will have higher score at private lb?
", "votes": 0, "replies": [] }, { "id": 541812, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-06-03T03:53:14.393000", "content": "What you said is not necessarily true.\n1) Better LB scores on low ttf scores will most probably lead to better private LB score (assuming the model is strong, and the person doing machine learning does not do any manual manipulation which can cause overfitting on public LB). The improvement on private LB is slower. A model which cannot capture low ttf, will more probably cannot capture high ttf as well.\n2) High ttf samples are unpredictable. People doing CV with more focus on high ttf, will probably suffer, because their CV with high ttf is although good, it is overfitted with the specific types of EQs in train.\n3) There is no guarantee a submission with high mean or high median will lead to better LB, except with very near public LB scores. Given a submission with LB 1.350 and mean 5, and the other 1.450 with mean 5.5, I prefer the 1.350.
", "votes": 2, "replies": [] }, { "id": 541839, "author_name": "Massoud Hosseinali", "author_url": "", "post_date": "2019-06-03T05:10:35.500000", "content": "@takeiy not necessarily. It also depends on other factors. \n@khahuras
\n\n> A model which cannot capture low ttf, will more probably cannot capture high ttf as well.
\n\nI am afraid this wasn't the case for any of my models. I think other people observed the same.
\n\n> High ttf samples are unpredictable.
\n\nI remember cpmp had a model which was able to capture higher ttfs. I also have a model that captures high TTFs very well.
\n\n> There is no guarantee a submission with high mean or high median will lead to better LB
\n\nI agree.
\n\nPS. Given that test set as a whole has mean of 5.7 and public test set (which is 13% of the whole test set) has mean of 4.17 we can safely say that private test set (which is 87% of the whole test set) has a mean of 6. These are all our speculations after all and I can be totally wrong. We will find out in next 12 hours.
", "votes": 0, "replies": [] }, { "id": 541843, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-06-03T05:17:05.970000", "content": "My model can capture above ttf=12, but I still say that those are just estimations which are averaged by the models from all of the high ttf distributions. What I mean \"unpredictable\" here is that given all samples above 12s of ttf, no feature can discriminate that, such that we cannot predict when exactly the quake will occur after 100 years or 105 years.
", "votes": 0, "replies": [] }, { "id": 541911, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-06-03T07:27:28.843000", "content": "@khahuras, I have a few models that can predict ttf>12 and have one that can detect up to ttf=17. The model with the highest ttf has a cv of 2.13 so it is really hard to say that submission with high mean/median is the best answer. We will know tomorrow for sure. Good luck everyone :-)
", "votes": 1, "replies": [] }, { "id": 542707, "author_name": "Massoud Hosseinali", "author_url": "", "post_date": "2019-06-04T03:48:53.590000", "content": "@sheriytm do you still believe its hard to say submission with high mean is the best? ;)
", "votes": 0, "replies": [] } ] }, { "id": 542665, "author_name": "joejeo1", "author_url": "", "post_date": "2019-06-04T03:07:22.087000", "content": "Wish I read it.
", "votes": 1, "replies": [] }, { "id": 541469, "author_name": "Carlos Prades K.", "author_url": "", "post_date": "2019-06-02T13:53:23.887000", "content": "Answering to your question: I think so ;)
", "votes": 1, "replies": [ { "id": 541516, "author_name": "Rob Mulla", "author_url": "", "post_date": "2019-06-02T15:32:19.520000", "content": "😄 thanks for answering directly. Hopefully the winners will post a write up. I’m looking forward to reading about the creative ways used to exploit this leak.
", "votes": 1, "replies": [] } ] }, { "id": 541433, "author_name": "Rob Mulla", "author_url": "", "post_date": "2019-06-02T12:47:27.287000", "content": "So..... nobody is answering my question directly I'll ask it another way. Is this a leak? Is it okay (morally or against the rules) to- for example - count the pixels in this image to determine the best test set mean TTF and build your model around it? Do we think the winning/top teams will have done that?
\n\nI'm asking because my team is not doing that- but if everyone else is and it's understood to be okay, maybe we should.
", "votes": 1, "replies": [ { "id": 541435, "author_name": "", "author_url": "", "post_date": "2019-06-02T12:51:46.830000", "content": "", "votes": 1, "replies": [] }, { "id": 541453, "author_name": "CPMP", "author_url": "", "post_date": "2019-06-02T13:31:52.333000", "content": "This paper is listed here: https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/77240#523604 hence is disclosed enough to meet rules requirement.
", "votes": 0, "replies": [] }, { "id": 541454, "author_name": "CPMP", "author_url": "", "post_date": "2019-06-02T13:36:31.007000", "content": "\n\n\nIs this a leak?
\n
If a leak (are we sure test data is the same?) then it is rather indirect as you have no way to map any of the test segment to whatever is shown in the paper.
\n\nAssume you can compute the test mean from the paper picture. How would you use that information?
", "votes": 0, "replies": [] }, { "id": 541462, "author_name": "CPMP", "author_url": "", "post_date": "2019-06-02T13:42:11.697000", "content": "\n\n\nIs it okay (morally or against the rules)
\n
Which rule would be violated here?
\n\nMoral is another topic I prefer not to enter 35 hours before competition end ;)
\n\nHow to deal with leaks (if this is one) is one area where Kaggle and real world differ. If you find a leak in real world then you'd want to modify the data you use for building and testing models in order to remove the leak.
", "votes": 2, "replies": [] }, { "id": 541515, "author_name": "Rob Mulla", "author_url": "", "post_date": "2019-06-02T15:28:58.563000", "content": "> Assume you can compute the test mean from the paper picture. How would you use that information?
\n\nI haven’t spent time to figure out how. But I’m guessing the winning team will have. For one you could compute the distribution of the true target variable and plot it vs the distribution of your predictions. This would serve as extra validation of if features will improve the model on the full test set.
\n\nA huge advantage for teams that did this... assuming private test actually is the one in the image.
", "votes": 0, "replies": [] }, { "id": 541521, "author_name": "OverfitModel", "author_url": "", "post_date": "2019-06-02T15:49:49.733000", "content": "What do you think about my previous post above ? Will it work? let say, we just use EQ with TTF peak >10 for training the model.
", "votes": 0, "replies": [] }, { "id": 541533, "author_name": "Carlos Prades K.", "author_url": "", "post_date": "2019-06-02T16:08:42.453000", "content": "That is an asumption at the end. It involves some risk, because it is also possible that the test set is not the one in the picture
", "votes": 1, "replies": [] }, { "id": 541581, "author_name": "Rob Mulla", "author_url": "", "post_date": "2019-06-02T17:21:33.660000", "content": "I'm starting to think it would be risker not to at least assume it for one of the final submissions.
", "votes": 0, "replies": [] }, { "id": 541818, "author_name": "pete", "author_url": "", "post_date": "2019-06-03T04:24:29.353000", "content": "My guess (?):
\n\nNote that \"some\" means \"nearly all\" as overfitting the leaderboard, at least for one entry, is generally done
", "votes": 1, "replies": [] }, { "id": 541825, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-06-03T04:37:06.213000", "content": "@petewills I have done all those things, and never be successful even with public LB fitting. We all have only 2 submissions with a lot of CV strategies, models, and features to choose from (that does not use this kind of information). And finally, after the last two days agonizing on which should I choose for final judgement, I decided not to gamble with those assumptions, or not to waste submission file for that, as I don't want to regret after all.
", "votes": 1, "replies": [] }, { "id": 542128, "author_name": "CPMP", "author_url": "", "post_date": "2019-06-03T13:40:21.617000", "content": "\n\n\nWill it work?
\n
There is only one way to know: try it.
", "votes": 1, "replies": [] } ] }, { "id": 542654, "author_name": "Giba", "author_url": "", "post_date": "2019-06-04T02:52:13.673000", "content": "This post helped a lot of people, including me.
", "votes": 2, "replies": [ { "id": 542678, "author_name": "Rob Mulla", "author_url": "", "post_date": "2019-06-04T03:26:18.147000", "content": "I wish I had focused more on it. It did end up helping us in the end- but there wasn't enough time to gain as much from it as I wished.
", "votes": 1, "replies": [] }, { "id": 542730, "author_name": "Carlos Prades K.", "author_url": "", "post_date": "2019-06-04T04:10:16.893000", "content": "It didn't helped too much to the ones that knew it! Haha
", "votes": 0, "replies": [] } ] }, { "id": 541978, "author_name": "miguel perez", "author_url": "", "post_date": "2019-06-03T09:17:57.190000", "content": "\n\n\nIs this a leak?
\n
In my view if test is finally confirmed to be there then it is indeed a leak .
\n\nNot the kind of leak that renders a competition worthless for the organizers but easily important enough to condition strongly private leaderboard. If finally confirmed.
\n\nAbout kaggle rules, usually \"non comp. destructive leaks\" are allowed (encouraged?) to exploit. In this case doesn't seem easy but possibly some insights can be gained from a simple glance at the experiment.
", "votes": 2, "replies": [] }, { "id": 541367, "author_name": "Gideon Vos", "author_url": "", "post_date": "2019-06-02T10:10:15.957000", "content": "Well, at the cost of $50k you'd want to hope the sponsors get value for their money. Some of the kernels I've seen, geez, the features involved makes it commercially impractical in my very humble opinion. Makes me wonder how many kernels barely hit silver or bronze, yet commercially would have been a better result for the sponsors. Less accurate but 100 lines of neat, clean code that runs in a jiffy.
", "votes": 2, "replies": [ { "id": 541382, "author_name": "bluetrain", "author_url": "", "post_date": "2019-06-02T10:48:51.393000", "content": "Who joins a Kaggle competition to make value for the sponsor raise her/his hand ;)
", "votes": 2, "replies": [] }, { "id": 541395, "author_name": "Gideon Vos", "author_url": "", "post_date": "2019-06-02T11:24:11.407000", "content": "crickets chirping...
", "votes": 1, "replies": [] }, { "id": 541409, "author_name": "", "author_url": "", "post_date": "2019-06-02T11:58:17.143000", "content": "", "votes": 2, "replies": [] }, { "id": 541464, "author_name": "CPMP", "author_url": "", "post_date": "2019-06-02T13:44:33.533000", "content": "I think they will rather look at winning solutions, which probably will be (very) different from public kernels.
", "votes": 1, "replies": [] } ] }, { "id": 542625, "author_name": "Rob Mulla", "author_url": "", "post_date": "2019-06-04T02:26:17.680000", "content": "Did this post end up helping anyone?
", "votes": 0, "replies": [] }, { "id": 542410, "author_name": "Z. Liu", "author_url": "", "post_date": "2019-06-03T21:10:50.193000", "content": "I have a neat NN based solution, but does not work well although I've tried my best to tune it.
", "votes": 0, "replies": [] }, { "id": 541450, "author_name": "OverfitModel", "author_url": "", "post_date": "2019-06-02T13:26:09.460000", "content": "If this is true, the the remaining of test data/private data will be all the EQ with TTF > 10, as the public test already evaluate the EQ with TTF <10 and +-4 as Mykper said in another discussion. Am i correct ?
", "votes": 0, "replies": [] }, { "id": 541349, "author_name": "Hein", "author_url": "", "post_date": "2019-06-02T09:26:35.293000", "content": "Unique innovations in the field of Data Science, supported by a documented working protoype should win in my opinion
", "votes": 0, "replies": [ { "id": 541408, "author_name": "CPMP", "author_url": "", "post_date": "2019-06-02T11:53:58.200000", "content": "Winning a Kaggle competition is defined by an automated procedure without any human bias. Why do you want to add a human bias?
", "votes": 2, "replies": [] }, { "id": 541420, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-06-02T12:28:44.103000", "content": "\"without any human bias\": Do you still think it's the case all after some past competitions were won by LB probings and human tricks?
", "votes": 0, "replies": [] }, { "id": 541441, "author_name": "Kain", "author_url": "", "post_date": "2019-06-02T12:55:34.900000", "content": "Winning a Kaggle competition is defined by an automated procedure without any human bias.\nWell... maybe using is supposed to be defined instead of is defined would be more meaningful here.
What human bias happens when evaluating a submission?
", "votes": 1, "replies": [] }, { "id": 541485, "author_name": "Kain", "author_url": "", "post_date": "2019-06-02T14:22:21.547000", "content": "None, now that I think about it the evaluation procedure has nothing to do with human bias and is done pretty much objectively.
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "541157": "About a month ago a thread discussed the discovery that the [data is actually from P4677.](https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/90664#latest-535844)\n\n\nCredit @ilu000 for confirming with this image\n\nRecently I haven't seen much discussion about this- but I've always had it in the back of my head that the team(s) that use this information to their advantage will end up winning. Especially if it is confirmed that the test set is in fact the same as the image below.\n\n\n\nIf you know, even roughly, what kind of quakes (average TTF, number of miniquakes, etc) are in the public and private test set - you could optimize your model for these types of quakes without it being traceable. Just select parameters that improve your CV for similar types of quakes.\n\nPlease tell me if I'm misunderstanding something, because it would be a shame for a leak like this to decide the results.", "541163": "Yes, it looks like the same dataset :)\n\nIf the test set is the same as in the publication, it's a shame. It is demotivating.\n\n", "541651": "Yes, it does.\nI am almost sure that test set is exactly shuffled version of the one shown in that figure and I can tell you if the top teams pick their highest-LB-score submission they're gonna shake badly. \nThe reason is, the test split shown in that figure has a mean of 5.7 while the public portion of it has a mean of 4.17 (I am not sure about the exact numbers since it's been a while I decided not to compete in this competition anymore).\nTo my understanding the models (at least the ones discussed publicly) fail to capture extreme TTFs. Hence, if your model is good on the public leaderboard then it should be good at capturing lower TTFs so it would fail to capture highest TTFs and you are gonna fail on the private leaderboard. \n\nDisclaimer: I am sure there are teams with models capable of capturing the whole range of TTFs. Good luck to you all.", "542665": "Wish I read it.", "541469": "Answering to your question: I think so ;)", "541433": "So..... nobody is answering my question directly I'll ask it another way. Is this a leak? Is it okay (morally or against the rules) to- for example - count the pixels in this image to determine the best test set mean TTF and build your model around it? Do we think the winning/top teams will have done that?\n\nI'm asking because my team is not doing that- but if everyone else is and it's understood to be okay, maybe we should.", "542654": "[This](https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/90664#latest-535844) post helped a lot of people, including me.", "541978": ">Is this a leak?\n\nIn my view if test is finally confirmed to be there then it is indeed a leak . \n\nNot the kind of leak that renders a competition worthless for the organizers but easily important enough to condition strongly private leaderboard. If finally confirmed.\n\nAbout kaggle rules, usually \"non comp. destructive leaks\" are allowed (encouraged?) to exploit. In this case doesn't seem easy but possibly some insights can be gained from a simple glance at the experiment.\n", "541367": "Well, at the cost of $50k you'd want to hope the sponsors get value for their money. Some of the kernels I've seen, geez, the features involved makes it commercially impractical in my *very* humble opinion. Makes me wonder how many kernels barely hit silver or bronze, yet commercially would have been a better result for the sponsors. Less accurate but 100 lines of neat, clean code that runs in a jiffy.", "542625": "Did this post end up helping anyone?", "542410": "I have a neat NN based solution, but does not work well although I've tried my best to tune it. ", "541450": "If this is true, the the remaining of test data/private data will be all the EQ with TTF > 10, as the public test already evaluate the EQ with TTF <10 and +-4 as Mykper said in another discussion. Am i correct ? ", "541349": "Unique innovations in the field of Data Science, supported by a documented working protoype should win in my opinion \n" } }