Going to share my take of how I feel about the current state of the leaderboard.
\nI have 3 strong models which:
\nThat's good enough for ~#30 place on the current state of LB.
\nThen I average my best CV ensemble with best public notebook submission - submit multiple times to optimize ensemble weights to maximize public LB score - current #11th place.
\nBased on current standings I would extrapolate that to LB score of 0.8007 public LB score.
I am very confident that the public LB model did not improve my model, as I have reused it in my current models on my folds.
\nWhat does this mean?
\nGoing to share my take of how I feel about the current state of the leaderboard.
\nI have 3 strong models which:
\nThat's good enough for ~#30 place on the current state of LB.
\nThen I average my best CV ensemble with best public notebook submission - submit multiple times to optimize ensemble weights to maximize public LB score - current #11th place.
\nBased on current standings I would extrapolate that to LB score of 0.8007 public LB score.
I am very confident that the public LB model did not improve my model, as I have reused it in my current models on my folds.
\nWhat does this mean?
\nIn fact, if your PC has enough RAM or batch training for you is feasible, you can simply add all the useful features from the public notebook to your own model. If you can get to 0.800+ through blend, that should already give you a score of 0.800+ with extra features. As for blend, you must carefully analyze the correlation between the public notebook output and the output of your original model. Even so, we should be careful.
", "rawMarkdown": "In fact, if your PC has enough RAM or batch training for you is feasible, you can simply add all the useful features from the public notebook to your own model. If you can get to 0.800+ through blend, that should already give you a score of 0.800+ with extra features. As for blend, you must carefully analyze the correlation between the public notebook output and the output of your original model. Even so, we should be careful. ", "votes": 4 }, { "id": 1872937, "postDate": "2022-07-27T10:36:29.387Z", "content": "Don’t forget about covid…. feels weird that everyone is fighting for the 4th decimal place on CV/LB while the private will be affected by the health crisis of the century.
", "rawMarkdown": "Don’t forget about covid.... feels weird that everyone is fighting for the 4th decimal place on CV/LB while the private will be affected by the health crisis of the century.", "votes": 1 }, { "id": 1868329, "postDate": "2022-07-23T23:10:59.710Z", "content": "Great insights! I think you've really hit the nail on the head with regard to the current state of the leaderboard.
\nI think you're also right that it's very hard to measure how much you've overfit on the public LB, and that it's probably not worth the risk in the end.
good job !!!
", "rawMarkdown": "good job !!!\n", "votes": 1 }, { "id": 1864957, "postDate": "2022-07-21T11:59:27.840Z", "content": "yes yes yes !!!1
", "rawMarkdown": "yes yes yes !!!1", "votes": 1 }, { "id": 1862804, "postDate": "2022-07-20T02:39:16.737Z", "content": "The best score in the public notebook is using DART, your three models, are you using DART? 😯
", "rawMarkdown": "The best score in the public notebook is using DART, your three models, are you using DART? 😯", "votes": 1, "replies": [ { "id": 1863200, "postDate": "2022-07-20T08:00:15.910Z", "content": "one of them - yes
", "rawMarkdown": "one of them - yes", "votes": 5 } ] }, { "id": 1864227, "postDate": "2022-07-20T21:08:58.217Z", "content": "Do you think that optimizing models must be only by cv scores?
", "rawMarkdown": "Do you think that optimizing models must be only by cv scores?", "votes": 2, "replies": [ { "id": 1864239, "postDate": "2022-07-20T21:14:51.117Z", "content": "For this competition - 100% sure
", "rawMarkdown": "For this competition - 100% sure", "votes": 7 }, { "id": 1864254, "postDate": "2022-07-20T21:27:01.337Z", "content": "I agree with you that if the proportion of positive and negative samples in the training and test sets is basically the same, and the data distribution of public and private LB are also similar. Then you should trust cv. But you may want to set a tolerance standard deviation of about 0.0001. One more thing, 51% of the data is in the public LB, it should not be ignored.
", "rawMarkdown": "I agree with you that if the proportion of positive and negative samples in the training and test sets is basically the same, and the data distribution of public and private LB are also similar. Then you should trust cv. But you may want to set a tolerance standard deviation of about 0.0001. One more thing, 51% of the data is in the public LB, it should not be ignored.", "votes": 6 } ] }, { "id": 1862176, "postDate": "2022-07-19T13:55:55.220Z", "content": "We are are having something similar with our best CV ensemble. If I may ask based on OOF are you just doing a average ensemble or also trying weighted or stacking strategies (all based on oof of course) to boost CV .
", "rawMarkdown": "We are are having something similar with our best CV ensemble. If I may ask based on OOF are you just doing a average ensemble or also trying weighted or stacking strategies (all based on oof of course) to boost CV .", "votes": 2, "replies": [ { "id": 1862357, "postDate": "2022-07-19T16:15:01.280Z", "content": "simple weighted average
", "rawMarkdown": "simple weighted average", "votes": 8 }, { "id": 1862491, "postDate": "2022-07-19T18:33:30.133Z", "content": "May I ask if you are averaging probabilities, ranked probabilities or the raw output from the models? 😸
", "rawMarkdown": "May I ask if you are averaging probabilities, ranked probabilities or the raw output from the models? 😸", "votes": 1 }, { "id": 1862537, "postDate": "2022-07-19T19:21:55.067Z", "content": "ranked probabilities
", "rawMarkdown": "ranked probabilities", "votes": 7 }, { "id": 1864217, "postDate": "2022-07-20T20:57:59.390Z", "content": "sorry, what do you mean by ranked probabilities?
", "rawMarkdown": "sorry, what do you mean by ranked probabilities?", "votes": 1 }, { "id": 1864237, "postDate": "2022-07-20T21:14:29.250Z", "content": "https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.rankdata.html
", "rawMarkdown": "https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.rankdata.html", "votes": 5 } ] }, { "id": 1862003, "postDate": "2022-07-19T11:40:21.310Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 1862702, "author_name": "Mengfei Li", "author_url": "", "post_date": "2022-07-19T22:59:38.583000", "content": "In fact, if your PC has enough RAM or batch training for you is feasible, you can simply add all the useful features from the public notebook to your own model. If you can get to 0.800+ through blend, that should already give you a score of 0.800+ with extra features. As for blend, you must carefully analyze the correlation between the public notebook output and the output of your original model. Even so, we should be careful.
", "votes": 4, "replies": [] }, { "id": 1872937, "author_name": "Lucas Morin", "author_url": "", "post_date": "2022-07-27T10:36:29.387000", "content": "Don’t forget about covid…. feels weird that everyone is fighting for the 4th decimal place on CV/LB while the private will be affected by the health crisis of the century.
", "votes": 1, "replies": [] }, { "id": 1868329, "author_name": "The Devastator", "author_url": "", "post_date": "2022-07-23T23:10:59.710000", "content": "Great insights! I think you've really hit the nail on the head with regard to the current state of the leaderboard.
\nI think you're also right that it's very hard to measure how much you've overfit on the public LB, and that it's probably not worth the risk in the end.
good job !!!
", "votes": 1, "replies": [] }, { "id": 1864957, "author_name": "Pratik Bharamgude", "author_url": "", "post_date": "2022-07-21T11:59:27.840000", "content": "yes yes yes !!!1
", "votes": 1, "replies": [] }, { "id": 1862804, "author_name": "kgxiao", "author_url": "", "post_date": "2022-07-20T02:39:16.737000", "content": "The best score in the public notebook is using DART, your three models, are you using DART? 😯
", "votes": 1, "replies": [ { "id": 1863200, "author_name": "raddar", "author_url": "", "post_date": "2022-07-20T08:00:15.910000", "content": "one of them - yes
", "votes": 5, "replies": [] } ] }, { "id": 1864227, "author_name": "Oday Mourad", "author_url": "", "post_date": "2022-07-20T21:08:58.217000", "content": "Do you think that optimizing models must be only by cv scores?
", "votes": 2, "replies": [ { "id": 1864239, "author_name": "raddar", "author_url": "", "post_date": "2022-07-20T21:14:51.117000", "content": "For this competition - 100% sure
", "votes": 7, "replies": [] }, { "id": 1864254, "author_name": "Mengfei Li", "author_url": "", "post_date": "2022-07-20T21:27:01.337000", "content": "I agree with you that if the proportion of positive and negative samples in the training and test sets is basically the same, and the data distribution of public and private LB are also similar. Then you should trust cv. But you may want to set a tolerance standard deviation of about 0.0001. One more thing, 51% of the data is in the public LB, it should not be ignored.
", "votes": 6, "replies": [] } ] }, { "id": 1862176, "author_name": "Gaurav Rawat", "author_url": "", "post_date": "2022-07-19T13:55:55.220000", "content": "We are are having something similar with our best CV ensemble. If I may ask based on OOF are you just doing a average ensemble or also trying weighted or stacking strategies (all based on oof of course) to boost CV .
", "votes": 2, "replies": [ { "id": 1862357, "author_name": "raddar", "author_url": "", "post_date": "2022-07-19T16:15:01.280000", "content": "simple weighted average
", "votes": 8, "replies": [] }, { "id": 1862491, "author_name": "delai50", "author_url": "", "post_date": "2022-07-19T18:33:30.133000", "content": "May I ask if you are averaging probabilities, ranked probabilities or the raw output from the models? 😸
", "votes": 1, "replies": [] }, { "id": 1862537, "author_name": "raddar", "author_url": "", "post_date": "2022-07-19T19:21:55.067000", "content": "ranked probabilities
", "votes": 7, "replies": [] }, { "id": 1864217, "author_name": "mavillan", "author_url": "", "post_date": "2022-07-20T20:57:59.390000", "content": "sorry, what do you mean by ranked probabilities?
", "votes": 1, "replies": [] }, { "id": 1864237, "author_name": "raddar", "author_url": "", "post_date": "2022-07-20T21:14:29.250000", "content": "https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.rankdata.html
", "votes": 5, "replies": [] } ] }, { "id": 1862003, "author_name": "", "author_url": "", "post_date": "2022-07-19T11:40:21.310000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1861935": "Going to share my take of how I feel about the current state of the leaderboard.\n\nI have 3 strong models which:\n- ensemble scores 0.8001 on CV\n- LB score 0.7999 (quite confident about 4th digit as its the best 0.799 model I have submitted when ranked in submission page)\n\nThat's good enough for ~#30 place on the current state of LB.\n\nThen I average my best CV ensemble with best public notebook submission - submit multiple times to optimize ensemble weights to maximize public LB score - current #11th place. \nBased on current standings I would extrapolate that to LB score of 0.8007 public LB score.\n\nI am very confident that the public LB model did not improve my model, as I have reused it in my current models on my folds. \n\nWhat does this mean?\n- I simply gained +0.0008 on LB by using the available public kernels. I am fairly confident that this gain will not hold on the private LB - in fact I am not going to choose this submission as my final submissions at the end of the competition\n- If you use several models weights optimized on public LB ranking, you are essentially doing the very same thing as I did with my experiment. It's even worse as it is even much harder to measure how much you have overfitted and might make you overconfident of your LB rankings. \n- Ignore available public submission files - they were not trained on your CV folds - you can still take the ideas from them, or retrain on your folds :)\n- Optimize model weights based on CV score. Don't optimize model ensembles based on the leaderboard results.\n\n\n", "1862702": "In fact, if your PC has enough RAM or batch training for you is feasible, you can simply add all the useful features from the public notebook to your own model. If you can get to 0.800+ through blend, that should already give you a score of 0.800+ with extra features. As for blend, you must carefully analyze the correlation between the public notebook output and the output of your original model. Even so, we should be careful. ", "1872937": "Don’t forget about covid.... feels weird that everyone is fighting for the 4th decimal place on CV/LB while the private will be affected by the health crisis of the century.", "1868329": "Great insights! I think you've really hit the nail on the head with regard to the current state of the leaderboard.\nI think you're also right that it's very hard to measure how much you've overfit on the public LB, and that it's probably not worth the risk in the end.\n\n", "1865078": "good job !!!\n", "1864957": "yes yes yes !!!1", "1862804": "The best score in the public notebook is using DART, your three models, are you using DART? 😯", "1864227": "Do you think that optimizing models must be only by cv scores?", "1862176": "We are are having something similar with our best CV ensemble. If I may ask based on OOF are you just doing a average ensemble or also trying weighted or stacking strategies (all based on oof of course) to boost CV .", "1862003": "" } }