![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2730705%2F6d4a5e3a425723baf8688546886791b4%2Fshake.jpg?generation=1597625830912705&alt=media\")
Hello everyone,
\nAre we going to face a huge shakeup like M5 forecasting again?
\nAnd what would be the possible things we can do to avoid this?
\nAny thoughts?
Hello everyone,
\nAre we going to face a huge shakeup like M5 forecasting again?
\nAnd what would be the possible things we can do to avoid this?
\nAny thoughts?
When a competition shakes like M5, you can basically do nothing I think
", "rawMarkdown": "When a competition shakes like M5, you can basically do nothing I think", "votes": null }, { "id": "969427", "postDate": "08/13/2020 17:08:25", "content": "Try to wisely choose 3 submissions, different approaches. For most of us the final place is unpredictable, think it will be private lb = public lb +- 200
", "rawMarkdown": "Try to wisely choose 3 submissions, different approaches. For most of us the final place is unpredictable, think it will be private lb = public lb +- 200", "votes": null }, { "id": "969453", "postDate": "08/13/2020 17:39:29", "content": "Maximizing CV, ensembling⦠This list is pretty standard. But somehow I feel that randomness is going to play a significant role in this competition. So, you can try adding \"praying\" to this list⦠:-)
", "rawMarkdown": "Maximizing CV, ensembling... This list is pretty standard. But somehow I feel that randomness is going to play a significant role in this competition. So, you can try adding \"praying\" to this list... :-)", "votes": null }, { "id": "969464", "postDate": "08/13/2020 17:46:54", "content": "@jielu0728
\nYeah, hoping for the best now, let's see.
@aybatov
\nYeah, let's see
Haha, yeah, I wish 'praying' can help that much!
\nI agree with you about randomness.
\n\nHaha, yeah, I wish 'praying' can help that much!
\n
One thing is for sure -- it can't hurt :-).
", "rawMarkdown": "> Haha, yeah, I wish 'praying' can help that much!\n\nOne thing is for sure -- it can't hurt :-).", "votes": null }, { "id": "969549", "postDate": "08/13/2020 19:07:01", "content": "Crossing fingers also helps π€
", "rawMarkdown": "Crossing fingers also helps π€", "votes": null }, { "id": "969734", "postDate": "08/13/2020 22:26:14", "content": "I've found drinking tends to help too π» :)
", "rawMarkdown": "I've found drinking tends to help too π» :)", "votes": null }, { "id": "969804", "postDate": "08/14/2020 00:41:45", "content": "why even think about the shakeup ? do your best and believe in your validation, not in the LB.
", "rawMarkdown": "why even think about the shakeup ? do your best and believe in your validation, not in the LB.", "votes": null }, { "id": "969847", "postDate": "08/14/2020 01:58:15", "content": "If this is like M5, that means that everyone has a gold medal in their submissions. Choose carefully!
", "rawMarkdown": "If this is like M5, that means that everyone has a gold medal in their submissions. Choose carefully!", "votes": null }, { "id": "969916", "postDate": "08/14/2020 04:07:51", "content": "\nHaha, let's drink(coffee for meπ) and watch the show!
", "rawMarkdown": "robikscube \n\nHaha, let's drink(coffee for meπ) and watch the show!", "votes": null }, { "id": "969921", "postDate": "08/14/2020 04:14:43", "content": "\nJust thinking, since the public LB is based on just 30% of data, so there is a good chance of shake-up!
\nAnd yeah, we should focus more on the CV than LB, let's see.
\nAll the best, man π
Yeah, I think, choosing final submissions will be playing a crucial part in this competition!
\nAnd, thanks a lot Chris, you are an inspiration for many of us β€οΈ
", "rawMarkdown": "cdeotte \n\nYeah, I think, choosing final submissions will be playing a crucial part in this competition!\n\nAnd, thanks a lot Chris, you are an inspiration for many of us β€οΈ", "votes": null }, { "id": "970011", "postDate": "08/14/2020 05:54:34", "content": "Have a very good baseline supported by a solid validation strategy.
", "rawMarkdown": "Have a very good baseline supported by a solid validation strategy.", "votes": null }, { "id": "970056", "postDate": "08/14/2020 06:48:28", "content": "That's where I am focusing on now, @pratikkumar
", "rawMarkdown": "That's where I am focusing on now, @pratikkumar", "votes": null }, { "id": "970124", "postDate": "08/14/2020 08:15:26", "content": "I don't think it will be like M5. M5 had only 1 sub and no public test data (for 2nd phase) which meant that you had only your CV to trust.
\nHere we have 3 subs and all test images are public. I think randomness will play its part (variance in models, seeds etc.)
\nMy mantra for the next few days - \"The universe tends to unfold as it should.\"
", "rawMarkdown": "I don't think it will be like M5. M5 had only 1 sub and no public test data (for 2nd phase) which meant that you had only your CV to trust. \n\nHere we have 3 subs and all test images are public. I think randomness will play its part (variance in models, seeds etc.)\n\nMy mantra for the next few days - **\"The universe tends to unfold as it should.\"**", "votes": null }, { "id": "970127", "postDate": "08/14/2020 08:25:20", "content": "\nYeah, let's hope for the best
", "rawMarkdown": "manjeshg03 \n\nYeah, let's hope for the best", "votes": null }, { "id": "970986", "postDate": "08/15/2020 04:26:00", "content": "Gonna try it this weekend and perhaps post the results after the competition ends π
", "rawMarkdown": "Gonna try it this weekend and perhaps post the results after the competition ends π", "votes": null }, { "id": "970989", "postDate": "08/15/2020 04:33:00", "content": "There will be as most of score from LB rank 15 to ~600 are in .97 to .96 .
\nBut how much we dont know , could be like Panda one which recently closed on 23rd July.
\nGlobal leader in score .9619 will be interesting to observe. There are almost 200 entry of this.
\nJust 3 days to go β¦
it was funny competition, small data size (this time target1 are 200 orso) is always unstable & dangerous for shakeup haha
I think the shakeup will be big. It's informative that many Grandmasters have only submitted LB solutions around position 500. That says something about public LB overfitting.
", "rawMarkdown": "I think the shakeup will be big. It's informative that many Grandmasters have only submitted LB solutions around position 500. That says something about public LB overfitting.", "votes": null }, { "id": "972922", "postDate": "08/17/2020 01:23:29", "content": "i think it's because of public LB proving,
\ni found i could get lb + 0.01 score with simple post processing, but i'm not using it cuz it's so danger & make subs overfit to lb
\n(although basically i think this competition will be danger then made little effort)
Most of the people are using ensembling, multiple models chances are kind of less, but can't deny. Let's wait for one more day to see the final output.
", "rawMarkdown": "Most of the people are using ensembling, multiple models chances are kind of less, but can't deny. Let's wait for one more day to see the final output.", "votes": null }, { "id": "973025", "postDate": "08/17/2020 03:49:21", "content": "@cdeotte
\nHow should we choose the final 3? Any strategy?
ππ
\nNow, let's wait to see how big it is!
There are many strategies and I give my specific opinion after the competition finishes. (Hopefully my strategy works π) For every choice you must ask yourself, \"what is the best guess of this model's (or ensemble's) expected private LB score and with what variance, i.e. confidence interval?\".
\nThen you either choose the highest expected value if you're conservative or the highest variance if you're a gambler. π(or since we have 3 submissions, choose a combination of both).
", "rawMarkdown": "There are many strategies and I give my specific opinion after the competition finishes. (Hopefully my strategy works π) For every choice you must ask yourself, \"what is the best guess of this model's (or ensemble's) expected private LB score and with what variance, i.e. confidence interval?\".\n\nThen you either choose the highest expected value if you're conservative or the highest variance if you're a gambler. π(or since we have 3 submissions, choose a combination of both).", "votes": null }, { "id": "973052", "postDate": "08/17/2020 04:37:03", "content": "what you think about geometric mean ensemble of predictions from various models.
", "rawMarkdown": "what you think about geometric mean ensemble of predictions from various models.", "votes": null }, { "id": "973077", "postDate": "08/17/2020 05:03:22", "content": "I would test geometric mean on CV. If it increases CV more than arithmetic mean, then go for it.
", "rawMarkdown": "I would test geometric mean on CV. If it increases CV more than arithmetic mean, then go for it.", "votes": null }, { "id": "973193", "postDate": "08/17/2020 06:54:04", "content": "I submitted best public just to feed my ego, I suppose) For final I choose much more stable models (I hope). And thanks, @cdeotte for your brilliant notebooks!
", "rawMarkdown": "I submitted best public just to feed my ego, I suppose) For final I choose much more stable models (I hope). And thanks, @cdeotte for your brilliant notebooks!", "votes": null }, { "id": "973374", "postDate": "08/17/2020 09:17:08", "content": "Agreed. Big. Eg, I have a single model yielding CV .955 +- .015, and that's the smallest error due to sampling. In my world it is pointless to submit this.
", "rawMarkdown": "Agreed. Big. Eg, I have a single model yielding CV .955 +- .015, and that's the smallest error due to sampling. In my world it is pointless to submit this.", "votes": null }, { "id": "973388", "postDate": "08/17/2020 09:24:54", "content": "@rahim3 ,
\nI think the shakeup will be more than the M5 and the final score will be so close. Few more hours to go. We will see the result.
Yeah, let's see!
\nIt's going to be another sleepless night for me like the M5!
@glimmung
\n0.955 π―
\nWow, that's great for a single model!
What was your highest single model CV and LB, @vin1234?
", "rawMarkdown": "What was your highest single model CV and LB, @vin1234?", "votes": null }, { "id": "975832", "postDate": "08/18/2020 13:41:58", "content": "Congratulations @jielu0728, Well Done π
", "rawMarkdown": "Congratulations @jielu0728, Well Done π", "votes": null }, { "id": "976058", "postDate": "08/18/2020 15:58:33", "content": "Well, with the error, consider from .94 to .97, ranging from 31st to 1363rd spot. Now, it's over, I will submit post competition and checkβ¦
", "rawMarkdown": "Well, with the error, consider from .94 to .97, ranging from 31st to 1363rd spot. Now, it's over, I will submit post competition and check...", "votes": null }, { "id": "976089", "postDate": "08/18/2020 16:17:25", "content": "Thanks!! @rahim3 π
", "rawMarkdown": "Thanks!! @rahim3 π", "votes": null } ], "comments": [ { "id": 969384, "author_name": "jielu0728", "author_url": "", "post_date": "08/13/2020 16:40:31", "content": "When a competition shakes like M5, you can basically do nothing I think
", "votes": null, "replies": [ { "id": 969464, "author_name": "rahim3", "author_url": "", "post_date": "08/13/2020 17:46:54", "content": "@jielu0728
\nYeah, hoping for the best now, let's see.
Congratulations @jielu0728, Well Done π
", "votes": null, "replies": [] }, { "id": 976089, "author_name": "jielu0728", "author_url": "", "post_date": "08/18/2020 16:17:25", "content": "Thanks!! @rahim3 π
", "votes": null, "replies": [] } ] }, { "id": 969427, "author_name": "aybatov", "author_url": "", "post_date": "08/13/2020 17:08:25", "content": "Try to wisely choose 3 submissions, different approaches. For most of us the final place is unpredictable, think it will be private lb = public lb +- 200
", "votes": null, "replies": [ { "id": 969466, "author_name": "rahim3", "author_url": "", "post_date": "08/13/2020 17:50:27", "content": "@aybatov
\nYeah, let's see
Maximizing CV, ensembling⦠This list is pretty standard. But somehow I feel that randomness is going to play a significant role in this competition. So, you can try adding \"praying\" to this list⦠:-)
", "votes": null, "replies": [ { "id": 969474, "author_name": "rahim3", "author_url": "", "post_date": "08/13/2020 17:53:44", "content": "\nHaha, yeah, I wish 'praying' can help that much!
\nI agree with you about randomness.
\n\nHaha, yeah, I wish 'praying' can help that much!
\n
One thing is for sure -- it can't hurt :-).
", "votes": null, "replies": [] }, { "id": 969549, "author_name": "santiviquez", "author_url": "", "post_date": "08/13/2020 19:07:01", "content": "Crossing fingers also helps π€
", "votes": null, "replies": [] }, { "id": 969734, "author_name": "robikscube", "author_url": "", "post_date": "08/13/2020 22:26:14", "content": "I've found drinking tends to help too π» :)
", "votes": null, "replies": [ { "id": 969916, "author_name": "rahim3", "author_url": "", "post_date": "08/14/2020 04:07:51", "content": "\nHaha, let's drink(coffee for meπ) and watch the show!
", "votes": null, "replies": [] } ] }, { "id": 970986, "author_name": "santiviquez", "author_url": "", "post_date": "08/15/2020 04:26:00", "content": "Gonna try it this weekend and perhaps post the results after the competition ends π
", "votes": null, "replies": [] } ] }, { "id": 969804, "author_name": "yash612", "author_url": "", "post_date": "08/14/2020 00:41:45", "content": "why even think about the shakeup ? do your best and believe in your validation, not in the LB.
", "votes": null, "replies": [ { "id": 969921, "author_name": "rahim3", "author_url": "", "post_date": "08/14/2020 04:14:43", "content": "\nJust thinking, since the public LB is based on just 30% of data, so there is a good chance of shake-up!
\nAnd yeah, we should focus more on the CV than LB, let's see.
\nAll the best, man π
If this is like M5, that means that everyone has a gold medal in their submissions. Choose carefully!
", "votes": null, "replies": [ { "id": 969928, "author_name": "rahim3", "author_url": "", "post_date": "08/14/2020 04:19:52", "content": "\nYeah, I think, choosing final submissions will be playing a crucial part in this competition!
\nAnd, thanks a lot Chris, you are an inspiration for many of us β€οΈ
", "votes": null, "replies": [] } ] }, { "id": 970124, "author_name": "manjeshg03", "author_url": "", "post_date": "08/14/2020 08:15:26", "content": "I don't think it will be like M5. M5 had only 1 sub and no public test data (for 2nd phase) which meant that you had only your CV to trust.
\nHere we have 3 subs and all test images are public. I think randomness will play its part (variance in models, seeds etc.)
\nMy mantra for the next few days - \"The universe tends to unfold as it should.\"
", "votes": null, "replies": [ { "id": 970127, "author_name": "rahim3", "author_url": "", "post_date": "08/14/2020 08:25:20", "content": "\nYeah, let's hope for the best
", "votes": null, "replies": [] } ] }, { "id": 970989, "author_name": "rajnishe", "author_url": "", "post_date": "08/15/2020 04:33:00", "content": "There will be as most of score from LB rank 15 to ~600 are in .97 to .96 .
\nBut how much we dont know , could be like Panda one which recently closed on 23rd July.
\nGlobal leader in score .9619 will be interesting to observe. There are almost 200 entry of this.
\nJust 3 days to go β¦
it was funny competition, small data size (this time target1 are 200 orso) is always unstable & dangerous for shakeup haha
ππ
\nNow, let's wait to see how big it is!
I think the shakeup will be big. It's informative that many Grandmasters have only submitted LB solutions around position 500. That says something about public LB overfitting.
", "votes": null, "replies": [ { "id": 972922, "author_name": "ryunosukeishizaki", "author_url": "", "post_date": "08/17/2020 01:23:29", "content": "i think it's because of public LB proving,
\ni found i could get lb + 0.01 score with simple post processing, but i'm not using it cuz it's so danger & make subs overfit to lb
\n(although basically i think this competition will be danger then made little effort)
@cdeotte
\nHow should we choose the final 3? Any strategy?
There are many strategies and I give my specific opinion after the competition finishes. (Hopefully my strategy works π) For every choice you must ask yourself, \"what is the best guess of this model's (or ensemble's) expected private LB score and with what variance, i.e. confidence interval?\".
\nThen you either choose the highest expected value if you're conservative or the highest variance if you're a gambler. π(or since we have 3 submissions, choose a combination of both).
", "votes": null, "replies": [] }, { "id": 973052, "author_name": "rajnishe", "author_url": "", "post_date": "08/17/2020 04:37:03", "content": "what you think about geometric mean ensemble of predictions from various models.
", "votes": null, "replies": [] }, { "id": 973077, "author_name": "cdeotte", "author_url": "", "post_date": "08/17/2020 05:03:22", "content": "I would test geometric mean on CV. If it increases CV more than arithmetic mean, then go for it.
", "votes": null, "replies": [] }, { "id": 973193, "author_name": "uspangaliev", "author_url": "", "post_date": "08/17/2020 06:54:04", "content": "I submitted best public just to feed my ego, I suppose) For final I choose much more stable models (I hope). And thanks, @cdeotte for your brilliant notebooks!
", "votes": null, "replies": [] }, { "id": 973374, "author_name": "glimmung", "author_url": "", "post_date": "08/17/2020 09:17:08", "content": "Agreed. Big. Eg, I have a single model yielding CV .955 +- .015, and that's the smallest error due to sampling. In my world it is pointless to submit this.
", "votes": null, "replies": [] }, { "id": 973408, "author_name": "rahim3", "author_url": "", "post_date": "08/17/2020 09:32:35", "content": "@glimmung
\n0.955 π―
\nWow, that's great for a single model!
Well, with the error, consider from .94 to .97, ranging from 31st to 1363rd spot. Now, it's over, I will submit post competition and checkβ¦
", "votes": null, "replies": [] } ] }, { "id": 972949, "author_name": "vin1234", "author_url": "", "post_date": "08/17/2020 01:47:52", "content": "Most of the people are using ensembling, multiple models chances are kind of less, but can't deny. Let's wait for one more day to see the final output.
", "votes": null, "replies": [ { "id": 973423, "author_name": "rahim3", "author_url": "", "post_date": "08/17/2020 09:41:25", "content": "What was your highest single model CV and LB, @vin1234?
", "votes": null, "replies": [] } ] }, { "id": 973388, "author_name": "mdfahimreshm", "author_url": "", "post_date": "08/17/2020 09:24:54", "content": "@rahim3 ,
\nI think the shakeup will be more than the M5 and the final score will be so close. Few more hours to go. We will see the result.
Yeah, let's see!
\nIt's going to be another sleepless night for me like the M5!
Have a very good baseline supported by a solid validation strategy.
", "votes": null, "replies": [ { "id": 970056, "author_name": "rahim3", "author_url": "", "post_date": "08/14/2020 06:48:28", "content": "That's where I am focusing on now, @pratikkumar
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "969354": "Hello everyone,\n\nAre we going to face a huge **shakeup** like M5 forecasting again? \nAnd what would be the possible things we can do to avoid this? \nAny thoughts?", "969384": "When a competition shakes like M5, you can basically do nothing I think", "969427": "Try to wisely choose 3 submissions, different approaches. For most of us the final place is unpredictable, think it will be private lb = public lb +- 200", "969453": "Maximizing CV, ensembling... This list is pretty standard. But somehow I feel that randomness is going to play a significant role in this competition. So, you can try adding \"praying\" to this list... :-)", "969464": "jielu0728 \nYeah, hoping for the best now, let's see.", "969466": "aybatov \nYeah, let's see", "969474": "graf10a \n\nHaha, yeah, I wish 'praying' can help that much!\nI agree with you about randomness.", "969545": "> Haha, yeah, I wish 'praying' can help that much!\n\nOne thing is for sure -- it can't hurt :-).", "969549": "Crossing fingers also helps π€", "969734": "I've found drinking tends to help too π» :)", "969804": "why even think about the shakeup ? do your best and believe in your validation, not in the LB.", "969847": "If this is like M5, that means that everyone has a gold medal in their submissions. Choose carefully!", "969916": "robikscube \n\nHaha, let's drink(coffee for meπ) and watch the show!", "969921": "yash612 \n\nJust thinking, since the public LB is based on just 30% of data, so there is a good chance of shake-up!\nAnd yeah, we should focus more on the CV than LB, let's see. \nAll the best, man π", "969928": "cdeotte \n\nYeah, I think, choosing final submissions will be playing a crucial part in this competition!\n\nAnd, thanks a lot Chris, you are an inspiration for many of us β€οΈ", "970011": "Have a very good baseline supported by a solid validation strategy.", "970056": "That's where I am focusing on now, @pratikkumar", "970124": "I don't think it will be like M5. M5 had only 1 sub and no public test data (for 2nd phase) which meant that you had only your CV to trust. \n\nHere we have 3 subs and all test images are public. I think randomness will play its part (variance in models, seeds etc.)\n\nMy mantra for the next few days - **\"The universe tends to unfold as it should.\"**", "970127": "manjeshg03 \n\nYeah, let's hope for the best", "970986": "Gonna try it this weekend and perhaps post the results after the competition ends π", "970989": "There will be as most of score from LB rank 15 to ~600 are in .97 to .96 .\nBut how much we dont know , could be like Panda one which recently closed on 23rd July.\nGlobal leader in score .9619 will be interesting to observe. There are almost 200 entry of this.\nJust 3 days to go ...", "972905": "it was funny competition, small data size (this time target1 are 200 orso) is always unstable & dangerous for shakeup haha", "972916": "I think the shakeup will be big. It's informative that many Grandmasters have only submitted LB solutions around position 500. That says something about public LB overfitting.", "972922": "i think it's because of public LB proving,\ni found i could get lb + 0.01 score with simple post processing, but i'm not using it cuz it's so danger & make subs overfit to lb\n(although basically i think this competition will be danger then made little effort)", "972949": "Most of the people are using ensembling, multiple models chances are kind of less, but can't deny. Let's wait for one more day to see the final output.", "973025": "cdeotte \nHow should we choose the final 3? Any strategy?", "973028": "ππ\nNow, let's wait to see how big it is!", "973048": "There are many strategies and I give my specific opinion after the competition finishes. (Hopefully my strategy works π) For every choice you must ask yourself, \"what is the best guess of this model's (or ensemble's) expected private LB score and with what variance, i.e. confidence interval?\".\n\nThen you either choose the highest expected value if you're conservative or the highest variance if you're a gambler. π(or since we have 3 submissions, choose a combination of both).", "973052": "what you think about geometric mean ensemble of predictions from various models.", "973077": "I would test geometric mean on CV. If it increases CV more than arithmetic mean, then go for it.", "973193": "I submitted best public just to feed my ego, I suppose) For final I choose much more stable models (I hope). And thanks, @cdeotte for your brilliant notebooks!", "973374": "Agreed. Big. Eg, I have a single model yielding CV .955 +- .015, and that's the smallest error due to sampling. In my world it is pointless to submit this.", "973388": "rahim3 ,\nI think the **shakeup** will be more than the **M5** and the final score will be **so close**. Few more hours to go. We will see the result.", "973400": "Yeah, let's see!\nIt's going to be another sleepless night for me like the M5!", "973408": "glimmung \n0.955 π― \nWow, that's great for a single model!", "973423": "What was your highest single model CV and LB, @vin1234?", "975832": "Congratulations @jielu0728, Well Done π", "976058": "Well, with the error, consider from .94 to .97, ranging from 31st to 1363rd spot. Now, it's over, I will submit post competition and check...", "976089": "Thanks!! @rahim3 π" }, "source": "meta" }