Competition is about to end soon. I would really like to know other's view on shakeup. I believe my team didn't overfit public LB much. Is there any possibilities that a shakeup would occur so that I could maintain some hope of grabbing a medal??😅😅
\nI noticed some of the models are having a large difference between CV and LB(ex: our Efficient net b0 had LB score +0.009 while b3 had just +0.003 in comparison with Cv score. Can you explain that too??? Which submission should I prefer considering the above situation?? the one with better cv score or Lb score???
\nThank you.
Agree with @philippsinger . There might be score shakeup, but position shakeup may not be that wild. A local reshuffle is more likely to happen.
", "rawMarkdown": "Agree with @philippsinger . There might be score shakeup, but position shakeup may not be that wild. A local reshuffle is more likely to happen.", "votes": 4 }, { "id": 935830, "postDate": "2020-07-19T17:21:50.903Z", "content": "There will be huge shake-up but not for everyone.
", "rawMarkdown": "There will be huge shake-up but not for everyone.", "votes": 3, "replies": [ { "id": 936496, "postDate": "2020-07-20T09:14:04.737Z", "content": "I am curious, according to you, what would be the cause of a huge shake-up ?
", "rawMarkdown": "I am curious, according to you, what would be the cause of a huge shake-up ?" } ] }, { "id": 935602, "postDate": "2020-07-19T14:14:31.263Z", "content": "There was some analysis done on this topic already and short answer... yes
", "rawMarkdown": "There was some analysis done on this topic already and short answer... [yes](https://www.kaggle.com/c/alaska2-image-steganalysis/discussion/162936)\n", "votes": 3, "replies": [ { "id": 935770, "postDate": "2020-07-19T16:07:23.780Z", "content": "Just because there is a variance between subsamples doesnt mean there is shakeup. This is the wrong conclusion. It just means that some subsamples are harder to predict than others.
\n\nNot saying there wont be any here.
", "rawMarkdown": "Just because there is a variance between subsamples doesnt mean there is shakeup. This is the wrong conclusion. It just means that some subsamples are harder to predict than others. \n\nNot saying there wont be any here.", "votes": 2 }, { "id": 935847, "postDate": "2020-07-19T17:46:15.763Z", "content": "Your point is valid. My thinking is that if you look at the variance between subsamples in the analysis vs the variance of top leaderboard scores then some shakeup is to be expected. My guess is it will be moderate.
", "rawMarkdown": "Your point is valid. My thinking is that if you look at the variance between subsamples in the analysis vs the variance of top leaderboard scores then some shakeup is to be expected. My guess is it will be moderate." }, { "id": 935975, "postDate": "2020-07-19T21:09:08.547Z", "content": "how about to take models A,B,C,D and all of their possible ensemble combinations, then predict on subsamples and analyse the variance between models and ensembles across subsamples?
", "rawMarkdown": "how about to take models A,B,C,D and all of their possible ensemble combinations, then predict on subsamples and analyse the variance between models and ensembles across subsamples?", "votes": 1 }, { "id": 935986, "postDate": "2020-07-19T21:25:01.857Z", "content": "Yes, that's the way to go.
", "rawMarkdown": "Yes, that's the way to go." }, { "id": 936082, "postDate": "2020-07-20T00:47:22.283Z", "content": "For me, it makes sense. Thanks for your advice!
", "rawMarkdown": "For me, it makes sense. Thanks for your advice!", "votes": 1 } ] }, { "id": 935281, "postDate": "2020-07-19T08:21:46.763Z", "content": "Competition is about to end soon. I would really like to know other's view on shakeup. I believe my team didn't overfit public LB much. Is there any possibilities that a shakeup would occur so that I could maintain some hope of grabbing a medal??😅😅
\nI noticed some of the models are having a large difference between CV and LB(ex: our Efficient net b0 had LB score +0.009 while b3 had just +0.003 in comparison with Cv score. Can you explain that too??? Which submission should I prefer considering the above situation?? the one with better cv score or Lb score???
\nThank you.
I also agree with @sgalib. The CV and LB correlate fairly well so far. If there is no hidden trap in the testing data I am not expecting to see a good shakeup.
", "rawMarkdown": "I also agree with @sgalib. The CV and LB correlate fairly well so far. If there is no hidden trap in the testing data I am not expecting to see a good shakeup. ", "votes": 2, "replies": [ { "id": 935817, "postDate": "2020-07-19T16:49:59.913Z", "rawMarkdown": "", "isDeleted": true }, { "id": 935913, "postDate": "2020-07-19T19:15:58.490Z", "content": "For me, the CV and LB has a difference of around ~0.002. I guess it depends on luck and maybe batch size. What is your difference?
", "rawMarkdown": "For me, the CV and LB has a difference of around ~0.002. I guess it depends on luck and maybe batch size. What is your difference? ", "votes": 1 }, { "id": 935916, "postDate": "2020-07-19T19:17:09.533Z", "content": "same for me, and for all model architectures.
", "rawMarkdown": "same for me, and for all model architectures." }, { "id": 937103, "postDate": "2020-07-20T18:12:07.040Z", "content": "Yeah. Let's see what happens in the private.
", "rawMarkdown": "Yeah. Let's see what happens in the private. ", "votes": 1 } ] }, { "id": 935642, "postDate": "2020-07-19T14:35:44.170Z", "content": "Since you have a chance to choose 2 submissions, you can try one with best lb score, and one with best cv score in case they are not the same sub :)
", "rawMarkdown": "Since you have a chance to choose 2 submissions, you can try one with best lb score, and one with best cv score in case they are not the same sub :)", "votes": 2 }, { "id": 936414, "postDate": "2020-07-20T07:48:29.800Z", "content": "Models based on DCT or SRNet may be more robust, which makes me have to use one submission to fuse a related model. I guess due to the problem of cover-source mismatch, there will be a shakeup for some non-efficientnet models...
", "rawMarkdown": "Models based on DCT or SRNet may be more robust, which makes me have to use one submission to fuse a related model. I guess due to the problem of cover-source mismatch, there will be a shakeup for some non-efficientnet models..." } ], "comments": [ { "id": 935794, "author_name": "Shai", "author_url": "", "post_date": "2020-07-19T16:29:18.407000", "content": "Agree with @philippsinger . There might be score shakeup, but position shakeup may not be that wild. A local reshuffle is more likely to happen.
", "votes": 4, "replies": [] }, { "id": 935830, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2020-07-19T17:21:50.903000", "content": "There will be huge shake-up but not for everyone.
", "votes": 3, "replies": [ { "id": 936496, "author_name": "Ppo", "author_url": "", "post_date": "2020-07-20T09:14:04.737000", "content": "I am curious, according to you, what would be the cause of a huge shake-up ?
", "votes": 0, "replies": [] } ] }, { "id": 935602, "author_name": "David Austin", "author_url": "", "post_date": "2020-07-19T14:14:31.263000", "content": "There was some analysis done on this topic already and short answer... yes
", "votes": 3, "replies": [ { "id": 935770, "author_name": "Psi", "author_url": "", "post_date": "2020-07-19T16:07:23.780000", "content": "Just because there is a variance between subsamples doesnt mean there is shakeup. This is the wrong conclusion. It just means that some subsamples are harder to predict than others.
\n\nNot saying there wont be any here.
", "votes": 2, "replies": [] }, { "id": 935847, "author_name": "David Austin", "author_url": "", "post_date": "2020-07-19T17:46:15.763000", "content": "Your point is valid. My thinking is that if you look at the variance between subsamples in the analysis vs the variance of top leaderboard scores then some shakeup is to be expected. My guess is it will be moderate.
", "votes": 0, "replies": [] }, { "id": 935975, "author_name": "Miroslav Valan", "author_url": "", "post_date": "2020-07-19T21:09:08.547000", "content": "how about to take models A,B,C,D and all of their possible ensemble combinations, then predict on subsamples and analyse the variance between models and ensembles across subsamples?
", "votes": 1, "replies": [] }, { "id": 935986, "author_name": "Psi", "author_url": "", "post_date": "2020-07-19T21:25:01.857000", "content": "Yes, that's the way to go.
", "votes": 0, "replies": [] }, { "id": 936082, "author_name": "Luck", "author_url": "", "post_date": "2020-07-20T00:47:22.283000", "content": "For me, it makes sense. Thanks for your advice!
", "votes": 1, "replies": [] } ] }, { "id": 935813, "author_name": "Md Yasin Kabir", "author_url": "", "post_date": "2020-07-19T16:44:01.343000", "content": "I also agree with @sgalib. The CV and LB correlate fairly well so far. If there is no hidden trap in the testing data I am not expecting to see a good shakeup.
", "votes": 2, "replies": [ { "id": 935817, "author_name": "", "author_url": "", "post_date": "2020-07-19T16:49:59.913000", "content": "", "votes": 0, "replies": [] }, { "id": 935913, "author_name": "Md Yasin Kabir", "author_url": "", "post_date": "2020-07-19T19:15:58.490000", "content": "For me, the CV and LB has a difference of around ~0.002. I guess it depends on luck and maybe batch size. What is your difference?
", "votes": 1, "replies": [] }, { "id": 935916, "author_name": "[ods.ai]Uladzimir Tumanau", "author_url": "", "post_date": "2020-07-19T19:17:09.533000", "content": "same for me, and for all model architectures.
", "votes": 0, "replies": [] }, { "id": 937103, "author_name": "Md Yasin Kabir", "author_url": "", "post_date": "2020-07-20T18:12:07.040000", "content": "Yeah. Let's see what happens in the private.
", "votes": 1, "replies": [] } ] }, { "id": 935642, "author_name": "[ods.ai]Uladzimir Tumanau", "author_url": "", "post_date": "2020-07-19T14:35:44.170000", "content": "Since you have a chance to choose 2 submissions, you can try one with best lb score, and one with best cv score in case they are not the same sub :)
", "votes": 2, "replies": [] }, { "id": 936414, "author_name": "ZHANG Zhi", "author_url": "", "post_date": "2020-07-20T07:48:29.800000", "content": "Models based on DCT or SRNet may be more robust, which makes me have to use one submission to fuse a related model. I guess due to the problem of cover-source mismatch, there will be a shakeup for some non-efficientnet models...
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "935794": "Agree with @philippsinger . There might be score shakeup, but position shakeup may not be that wild. A local reshuffle is more likely to happen.", "935830": "There will be huge shake-up but not for everyone.", "935602": "There was some analysis done on this topic already and short answer... [yes](https://www.kaggle.com/c/alaska2-image-steganalysis/discussion/162936)\n", "935281": "Competition is about to end soon. I would really like to know other's view on shakeup. I believe my team didn't overfit public LB much. Is there any possibilities that a shakeup would occur so that I could maintain some hope of grabbing a medal??😅😅\n\nI noticed some of the models are having a large difference between CV and LB(ex: our Efficient net b0 had LB score +0.009 while b3 had just +0.003 in comparison with Cv score. Can you explain that too??? Which submission should I prefer considering the above situation?? the one with better cv score or Lb score??? \nThank you.", "935813": "I also agree with @sgalib. The CV and LB correlate fairly well so far. If there is no hidden trap in the testing data I am not expecting to see a good shakeup. ", "935642": "Since you have a chance to choose 2 submissions, you can try one with best lb score, and one with best cv score in case they are not the same sub :)", "936414": "Models based on DCT or SRNet may be more robust, which makes me have to use one submission to fuse a related model. I guess due to the problem of cover-source mismatch, there will be a shakeup for some non-efficientnet models..." } }