{
  "id": 136877,
  "title": "257th Place Solution and Observations (public LB 55)",
  "url": "/competitions/bengaliai-cv19/writeups/257th-place-solution-and-observations-public-lb-55",
  "author_name": "",
  "post_date": "2020-03-18T05:35:59.339320400Z",
  "votes": 7,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Congratulation to all winners! I was definitely shocked by the shake and losing my first solo silver. I hope to share my solution as offering a comparison to winner solutions and looking forward to hear you about what might went wrong.</p>\n\n<p>I built my solution based on <a href=\"/corochann\">@corochann</a>'s kernel (<a href=\"https://www.kaggle.com/corochann/bengali-seresnext-prediction-with-pytorch\">https://www.kaggle.com/corochann/bengali-seresnext-prediction-with-pytorch</a>).</p>\n\n<p>I first tried around with seresnext50 and some basic augmentations, but the public LB score wandered under 0.967x. Then I stopped cropping images and everything became bright.</p>\n\n<p>Next, I experimented with different augmentation combinations and found cutmix only to work the best and sticked to it in different model testing process.</p>\n\n<p>Afterwards, I tested with models including seresnext50, seresnext101, densenet121, and res2net using 256x256 images.</p>\n\n<p>My final submissions consist of weighted ensembles of seresnext50 and res2net. The highest public LB score resulted is 0.9884 but with private score only 0.9290. </p>\n\n<p>I used 80/20 split for all training and never used all data for one single model. I'm not sure that is a problem. In the 0.9884 ensemble, I used 2 splits of res2net and 1 split of seresnext. I used stepLR with milestone of 75 and 90.</p>\n\n<p>Interestingly, I found that my highest private LB score is produced by densenet121. The ensemble of 3 different epochs of densenet121 (same split) resulted in public LB of 0.9799 but private LB of 0.9381. I also saw that efficientnet worked poorly on public LB but well on private LB in other people's solutions. Not sure if these simple model would have better performance on private LB. I also observed that densenet has a smaller cv/lb gap of around 0.5%, while seresnext and res2net all have cv/lb gap around 1.3%.</p>\n\n<p>I also found that local CV does not correlate with private LB nor public LB in my case. The densenet121 model never made its local CV above 0.99, while seresnext and res2net both achieved 0.995-6. So I guess \"trusting your CV\" is not a valid strategy for this competition.</p>\n\n<p>All in all, an unexpected competition experience. Definitely a bit sad losing a silver medal. Will try my best to get good score in the rest competitions before summer. </p>\n\n<p>Please comment below if you see any potential problem and upvote if you find this post interesting.</p>\n\n<p>Happy kaggling.</p>\n\n<p>Below is my full training/testing record:</p>\n\n<p>'Se_resnext50_32x4d_kernel_baseline_newlr' mixup/cutmix\nCv: 0.9868\nEpoch 102\nLB: 0.9646\ndiff : 0.0222</p>\n\n<p>Se_resnext50_32x4d_kernel_baseline_augmentation_randomseed666_8020split\nEpoch 35\nCv 0.9775\nLb 0.9641\nDiff = 0.0134</p>\n\n<p>Epoch 44\nCv 0.978\nLb 0.9646\nDff: 0.0134</p>\n\n<p>Epoch 84\nCv 0.9824\nLb 0.9654\nDiff = 0.017</p>\n\n<p>Milestone lr scheduler</p>\n\n<p>epoch 22\nCv 0.9783\nLb 0.9665\nDiff 0.0118</p>\n\n<p>Epoch 30\nCv 0.98\nLb 0.9674\nDiff 0.0126</p>\n\n<p>Epoch 42\nCv 0.9804\nLb 0.9678\nDiff 0.0126</p>\n\n<p>Epoch 53\nCv 0.9805\nLb 0.9676\nDiff 0.0129</p>\n\n<p>Milestone lr, ohem loss, stratified</p>\n\n<p>Epoch 17\nCv 0.9717\nLb 0.9527\nDiff 0.019</p>\n\n<p>Epoch 29\nCv 0.9739\nLb 0.9588\nDiff 0.015</p>\n\n<p>Epoch 39\nCv 0.9751\nLb 0.9584\nDiff 0.0167</p>\n\n<p>Multilabel stratified - cross entropy loss</p>\n\n<p>Epoch 87\nCv 0.9869\nLb 0.9679\nDiff 0.019</p>\n\n<p>Epoch 135\nCv 0.9886\nLb 0.9679\nDiff 0.0207</p>\n\n<p>Noconv0</p>\n\n<p>Epoch 60\nCv 0.9833\nLb 0.9667\nDiff 0.0166</p>\n\n<p>Epoch 76\nCv 0.9852\nLb 0.9670\nDff 0.0182</p>\n\n<p>Cnn - tail - noaffine\nEpoch 92\nCv 0.9892\nLb 0.9668\nDiff 0.0224</p>\n\n<p>Epoch 77\nCv 0.9872\nLb 0.9665\nDiff 0.0187</p>\n\n<p>Epoch 62\nCv 0.9849\nLb 0.9621\nDiff 0.0228</p>\n\n<p>Tail + affine\nEpoch 96\nCv 0.9866\nLb 0.9668\nDiff 0.0198</p>\n\n<p>Epoch 86\nCv 0.9858\nLb 0.9664\nDiff 0.0194</p>\n\n<p>Epoch 120\nCv 0.9876\nLB 0.9671\nDiff 0.0205</p>\n\n<p>Size 256\nEpoch 134\nCv 0.9896\nLb 0.9694\nDiff 0.0202</p>\n\n<p>Epoch 126\nCv 0.9896\nLb 0.9689\nDiff0.0207</p>\n\n<p>Epoch 114\nCv 0.989\nLb 0.9690\nDiff 0.020</p>\n\n<p>Epoch 107\nCv 0.9886\nLb 0.9688\nDiff 0.0198</p>\n\n<p>Epoch 98\nCv 0.9884\nLb 0.9691\nDiff 0.0193</p>\n\n<p>Epoch 48\nCv 0.981\nLb 0.9672\nDiff 0.0138</p>\n\n<p>Ensemble\nEpoch 134, 114, 98\nLb 0.9695</p>\n\n<p>Resnext 101 size 256\nEpoch 81 crop\nCv 0.9888\nLb 0.9680\nDiff 0.0208</p>\n\n<p>Epoch 81 no_crop\nCv 0.9888\nLb 0.9651\nDiff 0.0237</p>\n\n<p>Epoch 78 crop\nCv 0.9881\nLb 0.9688\nDiff 0.0193</p>\n\n<p>Epoch 65 crop\nCv 0.9851\nLb 0.9659\nDiff 0.0192</p>\n\n<p>Resnext50 size 256 no crop</p>\n\n<p>Epoch 109\nCv 0.9924\nLb 0.9766\nDiff 0.0158</p>\n\n<p>Epoch 98\nCv 0.9924\nLb 0.9762\nDiff 0.0162</p>\n\n<p>Epoch 84\nCv 0.9911\nLb 0.9759\nDiff 0.0152</p>\n\n<p>Epoch 79\nCv 0.9899\nLb 0.9750\nDiff 0.0149</p>\n\n<p>Ensemble epoch 84， 98， 109\nLb 0.9767</p>\n\n<p>Epoch 164\nCv 0.9937\nLb 0.9769\nDiff 0.0168</p>\n\n<p>Epoch 155\nCv 0.9934\nLb 0.9765\nDiff 0.0169</p>\n\n<p>Epoch 143\nCv 0.9928\nLb 0.9756\nDiff 0.0172</p>\n\n<p>Epoch 130\nCv 0.9928\nLb 0.9767\nDiff 0.161</p>\n\n<p>ensemble of epoch 164, 130, 109\nLb 0.9770</p>\n\n<p>Ensemble  (epoch 164)\n0.9779</p>\n\n<p>Epoch ～180\nCv 0.9940\nLb 0.9767\nDiff 0.0173</p>\n\n<p>Ensemble of  epoch 164, 130, 109\nLb 0.9779</p>\n\n<p>Seresnext50 autoaugment\nEpoch 93\nCv 0.9897\nLb 0.9734\nDiff 0.0163</p>\n\n<p>Epoch 85\nCv 0.9892\nLb 0.9742\nDiff 0.015</p>\n\n<p>Epoch 77\nCv 0.9872\nLb 0.9732\nDiff 0.014</p>\n\n<p>Epoch 65\nCv 0.9852\nLb 0.9709\nDiff 0.0143</p>\n\n<p>Ensemble: epoch 77 auto augment, epoch 164 no crop: 0.9784</p>\n\n<p>Epoch 139\nCv 0.9908\nLb 0.9741\nDiff 0.0167</p>\n\n<p>Epoch 122\nCv 0.9908\nLb 0.9737\nDiff 0.0171</p>\n\n<p>Epoch 108\nCv 0.9903\nLb 0.9735\nDiff 0.0168</p>\n\n<p>Densenet121 no cutmix</p>\n\n<p>Epoch 119\nCv 0.9862\nLb 0.9799\nDiff 0.0053</p>\n\n<p>Epoch 92\nCv 0.9861\nLb 0.9796\nDiff 0.0065</p>\n\n<p>Epoch 82\nCv 0.9859\nLb 0.9789\nDiff 0.007</p>\n\n<p>Ensemble of epoch 119, 92, 82\nLb 0.9799</p>\n\n<p>Seresnext 50 only cutmix</p>\n\n<p>Epoch 65\nCv 0.9947\nLb 0.9831\nDiff 0.0116</p>\n\n<p>Epoch 48\nCv 0.993\nLb 0.981\nDiff 0.012</p>\n\n<p>Epoch 120\nCv 0.9963\nLb 0.9849\nDiff 0.0114</p>\n\n<p>Epoch 118\nCv 0.9963\nLb 0.9849\nDiff 0.0114</p>\n\n<p>Epoch 100\nCv 0.996\nLb 0.9847\nDiff 0.0113</p>\n\n<p>seresnext 50 only cutmix Fold 0</p>\n\n<p>Epoch 112\nCv 0.9996\nLb 0.9862\nDiff 0.0134</p>\n\n<p>Epoch 91\nCv 0.9955\nLb  0.9854\nDiff 0.0101</p>\n\n<p>Epoch 81\nCv 0.9953\nLb 0.9851\nDiff 0.0102</p>\n\n<p>Ensemble of epoch 112, 91, shuffle120\nLb 0.9869</p>\n\n<p>Epoch 118\nCv 0.9996\nLb 0.9867\nDiff 0.0129</p>\n\n<p>Ensemble of epoch 112, 91, shuffle120, epoch 118\nLb 0.9869</p>\n\n<p>Ensemble of shuffle 120 epoch118 weighted 0.4 0.6\nLb 0.9873</p>\n\n<p>Weighted0.3, 0.7\nLb 0.9871</p>\n\n<p>Ensemble epoch 112, epoch 118 weighted 0.4 0.6\nLb 0.9865</p>\n\n<p>Res2net fold 0</p>\n\n<p>Epoch 77\nCv 0.9969\nLb 0.9858\nDiff 0.0111</p>\n\n<p>Epoch 67\nCv 0.9954\nLb 0.9853\nDiff 0.0101</p>\n\n<p>Epoch 53\nCv 0.9943\nLb 0.9829\nDiff 0.0114</p>\n\n<p>Ensemble weighted 0.4 0.6 r2n lb0.9858 seresnext_fold0 lb0.0.9867\nLb 0.9880</p>\n\n<p>res2net_fold1_new_fromepoch66_epochrange80(cv0.9996e85).pt</p>\n\n<p>Fold 1\nEpoch 85\nCv 0.9996\nDiff 0.9865</p>\n\n<p>Epoch 70\nCv 0.9991\nDiff 0.9858</p>\n\n<p>ensemble weighted 0.3 0.35 0.35 r2n lb0.9858 fold0 r2n lb0.9865 fold1 seresnext lb0.0.9867\n0.9881</p>\n\n<p>0.3 0.3 0.4 0.9884</p>",
  "messages": [
    {
      "id": "778052",
      "postDate": "03/18/2020 05:35:59",
      "content": "<p>Congratulation to all winners! I was definitely shocked by the shake and losing my first solo silver. I hope to share my solution as offering a comparison to winner solutions and looking forward to hear you about what might went wrong.</p>\n\n<p>I built my solution based on <a href=\"/corochann\">@corochann</a>'s kernel (<a href=\"https://www.kaggle.com/corochann/bengali-seresnext-prediction-with-pytorch\">https://www.kaggle.com/corochann/bengali-seresnext-prediction-with-pytorch</a>).</p>\n\n<p>I first tried around with seresnext50 and some basic augmentations, but the public LB score wandered under 0.967x. Then I stopped cropping images and everything became bright.</p>\n\n<p>Next, I experimented with different augmentation combinations and found cutmix only to work the best and sticked to it in different model testing process.</p>\n\n<p>Afterwards, I tested with models including seresnext50, seresnext101, densenet121, and res2net using 256x256 images.</p>\n\n<p>My final submissions consist of weighted ensembles of seresnext50 and res2net. The highest public LB score resulted is 0.9884 but with private score only 0.9290. </p>\n\n<p>I used 80/20 split for all training and never used all data for one single model. I'm not sure that is a problem. In the 0.9884 ensemble, I used 2 splits of res2net and 1 split of seresnext. I used stepLR with milestone of 75 and 90.</p>\n\n<p>Interestingly, I found that my highest private LB score is produced by densenet121. The ensemble of 3 different epochs of densenet121 (same split) resulted in public LB of 0.9799 but private LB of 0.9381. I also saw that efficientnet worked poorly on public LB but well on private LB in other people's solutions. Not sure if these simple model would have better performance on private LB. I also observed that densenet has a smaller cv/lb gap of around 0.5%, while seresnext and res2net all have cv/lb gap around 1.3%.</p>\n\n<p>I also found that local CV does not correlate with private LB nor public LB in my case. The densenet121 model never made its local CV above 0.99, while seresnext and res2net both achieved 0.995-6. So I guess \"trusting your CV\" is not a valid strategy for this competition.</p>\n\n<p>All in all, an unexpected competition experience. Definitely a bit sad losing a silver medal. Will try my best to get good score in the rest competitions before summer. </p>\n\n<p>Please comment below if you see any potential problem and upvote if you find this post interesting.</p>\n\n<p>Happy kaggling.</p>\n\n<p>Below is my full training/testing record:</p>\n\n<p>'Se_resnext50_32x4d_kernel_baseline_newlr' mixup/cutmix\nCv: 0.9868\nEpoch 102\nLB: 0.9646\ndiff : 0.0222</p>\n\n<p>Se_resnext50_32x4d_kernel_baseline_augmentation_randomseed666_8020split\nEpoch 35\nCv 0.9775\nLb 0.9641\nDiff = 0.0134</p>\n\n<p>Epoch 44\nCv 0.978\nLb 0.9646\nDff: 0.0134</p>\n\n<p>Epoch 84\nCv 0.9824\nLb 0.9654\nDiff = 0.017</p>\n\n<p>Milestone lr scheduler</p>\n\n<p>epoch 22\nCv 0.9783\nLb 0.9665\nDiff 0.0118</p>\n\n<p>Epoch 30\nCv 0.98\nLb 0.9674\nDiff 0.0126</p>\n\n<p>Epoch 42\nCv 0.9804\nLb 0.9678\nDiff 0.0126</p>\n\n<p>Epoch 53\nCv 0.9805\nLb 0.9676\nDiff 0.0129</p>\n\n<p>Milestone lr, ohem loss, stratified</p>\n\n<p>Epoch 17\nCv 0.9717\nLb 0.9527\nDiff 0.019</p>\n\n<p>Epoch 29\nCv 0.9739\nLb 0.9588\nDiff 0.015</p>\n\n<p>Epoch 39\nCv 0.9751\nLb 0.9584\nDiff 0.0167</p>\n\n<p>Multilabel stratified - cross entropy loss</p>\n\n<p>Epoch 87\nCv 0.9869\nLb 0.9679\nDiff 0.019</p>\n\n<p>Epoch 135\nCv 0.9886\nLb 0.9679\nDiff 0.0207</p>\n\n<p>Noconv0</p>\n\n<p>Epoch 60\nCv 0.9833\nLb 0.9667\nDiff 0.0166</p>\n\n<p>Epoch 76\nCv 0.9852\nLb 0.9670\nDff 0.0182</p>\n\n<p>Cnn - tail - noaffine\nEpoch 92\nCv 0.9892\nLb 0.9668\nDiff 0.0224</p>\n\n<p>Epoch 77\nCv 0.9872\nLb 0.9665\nDiff 0.0187</p>\n\n<p>Epoch 62\nCv 0.9849\nLb 0.9621\nDiff 0.0228</p>\n\n<p>Tail + affine\nEpoch 96\nCv 0.9866\nLb 0.9668\nDiff 0.0198</p>\n\n<p>Epoch 86\nCv 0.9858\nLb 0.9664\nDiff 0.0194</p>\n\n<p>Epoch 120\nCv 0.9876\nLB 0.9671\nDiff 0.0205</p>\n\n<p>Size 256\nEpoch 134\nCv 0.9896\nLb 0.9694\nDiff 0.0202</p>\n\n<p>Epoch 126\nCv 0.9896\nLb 0.9689\nDiff0.0207</p>\n\n<p>Epoch 114\nCv 0.989\nLb 0.9690\nDiff 0.020</p>\n\n<p>Epoch 107\nCv 0.9886\nLb 0.9688\nDiff 0.0198</p>\n\n<p>Epoch 98\nCv 0.9884\nLb 0.9691\nDiff 0.0193</p>\n\n<p>Epoch 48\nCv 0.981\nLb 0.9672\nDiff 0.0138</p>\n\n<p>Ensemble\nEpoch 134, 114, 98\nLb 0.9695</p>\n\n<p>Resnext 101 size 256\nEpoch 81 crop\nCv 0.9888\nLb 0.9680\nDiff 0.0208</p>\n\n<p>Epoch 81 no_crop\nCv 0.9888\nLb 0.9651\nDiff 0.0237</p>\n\n<p>Epoch 78 crop\nCv 0.9881\nLb 0.9688\nDiff 0.0193</p>\n\n<p>Epoch 65 crop\nCv 0.9851\nLb 0.9659\nDiff 0.0192</p>\n\n<p>Resnext50 size 256 no crop</p>\n\n<p>Epoch 109\nCv 0.9924\nLb 0.9766\nDiff 0.0158</p>\n\n<p>Epoch 98\nCv 0.9924\nLb 0.9762\nDiff 0.0162</p>\n\n<p>Epoch 84\nCv 0.9911\nLb 0.9759\nDiff 0.0152</p>\n\n<p>Epoch 79\nCv 0.9899\nLb 0.9750\nDiff 0.0149</p>\n\n<p>Ensemble epoch 84， 98， 109\nLb 0.9767</p>\n\n<p>Epoch 164\nCv 0.9937\nLb 0.9769\nDiff 0.0168</p>\n\n<p>Epoch 155\nCv 0.9934\nLb 0.9765\nDiff 0.0169</p>\n\n<p>Epoch 143\nCv 0.9928\nLb 0.9756\nDiff 0.0172</p>\n\n<p>Epoch 130\nCv 0.9928\nLb 0.9767\nDiff 0.161</p>\n\n<p>ensemble of epoch 164, 130, 109\nLb 0.9770</p>\n\n<p>Ensemble  (epoch 164)\n0.9779</p>\n\n<p>Epoch ～180\nCv 0.9940\nLb 0.9767\nDiff 0.0173</p>\n\n<p>Ensemble of  epoch 164, 130, 109\nLb 0.9779</p>\n\n<p>Seresnext50 autoaugment\nEpoch 93\nCv 0.9897\nLb 0.9734\nDiff 0.0163</p>\n\n<p>Epoch 85\nCv 0.9892\nLb 0.9742\nDiff 0.015</p>\n\n<p>Epoch 77\nCv 0.9872\nLb 0.9732\nDiff 0.014</p>\n\n<p>Epoch 65\nCv 0.9852\nLb 0.9709\nDiff 0.0143</p>\n\n<p>Ensemble: epoch 77 auto augment, epoch 164 no crop: 0.9784</p>\n\n<p>Epoch 139\nCv 0.9908\nLb 0.9741\nDiff 0.0167</p>\n\n<p>Epoch 122\nCv 0.9908\nLb 0.9737\nDiff 0.0171</p>\n\n<p>Epoch 108\nCv 0.9903\nLb 0.9735\nDiff 0.0168</p>\n\n<p>Densenet121 no cutmix</p>\n\n<p>Epoch 119\nCv 0.9862\nLb 0.9799\nDiff 0.0053</p>\n\n<p>Epoch 92\nCv 0.9861\nLb 0.9796\nDiff 0.0065</p>\n\n<p>Epoch 82\nCv 0.9859\nLb 0.9789\nDiff 0.007</p>\n\n<p>Ensemble of epoch 119, 92, 82\nLb 0.9799</p>\n\n<p>Seresnext 50 only cutmix</p>\n\n<p>Epoch 65\nCv 0.9947\nLb 0.9831\nDiff 0.0116</p>\n\n<p>Epoch 48\nCv 0.993\nLb 0.981\nDiff 0.012</p>\n\n<p>Epoch 120\nCv 0.9963\nLb 0.9849\nDiff 0.0114</p>\n\n<p>Epoch 118\nCv 0.9963\nLb 0.9849\nDiff 0.0114</p>\n\n<p>Epoch 100\nCv 0.996\nLb 0.9847\nDiff 0.0113</p>\n\n<p>seresnext 50 only cutmix Fold 0</p>\n\n<p>Epoch 112\nCv 0.9996\nLb 0.9862\nDiff 0.0134</p>\n\n<p>Epoch 91\nCv 0.9955\nLb  0.9854\nDiff 0.0101</p>\n\n<p>Epoch 81\nCv 0.9953\nLb 0.9851\nDiff 0.0102</p>\n\n<p>Ensemble of epoch 112, 91, shuffle120\nLb 0.9869</p>\n\n<p>Epoch 118\nCv 0.9996\nLb 0.9867\nDiff 0.0129</p>\n\n<p>Ensemble of epoch 112, 91, shuffle120, epoch 118\nLb 0.9869</p>\n\n<p>Ensemble of shuffle 120 epoch118 weighted 0.4 0.6\nLb 0.9873</p>\n\n<p>Weighted0.3, 0.7\nLb 0.9871</p>\n\n<p>Ensemble epoch 112, epoch 118 weighted 0.4 0.6\nLb 0.9865</p>\n\n<p>Res2net fold 0</p>\n\n<p>Epoch 77\nCv 0.9969\nLb 0.9858\nDiff 0.0111</p>\n\n<p>Epoch 67\nCv 0.9954\nLb 0.9853\nDiff 0.0101</p>\n\n<p>Epoch 53\nCv 0.9943\nLb 0.9829\nDiff 0.0114</p>\n\n<p>Ensemble weighted 0.4 0.6 r2n lb0.9858 seresnext_fold0 lb0.0.9867\nLb 0.9880</p>\n\n<p>res2net_fold1_new_fromepoch66_epochrange80(cv0.9996e85).pt</p>\n\n<p>Fold 1\nEpoch 85\nCv 0.9996\nDiff 0.9865</p>\n\n<p>Epoch 70\nCv 0.9991\nDiff 0.9858</p>\n\n<p>ensemble weighted 0.3 0.35 0.35 r2n lb0.9858 fold0 r2n lb0.9865 fold1 seresnext lb0.0.9867\n0.9881</p>\n\n<p>0.3 0.3 0.4 0.9884</p>",
      "rawMarkdown": "Congratulation to all winners! I was definitely shocked by the shake and losing my first solo silver. I hope to share my solution as offering a comparison to winner solutions and looking forward to hear you about what might went wrong.\n\nI built my solution based on @corochann's kernel (https://www.kaggle.com/corochann/bengali-seresnext-prediction-with-pytorch).\n\nI first tried around with seresnext50 and some basic augmentations, but the public LB score wandered under 0.967x. Then I stopped cropping images and everything became bright.\n\nNext, I experimented with different augmentation combinations and found cutmix only to work the best and sticked to it in different model testing process.\n\n Afterwards, I tested with models including seresnext50, seresnext101, densenet121, and res2net using 256x256 images.\n\nMy final submissions consist of weighted ensembles of seresnext50 and res2net. The highest public LB score resulted is 0.9884 but with private score only 0.9290. \n\nI used 80/20 split for all training and never used all data for one single model. I'm not sure that is a problem. In the 0.9884 ensemble, I used 2 splits of res2net and 1 split of seresnext. I used stepLR with milestone of 75 and 90.\n\nInterestingly, I found that my highest private LB score is produced by densenet121. The ensemble of 3 different epochs of densenet121 (same split) resulted in public LB of 0.9799 but private LB of 0.9381. I also saw that efficientnet worked poorly on public LB but well on private LB in other people's solutions. Not sure if these simple model would have better performance on private LB. I also observed that densenet has a smaller cv/lb gap of around 0.5%, while seresnext and res2net all have cv/lb gap around 1.3%.\n\nI also found that local CV does not correlate with private LB nor public LB in my case. The densenet121 model never made its local CV above 0.99, while seresnext and res2net both achieved 0.995-6. So I guess \"trusting your CV\" is not a valid strategy for this competition.\n\nAll in all, an unexpected competition experience. Definitely a bit sad losing a silver medal. Will try my best to get good score in the rest competitions before summer. \n\nPlease comment below if you see any potential problem and upvote if you find this post interesting.\n\nHappy kaggling.\n\nBelow is my full training/testing record:\n\n'Se_resnext50_32x4d_kernel_baseline_newlr' mixup/cutmix\nCv: 0.9868\nEpoch 102\nLB: 0.9646\ndiff : 0.0222\n\nSe_resnext50_32x4d_kernel_baseline_augmentation_randomseed666_8020split\nEpoch 35\nCv 0.9775\nLb 0.9641\nDiff = 0.0134\n\nEpoch 44\nCv 0.978\nLb 0.9646\nDff: 0.0134\n\nEpoch 84\nCv 0.9824\nLb 0.9654\nDiff = 0.017\n\n\nMilestone lr scheduler\n\nepoch 22\nCv 0.9783\nLb 0.9665\nDiff 0.0118\n\nEpoch 30\nCv 0.98\nLb 0.9674\nDiff 0.0126\n\nEpoch 42\nCv 0.9804\nLb 0.9678\nDiff 0.0126\n\nEpoch 53\nCv 0.9805\nLb 0.9676\nDiff 0.0129\n\nMilestone lr, ohem loss, stratified\n\nEpoch 17\nCv 0.9717\nLb 0.9527\nDiff 0.019\n\nEpoch 29\nCv 0.9739\nLb 0.9588\nDiff 0.015\n\nEpoch 39\nCv 0.9751\nLb 0.9584\nDiff 0.0167\n\nMultilabel stratified - cross entropy loss\n\nEpoch 87\nCv 0.9869\nLb 0.9679\nDiff 0.019\n\nEpoch 135\nCv 0.9886\nLb 0.9679\nDiff 0.0207\n \nNoconv0\n\nEpoch 60\nCv 0.9833\nLb 0.9667\nDiff 0.0166\n\nEpoch 76\nCv 0.9852\nLb 0.9670\nDff 0.0182\n\nCnn - tail - noaffine\nEpoch 92\nCv 0.9892\nLb 0.9668\nDiff 0.0224\n\nEpoch 77\nCv 0.9872\nLb 0.9665\nDiff 0.0187\n\nEpoch 62\nCv 0.9849\nLb 0.9621\nDiff 0.0228\n\nTail + affine\nEpoch 96\nCv 0.9866\nLb 0.9668\nDiff 0.0198\n\nEpoch 86\nCv 0.9858\nLb 0.9664\nDiff 0.0194\n\nEpoch 120\nCv 0.9876\nLB 0.9671\nDiff 0.0205\n\nSize 256\nEpoch 134\nCv 0.9896\nLb 0.9694\nDiff 0.0202\n\nEpoch 126\nCv 0.9896\nLb 0.9689\nDiff0.0207\n\nEpoch 114\nCv 0.989\nLb 0.9690\nDiff 0.020\n\nEpoch 107\nCv 0.9886\nLb 0.9688\nDiff 0.0198\n\nEpoch 98\nCv 0.9884\nLb 0.9691\nDiff 0.0193\n\nEpoch 48\nCv 0.981\nLb 0.9672\nDiff 0.0138\n\nEnsemble\nEpoch 134, 114, 98\nLb 0.9695\n\nResnext 101 size 256\nEpoch 81 crop\nCv 0.9888\nLb 0.9680\nDiff 0.0208\n\nEpoch 81 no_crop\nCv 0.9888\nLb 0.9651\nDiff 0.0237\n\nEpoch 78 crop\nCv 0.9881\nLb 0.9688\nDiff 0.0193\n\nEpoch 65 crop\nCv 0.9851\nLb 0.9659\nDiff 0.0192\n\nResnext50 size 256 no crop\n\nEpoch 109\nCv 0.9924\nLb 0.9766\nDiff 0.0158\n\nEpoch 98\nCv 0.9924\nLb 0.9762\nDiff 0.0162\n\nEpoch 84\nCv 0.9911\nLb 0.9759\nDiff 0.0152\n\nEpoch 79\nCv 0.9899\nLb 0.9750\nDiff 0.0149\n\nEnsemble epoch 84， 98， 109\nLb 0.9767\n\nEpoch 164\nCv 0.9937\nLb 0.9769\nDiff 0.0168\n\nEpoch 155\nCv 0.9934\nLb 0.9765\nDiff 0.0169\n\nEpoch 143\nCv 0.9928\nLb 0.9756\nDiff 0.0172\n\nEpoch 130\nCv 0.9928\nLb 0.9767\nDiff 0.161\n\nensemble of epoch 164, 130, 109\nLb 0.9770\n\nEnsemble  (epoch 164)\n0.9779\n\nEpoch ～180\nCv 0.9940\nLb 0.9767\nDiff 0.0173\n\n\nEnsemble of  epoch 164, 130, 109\nLb 0.9779\n\n\nSeresnext50 autoaugment\nEpoch 93\nCv 0.9897\nLb 0.9734\nDiff 0.0163\n\nEpoch 85\nCv 0.9892\nLb 0.9742\nDiff 0.015\n\nEpoch 77\nCv 0.9872\nLb 0.9732\nDiff 0.014\n\nEpoch 65\nCv 0.9852\nLb 0.9709\nDiff 0.0143\n\n\nEnsemble: epoch 77 auto augment, epoch 164 no crop: 0.9784\n\nEpoch 139\nCv 0.9908\nLb 0.9741\nDiff 0.0167\n\nEpoch 122\nCv 0.9908\nLb 0.9737\nDiff 0.0171\n\nEpoch 108\nCv 0.9903\nLb 0.9735\nDiff 0.0168\n\nDensenet121 no cutmix\n\nEpoch 119\nCv 0.9862\nLb 0.9799\nDiff 0.0053\n\nEpoch 92\nCv 0.9861\nLb 0.9796\nDiff 0.0065\n\nEpoch 82\nCv 0.9859\nLb 0.9789\nDiff 0.007\n\nEnsemble of epoch 119, 92, 82\nLb 0.9799\n\n\nSeresnext 50 only cutmix\n\nEpoch 65\nCv 0.9947\nLb 0.9831\nDiff 0.0116\n\nEpoch 48\nCv 0.993\nLb 0.981\nDiff 0.012\n\nEpoch 120\nCv 0.9963\nLb 0.9849\nDiff 0.0114\n\nEpoch 118\nCv 0.9963\nLb 0.9849\nDiff 0.0114\n\nEpoch 100\nCv 0.996\nLb 0.9847\nDiff 0.0113\n\n\nseresnext 50 only cutmix Fold 0\n\nEpoch 112\nCv 0.9996\nLb 0.9862\nDiff 0.0134\n\nEpoch 91\nCv 0.9955\nLb  0.9854\nDiff 0.0101\n\nEpoch 81\nCv 0.9953\nLb 0.9851\nDiff 0.0102\n\nEnsemble of epoch 112, 91, shuffle120\nLb 0.9869\n\nEpoch 118\nCv 0.9996\nLb 0.9867\nDiff 0.0129\n\nEnsemble of epoch 112, 91, shuffle120, epoch 118\nLb 0.9869\n\nEnsemble of shuffle 120 epoch118 weighted 0.4 0.6\nLb 0.9873\n\nWeighted0.3, 0.7\nLb 0.9871\n\nEnsemble epoch 112, epoch 118 weighted 0.4 0.6\nLb 0.9865\n\nRes2net fold 0\n\nEpoch 77\nCv 0.9969\nLb 0.9858\nDiff 0.0111\n\nEpoch 67\nCv 0.9954\nLb 0.9853\nDiff 0.0101\n\nEpoch 53\nCv 0.9943\nLb 0.9829\nDiff 0.0114\n\nEnsemble weighted 0.4 0.6 r2n lb0.9858 seresnext_fold0 lb0.0.9867\nLb 0.9880\n\nres2net_fold1_new_fromepoch66_epochrange80(cv0.9996e85).pt\n\nFold 1\nEpoch 85\nCv 0.9996\nDiff 0.9865\n\nEpoch 70\nCv 0.9991\nDiff 0.9858\n\nensemble weighted 0.3 0.35 0.35 r2n lb0.9858 fold0 r2n lb0.9865 fold1 seresnext lb0.0.9867\n0.9881\n\n0.3 0.3 0.4 0.9884",
      "votes": null
    },
    {
      "id": "778056",
      "postDate": "03/18/2020 05:41:57",
      "content": "<p>Thanks for sharing and utilizing the kernel :). </p>",
      "rawMarkdown": "Thanks for sharing and utilizing the kernel :).",
      "votes": null
    },
    {
      "id": "778060",
      "postDate": "03/18/2020 05:44:57",
      "content": "<p><a href=\"/corochann\">@corochann</a> Thank you for writing the kernel! It's really helpful!</p>",
      "rawMarkdown": "corochann Thank you for writing the kernel! It's really helpful!",
      "votes": null
    },
    {
      "id": "778064",
      "postDate": "03/18/2020 05:51:52",
      "content": "<p>\"trusting your CV\"  makes sense when you have a good setup where cv matches LB. Here you had the opposite. Next time you should try to understand where the discrepancy comes from and take that into account.</p>",
      "rawMarkdown": "\"trusting your CV\"  makes sense when you have a good setup where cv matches LB. Here you had the opposite. Next time you should try to understand where the discrepancy comes from and take that into account.",
      "votes": null
    },
    {
      "id": "778085",
      "postDate": "03/18/2020 06:17:15",
      "content": "<p><a href=\"/christofhenkel\">@christofhenkel</a> Thanks for pointing that out! I saw this in many other posts too, and many of them referred to R,C,V LB script. However, I still don't understand how that helps invetigaing the discrepancy. Could you explain that? Thank you very much!</p>",
      "rawMarkdown": "christofhenkel Thanks for pointing that out! I saw this in many other posts too, and many of them referred to R,C,V LB script. However, I still don't understand how that helps invetigaing the discrepancy. Could you explain that? Thank you very much!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 778056,
      "author_name": "corochann",
      "author_url": "",
      "post_date": "03/18/2020 05:41:57",
      "content": "<p>Thanks for sharing and utilizing the kernel :). </p>",
      "votes": null,
      "replies": [
        {
          "id": 778060,
          "author_name": "tonychenxyz",
          "author_url": "",
          "post_date": "03/18/2020 05:44:57",
          "content": "<p><a href=\"/corochann\">@corochann</a> Thank you for writing the kernel! It's really helpful!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 778064,
      "author_name": "christofhenkel",
      "author_url": "",
      "post_date": "03/18/2020 05:51:52",
      "content": "<p>\"trusting your CV\"  makes sense when you have a good setup where cv matches LB. Here you had the opposite. Next time you should try to understand where the discrepancy comes from and take that into account.</p>",
      "votes": null,
      "replies": [
        {
          "id": 778085,
          "author_name": "tonychenxyz",
          "author_url": "",
          "post_date": "03/18/2020 06:17:15",
          "content": "<p><a href=\"/christofhenkel\">@christofhenkel</a> Thanks for pointing that out! I saw this in many other posts too, and many of them referred to R,C,V LB script. However, I still don't understand how that helps invetigaing the discrepancy. Could you explain that? Thank you very much!</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "778052": "Congratulation to all winners! I was definitely shocked by the shake and losing my first solo silver. I hope to share my solution as offering a comparison to winner solutions and looking forward to hear you about what might went wrong.\n\nI built my solution based on @corochann's kernel (https://www.kaggle.com/corochann/bengali-seresnext-prediction-with-pytorch).\n\nI first tried around with seresnext50 and some basic augmentations, but the public LB score wandered under 0.967x. Then I stopped cropping images and everything became bright.\n\nNext, I experimented with different augmentation combinations and found cutmix only to work the best and sticked to it in different model testing process.\n\n Afterwards, I tested with models including seresnext50, seresnext101, densenet121, and res2net using 256x256 images.\n\nMy final submissions consist of weighted ensembles of seresnext50 and res2net. The highest public LB score resulted is 0.9884 but with private score only 0.9290. \n\nI used 80/20 split for all training and never used all data for one single model. I'm not sure that is a problem. In the 0.9884 ensemble, I used 2 splits of res2net and 1 split of seresnext. I used stepLR with milestone of 75 and 90.\n\nInterestingly, I found that my highest private LB score is produced by densenet121. The ensemble of 3 different epochs of densenet121 (same split) resulted in public LB of 0.9799 but private LB of 0.9381. I also saw that efficientnet worked poorly on public LB but well on private LB in other people's solutions. Not sure if these simple model would have better performance on private LB. I also observed that densenet has a smaller cv/lb gap of around 0.5%, while seresnext and res2net all have cv/lb gap around 1.3%.\n\nI also found that local CV does not correlate with private LB nor public LB in my case. The densenet121 model never made its local CV above 0.99, while seresnext and res2net both achieved 0.995-6. So I guess \"trusting your CV\" is not a valid strategy for this competition.\n\nAll in all, an unexpected competition experience. Definitely a bit sad losing a silver medal. Will try my best to get good score in the rest competitions before summer. \n\nPlease comment below if you see any potential problem and upvote if you find this post interesting.\n\nHappy kaggling.\n\nBelow is my full training/testing record:\n\n'Se_resnext50_32x4d_kernel_baseline_newlr' mixup/cutmix\nCv: 0.9868\nEpoch 102\nLB: 0.9646\ndiff : 0.0222\n\nSe_resnext50_32x4d_kernel_baseline_augmentation_randomseed666_8020split\nEpoch 35\nCv 0.9775\nLb 0.9641\nDiff = 0.0134\n\nEpoch 44\nCv 0.978\nLb 0.9646\nDff: 0.0134\n\nEpoch 84\nCv 0.9824\nLb 0.9654\nDiff = 0.017\n\n\nMilestone lr scheduler\n\nepoch 22\nCv 0.9783\nLb 0.9665\nDiff 0.0118\n\nEpoch 30\nCv 0.98\nLb 0.9674\nDiff 0.0126\n\nEpoch 42\nCv 0.9804\nLb 0.9678\nDiff 0.0126\n\nEpoch 53\nCv 0.9805\nLb 0.9676\nDiff 0.0129\n\nMilestone lr, ohem loss, stratified\n\nEpoch 17\nCv 0.9717\nLb 0.9527\nDiff 0.019\n\nEpoch 29\nCv 0.9739\nLb 0.9588\nDiff 0.015\n\nEpoch 39\nCv 0.9751\nLb 0.9584\nDiff 0.0167\n\nMultilabel stratified - cross entropy loss\n\nEpoch 87\nCv 0.9869\nLb 0.9679\nDiff 0.019\n\nEpoch 135\nCv 0.9886\nLb 0.9679\nDiff 0.0207\n \nNoconv0\n\nEpoch 60\nCv 0.9833\nLb 0.9667\nDiff 0.0166\n\nEpoch 76\nCv 0.9852\nLb 0.9670\nDff 0.0182\n\nCnn - tail - noaffine\nEpoch 92\nCv 0.9892\nLb 0.9668\nDiff 0.0224\n\nEpoch 77\nCv 0.9872\nLb 0.9665\nDiff 0.0187\n\nEpoch 62\nCv 0.9849\nLb 0.9621\nDiff 0.0228\n\nTail + affine\nEpoch 96\nCv 0.9866\nLb 0.9668\nDiff 0.0198\n\nEpoch 86\nCv 0.9858\nLb 0.9664\nDiff 0.0194\n\nEpoch 120\nCv 0.9876\nLB 0.9671\nDiff 0.0205\n\nSize 256\nEpoch 134\nCv 0.9896\nLb 0.9694\nDiff 0.0202\n\nEpoch 126\nCv 0.9896\nLb 0.9689\nDiff0.0207\n\nEpoch 114\nCv 0.989\nLb 0.9690\nDiff 0.020\n\nEpoch 107\nCv 0.9886\nLb 0.9688\nDiff 0.0198\n\nEpoch 98\nCv 0.9884\nLb 0.9691\nDiff 0.0193\n\nEpoch 48\nCv 0.981\nLb 0.9672\nDiff 0.0138\n\nEnsemble\nEpoch 134, 114, 98\nLb 0.9695\n\nResnext 101 size 256\nEpoch 81 crop\nCv 0.9888\nLb 0.9680\nDiff 0.0208\n\nEpoch 81 no_crop\nCv 0.9888\nLb 0.9651\nDiff 0.0237\n\nEpoch 78 crop\nCv 0.9881\nLb 0.9688\nDiff 0.0193\n\nEpoch 65 crop\nCv 0.9851\nLb 0.9659\nDiff 0.0192\n\nResnext50 size 256 no crop\n\nEpoch 109\nCv 0.9924\nLb 0.9766\nDiff 0.0158\n\nEpoch 98\nCv 0.9924\nLb 0.9762\nDiff 0.0162\n\nEpoch 84\nCv 0.9911\nLb 0.9759\nDiff 0.0152\n\nEpoch 79\nCv 0.9899\nLb 0.9750\nDiff 0.0149\n\nEnsemble epoch 84， 98， 109\nLb 0.9767\n\nEpoch 164\nCv 0.9937\nLb 0.9769\nDiff 0.0168\n\nEpoch 155\nCv 0.9934\nLb 0.9765\nDiff 0.0169\n\nEpoch 143\nCv 0.9928\nLb 0.9756\nDiff 0.0172\n\nEpoch 130\nCv 0.9928\nLb 0.9767\nDiff 0.161\n\nensemble of epoch 164, 130, 109\nLb 0.9770\n\nEnsemble  (epoch 164)\n0.9779\n\nEpoch ～180\nCv 0.9940\nLb 0.9767\nDiff 0.0173\n\n\nEnsemble of  epoch 164, 130, 109\nLb 0.9779\n\n\nSeresnext50 autoaugment\nEpoch 93\nCv 0.9897\nLb 0.9734\nDiff 0.0163\n\nEpoch 85\nCv 0.9892\nLb 0.9742\nDiff 0.015\n\nEpoch 77\nCv 0.9872\nLb 0.9732\nDiff 0.014\n\nEpoch 65\nCv 0.9852\nLb 0.9709\nDiff 0.0143\n\n\nEnsemble: epoch 77 auto augment, epoch 164 no crop: 0.9784\n\nEpoch 139\nCv 0.9908\nLb 0.9741\nDiff 0.0167\n\nEpoch 122\nCv 0.9908\nLb 0.9737\nDiff 0.0171\n\nEpoch 108\nCv 0.9903\nLb 0.9735\nDiff 0.0168\n\nDensenet121 no cutmix\n\nEpoch 119\nCv 0.9862\nLb 0.9799\nDiff 0.0053\n\nEpoch 92\nCv 0.9861\nLb 0.9796\nDiff 0.0065\n\nEpoch 82\nCv 0.9859\nLb 0.9789\nDiff 0.007\n\nEnsemble of epoch 119, 92, 82\nLb 0.9799\n\n\nSeresnext 50 only cutmix\n\nEpoch 65\nCv 0.9947\nLb 0.9831\nDiff 0.0116\n\nEpoch 48\nCv 0.993\nLb 0.981\nDiff 0.012\n\nEpoch 120\nCv 0.9963\nLb 0.9849\nDiff 0.0114\n\nEpoch 118\nCv 0.9963\nLb 0.9849\nDiff 0.0114\n\nEpoch 100\nCv 0.996\nLb 0.9847\nDiff 0.0113\n\n\nseresnext 50 only cutmix Fold 0\n\nEpoch 112\nCv 0.9996\nLb 0.9862\nDiff 0.0134\n\nEpoch 91\nCv 0.9955\nLb  0.9854\nDiff 0.0101\n\nEpoch 81\nCv 0.9953\nLb 0.9851\nDiff 0.0102\n\nEnsemble of epoch 112, 91, shuffle120\nLb 0.9869\n\nEpoch 118\nCv 0.9996\nLb 0.9867\nDiff 0.0129\n\nEnsemble of epoch 112, 91, shuffle120, epoch 118\nLb 0.9869\n\nEnsemble of shuffle 120 epoch118 weighted 0.4 0.6\nLb 0.9873\n\nWeighted0.3, 0.7\nLb 0.9871\n\nEnsemble epoch 112, epoch 118 weighted 0.4 0.6\nLb 0.9865\n\nRes2net fold 0\n\nEpoch 77\nCv 0.9969\nLb 0.9858\nDiff 0.0111\n\nEpoch 67\nCv 0.9954\nLb 0.9853\nDiff 0.0101\n\nEpoch 53\nCv 0.9943\nLb 0.9829\nDiff 0.0114\n\nEnsemble weighted 0.4 0.6 r2n lb0.9858 seresnext_fold0 lb0.0.9867\nLb 0.9880\n\nres2net_fold1_new_fromepoch66_epochrange80(cv0.9996e85).pt\n\nFold 1\nEpoch 85\nCv 0.9996\nDiff 0.9865\n\nEpoch 70\nCv 0.9991\nDiff 0.9858\n\nensemble weighted 0.3 0.35 0.35 r2n lb0.9858 fold0 r2n lb0.9865 fold1 seresnext lb0.0.9867\n0.9881\n\n0.3 0.3 0.4 0.9884",
    "778056": "Thanks for sharing and utilizing the kernel :).",
    "778060": "corochann Thank you for writing the kernel! It's really helpful!",
    "778064": "\"trusting your CV\"  makes sense when you have a good setup where cv matches LB. Here you had the opposite. Next time you should try to understand where the discrepancy comes from and take that into account.",
    "778085": "christofhenkel Thanks for pointing that out! I saw this in many other posts too, and many of them referred to R,C,V LB script. However, I still don't understand how that helps invetigaing the discrepancy. Could you explain that? Thank you very much!"
  },
  "source": "meta"
}