{
  "id": 175744,
  "title": "176th Place Write-up",
  "url": "/competitions/siim-isic-melanoma-classification/writeups/sai-ram-176th-place-write-up",
  "author_name": "",
  "post_date": "2020-08-19T11:40:27.417Z",
  "votes": 11,
  "comment_count": 4,
  "views": 0,
  "content": "<p>First of all, I would like to thank kaggle and the organizers for hosting such interesting competition and thank for my team <a href=\"https://www.kaggle.com/seshurajup\" target=\"_blank\">@seshurajup</a> <a href=\"https://www.kaggle.com/shravankoninti\" target=\"_blank\">@shravankoninti</a> <a href=\"https://www.kaggle.com/jakkulasasikiran\" target=\"_blank\">@jakkulasasikiran</a> <a href=\"https://www.kaggle.com/mounikabellamkonda\" target=\"_blank\">@mounikabellamkonda</a>. This was great learning and collaboration.</p>\n<p>special thanks to <a href=\"https://www.kaggle.com/shravankoninti\" target=\"_blank\">@shravankoninti</a> we worked hard for the last two months.</p>\n<h2>[Summary]</h2>\n<ul>\n<li>First, we trained a simple model using <strong>E1 256X256</strong> using 5Fold <strong>CV: 0.88</strong></li>\n<li>Then we analyze the predictions mostly <strong>FP</strong> and <strong>FN</strong></li>\n</ul>\n<p><strong>FP</strong> <code>Target = 1 and Predict &lt; 0.2</code><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2058044%2F6d599f0c9466d695b86a10bad076b821%2FFP.png?generation=1597827587126099&amp;alt=media\" alt=\"\"><br>\n<strong>FN</strong> <code>Target = 0 and Predict &gt; 0.8</code><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2058044%2Fc123c7520e87fcb326cb52a2cc3dd1c6%2FFN.png?generation=1597827662475852&amp;alt=media\" alt=\"\"></p>\n<ul>\n<li>we removed <strong>FP</strong> and <strong>FN</strong> base on the threshold </li>\n<li>trained the same model for training, we exclude <strong>FP</strong>  and <strong>FN</strong> cv increased to <strong>0.9012</strong></li>\n<li>using same <strong>E1</strong> model to find <strong>FP</strong> and <strong>FN</strong> on <strong>2017-2018-2019</strong> data</li>\n<li>we removed <strong>FP</strong> and <strong>FN</strong> on <strong>2017-2018-2019</strong>  </li>\n<li>same model trained again on <strong>2017-2018-2019-2020</strong> we get CV:0.92+ LB:0.95+ PLB:0.925+</li>\n</ul>\n<h2>[PyTorch]</h2>\n<p>for PyTorch setup, we are used DeepFake Winners solution scripts: <a href=\"https://github.com/selimsef/dfdc_deepfake_challenge\" target=\"_blank\">link</a> Big thanks to <a href=\"https://www.kaggle.com/selimsef\" target=\"_blank\">@selimsef</a></p>\n<ul>\n<li>For CV we are used <a href=\"https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords\" target=\"_blank\">Triple Stratified Leak-Free KFold CV</a> thanks to <a href=\"https://www.kaggle.com/cdotte\" target=\"_blank\">@cdotte</a> </li>\n<li>For training, we are used Pytorch 1.6 Native AMP. Notebook: <a href=\"https://www.kaggle.com/gopidurgaprasad/siim-pytorch-1-6-native-amp\" target=\"_blank\">link</a></li>\n<li>Optimizer: <code>Adam</code></li>\n<li>Schedule : <code>LearningRateScheduler</code> for <a href=\"https://www.kaggle.com/cdotte\" target=\"_blank\">@cdotte</a> notebook</li>\n<li>CUTMIX + MIXUP it working some time </li>\n<li>Sampling:  Upsample(US) + Downsample(DS) + Nosampling(NS) for same model at the end we take simpleaverg for those. <br>\n<code>sampler = BalanceClassSampler(labels=train_dataset.__get_labels__(), mode=\"downsampling\")</code><br>\nyou found WeightedClassSampler from this notebook : <a href=\"https://www.kaggle.com/gopidurgaprasad/pytorch-weightedclasssampler\" target=\"_blank\">link</a></li>\n<li>Loss: simply<code>BinaryCrossentropy</code></li>\n<li>Augmentations:</li>\n</ul>\n<pre><code>A.Compose([\n        A.OneOf([\n            A.VerticalFlip(),\n            A.HorizontalFlip(),\n            A.Flip(),\n            A.Rotate()\n        ], p=0.5),\n        A.Transpose(p=0.2),\n        A.ShiftScaleRotate(shift_limit=0.0625, scale_limit=0.1, rotate_limit=15),\n        A.RandomRotate90(p=0.1),\n        A.OneOf([\n            A.RandomGridShuffle(p=0.1),\n            A.Cutout(num_holes=8, max_h_size=size//8, max_w_size=size//8, fill_value=0, p=0.2),\n            A.CoarseDropout(max_holes=4, max_height=size//8, max_width=size//8, p=0.2),\n            A.GridDropout(p=0.2),\n            RandomEraser(p=0.2),\n            BitMask(size=size, p=0.1),\n        ], p=0.11),\n        A.RandomBrightness(limit=(-0.2,0.2), p=0.1),\n        A.OneOf([\n            IsotropicResize(max_side=size, interpolation_down=cv2.INTER_AREA, interpolation_up=cv2.INTER_CUBIC),\n            IsotropicResize(max_side=size, interpolation_down=cv2.INTER_AREA, interpolation_up=cv2.INTER_LINEAR),\n            IsotropicResize(max_side=size, interpolation_down=cv2.INTER_LINEAR, interpolation_up=cv2.INTER_LINEAR),\n        ], p=1),\n        A.PadIfNeeded(min_height=size, min_width=size, border_mode=cv2.BORDER_CONSTANT)\n</code></pre>\n<h4>Pytorch Models</h4>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>SIZE</th>\n<th>RESIZE</th>\n<th>SAMPLE</th>\n<th>FOLDS</th>\n<th>CV</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>E4</td>\n<td>512</td>\n<td>380</td>\n<td>DS</td>\n<td>5</td>\n<td>0.9175</td>\n</tr>\n<tr>\n<td>Sk50</td>\n<td>512</td>\n<td>512</td>\n<td>DS</td>\n<td>5</td>\n<td>0.9178</td>\n</tr>\n<tr>\n<td>E6</td>\n<td>768</td>\n<td>528</td>\n<td>DS</td>\n<td>10</td>\n<td>0.9266</td>\n</tr>\n<tr>\n<td>E5</td>\n<td>512</td>\n<td>456</td>\n<td>DS</td>\n<td>5</td>\n<td>0.9288</td>\n</tr>\n<tr>\n<td>E7</td>\n<td>512</td>\n<td>512</td>\n<td>DS</td>\n<td>5</td>\n<td>0.9224</td>\n</tr>\n<tr>\n<td>Dpn92</td>\n<td>512</td>\n<td>512</td>\n<td>DS</td>\n<td>5</td>\n<td>0.9218</td>\n</tr>\n<tr>\n<td>E5</td>\n<td>512</td>\n<td>456</td>\n<td>DS</td>\n<td>10</td>\n<td>0.9229</td>\n</tr>\n<tr>\n<td>D201</td>\n<td>512</td>\n<td>512</td>\n<td>DS</td>\n<td>5</td>\n<td>0.9266</td>\n</tr>\n<tr>\n<td>D201</td>\n<td>512</td>\n<td>512</td>\n<td>US</td>\n<td>5</td>\n<td>0.9240</td>\n</tr>\n<tr>\n<td>E6</td>\n<td>768</td>\n<td>528</td>\n<td>US</td>\n<td>5</td>\n<td>0.9203</td>\n</tr>\n<tr>\n<td>E4</td>\n<td>512</td>\n<td>380</td>\n<td>US</td>\n<td>10</td>\n<td>0.9242</td>\n</tr>\n<tr>\n<td>E5</td>\n<td>512</td>\n<td>456</td>\n<td>NS</td>\n<td>5</td>\n<td>0.9153</td>\n</tr>\n<tr>\n<td>D201</td>\n<td>512</td>\n<td>224</td>\n<td>NS</td>\n<td>5</td>\n<td>0.9185</td>\n</tr>\n<tr>\n<td>B5</td>\n<td>512</td>\n<td>456</td>\n<td>US</td>\n<td>10</td>\n<td>0.9234</td>\n</tr>\n<tr>\n<td>D201</td>\n<td>512</td>\n<td>224</td>\n<td>US</td>\n<td>5</td>\n<td>0.9068</td>\n</tr>\n</tbody>\n</table>\n<h4>SimpleAverage</h4>\n<ul>\n<li>simply average all models CV:<strong>0.9372</strong> LB:<strong>0.9544</strong> PLB<strong>0.9388</strong></li>\n</ul>\n<h4>OptimizeAUC</h4>\n<ul>\n<li>for optimizing OOF AUC we are using OptimizeAUC function from  <a href=\"https://www.amazon.in/dp/B089P13QHT/ref=dp-kindle-redirect?_encoding=UTF8&amp;btkr=1#:~:text=This%20book%20is%20for%20people,learning%20and%20deep%20learning%20problems.\" target=\"_blank\">Approaching (Almost) Any Machine Learning Problem</a> by <a href=\"https://www.kaggle.com/abhishek\" target=\"_blank\">@abhishek</a> Big thanks to him.</li>\n<li>you found the OptimizeAUC in this notebook: <a href=\"https://www.kaggle.com/gopidurgaprasad/optimize-auc-using-oof\" target=\"_blank\">link</a></li>\n<li>CV:<strong>0.94741</strong> LB:<strong>0.9492</strong> PLB<strong>0.9344</strong></li>\n</ul>\n<p><strong>Pytorch Training Notebook:</strong> <a href=\"https://www.kaggle.com/gopidurgaprasad/siim-final-d201-ns\" target=\"_blank\">link</a></p>\n<h2>[TF]</h2>\n<ul>\n<li>we are using <a href=\"https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords\" target=\"_blank\">Triple Stratified KFold with TFRecords</a> again Big thanks to <a href=\"https://www.kaggle.com/cdotte\" target=\"_blank\">@cdotte</a></li>\n<li>thanks to <a href=\"https://www.kaggle.com/shravankoninti\" target=\"_blank\">@shravankoninti</a> running those TF models one by one every data</li>\n</ul>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>SIZE</th>\n<th>FOLDS</th>\n<th>CV</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>E7</td>\n<td>512</td>\n<td>15</td>\n<td>0.8826</td>\n</tr>\n<tr>\n<td>E6</td>\n<td>512</td>\n<td>15</td>\n<td>0.9194</td>\n</tr>\n<tr>\n<td>E5</td>\n<td>512</td>\n<td>15</td>\n<td>0.9243</td>\n</tr>\n<tr>\n<td>E7</td>\n<td>512</td>\n<td>10</td>\n<td>0.9163</td>\n</tr>\n<tr>\n<td>E6</td>\n<td>512</td>\n<td>10</td>\n<td>0.9027</td>\n</tr>\n<tr>\n<td>E5</td>\n<td>512</td>\n<td>10</td>\n<td>0.9275</td>\n</tr>\n<tr>\n<td>E4</td>\n<td>512</td>\n<td>10</td>\n<td>0.9114</td>\n</tr>\n<tr>\n<td>E3</td>\n<td>512</td>\n<td>10</td>\n<td>0.9133</td>\n</tr>\n<tr>\n<td>E2</td>\n<td>512</td>\n<td>10</td>\n<td>0.9154</td>\n</tr>\n<tr>\n<td>E1</td>\n<td>512</td>\n<td>15</td>\n<td>0.9142</td>\n</tr>\n<tr>\n<td>E7</td>\n<td>768</td>\n<td>5</td>\n<td>0.9076</td>\n</tr>\n<tr>\n<td>E6</td>\n<td>768</td>\n<td>5</td>\n<td>0.9161</td>\n</tr>\n<tr>\n<td>E5</td>\n<td>768</td>\n<td>5</td>\n<td>0.9063</td>\n</tr>\n<tr>\n<td>E4</td>\n<td>768</td>\n<td>5</td>\n<td>0.9144</td>\n</tr>\n<tr>\n<td>E3</td>\n<td>768</td>\n<td>5</td>\n<td>0.9019</td>\n</tr>\n<tr>\n<td>E2</td>\n<td>768</td>\n<td>5</td>\n<td>0.8917</td>\n</tr>\n<tr>\n<td>E1</td>\n<td>768</td>\n<td>5</td>\n<td>0.8866</td>\n</tr>\n<tr>\n<td>E7</td>\n<td>384</td>\n<td>5</td>\n<td>0.9212</td>\n</tr>\n<tr>\n<td>E6</td>\n<td>384</td>\n<td>5</td>\n<td>0.9237</td>\n</tr>\n<tr>\n<td>E5</td>\n<td>384</td>\n<td>5</td>\n<td>0.9278</td>\n</tr>\n</tbody>\n</table>\n<h4>SimpleAverage</h4>\n<ul>\n<li>simply average all TF models CV:<strong>0.9301</strong> LB:<strong>0.9503</strong> PLB<strong>0.9376</strong></li>\n</ul>\n<h4>OptimizeAUC</h4>\n<ul>\n<li>CV:<strong>0.9446</strong> LB:<strong>0.9538</strong> PLB<strong>0.9348</strong></li>\n</ul>\n<h2>[PyTorch + TF]</h2>\n<h4>SimpleAverage</h4>\n<ul>\n<li>simply average all PyTorch and TF models CV:<strong>0.9402</strong> LB:<strong>0.9538</strong> PLB:<strong>0.9399</strong></li>\n<li>This is our <strong>best-selected submission</strong></li>\n</ul>\n<h4>OptimizeAUC</h4>\n<ul>\n<li>CV:<strong>0.949354</strong> LB:<strong>0.9496</strong> PLB:<strong>0.9362</strong></li>\n</ul>\n<h2>🎊 Congratulations  to all Winners and Learners 🎊</h2>",
  "messages": [
    {
      "id": "977045",
      "postDate": "08/19/2020 09:01:30",
      "content": "<p>First of all, I would like to thank kaggle and the organizers for hosting such interesting competition and thank for my team <a href=\"https://www.kaggle.com/seshurajup\" target=\"_blank\">@seshurajup</a> <a href=\"https://www.kaggle.com/shravankoninti\" target=\"_blank\">@shravankoninti</a> <a href=\"https://www.kaggle.com/jakkulasasikiran\" target=\"_blank\">@jakkulasasikiran</a> <a href=\"https://www.kaggle.com/mounikabellamkonda\" target=\"_blank\">@mounikabellamkonda</a>. This was great learning and collaboration.</p>\n<p>special thanks to <a href=\"https://www.kaggle.com/shravankoninti\" target=\"_blank\">@shravankoninti</a> we worked hard for the last two months.</p>\n<h2>[Summary]</h2>\n<ul>\n<li>First, we trained a simple model using <strong>E1 256X256</strong> using 5Fold <strong>CV: 0.88</strong></li>\n<li>Then we analyze the predictions mostly <strong>FP</strong> and <strong>FN</strong></li>\n</ul>\n<p><strong>FP</strong> <code>Target = 1 and Predict &lt; 0.2</code><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2058044%2F6d599f0c9466d695b86a10bad076b821%2FFP.png?generation=1597827587126099&amp;alt=media\" alt=\"\"><br>\n<strong>FN</strong> <code>Target = 0 and Predict &gt; 0.8</code><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2058044%2Fc123c7520e87fcb326cb52a2cc3dd1c6%2FFN.png?generation=1597827662475852&amp;alt=media\" alt=\"\"></p>\n<ul>\n<li>we removed <strong>FP</strong> and <strong>FN</strong> base on the threshold </li>\n<li>trained the same model for training, we exclude <strong>FP</strong>  and <strong>FN</strong> cv increased to <strong>0.9012</strong></li>\n<li>using same <strong>E1</strong> model to find <strong>FP</strong> and <strong>FN</strong> on <strong>2017-2018-2019</strong> data</li>\n<li>we removed <strong>FP</strong> and <strong>FN</strong> on <strong>2017-2018-2019</strong>  </li>\n<li>same model trained again on <strong>2017-2018-2019-2020</strong> we get CV:0.92+ LB:0.95+ PLB:0.925+</li>\n</ul>\n<h2>[PyTorch]</h2>\n<p>for PyTorch setup, we are used DeepFake Winners solution scripts: <a href=\"https://github.com/selimsef/dfdc_deepfake_challenge\" target=\"_blank\">link</a> Big thanks to <a href=\"https://www.kaggle.com/selimsef\" target=\"_blank\">@selimsef</a></p>\n<ul>\n<li>For CV we are used <a href=\"https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords\" target=\"_blank\">Triple Stratified Leak-Free KFold CV</a> thanks to <a href=\"https://www.kaggle.com/cdotte\" target=\"_blank\">@cdotte</a> </li>\n<li>For training, we are used Pytorch 1.6 Native AMP. Notebook: <a href=\"https://www.kaggle.com/gopidurgaprasad/siim-pytorch-1-6-native-amp\" target=\"_blank\">link</a></li>\n<li>Optimizer: <code>Adam</code></li>\n<li>Schedule : <code>LearningRateScheduler</code> for <a href=\"https://www.kaggle.com/cdotte\" target=\"_blank\">@cdotte</a> notebook</li>\n<li>CUTMIX + MIXUP it working some time </li>\n<li>Sampling:  Upsample(US) + Downsample(DS) + Nosampling(NS) for same model at the end we take simpleaverg for those. <br>\n<code>sampler = BalanceClassSampler(labels=train_dataset.__get_labels__(), mode=\"downsampling\")</code><br>\nyou found WeightedClassSampler from this notebook : <a href=\"https://www.kaggle.com/gopidurgaprasad/pytorch-weightedclasssampler\" target=\"_blank\">link</a></li>\n<li>Loss: simply<code>BinaryCrossentropy</code></li>\n<li>Augmentations:</li>\n</ul>\n<pre><code>A.Compose([\n        A.OneOf([\n            A.VerticalFlip(),\n            A.HorizontalFlip(),\n            A.Flip(),\n            A.Rotate()\n        ], p=0.5),\n        A.Transpose(p=0.2),\n        A.ShiftScaleRotate(shift_limit=0.0625, scale_limit=0.1, rotate_limit=15),\n        A.RandomRotate90(p=0.1),\n        A.OneOf([\n            A.RandomGridShuffle(p=0.1),\n            A.Cutout(num_holes=8, max_h_size=size//8, max_w_size=size//8, fill_value=0, p=0.2),\n            A.CoarseDropout(max_holes=4, max_height=size//8, max_width=size//8, p=0.2),\n            A.GridDropout(p=0.2),\n            RandomEraser(p=0.2),\n            BitMask(size=size, p=0.1),\n        ], p=0.11),\n        A.RandomBrightness(limit=(-0.2,0.2), p=0.1),\n        A.OneOf([\n            IsotropicResize(max_side=size, interpolation_down=cv2.INTER_AREA, interpolation_up=cv2.INTER_CUBIC),\n            IsotropicResize(max_side=size, interpolation_down=cv2.INTER_AREA, interpolation_up=cv2.INTER_LINEAR),\n            IsotropicResize(max_side=size, interpolation_down=cv2.INTER_LINEAR, interpolation_up=cv2.INTER_LINEAR),\n        ], p=1),\n        A.PadIfNeeded(min_height=size, min_width=size, border_mode=cv2.BORDER_CONSTANT)\n</code></pre>\n<h4>Pytorch Models</h4>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>SIZE</th>\n<th>RESIZE</th>\n<th>SAMPLE</th>\n<th>FOLDS</th>\n<th>CV</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>E4</td>\n<td>512</td>\n<td>380</td>\n<td>DS</td>\n<td>5</td>\n<td>0.9175</td>\n</tr>\n<tr>\n<td>Sk50</td>\n<td>512</td>\n<td>512</td>\n<td>DS</td>\n<td>5</td>\n<td>0.9178</td>\n</tr>\n<tr>\n<td>E6</td>\n<td>768</td>\n<td>528</td>\n<td>DS</td>\n<td>10</td>\n<td>0.9266</td>\n</tr>\n<tr>\n<td>E5</td>\n<td>512</td>\n<td>456</td>\n<td>DS</td>\n<td>5</td>\n<td>0.9288</td>\n</tr>\n<tr>\n<td>E7</td>\n<td>512</td>\n<td>512</td>\n<td>DS</td>\n<td>5</td>\n<td>0.9224</td>\n</tr>\n<tr>\n<td>Dpn92</td>\n<td>512</td>\n<td>512</td>\n<td>DS</td>\n<td>5</td>\n<td>0.9218</td>\n</tr>\n<tr>\n<td>E5</td>\n<td>512</td>\n<td>456</td>\n<td>DS</td>\n<td>10</td>\n<td>0.9229</td>\n</tr>\n<tr>\n<td>D201</td>\n<td>512</td>\n<td>512</td>\n<td>DS</td>\n<td>5</td>\n<td>0.9266</td>\n</tr>\n<tr>\n<td>D201</td>\n<td>512</td>\n<td>512</td>\n<td>US</td>\n<td>5</td>\n<td>0.9240</td>\n</tr>\n<tr>\n<td>E6</td>\n<td>768</td>\n<td>528</td>\n<td>US</td>\n<td>5</td>\n<td>0.9203</td>\n</tr>\n<tr>\n<td>E4</td>\n<td>512</td>\n<td>380</td>\n<td>US</td>\n<td>10</td>\n<td>0.9242</td>\n</tr>\n<tr>\n<td>E5</td>\n<td>512</td>\n<td>456</td>\n<td>NS</td>\n<td>5</td>\n<td>0.9153</td>\n</tr>\n<tr>\n<td>D201</td>\n<td>512</td>\n<td>224</td>\n<td>NS</td>\n<td>5</td>\n<td>0.9185</td>\n</tr>\n<tr>\n<td>B5</td>\n<td>512</td>\n<td>456</td>\n<td>US</td>\n<td>10</td>\n<td>0.9234</td>\n</tr>\n<tr>\n<td>D201</td>\n<td>512</td>\n<td>224</td>\n<td>US</td>\n<td>5</td>\n<td>0.9068</td>\n</tr>\n</tbody>\n</table>\n<h4>SimpleAverage</h4>\n<ul>\n<li>simply average all models CV:<strong>0.9372</strong> LB:<strong>0.9544</strong> PLB<strong>0.9388</strong></li>\n</ul>\n<h4>OptimizeAUC</h4>\n<ul>\n<li>for optimizing OOF AUC we are using OptimizeAUC function from  <a href=\"https://www.amazon.in/dp/B089P13QHT/ref=dp-kindle-redirect?_encoding=UTF8&amp;btkr=1#:~:text=This%20book%20is%20for%20people,learning%20and%20deep%20learning%20problems.\" target=\"_blank\">Approaching (Almost) Any Machine Learning Problem</a> by <a href=\"https://www.kaggle.com/abhishek\" target=\"_blank\">@abhishek</a> Big thanks to him.</li>\n<li>you found the OptimizeAUC in this notebook: <a href=\"https://www.kaggle.com/gopidurgaprasad/optimize-auc-using-oof\" target=\"_blank\">link</a></li>\n<li>CV:<strong>0.94741</strong> LB:<strong>0.9492</strong> PLB<strong>0.9344</strong></li>\n</ul>\n<p><strong>Pytorch Training Notebook:</strong> <a href=\"https://www.kaggle.com/gopidurgaprasad/siim-final-d201-ns\" target=\"_blank\">link</a></p>\n<h2>[TF]</h2>\n<ul>\n<li>we are using <a href=\"https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords\" target=\"_blank\">Triple Stratified KFold with TFRecords</a> again Big thanks to <a href=\"https://www.kaggle.com/cdotte\" target=\"_blank\">@cdotte</a></li>\n<li>thanks to <a href=\"https://www.kaggle.com/shravankoninti\" target=\"_blank\">@shravankoninti</a> running those TF models one by one every data</li>\n</ul>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>SIZE</th>\n<th>FOLDS</th>\n<th>CV</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>E7</td>\n<td>512</td>\n<td>15</td>\n<td>0.8826</td>\n</tr>\n<tr>\n<td>E6</td>\n<td>512</td>\n<td>15</td>\n<td>0.9194</td>\n</tr>\n<tr>\n<td>E5</td>\n<td>512</td>\n<td>15</td>\n<td>0.9243</td>\n</tr>\n<tr>\n<td>E7</td>\n<td>512</td>\n<td>10</td>\n<td>0.9163</td>\n</tr>\n<tr>\n<td>E6</td>\n<td>512</td>\n<td>10</td>\n<td>0.9027</td>\n</tr>\n<tr>\n<td>E5</td>\n<td>512</td>\n<td>10</td>\n<td>0.9275</td>\n</tr>\n<tr>\n<td>E4</td>\n<td>512</td>\n<td>10</td>\n<td>0.9114</td>\n</tr>\n<tr>\n<td>E3</td>\n<td>512</td>\n<td>10</td>\n<td>0.9133</td>\n</tr>\n<tr>\n<td>E2</td>\n<td>512</td>\n<td>10</td>\n<td>0.9154</td>\n</tr>\n<tr>\n<td>E1</td>\n<td>512</td>\n<td>15</td>\n<td>0.9142</td>\n</tr>\n<tr>\n<td>E7</td>\n<td>768</td>\n<td>5</td>\n<td>0.9076</td>\n</tr>\n<tr>\n<td>E6</td>\n<td>768</td>\n<td>5</td>\n<td>0.9161</td>\n</tr>\n<tr>\n<td>E5</td>\n<td>768</td>\n<td>5</td>\n<td>0.9063</td>\n</tr>\n<tr>\n<td>E4</td>\n<td>768</td>\n<td>5</td>\n<td>0.9144</td>\n</tr>\n<tr>\n<td>E3</td>\n<td>768</td>\n<td>5</td>\n<td>0.9019</td>\n</tr>\n<tr>\n<td>E2</td>\n<td>768</td>\n<td>5</td>\n<td>0.8917</td>\n</tr>\n<tr>\n<td>E1</td>\n<td>768</td>\n<td>5</td>\n<td>0.8866</td>\n</tr>\n<tr>\n<td>E7</td>\n<td>384</td>\n<td>5</td>\n<td>0.9212</td>\n</tr>\n<tr>\n<td>E6</td>\n<td>384</td>\n<td>5</td>\n<td>0.9237</td>\n</tr>\n<tr>\n<td>E5</td>\n<td>384</td>\n<td>5</td>\n<td>0.9278</td>\n</tr>\n</tbody>\n</table>\n<h4>SimpleAverage</h4>\n<ul>\n<li>simply average all TF models CV:<strong>0.9301</strong> LB:<strong>0.9503</strong> PLB<strong>0.9376</strong></li>\n</ul>\n<h4>OptimizeAUC</h4>\n<ul>\n<li>CV:<strong>0.9446</strong> LB:<strong>0.9538</strong> PLB<strong>0.9348</strong></li>\n</ul>\n<h2>[PyTorch + TF]</h2>\n<h4>SimpleAverage</h4>\n<ul>\n<li>simply average all PyTorch and TF models CV:<strong>0.9402</strong> LB:<strong>0.9538</strong> PLB:<strong>0.9399</strong></li>\n<li>This is our <strong>best-selected submission</strong></li>\n</ul>\n<h4>OptimizeAUC</h4>\n<ul>\n<li>CV:<strong>0.949354</strong> LB:<strong>0.9496</strong> PLB:<strong>0.9362</strong></li>\n</ul>\n<h2>🎊 Congratulations  to all Winners and Learners 🎊</h2>",
      "rawMarkdown": "First of all, I would like to thank kaggle and the organizers for hosting such interesting competition and thank for my team @seshurajup @shravankoninti @jakkulasasikiran @mounikabellamkonda. This was great learning and collaboration.\n\nspecial thanks to @shravankoninti we worked hard for the last two months.\n\n## [Summary]\n- First, we trained a simple model using **E1 256X256** using 5Fold **CV: 0.88**\n- Then we analyze the predictions mostly **FP** and **FN**\n\n**FP** `Target = 1 and Predict < 0.2`\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2058044%2F6d599f0c9466d695b86a10bad076b821%2FFP.png?generation=1597827587126099&alt=media)\n**FN** `Target = 0 and Predict > 0.8`\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2058044%2Fc123c7520e87fcb326cb52a2cc3dd1c6%2FFN.png?generation=1597827662475852&alt=media)\n\n- we removed **FP** and **FN** base on the threshold \n- trained the same model for training, we exclude **FP**  and **FN** cv increased to **0.9012**\n- using same **E1** model to find **FP** and **FN** on **2017-2018-2019** data\n- we removed **FP** and **FN** on **2017-2018-2019**  \n- same model trained again on **2017-2018-2019-2020** we get CV:0.92+ LB:0.95+ PLB:0.925+\n\n## [PyTorch]\n\nfor PyTorch setup, we are used DeepFake Winners solution scripts: [link](https://github.com/selimsef/dfdc_deepfake_challenge) Big thanks to @selimsef\n\n- For CV we are used [Triple Stratified Leak-Free KFold CV](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords) thanks to @cdotte \n- For training, we are used Pytorch 1.6 Native AMP. Notebook: [link](https://www.kaggle.com/gopidurgaprasad/siim-pytorch-1-6-native-amp)\n- Optimizer: `Adam`\n- Schedule : `LearningRateScheduler` for @cdotte notebook\n- CUTMIX + MIXUP it working some time \n- Sampling:  Upsample(US) + Downsample(DS) + Nosampling(NS) for same model at the end we take simpleaverg for those. \n `sampler = BalanceClassSampler(labels=train_dataset.__get_labels__(), mode=\"downsampling\")`\n you found WeightedClassSampler from this notebook : [link](https://www.kaggle.com/gopidurgaprasad/pytorch-weightedclasssampler)\n- Loss: simply`BinaryCrossentropy`\n- Augmentations:\n```\nA.Compose([\n        A.OneOf([\n            A.VerticalFlip(),\n            A.HorizontalFlip(),\n            A.Flip(),\n            A.Rotate()\n        ], p=0.5),\n        A.Transpose(p=0.2),\n        A.ShiftScaleRotate(shift_limit=0.0625, scale_limit=0.1, rotate_limit=15),\n        A.RandomRotate90(p=0.1),\n        A.OneOf([\n            A.RandomGridShuffle(p=0.1),\n            A.Cutout(num_holes=8, max_h_size=size//8, max_w_size=size//8, fill_value=0, p=0.2),\n            A.CoarseDropout(max_holes=4, max_height=size//8, max_width=size//8, p=0.2),\n            A.GridDropout(p=0.2),\n            RandomEraser(p=0.2),\n            BitMask(size=size, p=0.1),\n        ], p=0.11),\n        A.RandomBrightness(limit=(-0.2,0.2), p=0.1),\n        A.OneOf([\n            IsotropicResize(max_side=size, interpolation_down=cv2.INTER_AREA, interpolation_up=cv2.INTER_CUBIC),\n            IsotropicResize(max_side=size, interpolation_down=cv2.INTER_AREA, interpolation_up=cv2.INTER_LINEAR),\n            IsotropicResize(max_side=size, interpolation_down=cv2.INTER_LINEAR, interpolation_up=cv2.INTER_LINEAR),\n        ], p=1),\n        A.PadIfNeeded(min_height=size, min_width=size, border_mode=cv2.BORDER_CONSTANT)\n```\n#### Pytorch Models\n| Model | SIZE |RESIZE|SAMPLE|FOLDS|CV|\n| --- | --- |---|---|---|\n| E4| 512 | 380 |DS|5|0.9175|\n| Sk50| 512 | 512 |DS|5|0.9178|\n| E6| 768 | 528 |DS|10|0.9266|\n| E5| 512 | 456 |DS|5|0.9288|\n| E7| 512 | 512 |DS|5|0.9224|\n| Dpn92| 512 | 512 |DS|5|0.9218|\n| E5| 512 | 456 |DS|10|0.9229|\n| D201| 512 | 512 |DS|5|0.9266|\n| D201| 512 | 512 |US|5|0.9240|\n| E6| 768 | 528 |US|5|0.9203|\n| E4| 512 | 380 |US|10|0.9242|\n| E5| 512 | 456 |NS|5|0.9153|\n| D201| 512 | 224 |NS|5|0.9185|\n| B5| 512 | 456 |US|10|0.9234|\n| D201| 512 | 224 |US|5|0.9068|\n\n#### SimpleAverage\n- simply average all models CV:**0.9372** LB:**0.9544** PLB**0.9388**\n\n#### OptimizeAUC\n- for optimizing OOF AUC we are using OptimizeAUC function from  [Approaching (Almost) Any Machine Learning Problem](https://www.amazon.in/dp/B089P13QHT/ref=dp-kindle-redirect?_encoding=UTF8&btkr=1#:~:text=This%20book%20is%20for%20people,learning%20and%20deep%20learning%20problems.) by @abhishek Big thanks to him.\n- you found the OptimizeAUC in this notebook: [link](https://www.kaggle.com/gopidurgaprasad/optimize-auc-using-oof)\n- CV:**0.94741** LB:**0.9492** PLB**0.9344**\n\n**Pytorch Training Notebook:** [link](https://www.kaggle.com/gopidurgaprasad/siim-final-d201-ns)\n\n\n## [TF]\n\n- we are using [Triple Stratified KFold with TFRecords](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords) again Big thanks to @cdotte\n- thanks to @shravankoninti running those TF models one by one every data\n\n| Model | SIZE |FOLDS|CV|\n| --- | --- |---|---|\n| E7| 512 |15|0.8826|\n| E6| 512 |15|0.9194|\n| E5| 512 |15|0.9243|\n| E7| 512 |10|0.9163|\n| E6| 512 |10|0.9027|\n| E5| 512 |10|0.9275|\n| E4| 512 |10|0.9114|\n| E3| 512 |10|0.9133|\n| E2| 512 |10|0.9154|\n| E1| 512 |15|0.9142|\n| E7| 768 |5|0.9076|\n| E6| 768 |5|0.9161|\n| E5| 768 |5|0.9063|\n| E4| 768 |5|0.9144|\n| E3| 768 |5|0.9019|\n| E2| 768 |5|0.8917|\n| E1| 768 |5|0.8866|\n| E7| 384|5|0.9212|\n| E6| 384|5|0.9237|\n| E5| 384|5|0.9278|\n\n#### SimpleAverage\n- simply average all TF models CV:**0.9301** LB:**0.9503** PLB**0.9376**\n\n#### OptimizeAUC\n- CV:**0.9446** LB:**0.9538** PLB**0.9348**\n\n## [PyTorch + TF]\n\n#### SimpleAverage\n- simply average all PyTorch and TF models CV:**0.9402** LB:**0.9538** PLB:**0.9399**\n- This is our **best-selected submission**\n\n#### OptimizeAUC\n- CV:**0.949354** LB:**0.9496** PLB:**0.9362**\n\n\n##🎊 Congratulations  to all Winners and Learners 🎊",
      "votes": null
    },
    {
      "id": "977247",
      "postDate": "08/19/2020 11:24:18",
      "content": "<p>Good explanation and useful link!<br>\nThanks. <a href=\"https://www.kaggle.com/gopidurgaprasad\" target=\"_blank\">@gopidurgaprasad</a> </p>",
      "rawMarkdown": "Good explanation and useful link!\nThanks. @gopidurgaprasad",
      "votes": null
    },
    {
      "id": "977276",
      "postDate": "08/19/2020 11:41:28",
      "content": "<p>Thank you 😍</p>",
      "rawMarkdown": "Thank you 😍",
      "votes": null
    },
    {
      "id": "977416",
      "postDate": "08/19/2020 13:13:20",
      "content": "<p>Congrats Brother for your medal! Thanks for a great explanation and for sharing the notebook.</p>",
      "rawMarkdown": "Congrats Brother for your medal! Thanks for a great explanation and for sharing the notebook.",
      "votes": null
    },
    {
      "id": "977421",
      "postDate": "08/19/2020 13:16:08",
      "content": "<p>Thanks brother 😍 keep rocking </p>",
      "rawMarkdown": "Thanks brother 😍 keep rocking",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 977247,
      "author_name": "piantic",
      "author_url": "",
      "post_date": "08/19/2020 11:24:18",
      "content": "<p>Good explanation and useful link!<br>\nThanks. <a href=\"https://www.kaggle.com/gopidurgaprasad\" target=\"_blank\">@gopidurgaprasad</a> </p>",
      "votes": null,
      "replies": [
        {
          "id": 977276,
          "author_name": "gopidurgaprasad",
          "author_url": "",
          "post_date": "08/19/2020 11:41:28",
          "content": "<p>Thank you 😍</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 977416,
      "author_name": "aakashveera",
      "author_url": "",
      "post_date": "08/19/2020 13:13:20",
      "content": "<p>Congrats Brother for your medal! Thanks for a great explanation and for sharing the notebook.</p>",
      "votes": null,
      "replies": [
        {
          "id": 977421,
          "author_name": "gopidurgaprasad",
          "author_url": "",
          "post_date": "08/19/2020 13:16:08",
          "content": "<p>Thanks brother 😍 keep rocking </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "977045": "First of all, I would like to thank kaggle and the organizers for hosting such interesting competition and thank for my team @seshurajup @shravankoninti @jakkulasasikiran @mounikabellamkonda. This was great learning and collaboration.\n\nspecial thanks to @shravankoninti we worked hard for the last two months.\n\n## [Summary]\n- First, we trained a simple model using **E1 256X256** using 5Fold **CV: 0.88**\n- Then we analyze the predictions mostly **FP** and **FN**\n\n**FP** `Target = 1 and Predict < 0.2`\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2058044%2F6d599f0c9466d695b86a10bad076b821%2FFP.png?generation=1597827587126099&alt=media)\n**FN** `Target = 0 and Predict > 0.8`\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2058044%2Fc123c7520e87fcb326cb52a2cc3dd1c6%2FFN.png?generation=1597827662475852&alt=media)\n\n- we removed **FP** and **FN** base on the threshold \n- trained the same model for training, we exclude **FP**  and **FN** cv increased to **0.9012**\n- using same **E1** model to find **FP** and **FN** on **2017-2018-2019** data\n- we removed **FP** and **FN** on **2017-2018-2019**  \n- same model trained again on **2017-2018-2019-2020** we get CV:0.92+ LB:0.95+ PLB:0.925+\n\n## [PyTorch]\n\nfor PyTorch setup, we are used DeepFake Winners solution scripts: [link](https://github.com/selimsef/dfdc_deepfake_challenge) Big thanks to @selimsef\n\n- For CV we are used [Triple Stratified Leak-Free KFold CV](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords) thanks to @cdotte \n- For training, we are used Pytorch 1.6 Native AMP. Notebook: [link](https://www.kaggle.com/gopidurgaprasad/siim-pytorch-1-6-native-amp)\n- Optimizer: `Adam`\n- Schedule : `LearningRateScheduler` for @cdotte notebook\n- CUTMIX + MIXUP it working some time \n- Sampling:  Upsample(US) + Downsample(DS) + Nosampling(NS) for same model at the end we take simpleaverg for those. \n `sampler = BalanceClassSampler(labels=train_dataset.__get_labels__(), mode=\"downsampling\")`\n you found WeightedClassSampler from this notebook : [link](https://www.kaggle.com/gopidurgaprasad/pytorch-weightedclasssampler)\n- Loss: simply`BinaryCrossentropy`\n- Augmentations:\n```\nA.Compose([\n        A.OneOf([\n            A.VerticalFlip(),\n            A.HorizontalFlip(),\n            A.Flip(),\n            A.Rotate()\n        ], p=0.5),\n        A.Transpose(p=0.2),\n        A.ShiftScaleRotate(shift_limit=0.0625, scale_limit=0.1, rotate_limit=15),\n        A.RandomRotate90(p=0.1),\n        A.OneOf([\n            A.RandomGridShuffle(p=0.1),\n            A.Cutout(num_holes=8, max_h_size=size//8, max_w_size=size//8, fill_value=0, p=0.2),\n            A.CoarseDropout(max_holes=4, max_height=size//8, max_width=size//8, p=0.2),\n            A.GridDropout(p=0.2),\n            RandomEraser(p=0.2),\n            BitMask(size=size, p=0.1),\n        ], p=0.11),\n        A.RandomBrightness(limit=(-0.2,0.2), p=0.1),\n        A.OneOf([\n            IsotropicResize(max_side=size, interpolation_down=cv2.INTER_AREA, interpolation_up=cv2.INTER_CUBIC),\n            IsotropicResize(max_side=size, interpolation_down=cv2.INTER_AREA, interpolation_up=cv2.INTER_LINEAR),\n            IsotropicResize(max_side=size, interpolation_down=cv2.INTER_LINEAR, interpolation_up=cv2.INTER_LINEAR),\n        ], p=1),\n        A.PadIfNeeded(min_height=size, min_width=size, border_mode=cv2.BORDER_CONSTANT)\n```\n#### Pytorch Models\n| Model | SIZE |RESIZE|SAMPLE|FOLDS|CV|\n| --- | --- |---|---|---|\n| E4| 512 | 380 |DS|5|0.9175|\n| Sk50| 512 | 512 |DS|5|0.9178|\n| E6| 768 | 528 |DS|10|0.9266|\n| E5| 512 | 456 |DS|5|0.9288|\n| E7| 512 | 512 |DS|5|0.9224|\n| Dpn92| 512 | 512 |DS|5|0.9218|\n| E5| 512 | 456 |DS|10|0.9229|\n| D201| 512 | 512 |DS|5|0.9266|\n| D201| 512 | 512 |US|5|0.9240|\n| E6| 768 | 528 |US|5|0.9203|\n| E4| 512 | 380 |US|10|0.9242|\n| E5| 512 | 456 |NS|5|0.9153|\n| D201| 512 | 224 |NS|5|0.9185|\n| B5| 512 | 456 |US|10|0.9234|\n| D201| 512 | 224 |US|5|0.9068|\n\n#### SimpleAverage\n- simply average all models CV:**0.9372** LB:**0.9544** PLB**0.9388**\n\n#### OptimizeAUC\n- for optimizing OOF AUC we are using OptimizeAUC function from  [Approaching (Almost) Any Machine Learning Problem](https://www.amazon.in/dp/B089P13QHT/ref=dp-kindle-redirect?_encoding=UTF8&btkr=1#:~:text=This%20book%20is%20for%20people,learning%20and%20deep%20learning%20problems.) by @abhishek Big thanks to him.\n- you found the OptimizeAUC in this notebook: [link](https://www.kaggle.com/gopidurgaprasad/optimize-auc-using-oof)\n- CV:**0.94741** LB:**0.9492** PLB**0.9344**\n\n**Pytorch Training Notebook:** [link](https://www.kaggle.com/gopidurgaprasad/siim-final-d201-ns)\n\n\n## [TF]\n\n- we are using [Triple Stratified KFold with TFRecords](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords) again Big thanks to @cdotte\n- thanks to @shravankoninti running those TF models one by one every data\n\n| Model | SIZE |FOLDS|CV|\n| --- | --- |---|---|\n| E7| 512 |15|0.8826|\n| E6| 512 |15|0.9194|\n| E5| 512 |15|0.9243|\n| E7| 512 |10|0.9163|\n| E6| 512 |10|0.9027|\n| E5| 512 |10|0.9275|\n| E4| 512 |10|0.9114|\n| E3| 512 |10|0.9133|\n| E2| 512 |10|0.9154|\n| E1| 512 |15|0.9142|\n| E7| 768 |5|0.9076|\n| E6| 768 |5|0.9161|\n| E5| 768 |5|0.9063|\n| E4| 768 |5|0.9144|\n| E3| 768 |5|0.9019|\n| E2| 768 |5|0.8917|\n| E1| 768 |5|0.8866|\n| E7| 384|5|0.9212|\n| E6| 384|5|0.9237|\n| E5| 384|5|0.9278|\n\n#### SimpleAverage\n- simply average all TF models CV:**0.9301** LB:**0.9503** PLB**0.9376**\n\n#### OptimizeAUC\n- CV:**0.9446** LB:**0.9538** PLB**0.9348**\n\n## [PyTorch + TF]\n\n#### SimpleAverage\n- simply average all PyTorch and TF models CV:**0.9402** LB:**0.9538** PLB:**0.9399**\n- This is our **best-selected submission**\n\n#### OptimizeAUC\n- CV:**0.949354** LB:**0.9496** PLB:**0.9362**\n\n\n##🎊 Congratulations  to all Winners and Learners 🎊",
    "977247": "Good explanation and useful link!\nThanks. @gopidurgaprasad",
    "977276": "Thank you 😍",
    "977416": "Congrats Brother for your medal! Thanks for a great explanation and for sharing the notebook.",
    "977421": "Thanks brother 😍 keep rocking"
  },
  "source": "meta"
}