{
  "id": 207230,
  "title": "Multi-Head Approach Baseline [LB: 0.964 now ;) ]",
  "url": "/competitions/ranzcr-clip-catheter-line-classification/discussion/207230",
  "author_name": "Tawara",
  "post_date": "2020-12-28T18:49:16.643000",
  "votes": 85,
  "comment_count": 0,
  "views": 0,
  "content": "<p>I'm working on my idea in this topic:<br>\n<a href=\"https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/205208\" target=\"_blank\">https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/205208</a></p>\n<p>Unfortunately, this baseline has only achieved a low score (LB: 0.930) yet at the 1st version. <br>\nI will continue to improve this baseline every other week (I need more GPU quota 😓).</p>\n<table>\n<thead>\n<tr>\n<th>Ver</th>\n<th>Size</th>\n<th>Base</th>\n<th>Mult-Head</th>\n<th>#Folds</th>\n<th>CV</th>\n<th>LB</th>\n<th>LB(rank avg)</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>384x384</td>\n<td>ResNeXt50</td>\n<td>ResBlock4</td>\n<td>5</td>\n<td>0.9205</td>\n<td>0.929</td>\n<td>0.930</td>\n</tr>\n<tr>\n<td>2</td>\n<td>448x448</td>\n<td>ResNeXt50</td>\n<td>ResBlock4</td>\n<td>4</td>\n<td>0.9249</td>\n<td>0.937</td>\n<td>0.937</td>\n</tr>\n<tr>\n<td>3</td>\n<td>448x448</td>\n<td>ResNeXt50</td>\n<td>ResBlock4</td>\n<td>5</td>\n<td>0.9309</td>\n<td>0.938</td>\n<td>0.938</td>\n</tr>\n<tr>\n<td>4</td>\n<td>448x448</td>\n<td>ResNeXt50</td>\n<td>sSE Module</td>\n<td>5</td>\n<td>0.9298</td>\n<td>0.941</td>\n<td>0.941</td>\n</tr>\n<tr>\n<td>5</td>\n<td>448x448</td>\n<td>ResNeXt50</td>\n<td>sSE Module</td>\n<td>5</td>\n<td>0.9305</td>\n<td>0.943</td>\n<td>0.943</td>\n</tr>\n<tr>\n<td>6</td>\n<td>512x512</td>\n<td>ResNeXt50</td>\n<td>sSE Module</td>\n<td>4</td>\n<td>0.9230</td>\n<td>0.937</td>\n<td>0.937</td>\n</tr>\n<tr>\n<td>7</td>\n<td>448x448</td>\n<td>ResNeXt50</td>\n<td>sSE Module</td>\n<td>4</td>\n<td>0.9323</td>\n<td>0.943</td>\n<td>0.942</td>\n</tr>\n<tr>\n<td>8</td>\n<td>448x448</td>\n<td>ResNeXt50</td>\n<td><strong>no multi-head</strong></td>\n<td>5</td>\n<td>0.9246</td>\n<td>0.939</td>\n<td>0.939</td>\n</tr>\n<tr>\n<td>9</td>\n<td>448x448</td>\n<td>RegNetY032</td>\n<td><strong>no multi-head</strong></td>\n<td>5</td>\n<td>0.9378</td>\n<td>0.950</td>\n<td>0.950</td>\n</tr>\n<tr>\n<td>10</td>\n<td>448x448</td>\n<td>RegNetY032</td>\n<td><strong>no multi-head</strong></td>\n<td>5</td>\n<td>0.9372</td>\n<td>0.952</td>\n<td>0.951</td>\n</tr>\n<tr>\n<td>11</td>\n<td>448x448</td>\n<td>ResNeXt50</td>\n<td>Spatial-Attention</td>\n<td>5</td>\n<td>0.9337</td>\n<td>0.943</td>\n<td>0.943</td>\n</tr>\n<tr>\n<td>12</td>\n<td>448x448</td>\n<td>RegNetY032</td>\n<td>Spatial-Attention</td>\n<td>5</td>\n<td>0.9408</td>\n<td>0.948</td>\n<td>0.949</td>\n</tr>\n<tr>\n<td>13</td>\n<td>448x448</td>\n<td>RegNetY032</td>\n<td>Spatial-Attention</td>\n<td>5</td>\n<td>0.9385</td>\n<td>0.949</td>\n<td>0.949</td>\n</tr>\n<tr>\n<td>14</td>\n<td>448x448</td>\n<td>RegNetY032</td>\n<td>sSE Module</td>\n<td>5</td>\n<td>0.9386</td>\n<td>0.946</td>\n<td>0.947</td>\n</tr>\n<tr>\n<td>15</td>\n<td>512x512</td>\n<td>RegNetY032</td>\n<td>Spatial-Attention</td>\n<td>5</td>\n<td>0.9419</td>\n<td>0.954</td>\n<td>0.954</td>\n</tr>\n<tr>\n<td>16</td>\n<td>640x640</td>\n<td>RegNetY032</td>\n<td>Spatial-Attention</td>\n<td>5</td>\n<td>0.9368</td>\n<td>0.949</td>\n<td>0.948</td>\n</tr>\n<tr>\n<td>17</td>\n<td>512x512</td>\n<td>RegNetY032</td>\n<td>Spatial-Attention</td>\n<td>5</td>\n<td>0.9437</td>\n<td>0.955</td>\n<td>0.955</td>\n</tr>\n<tr>\n<td>18</td>\n<td>640x640</td>\n<td>RegNetY032</td>\n<td>Spatial-Attention</td>\n<td>5</td>\n<td>0.9484</td>\n<td>0.957</td>\n<td>0.958</td>\n</tr>\n<tr>\n<td>19</td>\n<td>640x640</td>\n<td>ECA-ResNet101D</td>\n<td>Spatial-Attention</td>\n<td>5</td>\n<td>0.9471</td>\n<td>0.960</td>\n<td>0.958</td>\n</tr>\n<tr>\n<td>20</td>\n<td>512x512</td>\n<td>ResNet200D</td>\n<td>Spatial-Attention</td>\n<td>5</td>\n<td>0.9608</td>\n<td>0.964</td>\n<td>not sub</td>\n</tr>\n</tbody>\n</table>\n<h5>Version1 (CV: 0.9205, LB: 0.930)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-training?scriptVersionId=50362460\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-inference?scriptVersionId=50462327\" target=\"_blank\">inference</a></li>\n<li>Size: 384x384</li>\n<li>Split: Multi-Label Stratified Group K-Fold 5 (K=5)</li>\n<li>Training for 8 epochs</li>\n<li>Base Model: ResNeXt50_32x4d</li>\n<li>Multi-Head:  <br>\nThe model branches at the 3rd ResBlock's output.<br>\n<strong>Separated</strong> 4th ResBlocks and Linear layers are prepared for each group(<code>ETT(3)</code>, <code>NGT(4)</code>, <code>CVC(3)</code>, and <code>Swan(1)</code>).<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F473234%2Fcaf909bdb97089e2e28f901b95263579%2Fmulti-head_resnext.png?generation=1609044658423223&amp;alt=media\"></li>\n</ul>\n<h5>Version2(CV: 0.9249, LB: 0.937)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=50816602\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-inference?scriptVersionId=50852856\" target=\"_blank\">inference</a></li>\n<li>Size: <strong>448x448</strong></li>\n<li>Split: Multi-Label Stratified Group K-Fold (<strong>K=4</strong>)</li>\n<li>Training for 8 epochs</li>\n<li>Base Model: ResNeXt50_32x4d</li>\n<li>Multi-Head: the same architecture as version 1</li>\n</ul>\n<h5>Version3(CV: 0.9309, LB: 0.938)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-training?scriptVersionId=51101435\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-inference?scriptVersionId=51147518\" target=\"_blank\">inference</a></li>\n<li>Size: 448x448</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: ResNeXt50_32x4d</li>\n<li>Training for 7 epochs <strong>with mixed precision</strong></li>\n<li>Multi-Head: the same architecture as version 1</li>\n</ul>\n<h5>Version4(CV: 0.9298, LB: 0.941)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-training?scriptVersionId=51418323\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-inference?scriptVersionId=51504643\" target=\"_blank\">inference</a></li>\n<li>Size: 448x448</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: ResNeXt50_32x4d</li>\n<li>Training for 9 epochs with mixed precision</li>\n<li>Multi-Head:<br>\nseparated <strong>sSE Module</strong> and Linear layer<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F473234%2F1bfc5ce7b12ba1447c533e5f3fb6a373%2Fmulti-head_approach_sSE.png?generation=1610117554980504&amp;alt=media\"></li>\n</ul>\n<h5>Version5(CV: 0.9305, LB: 0.943)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-training?scriptVersionId=51440725\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=51515553\" target=\"_blank\">inference</a></li>\n<li>Size: 448x448</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: ResNeXt50_32x4d</li>\n<li>Training for 9 epochs with mixed precision</li>\n<li>Multi-Head:<br>\nseparated sSE Module and <strong>MLP</strong>(Linear -&gt; ReLU -&gt; Dropout -&gt; Linear)<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F473234%2Fa7284b20fb1421be94880521413b3ceb%2Fmulti-head_approach_sSE_2.png?generation=1610251015348291&amp;alt=media\"></li>\n</ul>\n<h5>Version6(CV: 0.9230, LB: 0.937)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-training?scriptVersionId=51442070\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=51518379\" target=\"_blank\">inference</a></li>\n<li>Size: 512x512</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: ResNeXt50_32x4d</li>\n<li>Training for 7 epochs with mixed precision</li>\n<li>Multi-Head:  the same architecture as version 4</li>\n</ul>\n<h5>Version7(CV: 0.9323, LB: 0.943)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=51521616\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=51584614\" target=\"_blank\">inference</a></li>\n<li>Size: 448x448</li>\n<li>Split: Multi-Label Stratified Group K-Fold (<strong>K=4</strong>)</li>\n<li>Base Model: ResNeXt50_32x4d</li>\n<li>Training for <strong>12</strong> epochs with mixed precision</li>\n<li>Multi-Head: the same architecture as version 5</li>\n</ul>\n<h5>Version8(CV: 0.9246, LB: 0.939)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=51935970\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=51966667\" target=\"_blank\">inference</a></li>\n<li>Size: 448x448</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: ResNeXt50_32x4d</li>\n<li>Training for 9 epochs with mixed precision</li>\n<li>Multi-Head: <strong>This version is a baseline which doesn't use multi-head architecture</strong>.</li>\n</ul>\n<h5>Version9(CV: 0.9378, LB: 0.950)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=51936107\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=51967884\" target=\"_blank\">inference</a></li>\n<li>Size: 448x448</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: <strong>RegNetY032</strong></li>\n<li>Training for 9 epochs with mixed precision</li>\n<li>Multi-Head: <strong>This version is a baseline which doesn't use multi-head architecture</strong>.</li>\n</ul>\n<h5>Version10(CV: 0.9372, LB: 0.952)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=51963792\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52017732\" target=\"_blank\">inference</a></li>\n<li>Size: 448x448</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: <strong>RegNetY032</strong></li>\n<li>Training for <strong>11</strong> epochs with mixed precision</li>\n<li>Multi-Head: <strong>This version is a baseline which doesn't use multi-head architecture</strong>.</li>\n</ul>\n<h5>Version11(CV: 0.9337, LB: 0.943)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training/output?scriptVersionId=51987851\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52019491\" target=\"_blank\">inference</a></li>\n<li>Size: 448x448</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: ResNeXt50_32x4d</li>\n<li>Training for 9 epochs with mixed precision</li>\n<li>Multi-Head: <strong>Spatial Attention Module</strong> (cf: <a href=\"https://www.kaggle.com/ipythonx/tf-keras-ranzcr-multi-attention-efficientnet\" target=\"_blank\">[TF.Keras]: RANZCR: Multi-Attention EfficientNet</a>) and MLP</li>\n</ul>\n<h5>Version12(CV: 0.9408 LB: 0.949)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52021506\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52037572\" target=\"_blank\">inference</a></li>\n<li>Size: 448x448</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: <strong>RegNetY032</strong></li>\n<li>Training for 9 epochs with mixed precision</li>\n<li>Multi-Head: <strong>Spatial Attention Module</strong> (cf: <a href=\"https://www.kaggle.com/ipythonx/tf-keras-ranzcr-multi-attention-efficientnet\" target=\"_blank\">[TF.Keras]: RANZCR: Multi-Attention EfficientNet</a>) and MLP</li>\n</ul>\n<h5>Version13(CV: 0.9385 LB: 0.949)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52475294\" target=\"_blank\">training</a>, inference</li>\n<li>Size: 448x448</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: RegNetY032</li>\n<li>Training for 11 epochs with mixed precision</li>\n<li>Multi-Head: Spatial Attention Module and MLP</li>\n</ul>\n<h5>Version14(CV: 0.9386 LB: 0.947)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52477310\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52527304\" target=\"_blank\">inference</a></li>\n<li>Size: 448x448</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: RegNetY032</li>\n<li>Training for 11 epochs with mixed precision</li>\n<li>Multi-Head: **sSE Module ** and MLP</li>\n</ul>\n<h5>Version15(CV: 0.9419 LB: 0.954)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52531312\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52570804\" target=\"_blank\">inference</a></li>\n<li>Size: <strong>512x512</strong></li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: RegNetY032</li>\n<li>Training for 8 epochs with mixed precision</li>\n<li>Multi-Head: <strong>Spatial-Attention Module</strong> and MLP</li>\n</ul>\n<h5>Version16(CV: 0.9368 LB: 0.948)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52606857\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52646796\" target=\"_blank\">inference</a></li>\n<li>Size: <strong>640x640</strong></li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: RegNetY032</li>\n<li>Training for 5 epochs with mixed precision</li>\n<li>Multi-Head: Spatial-Attention Module and MLP</li>\n</ul>\n<h5>Version17(CV: 0.9437 LB: 0.955)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52614442\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52647496\" target=\"_blank\">inference</a></li>\n<li>Size: 512x512</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: RegNetY032</li>\n<li>Training for 8 epochs with mixed precision</li>\n<li>Difference from v15: <strong>add some augmentations</strong></li>\n<li>Multi-Head: Spatial-Attention Module and MLP</li>\n</ul>\n<h5>Version18(CV: 0.9484 LB: 0.958)</h5>\n<ul>\n<li>Notebook: training(<a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=53586035\" target=\"_blank\">fold0</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=53589585\" target=\"_blank\">fold1</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=53607319\" target=\"_blank\">fold2</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=53607347\" target=\"_blank\">fold3</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=53621592\" target=\"_blank\">fold4</a>), <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=53640206\" target=\"_blank\">inference</a></li>\n<li>Size: <strong>640x640</strong></li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: RegNetY032</li>\n<li>Training for <strong>16</strong> epochs with mixed precision</li>\n<li>Difference from v15: add some augmentations</li>\n<li>Multi-Head: Spatial-Attention Module and MLP</li>\n</ul>\n<h5>Version19(CV: 0.9471 LB: 0.960)</h5>\n<ul>\n<li>Notebook: training(<a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=54709974\" target=\"_blank\">fold0</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=54710036\" target=\"_blank\">fold1</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=54759689\" target=\"_blank\">fold2</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=54759731\" target=\"_blank\">fold3</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=54789057\" target=\"_blank\">fold4</a>), <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=54847456\" target=\"_blank\">inference</a></li>\n<li>Size: 640x640</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: <strong>ECA-ResNet101D</strong></li>\n<li>Training for 16 epochs with mixed precision</li>\n<li>Difference from v15: add some augmentations</li>\n<li>Multi-Head: Spatial-Attention Module and MLP</li>\n</ul>\n<h5>Version20(CV: 0.9608 LB: 0.964)</h5>\n<ul>\n<li>Notebook: training(<a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=55258318\" target=\"_blank\">fold0</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=55258391\" target=\"_blank\">fold1</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=55286561\" target=\"_blank\">fold2</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=55286597\" target=\"_blank\">fold3</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=55325612\" target=\"_blank\">fold4</a>), <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=55373372\" target=\"_blank\">inference</a></li>\n<li>Size: 512x512</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: <strong>ResNet200D</strong><ul>\n<li><strong>NOTE: I use <a href=\"https://www.kaggle.com/ammarali32/startingpointschestx\" target=\"_blank\">the pre-trained model</a> shared by <a href=\"https://www.kaggle.com/ammarali32\" target=\"_blank\">@ammarali32</a> .</strong> Thanks!</li></ul></li>\n<li>Training for 16 epochs with mixed precision</li>\n<li>Multi-Head: Spatial-Attention Module and MLP</li>\n</ul>\n<h5>bonus</h5>\n<p>I've published pre-processed dataset:<br>\n<a href=\"https://www.kaggle.com/ttahara/ranzcr-clip-train-numpy\" target=\"_blank\">https://www.kaggle.com/ttahara/ranzcr-clip-train-numpy</a></p>\n<p>Images are resized and consolidated in a <code>.npy</code> file by size (<code>256x256</code>, <code>320x320</code>, <code>384x384</code>, <code>448x448</code>, <code>512x512</code> and <code>640x640</code>).</p>\n<p>I wasn't sure if I'm permitted to share the dataset, but I decided to publish it after seeing this discussion:<br>\n<a href=\"https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/203342\" target=\"_blank\">https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/203342</a></p>\n<p>I hope that it will make your training models faster :)</p>",
  "messages": [
    {
      "id": 1130122,
      "postDate": "2020-12-28T18:49:16.643Z",
      "content": "<p>I'm working on my idea in this topic:<br>\n<a href=\"https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/205208\" target=\"_blank\">https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/205208</a></p>\n<p>Unfortunately, this baseline has only achieved a low score (LB: 0.930) yet at the 1st version. <br>\nI will continue to improve this baseline every other week (I need more GPU quota 😓).</p>\n<table>\n<thead>\n<tr>\n<th>Ver</th>\n<th>Size</th>\n<th>Base</th>\n<th>Mult-Head</th>\n<th>#Folds</th>\n<th>CV</th>\n<th>LB</th>\n<th>LB(rank avg)</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>384x384</td>\n<td>ResNeXt50</td>\n<td>ResBlock4</td>\n<td>5</td>\n<td>0.9205</td>\n<td>0.929</td>\n<td>0.930</td>\n</tr>\n<tr>\n<td>2</td>\n<td>448x448</td>\n<td>ResNeXt50</td>\n<td>ResBlock4</td>\n<td>4</td>\n<td>0.9249</td>\n<td>0.937</td>\n<td>0.937</td>\n</tr>\n<tr>\n<td>3</td>\n<td>448x448</td>\n<td>ResNeXt50</td>\n<td>ResBlock4</td>\n<td>5</td>\n<td>0.9309</td>\n<td>0.938</td>\n<td>0.938</td>\n</tr>\n<tr>\n<td>4</td>\n<td>448x448</td>\n<td>ResNeXt50</td>\n<td>sSE Module</td>\n<td>5</td>\n<td>0.9298</td>\n<td>0.941</td>\n<td>0.941</td>\n</tr>\n<tr>\n<td>5</td>\n<td>448x448</td>\n<td>ResNeXt50</td>\n<td>sSE Module</td>\n<td>5</td>\n<td>0.9305</td>\n<td>0.943</td>\n<td>0.943</td>\n</tr>\n<tr>\n<td>6</td>\n<td>512x512</td>\n<td>ResNeXt50</td>\n<td>sSE Module</td>\n<td>4</td>\n<td>0.9230</td>\n<td>0.937</td>\n<td>0.937</td>\n</tr>\n<tr>\n<td>7</td>\n<td>448x448</td>\n<td>ResNeXt50</td>\n<td>sSE Module</td>\n<td>4</td>\n<td>0.9323</td>\n<td>0.943</td>\n<td>0.942</td>\n</tr>\n<tr>\n<td>8</td>\n<td>448x448</td>\n<td>ResNeXt50</td>\n<td><strong>no multi-head</strong></td>\n<td>5</td>\n<td>0.9246</td>\n<td>0.939</td>\n<td>0.939</td>\n</tr>\n<tr>\n<td>9</td>\n<td>448x448</td>\n<td>RegNetY032</td>\n<td><strong>no multi-head</strong></td>\n<td>5</td>\n<td>0.9378</td>\n<td>0.950</td>\n<td>0.950</td>\n</tr>\n<tr>\n<td>10</td>\n<td>448x448</td>\n<td>RegNetY032</td>\n<td><strong>no multi-head</strong></td>\n<td>5</td>\n<td>0.9372</td>\n<td>0.952</td>\n<td>0.951</td>\n</tr>\n<tr>\n<td>11</td>\n<td>448x448</td>\n<td>ResNeXt50</td>\n<td>Spatial-Attention</td>\n<td>5</td>\n<td>0.9337</td>\n<td>0.943</td>\n<td>0.943</td>\n</tr>\n<tr>\n<td>12</td>\n<td>448x448</td>\n<td>RegNetY032</td>\n<td>Spatial-Attention</td>\n<td>5</td>\n<td>0.9408</td>\n<td>0.948</td>\n<td>0.949</td>\n</tr>\n<tr>\n<td>13</td>\n<td>448x448</td>\n<td>RegNetY032</td>\n<td>Spatial-Attention</td>\n<td>5</td>\n<td>0.9385</td>\n<td>0.949</td>\n<td>0.949</td>\n</tr>\n<tr>\n<td>14</td>\n<td>448x448</td>\n<td>RegNetY032</td>\n<td>sSE Module</td>\n<td>5</td>\n<td>0.9386</td>\n<td>0.946</td>\n<td>0.947</td>\n</tr>\n<tr>\n<td>15</td>\n<td>512x512</td>\n<td>RegNetY032</td>\n<td>Spatial-Attention</td>\n<td>5</td>\n<td>0.9419</td>\n<td>0.954</td>\n<td>0.954</td>\n</tr>\n<tr>\n<td>16</td>\n<td>640x640</td>\n<td>RegNetY032</td>\n<td>Spatial-Attention</td>\n<td>5</td>\n<td>0.9368</td>\n<td>0.949</td>\n<td>0.948</td>\n</tr>\n<tr>\n<td>17</td>\n<td>512x512</td>\n<td>RegNetY032</td>\n<td>Spatial-Attention</td>\n<td>5</td>\n<td>0.9437</td>\n<td>0.955</td>\n<td>0.955</td>\n</tr>\n<tr>\n<td>18</td>\n<td>640x640</td>\n<td>RegNetY032</td>\n<td>Spatial-Attention</td>\n<td>5</td>\n<td>0.9484</td>\n<td>0.957</td>\n<td>0.958</td>\n</tr>\n<tr>\n<td>19</td>\n<td>640x640</td>\n<td>ECA-ResNet101D</td>\n<td>Spatial-Attention</td>\n<td>5</td>\n<td>0.9471</td>\n<td>0.960</td>\n<td>0.958</td>\n</tr>\n<tr>\n<td>20</td>\n<td>512x512</td>\n<td>ResNet200D</td>\n<td>Spatial-Attention</td>\n<td>5</td>\n<td>0.9608</td>\n<td>0.964</td>\n<td>not sub</td>\n</tr>\n</tbody>\n</table>\n<h5>Version1 (CV: 0.9205, LB: 0.930)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-training?scriptVersionId=50362460\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-inference?scriptVersionId=50462327\" target=\"_blank\">inference</a></li>\n<li>Size: 384x384</li>\n<li>Split: Multi-Label Stratified Group K-Fold 5 (K=5)</li>\n<li>Training for 8 epochs</li>\n<li>Base Model: ResNeXt50_32x4d</li>\n<li>Multi-Head:  <br>\nThe model branches at the 3rd ResBlock's output.<br>\n<strong>Separated</strong> 4th ResBlocks and Linear layers are prepared for each group(<code>ETT(3)</code>, <code>NGT(4)</code>, <code>CVC(3)</code>, and <code>Swan(1)</code>).<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F473234%2Fcaf909bdb97089e2e28f901b95263579%2Fmulti-head_resnext.png?generation=1609044658423223&amp;alt=media\"></li>\n</ul>\n<h5>Version2(CV: 0.9249, LB: 0.937)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=50816602\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-inference?scriptVersionId=50852856\" target=\"_blank\">inference</a></li>\n<li>Size: <strong>448x448</strong></li>\n<li>Split: Multi-Label Stratified Group K-Fold (<strong>K=4</strong>)</li>\n<li>Training for 8 epochs</li>\n<li>Base Model: ResNeXt50_32x4d</li>\n<li>Multi-Head: the same architecture as version 1</li>\n</ul>\n<h5>Version3(CV: 0.9309, LB: 0.938)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-training?scriptVersionId=51101435\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-inference?scriptVersionId=51147518\" target=\"_blank\">inference</a></li>\n<li>Size: 448x448</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: ResNeXt50_32x4d</li>\n<li>Training for 7 epochs <strong>with mixed precision</strong></li>\n<li>Multi-Head: the same architecture as version 1</li>\n</ul>\n<h5>Version4(CV: 0.9298, LB: 0.941)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-training?scriptVersionId=51418323\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-inference?scriptVersionId=51504643\" target=\"_blank\">inference</a></li>\n<li>Size: 448x448</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: ResNeXt50_32x4d</li>\n<li>Training for 9 epochs with mixed precision</li>\n<li>Multi-Head:<br>\nseparated <strong>sSE Module</strong> and Linear layer<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F473234%2F1bfc5ce7b12ba1447c533e5f3fb6a373%2Fmulti-head_approach_sSE.png?generation=1610117554980504&amp;alt=media\"></li>\n</ul>\n<h5>Version5(CV: 0.9305, LB: 0.943)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-training?scriptVersionId=51440725\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=51515553\" target=\"_blank\">inference</a></li>\n<li>Size: 448x448</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: ResNeXt50_32x4d</li>\n<li>Training for 9 epochs with mixed precision</li>\n<li>Multi-Head:<br>\nseparated sSE Module and <strong>MLP</strong>(Linear -&gt; ReLU -&gt; Dropout -&gt; Linear)<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F473234%2Fa7284b20fb1421be94880521413b3ceb%2Fmulti-head_approach_sSE_2.png?generation=1610251015348291&amp;alt=media\"></li>\n</ul>\n<h5>Version6(CV: 0.9230, LB: 0.937)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-training?scriptVersionId=51442070\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=51518379\" target=\"_blank\">inference</a></li>\n<li>Size: 512x512</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: ResNeXt50_32x4d</li>\n<li>Training for 7 epochs with mixed precision</li>\n<li>Multi-Head:  the same architecture as version 4</li>\n</ul>\n<h5>Version7(CV: 0.9323, LB: 0.943)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=51521616\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=51584614\" target=\"_blank\">inference</a></li>\n<li>Size: 448x448</li>\n<li>Split: Multi-Label Stratified Group K-Fold (<strong>K=4</strong>)</li>\n<li>Base Model: ResNeXt50_32x4d</li>\n<li>Training for <strong>12</strong> epochs with mixed precision</li>\n<li>Multi-Head: the same architecture as version 5</li>\n</ul>\n<h5>Version8(CV: 0.9246, LB: 0.939)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=51935970\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=51966667\" target=\"_blank\">inference</a></li>\n<li>Size: 448x448</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: ResNeXt50_32x4d</li>\n<li>Training for 9 epochs with mixed precision</li>\n<li>Multi-Head: <strong>This version is a baseline which doesn't use multi-head architecture</strong>.</li>\n</ul>\n<h5>Version9(CV: 0.9378, LB: 0.950)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=51936107\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=51967884\" target=\"_blank\">inference</a></li>\n<li>Size: 448x448</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: <strong>RegNetY032</strong></li>\n<li>Training for 9 epochs with mixed precision</li>\n<li>Multi-Head: <strong>This version is a baseline which doesn't use multi-head architecture</strong>.</li>\n</ul>\n<h5>Version10(CV: 0.9372, LB: 0.952)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=51963792\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52017732\" target=\"_blank\">inference</a></li>\n<li>Size: 448x448</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: <strong>RegNetY032</strong></li>\n<li>Training for <strong>11</strong> epochs with mixed precision</li>\n<li>Multi-Head: <strong>This version is a baseline which doesn't use multi-head architecture</strong>.</li>\n</ul>\n<h5>Version11(CV: 0.9337, LB: 0.943)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training/output?scriptVersionId=51987851\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52019491\" target=\"_blank\">inference</a></li>\n<li>Size: 448x448</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: ResNeXt50_32x4d</li>\n<li>Training for 9 epochs with mixed precision</li>\n<li>Multi-Head: <strong>Spatial Attention Module</strong> (cf: <a href=\"https://www.kaggle.com/ipythonx/tf-keras-ranzcr-multi-attention-efficientnet\" target=\"_blank\">[TF.Keras]: RANZCR: Multi-Attention EfficientNet</a>) and MLP</li>\n</ul>\n<h5>Version12(CV: 0.9408 LB: 0.949)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52021506\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52037572\" target=\"_blank\">inference</a></li>\n<li>Size: 448x448</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: <strong>RegNetY032</strong></li>\n<li>Training for 9 epochs with mixed precision</li>\n<li>Multi-Head: <strong>Spatial Attention Module</strong> (cf: <a href=\"https://www.kaggle.com/ipythonx/tf-keras-ranzcr-multi-attention-efficientnet\" target=\"_blank\">[TF.Keras]: RANZCR: Multi-Attention EfficientNet</a>) and MLP</li>\n</ul>\n<h5>Version13(CV: 0.9385 LB: 0.949)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52475294\" target=\"_blank\">training</a>, inference</li>\n<li>Size: 448x448</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: RegNetY032</li>\n<li>Training for 11 epochs with mixed precision</li>\n<li>Multi-Head: Spatial Attention Module and MLP</li>\n</ul>\n<h5>Version14(CV: 0.9386 LB: 0.947)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52477310\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52527304\" target=\"_blank\">inference</a></li>\n<li>Size: 448x448</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: RegNetY032</li>\n<li>Training for 11 epochs with mixed precision</li>\n<li>Multi-Head: **sSE Module ** and MLP</li>\n</ul>\n<h5>Version15(CV: 0.9419 LB: 0.954)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52531312\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52570804\" target=\"_blank\">inference</a></li>\n<li>Size: <strong>512x512</strong></li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: RegNetY032</li>\n<li>Training for 8 epochs with mixed precision</li>\n<li>Multi-Head: <strong>Spatial-Attention Module</strong> and MLP</li>\n</ul>\n<h5>Version16(CV: 0.9368 LB: 0.948)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52606857\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52646796\" target=\"_blank\">inference</a></li>\n<li>Size: <strong>640x640</strong></li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: RegNetY032</li>\n<li>Training for 5 epochs with mixed precision</li>\n<li>Multi-Head: Spatial-Attention Module and MLP</li>\n</ul>\n<h5>Version17(CV: 0.9437 LB: 0.955)</h5>\n<ul>\n<li>Notebook: <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52614442\" target=\"_blank\">training</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52647496\" target=\"_blank\">inference</a></li>\n<li>Size: 512x512</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: RegNetY032</li>\n<li>Training for 8 epochs with mixed precision</li>\n<li>Difference from v15: <strong>add some augmentations</strong></li>\n<li>Multi-Head: Spatial-Attention Module and MLP</li>\n</ul>\n<h5>Version18(CV: 0.9484 LB: 0.958)</h5>\n<ul>\n<li>Notebook: training(<a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=53586035\" target=\"_blank\">fold0</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=53589585\" target=\"_blank\">fold1</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=53607319\" target=\"_blank\">fold2</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=53607347\" target=\"_blank\">fold3</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=53621592\" target=\"_blank\">fold4</a>), <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=53640206\" target=\"_blank\">inference</a></li>\n<li>Size: <strong>640x640</strong></li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: RegNetY032</li>\n<li>Training for <strong>16</strong> epochs with mixed precision</li>\n<li>Difference from v15: add some augmentations</li>\n<li>Multi-Head: Spatial-Attention Module and MLP</li>\n</ul>\n<h5>Version19(CV: 0.9471 LB: 0.960)</h5>\n<ul>\n<li>Notebook: training(<a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=54709974\" target=\"_blank\">fold0</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=54710036\" target=\"_blank\">fold1</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=54759689\" target=\"_blank\">fold2</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=54759731\" target=\"_blank\">fold3</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=54789057\" target=\"_blank\">fold4</a>), <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=54847456\" target=\"_blank\">inference</a></li>\n<li>Size: 640x640</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: <strong>ECA-ResNet101D</strong></li>\n<li>Training for 16 epochs with mixed precision</li>\n<li>Difference from v15: add some augmentations</li>\n<li>Multi-Head: Spatial-Attention Module and MLP</li>\n</ul>\n<h5>Version20(CV: 0.9608 LB: 0.964)</h5>\n<ul>\n<li>Notebook: training(<a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=55258318\" target=\"_blank\">fold0</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=55258391\" target=\"_blank\">fold1</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=55286561\" target=\"_blank\">fold2</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=55286597\" target=\"_blank\">fold3</a>, <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=55325612\" target=\"_blank\">fold4</a>), <a href=\"https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=55373372\" target=\"_blank\">inference</a></li>\n<li>Size: 512x512</li>\n<li>Split: Multi-Label Stratified Group K-Fold (K=5)</li>\n<li>Base Model: <strong>ResNet200D</strong><ul>\n<li><strong>NOTE: I use <a href=\"https://www.kaggle.com/ammarali32/startingpointschestx\" target=\"_blank\">the pre-trained model</a> shared by <a href=\"https://www.kaggle.com/ammarali32\" target=\"_blank\">@ammarali32</a> .</strong> Thanks!</li></ul></li>\n<li>Training for 16 epochs with mixed precision</li>\n<li>Multi-Head: Spatial-Attention Module and MLP</li>\n</ul>\n<h5>bonus</h5>\n<p>I've published pre-processed dataset:<br>\n<a href=\"https://www.kaggle.com/ttahara/ranzcr-clip-train-numpy\" target=\"_blank\">https://www.kaggle.com/ttahara/ranzcr-clip-train-numpy</a></p>\n<p>Images are resized and consolidated in a <code>.npy</code> file by size (<code>256x256</code>, <code>320x320</code>, <code>384x384</code>, <code>448x448</code>, <code>512x512</code> and <code>640x640</code>).</p>\n<p>I wasn't sure if I'm permitted to share the dataset, but I decided to publish it after seeing this discussion:<br>\n<a href=\"https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/203342\" target=\"_blank\">https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/203342</a></p>\n<p>I hope that it will make your training models faster :)</p>",
      "rawMarkdown": "I'm working on my idea in this topic:\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/205208\n\nUnfortunately, this baseline has only achieved a low score (LB: 0.930) yet at the 1st version. \nI will continue to improve this baseline every other week (I need more GPU quota 😓).\n\n| Ver | Size         | Base | Mult-Head |#Folds | CV | LB | LB(rank avg) |\n|:----:|:---------:|:-----------:|:-----------:|:------:|:---------:|:-------------:|:-------------:|\n| 1     | 384x384 | ResNeXt50 | ResBlock4 |   5      | 0.9205 | 0.929 |  0.930 |\n| 2     | 448x448 | ResNeXt50 |ResBlock4 |    4      | 0.9249 | 0.937 | 0.937 |  \n| 3     | 448x448 | ResNeXt50 |ResBlock4 |    5      | 0.9309 | 0.938 | 0.938 | \n| 4     | 448x448 | ResNeXt50 |sSE Module |    5      | 0.9298 | 0.941 | 0.941 |\n| 5     | 448x448 | ResNeXt50 |sSE Module |     5      | 0.9305 | 0.943  | 0.943 |\n| 6     | 512x512 | ResNeXt50 |sSE Module |    4      | 0.9230 | 0.937 | 0.937 |\n| 7     | 448x448 | ResNeXt50 |sSE Module |    4      | 0.9323 | 0.943 | 0.942 |\n| 8     | 448x448 | ResNeXt50 |**no multi-head** |      5      | 0.9246 | 0.939 | 0.939 | \n| 9     | 448x448 | RegNetY032 | **no multi-head** |     5      | 0.9378 | 0.950 | 0.950 | \n| 10   | 448x448 | RegNetY032 | **no multi-head** |     5      | 0.9372 | 0.952 | 0.951 | \n| 11    | 448x448 | ResNeXt50 | Spatial-Attention |     5      | 0.9337 | 0.943 | 0.943 | \n| 12    | 448x448 | RegNetY032 | Spatial-Attention |    5      | 0.9408 | 0.948 | 0.949 |\n| 13    | 448x448 | RegNetY032 | Spatial-Attention |    5      | 0.9385 | 0.949 | 0.949 |\n| 14    | 448x448 | RegNetY032 | sSE Module |    5      | 0.9386 | 0.946 | 0.947 |\n| 15    | 512x512 | RegNetY032 | Spatial-Attention |    5      | 0.9419 | 0.954 | 0.954 |\n| 16    | 640x640 | RegNetY032 | Spatial-Attention |    5      | 0.9368 | 0.949 | 0.948 |\n| 17    | 512x512 | RegNetY032 | Spatial-Attention |    5      | 0.9437 | 0.955 | 0.955 |\n| 18    | 640x640 | RegNetY032 | Spatial-Attention |    5      | 0.9484 | 0.957 | 0.958 |\n| 19    | 640x640 | ECA-ResNet101D | Spatial-Attention |    5      | 0.9471 | 0.960 | 0.958 |\n| 20    | 512x512 | ResNet200D | Spatial-Attention |    5      | 0.9608 | 0.964 | not sub |\n\n##### Version1 (CV: 0.9205, LB: 0.930)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-training?scriptVersionId=50362460), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-inference?scriptVersionId=50462327)\n* Size: 384x384\n* Split: Multi-Label Stratified Group K-Fold 5 (K=5)\n* Training for 8 epochs\n* Base Model: ResNeXt50_32x4d\n* Multi-Head:  \nThe model branches at the 3rd ResBlock's output.\n**Separated** 4th ResBlocks and Linear layers are prepared for each group(`ETT(3)`, `NGT(4)`, `CVC(3)`, and `Swan(1)`).\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F473234%2Fcaf909bdb97089e2e28f901b95263579%2Fmulti-head_resnext.png?generation=1609044658423223&alt=media\" width=55%>\n\n##### Version2(CV: 0.9249, LB: 0.937)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=50816602), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-inference?scriptVersionId=50852856)\n* Size: **448x448**\n* Split: Multi-Label Stratified Group K-Fold (**K=4**)\n* Training for 8 epochs\n* Base Model: ResNeXt50_32x4d\n* Multi-Head: the same architecture as version 1\n\n##### Version3(CV: 0.9309, LB: 0.938)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-training?scriptVersionId=51101435), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-inference?scriptVersionId=51147518)\n* Size: 448x448\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: ResNeXt50_32x4d\n* Training for 7 epochs **with mixed precision**\n* Multi-Head: the same architecture as version 1\n\n##### Version4(CV: 0.9298, LB: 0.941)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-training?scriptVersionId=51418323), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-inference?scriptVersionId=51504643)\n* Size: 448x448\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: ResNeXt50_32x4d\n* Training for 9 epochs with mixed precision\n* Multi-Head:\nseparated **sSE Module** and Linear layer\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F473234%2F1bfc5ce7b12ba1447c533e5f3fb6a373%2Fmulti-head_approach_sSE.png?generation=1610117554980504&alt=media\" width=55%>\n\n##### Version5(CV: 0.9305, LB: 0.943)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-training?scriptVersionId=51440725), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=51515553)\n* Size: 448x448\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: ResNeXt50_32x4d\n* Training for 9 epochs with mixed precision\n* Multi-Head:\nseparated sSE Module and **MLP**(Linear -> ReLU -> Dropout -> Linear)\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F473234%2Fa7284b20fb1421be94880521413b3ceb%2Fmulti-head_approach_sSE_2.png?generation=1610251015348291&alt=media\" width=55%>\n\n##### Version6(CV: 0.9230, LB: 0.937)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-training?scriptVersionId=51442070), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=51518379)\n* Size: 512x512\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: ResNeXt50_32x4d\n* Training for 7 epochs with mixed precision\n* Multi-Head:  the same architecture as version 4\n\n##### Version7(CV: 0.9323, LB: 0.943)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=51521616), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=51584614)\n* Size: 448x448\n* Split: Multi-Label Stratified Group K-Fold (**K=4**)\n* Base Model: ResNeXt50_32x4d\n* Training for **12** epochs with mixed precision\n* Multi-Head: the same architecture as version 5\n\n##### Version8(CV: 0.9246, LB: 0.939)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=51935970), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=51966667)\n* Size: 448x448\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: ResNeXt50_32x4d\n* Training for 9 epochs with mixed precision\n* Multi-Head: **This version is a baseline which doesn't use multi-head architecture**.\n\n##### Version9(CV: 0.9378, LB: 0.950)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=51936107), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=51967884)\n* Size: 448x448\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: **RegNetY032**\n* Training for 9 epochs with mixed precision\n* Multi-Head: **This version is a baseline which doesn't use multi-head architecture**.\n\n##### Version10(CV: 0.9372, LB: 0.952)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=51963792), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52017732)\n* Size: 448x448\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: **RegNetY032**\n* Training for **11** epochs with mixed precision\n* Multi-Head: **This version is a baseline which doesn't use multi-head architecture**.\n\n##### Version11(CV: 0.9337, LB: 0.943)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training/output?scriptVersionId=51987851), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52019491)\n* Size: 448x448\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: ResNeXt50_32x4d\n* Training for 9 epochs with mixed precision\n* Multi-Head: **Spatial Attention Module** (cf: [[TF.Keras]: RANZCR: Multi-Attention EfficientNet](https://www.kaggle.com/ipythonx/tf-keras-ranzcr-multi-attention-efficientnet)) and MLP\n\n##### Version12(CV: 0.9408 LB: 0.949)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52021506), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52037572)\n* Size: 448x448\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: **RegNetY032**\n* Training for 9 epochs with mixed precision\n* Multi-Head: **Spatial Attention Module** (cf: [[TF.Keras]: RANZCR: Multi-Attention EfficientNet](https://www.kaggle.com/ipythonx/tf-keras-ranzcr-multi-attention-efficientnet)) and MLP\n\n##### Version13(CV: 0.9385 LB: 0.949)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52475294), inference\n* Size: 448x448\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: RegNetY032\n* Training for 11 epochs with mixed precision\n* Multi-Head: Spatial Attention Module and MLP\n\n##### Version14(CV: 0.9386 LB: 0.947)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52477310), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52527304)\n* Size: 448x448\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: RegNetY032\n* Training for 11 epochs with mixed precision\n* Multi-Head: **sSE Module ** and MLP\n\n##### Version15(CV: 0.9419 LB: 0.954)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52531312), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52570804)\n* Size: **512x512**\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: RegNetY032\n* Training for 8 epochs with mixed precision\n* Multi-Head: **Spatial-Attention Module** and MLP\n\n##### Version16(CV: 0.9368 LB: 0.948)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52606857), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52646796)\n* Size: **640x640**\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: RegNetY032\n* Training for 5 epochs with mixed precision\n* Multi-Head: Spatial-Attention Module and MLP\n\n##### Version17(CV: 0.9437 LB: 0.955)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52614442), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52647496)\n* Size: 512x512\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: RegNetY032\n* Training for 8 epochs with mixed precision\n* Difference from v15: **add some augmentations**\n* Multi-Head: Spatial-Attention Module and MLP\n\n##### Version18(CV: 0.9484 LB: 0.958)\n* Notebook: training([fold0](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=53586035), [fold1](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=53589585), [fold2](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=53607319), [fold3](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=53607347), [fold4](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=53621592)), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=53640206)\n* Size: **640x640**\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: RegNetY032\n* Training for **16** epochs with mixed precision\n* Difference from v15: add some augmentations\n* Multi-Head: Spatial-Attention Module and MLP\n\n##### Version19(CV: 0.9471 LB: 0.960)\n* Notebook: training([fold0](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=54709974), [fold1](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=54710036), [fold2](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=54759689), [fold3](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=54759731), [fold4](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=54789057)), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=54847456)\n* Size: 640x640\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: **ECA-ResNet101D**\n* Training for 16 epochs with mixed precision\n* Difference from v15: add some augmentations\n* Multi-Head: Spatial-Attention Module and MLP\n\n##### Version20(CV: 0.9608 LB: 0.964)\n* Notebook: training([fold0](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=55258318), [fold1](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=55258391), [fold2](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=55286561), [fold3](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=55286597), [fold4](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=55325612)), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=55373372)\n* Size: 512x512\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: **ResNet200D**\n  * **NOTE: I use [the pre-trained model](https://www.kaggle.com/ammarali32/startingpointschestx) shared by @ammarali32 .** Thanks!\n* Training for 16 epochs with mixed precision\n* Multi-Head: Spatial-Attention Module and MLP\n\n##### bonus\n\nI've published pre-processed dataset:\nhttps://www.kaggle.com/ttahara/ranzcr-clip-train-numpy\n\nImages are resized and consolidated in a `.npy` file by size (`256x256`, `320x320`, `384x384`, `448x448`, `512x512` and `640x640`).\n\nI wasn't sure if I'm permitted to share the dataset, but I decided to publish it after seeing this discussion:\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/203342\n\nI hope that it will make your training models faster :)",
      "votes": 85
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1130122": "I'm working on my idea in this topic:\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/205208\n\nUnfortunately, this baseline has only achieved a low score (LB: 0.930) yet at the 1st version. \nI will continue to improve this baseline every other week (I need more GPU quota 😓).\n\n| Ver | Size         | Base | Mult-Head |#Folds | CV | LB | LB(rank avg) |\n|:----:|:---------:|:-----------:|:-----------:|:------:|:---------:|:-------------:|:-------------:|\n| 1     | 384x384 | ResNeXt50 | ResBlock4 |   5      | 0.9205 | 0.929 |  0.930 |\n| 2     | 448x448 | ResNeXt50 |ResBlock4 |    4      | 0.9249 | 0.937 | 0.937 |  \n| 3     | 448x448 | ResNeXt50 |ResBlock4 |    5      | 0.9309 | 0.938 | 0.938 | \n| 4     | 448x448 | ResNeXt50 |sSE Module |    5      | 0.9298 | 0.941 | 0.941 |\n| 5     | 448x448 | ResNeXt50 |sSE Module |     5      | 0.9305 | 0.943  | 0.943 |\n| 6     | 512x512 | ResNeXt50 |sSE Module |    4      | 0.9230 | 0.937 | 0.937 |\n| 7     | 448x448 | ResNeXt50 |sSE Module |    4      | 0.9323 | 0.943 | 0.942 |\n| 8     | 448x448 | ResNeXt50 |**no multi-head** |      5      | 0.9246 | 0.939 | 0.939 | \n| 9     | 448x448 | RegNetY032 | **no multi-head** |     5      | 0.9378 | 0.950 | 0.950 | \n| 10   | 448x448 | RegNetY032 | **no multi-head** |     5      | 0.9372 | 0.952 | 0.951 | \n| 11    | 448x448 | ResNeXt50 | Spatial-Attention |     5      | 0.9337 | 0.943 | 0.943 | \n| 12    | 448x448 | RegNetY032 | Spatial-Attention |    5      | 0.9408 | 0.948 | 0.949 |\n| 13    | 448x448 | RegNetY032 | Spatial-Attention |    5      | 0.9385 | 0.949 | 0.949 |\n| 14    | 448x448 | RegNetY032 | sSE Module |    5      | 0.9386 | 0.946 | 0.947 |\n| 15    | 512x512 | RegNetY032 | Spatial-Attention |    5      | 0.9419 | 0.954 | 0.954 |\n| 16    | 640x640 | RegNetY032 | Spatial-Attention |    5      | 0.9368 | 0.949 | 0.948 |\n| 17    | 512x512 | RegNetY032 | Spatial-Attention |    5      | 0.9437 | 0.955 | 0.955 |\n| 18    | 640x640 | RegNetY032 | Spatial-Attention |    5      | 0.9484 | 0.957 | 0.958 |\n| 19    | 640x640 | ECA-ResNet101D | Spatial-Attention |    5      | 0.9471 | 0.960 | 0.958 |\n| 20    | 512x512 | ResNet200D | Spatial-Attention |    5      | 0.9608 | 0.964 | not sub |\n\n##### Version1 (CV: 0.9205, LB: 0.930)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-training?scriptVersionId=50362460), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-inference?scriptVersionId=50462327)\n* Size: 384x384\n* Split: Multi-Label Stratified Group K-Fold 5 (K=5)\n* Training for 8 epochs\n* Base Model: ResNeXt50_32x4d\n* Multi-Head:  \nThe model branches at the 3rd ResBlock's output.\n**Separated** 4th ResBlocks and Linear layers are prepared for each group(`ETT(3)`, `NGT(4)`, `CVC(3)`, and `Swan(1)`).\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F473234%2Fcaf909bdb97089e2e28f901b95263579%2Fmulti-head_resnext.png?generation=1609044658423223&alt=media\" width=55%>\n\n##### Version2(CV: 0.9249, LB: 0.937)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=50816602), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-inference?scriptVersionId=50852856)\n* Size: **448x448**\n* Split: Multi-Label Stratified Group K-Fold (**K=4**)\n* Training for 8 epochs\n* Base Model: ResNeXt50_32x4d\n* Multi-Head: the same architecture as version 1\n\n##### Version3(CV: 0.9309, LB: 0.938)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-training?scriptVersionId=51101435), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-inference?scriptVersionId=51147518)\n* Size: 448x448\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: ResNeXt50_32x4d\n* Training for 7 epochs **with mixed precision**\n* Multi-Head: the same architecture as version 1\n\n##### Version4(CV: 0.9298, LB: 0.941)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-training?scriptVersionId=51418323), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-inference?scriptVersionId=51504643)\n* Size: 448x448\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: ResNeXt50_32x4d\n* Training for 9 epochs with mixed precision\n* Multi-Head:\nseparated **sSE Module** and Linear layer\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F473234%2F1bfc5ce7b12ba1447c533e5f3fb6a373%2Fmulti-head_approach_sSE.png?generation=1610117554980504&alt=media\" width=55%>\n\n##### Version5(CV: 0.9305, LB: 0.943)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-training?scriptVersionId=51440725), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=51515553)\n* Size: 448x448\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: ResNeXt50_32x4d\n* Training for 9 epochs with mixed precision\n* Multi-Head:\nseparated sSE Module and **MLP**(Linear -> ReLU -> Dropout -> Linear)\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F473234%2Fa7284b20fb1421be94880521413b3ceb%2Fmulti-head_approach_sSE_2.png?generation=1610251015348291&alt=media\" width=55%>\n\n##### Version6(CV: 0.9230, LB: 0.937)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-resnext50-32x4d-training?scriptVersionId=51442070), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=51518379)\n* Size: 512x512\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: ResNeXt50_32x4d\n* Training for 7 epochs with mixed precision\n* Multi-Head:  the same architecture as version 4\n\n##### Version7(CV: 0.9323, LB: 0.943)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=51521616), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=51584614)\n* Size: 448x448\n* Split: Multi-Label Stratified Group K-Fold (**K=4**)\n* Base Model: ResNeXt50_32x4d\n* Training for **12** epochs with mixed precision\n* Multi-Head: the same architecture as version 5\n\n##### Version8(CV: 0.9246, LB: 0.939)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=51935970), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=51966667)\n* Size: 448x448\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: ResNeXt50_32x4d\n* Training for 9 epochs with mixed precision\n* Multi-Head: **This version is a baseline which doesn't use multi-head architecture**.\n\n##### Version9(CV: 0.9378, LB: 0.950)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=51936107), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=51967884)\n* Size: 448x448\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: **RegNetY032**\n* Training for 9 epochs with mixed precision\n* Multi-Head: **This version is a baseline which doesn't use multi-head architecture**.\n\n##### Version10(CV: 0.9372, LB: 0.952)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=51963792), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52017732)\n* Size: 448x448\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: **RegNetY032**\n* Training for **11** epochs with mixed precision\n* Multi-Head: **This version is a baseline which doesn't use multi-head architecture**.\n\n##### Version11(CV: 0.9337, LB: 0.943)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training/output?scriptVersionId=51987851), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52019491)\n* Size: 448x448\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: ResNeXt50_32x4d\n* Training for 9 epochs with mixed precision\n* Multi-Head: **Spatial Attention Module** (cf: [[TF.Keras]: RANZCR: Multi-Attention EfficientNet](https://www.kaggle.com/ipythonx/tf-keras-ranzcr-multi-attention-efficientnet)) and MLP\n\n##### Version12(CV: 0.9408 LB: 0.949)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52021506), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52037572)\n* Size: 448x448\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: **RegNetY032**\n* Training for 9 epochs with mixed precision\n* Multi-Head: **Spatial Attention Module** (cf: [[TF.Keras]: RANZCR: Multi-Attention EfficientNet](https://www.kaggle.com/ipythonx/tf-keras-ranzcr-multi-attention-efficientnet)) and MLP\n\n##### Version13(CV: 0.9385 LB: 0.949)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52475294), inference\n* Size: 448x448\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: RegNetY032\n* Training for 11 epochs with mixed precision\n* Multi-Head: Spatial Attention Module and MLP\n\n##### Version14(CV: 0.9386 LB: 0.947)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52477310), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52527304)\n* Size: 448x448\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: RegNetY032\n* Training for 11 epochs with mixed precision\n* Multi-Head: **sSE Module ** and MLP\n\n##### Version15(CV: 0.9419 LB: 0.954)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52531312), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52570804)\n* Size: **512x512**\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: RegNetY032\n* Training for 8 epochs with mixed precision\n* Multi-Head: **Spatial-Attention Module** and MLP\n\n##### Version16(CV: 0.9368 LB: 0.948)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52606857), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52646796)\n* Size: **640x640**\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: RegNetY032\n* Training for 5 epochs with mixed precision\n* Multi-Head: Spatial-Attention Module and MLP\n\n##### Version17(CV: 0.9437 LB: 0.955)\n* Notebook: [training](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=52614442), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=52647496)\n* Size: 512x512\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: RegNetY032\n* Training for 8 epochs with mixed precision\n* Difference from v15: **add some augmentations**\n* Multi-Head: Spatial-Attention Module and MLP\n\n##### Version18(CV: 0.9484 LB: 0.958)\n* Notebook: training([fold0](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=53586035), [fold1](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=53589585), [fold2](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=53607319), [fold3](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=53607347), [fold4](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=53621592)), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=53640206)\n* Size: **640x640**\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: RegNetY032\n* Training for **16** epochs with mixed precision\n* Difference from v15: add some augmentations\n* Multi-Head: Spatial-Attention Module and MLP\n\n##### Version19(CV: 0.9471 LB: 0.960)\n* Notebook: training([fold0](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=54709974), [fold1](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=54710036), [fold2](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=54759689), [fold3](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=54759731), [fold4](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=54789057)), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=54847456)\n* Size: 640x640\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: **ECA-ResNet101D**\n* Training for 16 epochs with mixed precision\n* Difference from v15: add some augmentations\n* Multi-Head: Spatial-Attention Module and MLP\n\n##### Version20(CV: 0.9608 LB: 0.964)\n* Notebook: training([fold0](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=55258318), [fold1](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=55258391), [fold2](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=55286561), [fold3](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=55286597), [fold4](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-training?scriptVersionId=55325612)), [inference](https://www.kaggle.com/ttahara/ranzcr-multi-head-model-inference?scriptVersionId=55373372)\n* Size: 512x512\n* Split: Multi-Label Stratified Group K-Fold (K=5)\n* Base Model: **ResNet200D**\n  * **NOTE: I use [the pre-trained model](https://www.kaggle.com/ammarali32/startingpointschestx) shared by @ammarali32 .** Thanks!\n* Training for 16 epochs with mixed precision\n* Multi-Head: Spatial-Attention Module and MLP\n\n##### bonus\n\nI've published pre-processed dataset:\nhttps://www.kaggle.com/ttahara/ranzcr-clip-train-numpy\n\nImages are resized and consolidated in a `.npy` file by size (`256x256`, `320x320`, `384x384`, `448x448`, `512x512` and `640x640`).\n\nI wasn't sure if I'm permitted to share the dataset, but I decided to publish it after seeing this discussion:\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/203342\n\nI hope that it will make your training models faster :)"
  }
}