{
  "id": 563316,
  "title": "77th Place Solution & Late Submission (Single Model Focus)",
  "url": "/competitions/czii-cryo-et-object-identification/discussion/563316",
  "author_name": "Akima",
  "post_date": "2025-02-16T13:01:59.279000",
  "votes": 8,
  "comment_count": 0,
  "views": 0,
  "content": "<h1>1. 77th Place Solution</h1>\n<ul>\n<li><strong>Data:</strong> Hard label (radius × 0.7)  </li>\n<li><strong>Model:</strong> Ensemble of 2D-3D U-Net (Backbone: SEResNeXt26TN_32x4d, CoAtNet) &amp; YOLO (from <a href=\"https://www.kaggle.com/code/sersasj/czii-yolo11-submission-baseline-with-kdtree-update\" target=\"_blank\">public notebook</a>  )  </li>\n<li><strong>Training:</strong> U-Net pre-trained on simulated data -&gt; Fine-tuned on competition data</li>\n</ul>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>Data</th>\n<th>Train Size (z,x,y)</th>\n<th>Infer Size (z,x,y)</th>\n<th>Optimizer</th>\n<th>Learning Rate</th>\n<th>Scheduler</th>\n<th>Loss</th>\n<th>Valid</th>\n<th>TTA</th>\n<th>Public</th>\n<th>Private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>2d-3d U-Net (seresnext26tn_32x4d)</td>\n<td>Denoised only</td>\n<td>128x128x128</td>\n<td>128x128x128</td>\n<td>AdamW</td>\n<td>1e-3</td>\n<td>CosineAnnealingLR</td>\n<td>DiceLoss</td>\n<td>TS_69_2</td>\n<td>〇</td>\n<td>0.7134</td>\n<td>0.7060</td>\n</tr>\n<tr>\n<td>2d-3d U-Net (coatnet)</td>\n<td>Denoised only</td>\n<td>128x128x128</td>\n<td>128x128x128</td>\n<td>AdamW</td>\n<td>1e-3</td>\n<td>CosineAnnealingLR</td>\n<td>DiceLoss</td>\n<td>TS_69_2</td>\n<td>〇</td>\n<td>0.7032</td>\n<td>0.7016</td>\n</tr>\n<tr>\n<td>Ensemble (2d-3d U-Net×2, YOLO)</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>0.7373</td>\n<td>0.7264</td>\n</tr>\n</tbody>\n</table>\n<h1>2. Latesub results</h1>\n<p>I share the latesub experiments based on <strong>2th and 8th place solutions</strong>.   (Many thanks to the authors of these solutions for their valuable contributions!)  <br>\nMy main focus is <strong>single model results</strong> (without TTA or ensemble).  </p>\n<p>The experiments are categorized into two settings:</p>\n<ul>\n<li>(A): <a href=\"https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/561515\" target=\"_blank\">8th Place</a> based U-Net  </li>\n<li>(B): <a href=\"https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/561568\" target=\"_blank\">2th Place</a> based U-Net</li>\n</ul>\n<p>Below are the details and results of these experiments.</p>\n<h2><strong>2-1. Dataset and Model Config</strong></h2>\n<h3><strong>Dataset</strong></h3>\n<ul>\n<li><strong>Data:</strong> Hard label (radius × 0.5), only competition data used.</li>\n<li><strong>Valid ID:</strong> TS_69_2</li>\n</ul>\n<h3><strong>Model</strong></h3>\n<table>\n<thead>\n<tr>\n<th>Category</th>\n<th>Parameter</th>\n<th>(A) <a href=\"https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/561515\" target=\"_blank\">8th Place</a> based U-Net</th>\n<th>(B): <a href=\"https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/561568\" target=\"_blank\">2th Place</a> based U-Net</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><strong>Model</strong></td>\n<td>Architecture</td>\n<td>monai 3dU-Net</td>\n<td>monai 3dU-Net</td>\n</tr>\n<tr>\n<td></td>\n<td>Channels</td>\n<td><strong>(32, 64, 128, 256)</strong></td>\n<td><strong>(48, 64, 80, 80)</strong></td>\n</tr>\n<tr>\n<td></td>\n<td>Strides</td>\n<td>(2, 2, 1)</td>\n<td>(2, 2, 1)</td>\n</tr>\n<tr>\n<td></td>\n<td>Num Residual Units</td>\n<td>2</td>\n<td>2</td>\n</tr>\n<tr>\n<td><strong>Training</strong></td>\n<td>Train Size</td>\n<td><strong>128x128x128</strong></td>\n<td><strong>128x384x384</strong></td>\n</tr>\n<tr>\n<td></td>\n<td>Loss Function</td>\n<td><strong>DiceLoss</strong></td>\n<td><strong>Tversky+CE</strong></td>\n</tr>\n<tr>\n<td></td>\n<td>Optimizer</td>\n<td>AdamW</td>\n<td>AdamW</td>\n</tr>\n<tr>\n<td></td>\n<td>Learning Rate</td>\n<td>1e-3</td>\n<td>1e-3</td>\n</tr>\n<tr>\n<td></td>\n<td>Scheduler</td>\n<td><strong>CosineAnnealingLR</strong></td>\n<td><strong>No</strong></td>\n</tr>\n</tbody>\n</table>\n<h2><strong>2-2. Results</strong></h2>\n<h3><strong>(A) 8th Place based U-Net</strong></h3>\n<table>\n<thead>\n<tr>\n<th>No.</th>\n<th>Infer Size</th>\n<th>EMA</th>\n<th>Multi-tomo</th>\n<th>Public</th>\n<th>Private</th>\n<th>ΔPublic</th>\n<th>ΔPrivate</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>128x128x128</td>\n<td>❌</td>\n<td>❌</td>\n<td>0.6810</td>\n<td>0.6769</td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td>2</td>\n<td>184x512x512</td>\n<td>❌</td>\n<td>❌</td>\n<td>0.6976</td>\n<td>0.6969</td>\n<td>+0.0166</td>\n<td>+0.0200</td>\n</tr>\n<tr>\n<td>3</td>\n<td>128x128x128</td>\n<td>✅</td>\n<td>❌</td>\n<td>0.7136</td>\n<td>0.7093</td>\n<td>+0.0326</td>\n<td>+0.0323</td>\n</tr>\n<tr>\n<td>4</td>\n<td>184x512x512</td>\n<td>✅</td>\n<td>❌</td>\n<td><strong>0.7410</strong></td>\n<td><strong>0.7357</strong></td>\n<td><strong>+0.0600</strong></td>\n<td><strong>+0.0588</strong></td>\n</tr>\n<tr>\n<td>5</td>\n<td>184x512x512</td>\n<td>✅</td>\n<td>✅</td>\n<td>0.7324</td>\n<td>0.7305</td>\n<td>+0.0514</td>\n<td>+0.0536</td>\n</tr>\n</tbody>\n</table>\n<p>*Multi-tomo: Use denoised, wbp, ctfdeconvolved, isonetcorrected.❌ means only denoised</p>\n<ul>\n<li>Large infer size &amp; EMA improved score by <strong>+0.06</strong>.  </li>\n<li>The use of multi-tomo types was not effective.  </li>\n</ul>\n<h3><strong>(B) 2nd Place based U-Net</strong></h3>\n<table>\n<thead>\n<tr>\n<th>No.</th>\n<th>Infer Size</th>\n<th>LR Scheduler</th>\n<th>EMA</th>\n<th>Multi-tomo</th>\n<th>Public</th>\n<th>Private</th>\n<th>ΔPublic</th>\n<th>ΔPrivate</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>128x384x384</td>\n<td>No</td>\n<td>❌</td>\n<td>❌</td>\n<td>0.7381</td>\n<td>0.7290</td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td>2</td>\n<td>184x512x512</td>\n<td>No</td>\n<td>❌</td>\n<td>❌</td>\n<td>0.7467</td>\n<td>0.7423</td>\n<td>+0.0086</td>\n<td>+0.0133</td>\n</tr>\n<tr>\n<td>3</td>\n<td>128x384x384</td>\n<td>No</td>\n<td>✅</td>\n<td>❌</td>\n<td>0.7312</td>\n<td>0.7251</td>\n<td>-0.0069</td>\n<td>-0.0040</td>\n</tr>\n<tr>\n<td>4</td>\n<td>128x384x384</td>\n<td>No</td>\n<td>❌</td>\n<td>✅</td>\n<td>0.7479</td>\n<td>0.7415</td>\n<td>+0.0098</td>\n<td>+0.0125</td>\n</tr>\n<tr>\n<td>5</td>\n<td>184x512x512</td>\n<td>No</td>\n<td>✅</td>\n<td>❌</td>\n<td>0.7336</td>\n<td>0.7308</td>\n<td>-0.0045</td>\n<td>+0.0018</td>\n</tr>\n<tr>\n<td>6</td>\n<td>128x384x384</td>\n<td>CosineAnnealingLR</td>\n<td>✅</td>\n<td>✅</td>\n<td>0.7228</td>\n<td>0.7169</td>\n<td>-0.0153</td>\n<td>-0.0121</td>\n</tr>\n<tr>\n<td>7</td>\n<td>128x384x384</td>\n<td>No</td>\n<td>✅</td>\n<td>✅</td>\n<td>0.7543</td>\n<td>0.7460</td>\n<td>+0.0315</td>\n<td>+0.0291</td>\n</tr>\n<tr>\n<td>8</td>\n<td>184x512x512</td>\n<td>No</td>\n<td>✅</td>\n<td>✅</td>\n<td><strong>0.7642</strong></td>\n<td><strong>0.7562</strong></td>\n<td><strong>+0.0261</strong></td>\n<td><strong>+0.0272</strong></td>\n</tr>\n</tbody>\n</table>\n<p>*Multi-tomo: Use denoised, wbp, ctfdeconvolved, isonetcorrected.❌ means only denoised</p>\n<ul>\n<li><p>Lightweigh model achieve higher scores.  </p></li>\n<li><p>Large infer size and multi-tomo type was effective.  </p></li>\n<li><p>EMA alone did not work well, but it became effective when used together with multi-tomo type.</p></li>\n<li><p>The model performed better without a scheduler.</p>\n<ul>\n<li>Just replacing CosineAnnealingLR with No scheduler improved the score by 0.03.  </li>\n<li>Only \"CosineAnnealingLR\" was tested, and other schedulers were not explored. </li></ul>\n<table>\n<thead>\n<tr>\n<th>Scheduler</th>\n<th>Infer Size</th>\n<th>EMA</th>\n<th>Multi-tomo</th>\n<th>Public</th>\n<th>Private</th>\n<th>ΔPublic</th>\n<th>ΔPrivate</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><strong>CosineAnnealingLR</strong></td>\n<td>128x384x384</td>\n<td>✅</td>\n<td>✅</td>\n<td>0.7228</td>\n<td>0.7169</td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td><strong>No</strong></td>\n<td>128x384x384</td>\n<td>✅</td>\n<td>✅</td>\n<td><strong>0.7543</strong></td>\n<td><strong>0.7460</strong></td>\n<td><strong>+0.0315</strong></td>\n<td><strong>+0.0291</strong></td>\n</tr>\n</tbody>\n</table></li>\n</ul>\n<h3><strong>Summary</strong></h3>\n<ul>\n<li><p><strong>Large inference size consistently contributed to score improvements.</strong>  </p>\n<ul>\n<li>Both (A) and (B) showed a performance boost with larger inference sizes.  </li></ul></li>\n<li><p><strong>EMA was not always effective, but generally led to better scores.</strong>  </p>\n<ul>\n<li>In (A), EMA alone contributed to performance improvement.  </li>\n<li>In (B), EMA alone was ineffective, but combining it with multi-tomo types led to the best results.  </li>\n<li>I was unable to make EMA work effectively on 2D-3D U-Net (Backbone: SEResNeXt26TN_32x4d, CoAtNet) during the competition. It might be more suitable for simple models.  </li></ul></li>\n<li><p><strong>Lightweight models performed better.</strong>  </p>\n<ul>\n<li>(B), which had a smaller channel size, outperformed (A), suggesting that a more efficient architecture was beneficial.  </li></ul></li>\n<li><p><strong>Multi-tomo type usage was conditionally effective.</strong>  </p>\n<ul>\n<li>In (A), adding multi-tomo did not contribute to score improvements.  </li>\n<li>In (B), using multi-tomo <strong>with EMA</strong> significantly improved the score.  </li></ul></li>\n</ul>\n<h4><strong>Best Performing</strong></h4>\n<ul>\n<li><strong>(B) 2nd place-based U-Net</strong></li>\n<li><strong>Infer Size: 184x512x512</strong></li>\n<li><strong>EMA + Multi-Tomo</strong></li>\n<li><strong>Achieved Public: 0.7642 / Private: 0.7562</strong></li>\n</ul>\n<p>For more detailed experiment logs, you can refer to the <strong>CZII_2024_latesub_results.xlsx</strong>, where all experiments and results are recorded.</p>\n<h3><strong>(Additional) TTA &amp; Ensemble Results</strong></h3>\n<p>For the (B) setting,  </p>\n<ul>\n<li>Ensembling 2 models with different validation sets (TS_6_4 and TS_69_2) using TTA(90,180,270 rot and xyz flipe) achieved Public: 0.7734 / Private: 0.7632.</li>\n</ul>",
  "messages": [
    {
      "id": 3125650,
      "postDate": "2025-02-16T13:01:59.280Z",
      "content": "<h1>1. 77th Place Solution</h1>\n<ul>\n<li><strong>Data:</strong> Hard label (radius × 0.7)  </li>\n<li><strong>Model:</strong> Ensemble of 2D-3D U-Net (Backbone: SEResNeXt26TN_32x4d, CoAtNet) &amp; YOLO (from <a href=\"https://www.kaggle.com/code/sersasj/czii-yolo11-submission-baseline-with-kdtree-update\" target=\"_blank\">public notebook</a>  )  </li>\n<li><strong>Training:</strong> U-Net pre-trained on simulated data -&gt; Fine-tuned on competition data</li>\n</ul>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>Data</th>\n<th>Train Size (z,x,y)</th>\n<th>Infer Size (z,x,y)</th>\n<th>Optimizer</th>\n<th>Learning Rate</th>\n<th>Scheduler</th>\n<th>Loss</th>\n<th>Valid</th>\n<th>TTA</th>\n<th>Public</th>\n<th>Private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>2d-3d U-Net (seresnext26tn_32x4d)</td>\n<td>Denoised only</td>\n<td>128x128x128</td>\n<td>128x128x128</td>\n<td>AdamW</td>\n<td>1e-3</td>\n<td>CosineAnnealingLR</td>\n<td>DiceLoss</td>\n<td>TS_69_2</td>\n<td>〇</td>\n<td>0.7134</td>\n<td>0.7060</td>\n</tr>\n<tr>\n<td>2d-3d U-Net (coatnet)</td>\n<td>Denoised only</td>\n<td>128x128x128</td>\n<td>128x128x128</td>\n<td>AdamW</td>\n<td>1e-3</td>\n<td>CosineAnnealingLR</td>\n<td>DiceLoss</td>\n<td>TS_69_2</td>\n<td>〇</td>\n<td>0.7032</td>\n<td>0.7016</td>\n</tr>\n<tr>\n<td>Ensemble (2d-3d U-Net×2, YOLO)</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>0.7373</td>\n<td>0.7264</td>\n</tr>\n</tbody>\n</table>\n<h1>2. Latesub results</h1>\n<p>I share the latesub experiments based on <strong>2th and 8th place solutions</strong>.   (Many thanks to the authors of these solutions for their valuable contributions!)  <br>\nMy main focus is <strong>single model results</strong> (without TTA or ensemble).  </p>\n<p>The experiments are categorized into two settings:</p>\n<ul>\n<li>(A): <a href=\"https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/561515\" target=\"_blank\">8th Place</a> based U-Net  </li>\n<li>(B): <a href=\"https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/561568\" target=\"_blank\">2th Place</a> based U-Net</li>\n</ul>\n<p>Below are the details and results of these experiments.</p>\n<h2><strong>2-1. Dataset and Model Config</strong></h2>\n<h3><strong>Dataset</strong></h3>\n<ul>\n<li><strong>Data:</strong> Hard label (radius × 0.5), only competition data used.</li>\n<li><strong>Valid ID:</strong> TS_69_2</li>\n</ul>\n<h3><strong>Model</strong></h3>\n<table>\n<thead>\n<tr>\n<th>Category</th>\n<th>Parameter</th>\n<th>(A) <a href=\"https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/561515\" target=\"_blank\">8th Place</a> based U-Net</th>\n<th>(B): <a href=\"https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/561568\" target=\"_blank\">2th Place</a> based U-Net</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><strong>Model</strong></td>\n<td>Architecture</td>\n<td>monai 3dU-Net</td>\n<td>monai 3dU-Net</td>\n</tr>\n<tr>\n<td></td>\n<td>Channels</td>\n<td><strong>(32, 64, 128, 256)</strong></td>\n<td><strong>(48, 64, 80, 80)</strong></td>\n</tr>\n<tr>\n<td></td>\n<td>Strides</td>\n<td>(2, 2, 1)</td>\n<td>(2, 2, 1)</td>\n</tr>\n<tr>\n<td></td>\n<td>Num Residual Units</td>\n<td>2</td>\n<td>2</td>\n</tr>\n<tr>\n<td><strong>Training</strong></td>\n<td>Train Size</td>\n<td><strong>128x128x128</strong></td>\n<td><strong>128x384x384</strong></td>\n</tr>\n<tr>\n<td></td>\n<td>Loss Function</td>\n<td><strong>DiceLoss</strong></td>\n<td><strong>Tversky+CE</strong></td>\n</tr>\n<tr>\n<td></td>\n<td>Optimizer</td>\n<td>AdamW</td>\n<td>AdamW</td>\n</tr>\n<tr>\n<td></td>\n<td>Learning Rate</td>\n<td>1e-3</td>\n<td>1e-3</td>\n</tr>\n<tr>\n<td></td>\n<td>Scheduler</td>\n<td><strong>CosineAnnealingLR</strong></td>\n<td><strong>No</strong></td>\n</tr>\n</tbody>\n</table>\n<h2><strong>2-2. Results</strong></h2>\n<h3><strong>(A) 8th Place based U-Net</strong></h3>\n<table>\n<thead>\n<tr>\n<th>No.</th>\n<th>Infer Size</th>\n<th>EMA</th>\n<th>Multi-tomo</th>\n<th>Public</th>\n<th>Private</th>\n<th>ΔPublic</th>\n<th>ΔPrivate</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>128x128x128</td>\n<td>❌</td>\n<td>❌</td>\n<td>0.6810</td>\n<td>0.6769</td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td>2</td>\n<td>184x512x512</td>\n<td>❌</td>\n<td>❌</td>\n<td>0.6976</td>\n<td>0.6969</td>\n<td>+0.0166</td>\n<td>+0.0200</td>\n</tr>\n<tr>\n<td>3</td>\n<td>128x128x128</td>\n<td>✅</td>\n<td>❌</td>\n<td>0.7136</td>\n<td>0.7093</td>\n<td>+0.0326</td>\n<td>+0.0323</td>\n</tr>\n<tr>\n<td>4</td>\n<td>184x512x512</td>\n<td>✅</td>\n<td>❌</td>\n<td><strong>0.7410</strong></td>\n<td><strong>0.7357</strong></td>\n<td><strong>+0.0600</strong></td>\n<td><strong>+0.0588</strong></td>\n</tr>\n<tr>\n<td>5</td>\n<td>184x512x512</td>\n<td>✅</td>\n<td>✅</td>\n<td>0.7324</td>\n<td>0.7305</td>\n<td>+0.0514</td>\n<td>+0.0536</td>\n</tr>\n</tbody>\n</table>\n<p>*Multi-tomo: Use denoised, wbp, ctfdeconvolved, isonetcorrected.❌ means only denoised</p>\n<ul>\n<li>Large infer size &amp; EMA improved score by <strong>+0.06</strong>.  </li>\n<li>The use of multi-tomo types was not effective.  </li>\n</ul>\n<h3><strong>(B) 2nd Place based U-Net</strong></h3>\n<table>\n<thead>\n<tr>\n<th>No.</th>\n<th>Infer Size</th>\n<th>LR Scheduler</th>\n<th>EMA</th>\n<th>Multi-tomo</th>\n<th>Public</th>\n<th>Private</th>\n<th>ΔPublic</th>\n<th>ΔPrivate</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>128x384x384</td>\n<td>No</td>\n<td>❌</td>\n<td>❌</td>\n<td>0.7381</td>\n<td>0.7290</td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td>2</td>\n<td>184x512x512</td>\n<td>No</td>\n<td>❌</td>\n<td>❌</td>\n<td>0.7467</td>\n<td>0.7423</td>\n<td>+0.0086</td>\n<td>+0.0133</td>\n</tr>\n<tr>\n<td>3</td>\n<td>128x384x384</td>\n<td>No</td>\n<td>✅</td>\n<td>❌</td>\n<td>0.7312</td>\n<td>0.7251</td>\n<td>-0.0069</td>\n<td>-0.0040</td>\n</tr>\n<tr>\n<td>4</td>\n<td>128x384x384</td>\n<td>No</td>\n<td>❌</td>\n<td>✅</td>\n<td>0.7479</td>\n<td>0.7415</td>\n<td>+0.0098</td>\n<td>+0.0125</td>\n</tr>\n<tr>\n<td>5</td>\n<td>184x512x512</td>\n<td>No</td>\n<td>✅</td>\n<td>❌</td>\n<td>0.7336</td>\n<td>0.7308</td>\n<td>-0.0045</td>\n<td>+0.0018</td>\n</tr>\n<tr>\n<td>6</td>\n<td>128x384x384</td>\n<td>CosineAnnealingLR</td>\n<td>✅</td>\n<td>✅</td>\n<td>0.7228</td>\n<td>0.7169</td>\n<td>-0.0153</td>\n<td>-0.0121</td>\n</tr>\n<tr>\n<td>7</td>\n<td>128x384x384</td>\n<td>No</td>\n<td>✅</td>\n<td>✅</td>\n<td>0.7543</td>\n<td>0.7460</td>\n<td>+0.0315</td>\n<td>+0.0291</td>\n</tr>\n<tr>\n<td>8</td>\n<td>184x512x512</td>\n<td>No</td>\n<td>✅</td>\n<td>✅</td>\n<td><strong>0.7642</strong></td>\n<td><strong>0.7562</strong></td>\n<td><strong>+0.0261</strong></td>\n<td><strong>+0.0272</strong></td>\n</tr>\n</tbody>\n</table>\n<p>*Multi-tomo: Use denoised, wbp, ctfdeconvolved, isonetcorrected.❌ means only denoised</p>\n<ul>\n<li><p>Lightweigh model achieve higher scores.  </p></li>\n<li><p>Large infer size and multi-tomo type was effective.  </p></li>\n<li><p>EMA alone did not work well, but it became effective when used together with multi-tomo type.</p></li>\n<li><p>The model performed better without a scheduler.</p>\n<ul>\n<li>Just replacing CosineAnnealingLR with No scheduler improved the score by 0.03.  </li>\n<li>Only \"CosineAnnealingLR\" was tested, and other schedulers were not explored. </li></ul>\n<table>\n<thead>\n<tr>\n<th>Scheduler</th>\n<th>Infer Size</th>\n<th>EMA</th>\n<th>Multi-tomo</th>\n<th>Public</th>\n<th>Private</th>\n<th>ΔPublic</th>\n<th>ΔPrivate</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><strong>CosineAnnealingLR</strong></td>\n<td>128x384x384</td>\n<td>✅</td>\n<td>✅</td>\n<td>0.7228</td>\n<td>0.7169</td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td><strong>No</strong></td>\n<td>128x384x384</td>\n<td>✅</td>\n<td>✅</td>\n<td><strong>0.7543</strong></td>\n<td><strong>0.7460</strong></td>\n<td><strong>+0.0315</strong></td>\n<td><strong>+0.0291</strong></td>\n</tr>\n</tbody>\n</table></li>\n</ul>\n<h3><strong>Summary</strong></h3>\n<ul>\n<li><p><strong>Large inference size consistently contributed to score improvements.</strong>  </p>\n<ul>\n<li>Both (A) and (B) showed a performance boost with larger inference sizes.  </li></ul></li>\n<li><p><strong>EMA was not always effective, but generally led to better scores.</strong>  </p>\n<ul>\n<li>In (A), EMA alone contributed to performance improvement.  </li>\n<li>In (B), EMA alone was ineffective, but combining it with multi-tomo types led to the best results.  </li>\n<li>I was unable to make EMA work effectively on 2D-3D U-Net (Backbone: SEResNeXt26TN_32x4d, CoAtNet) during the competition. It might be more suitable for simple models.  </li></ul></li>\n<li><p><strong>Lightweight models performed better.</strong>  </p>\n<ul>\n<li>(B), which had a smaller channel size, outperformed (A), suggesting that a more efficient architecture was beneficial.  </li></ul></li>\n<li><p><strong>Multi-tomo type usage was conditionally effective.</strong>  </p>\n<ul>\n<li>In (A), adding multi-tomo did not contribute to score improvements.  </li>\n<li>In (B), using multi-tomo <strong>with EMA</strong> significantly improved the score.  </li></ul></li>\n</ul>\n<h4><strong>Best Performing</strong></h4>\n<ul>\n<li><strong>(B) 2nd place-based U-Net</strong></li>\n<li><strong>Infer Size: 184x512x512</strong></li>\n<li><strong>EMA + Multi-Tomo</strong></li>\n<li><strong>Achieved Public: 0.7642 / Private: 0.7562</strong></li>\n</ul>\n<p>For more detailed experiment logs, you can refer to the <strong>CZII_2024_latesub_results.xlsx</strong>, where all experiments and results are recorded.</p>\n<h3><strong>(Additional) TTA &amp; Ensemble Results</strong></h3>\n<p>For the (B) setting,  </p>\n<ul>\n<li>Ensembling 2 models with different validation sets (TS_6_4 and TS_69_2) using TTA(90,180,270 rot and xyz flipe) achieved Public: 0.7734 / Private: 0.7632.</li>\n</ul>",
      "rawMarkdown": "# 1. 77th Place Solution\n\n- **Data:** Hard label (radius × 0.7)  \n- **Model:** Ensemble of 2D-3D U-Net (Backbone: SEResNeXt26TN_32x4d, CoAtNet) & YOLO (from [public notebook](https://www.kaggle.com/code/sersasj/czii-yolo11-submission-baseline-with-kdtree-update)  )  \n- **Training:** U-Net pre-trained on simulated data -> Fine-tuned on competition data\n\n| Model                                | Data          | Train Size (z,x,y) | Infer Size (z,x,y) | Optimizer | Learning Rate | Scheduler          | Loss     | Valid   | TTA | Public | Private |\n|----|-----|----|---|----|---|---|---|---|---|---|---|\n| 2d-3d U-Net (seresnext26tn_32x4d)   | Denoised only | 128x128x128     | 128x128x128     | AdamW     | 1e-3      | CosineAnnealingLR | DiceLoss | TS_69_2 | 〇  | 0.7134 | 0.7060 |\n| 2d-3d U-Net (coatnet)               | Denoised only | 128x128x128     | 128x128x128     | AdamW     | 1e-3      | CosineAnnealingLR | DiceLoss | TS_69_2 |〇  | 0.7032 | 0.7016 |\n| Ensemble (2d-3d U-Net×2, YOLO)      | -            | -               | -               | -         | -             | -                  | -        | -       | -   | 0.7373 | 0.7264 |\n\n\n# 2. Latesub results\n\nI share the latesub experiments based on **2th and 8th place solutions**.   (Many thanks to the authors of these solutions for their valuable contributions!)  \nMy main focus is **single model results** (without TTA or ensemble).  \n\nThe experiments are categorized into two settings:\n- (A): [8th Place](https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/561515) based U-Net  \n- (B): [2th Place](https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/561568) based U-Net\n\nBelow are the details and results of these experiments.\n\n## **2-1. Dataset and Model Config**\n\n### **Dataset**\n\n- **Data:** Hard label (radius × 0.5), only competition data used.\n- **Valid ID:** TS_69_2\n\n### **Model**\n\n| Category        | Parameter            | (A) [8th Place](https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/561515) based U-Net                      | (B): [2th Place](https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/561568) based U-Net |\n|----------------|----------------------|---------------------------|---------------------------|\n| **Model**      | Architecture         | monai 3dU-Net             | monai 3dU-Net |  \n|                | Channels             | **(32, 64, 128, 256)**    | **(48, 64, 80, 80)** |\n|                | Strides              | (2, 2, 1)                 | (2, 2, 1) |\n|                | Num Residual Units   | 2                         | 2 |\n| **Training**   | Train Size           | **128x128x128**           | **128x384x384** |\n|                | Loss Function        | **DiceLoss**              | **Tversky+CE** |\n|                | Optimizer            | AdamW                     | AdamW |\n|                | Learning Rate        | 1e-3                      | 1e-3 |\n|                | Scheduler            | **CosineAnnealingLR**     | **No** |\n\n## **2-2. Results**\n\n### **(A) 8th Place based U-Net**\n\n| No. | Infer Size  | EMA | Multi-tomo | Public | Private | ΔPublic | ΔPrivate | \n| --- |----------------|:--------:|:-------:|--------|--------|--------|--------| \n|  1  | 128x128x128   | ❌  | ❌  | 0.6810 | 0.6769 |  |  |\n|  2  | 184x512x512   | ❌  | ❌  | 0.6976 | 0.6969 | +0.0166 | +0.0200 | \n|  3  | 128x128x128   | ✅  | ❌  | 0.7136 | 0.7093 | +0.0326 | +0.0323 | \n|  4  | 184x512x512   | ✅  | ❌  | **0.7410** | **0.7357** | **+0.0600** | **+0.0588** | \n|  5  | 184x512x512   | ✅  | ✅  | 0.7324 | 0.7305 | +0.0514 | +0.0536 | \n\n*Multi-tomo: Use denoised, wbp, ctfdeconvolved, isonetcorrected.❌ means only denoised\n\n- Large infer size & EMA improved score by **+0.06**.  \n- The use of multi-tomo types was not effective.  \n\n\n### **(B) 2nd Place based U-Net**\n\n| No. | Infer Size | LR Scheduler | EMA | Multi-tomo | Public | Private | ΔPublic | ΔPrivate |\n| --- |------------|-----------|-----|------------|--------|--------|--------|--------|\n|  1 | 128x384x384 | No  | ❌  | ❌  | 0.7381 | 0.7290 |  |  |\n|  2 | 184x512x512 | No  | ❌  | ❌  | 0.7467 | 0.7423 | +0.0086 | +0.0133 |\n|  3 | 128x384x384 | No  | ✅  | ❌  | 0.7312 | 0.7251 | -0.0069 | -0.0040 |\n|  4 | 128x384x384 | No  | ❌  | ✅  | 0.7479 | 0.7415 | +0.0098 | +0.0125 |\n|  5 | 184x512x512 | No  | ✅  | ❌  | 0.7336 | 0.7308 | -0.0045 | +0.0018 |\n|  6 | 128x384x384| CosineAnnealingLR  | ✅  | ✅  | 0.7228 | 0.7169 | -0.0153 | -0.0121 |\n|  7 | 128x384x384| No   | ✅  | ✅  | 0.7543 | 0.7460 | +0.0315 | +0.0291 |\n|  8 | 184x512x512 | No  | ✅  | ✅  | **0.7642** | **0.7562** | **+0.0261** | **+0.0272** |\n\n*Multi-tomo: Use denoised, wbp, ctfdeconvolved, isonetcorrected.❌ means only denoised\n\n\n- Lightweigh model achieve higher scores.  \n- Large infer size and multi-tomo type was effective.  \n- EMA alone did not work well, but it became effective when used together with multi-tomo type.\n- The model performed better without a scheduler.\n    - Just replacing CosineAnnealingLR with No scheduler improved the score by 0.03.  \n    - Only \"CosineAnnealingLR\" was tested, and other schedulers were not explored. \n\n    | Scheduler |Infer Size  | EMA | Multi-tomo| Public | Private | ΔPublic | ΔPrivate |\n    | --- |--- |--- |--- |--- |--- |--- |--- |\n    | **CosineAnnealingLR** | 128x384x384 | ✅  | ✅  | 0.7228 | 0.7169 |  |  |\n    | **No** | 128x384x384 | ✅  | ✅  | **0.7543** | **0.7460** | **+0.0315** | **+0.0291** |\n\n\n### **Summary**\n\n- **Large inference size consistently contributed to score improvements.**  \n  - Both (A) and (B) showed a performance boost with larger inference sizes.  \n\n- **EMA was not always effective, but generally led to better scores.**  \n  - In (A), EMA alone contributed to performance improvement.  \n  - In (B), EMA alone was ineffective, but combining it with multi-tomo types led to the best results.  \n  - I was unable to make EMA work effectively on 2D-3D U-Net (Backbone: SEResNeXt26TN_32x4d, CoAtNet) during the competition. It might be more suitable for simple models.  \n\n- **Lightweight models performed better.**  \n  - (B), which had a smaller channel size, outperformed (A), suggesting that a more efficient architecture was beneficial.  \n\n- **Multi-tomo type usage was conditionally effective.**  \n  - In (A), adding multi-tomo did not contribute to score improvements.  \n  - In (B), using multi-tomo **with EMA** significantly improved the score.  \n\n#### **Best Performing**\n- **(B) 2nd place-based U-Net**\n- **Infer Size: 184x512x512**\n- **EMA + Multi-Tomo**\n- **Achieved Public: 0.7642 / Private: 0.7562**\n\nFor more detailed experiment logs, you can refer to the **CZII_2024_latesub_results.xlsx**, where all experiments and results are recorded.\n\n### **(Additional) TTA & Ensemble Results**\n\nFor the (B) setting,  \n- Ensembling 2 models with different validation sets (TS_6_4 and TS_69_2) using TTA(90,180,270 rot and xyz flipe) achieved Public: 0.7734 / Private: 0.7632.\n",
      "votes": 8
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "3125650": "# 1. 77th Place Solution\n\n- **Data:** Hard label (radius × 0.7)  \n- **Model:** Ensemble of 2D-3D U-Net (Backbone: SEResNeXt26TN_32x4d, CoAtNet) & YOLO (from [public notebook](https://www.kaggle.com/code/sersasj/czii-yolo11-submission-baseline-with-kdtree-update)  )  \n- **Training:** U-Net pre-trained on simulated data -> Fine-tuned on competition data\n\n| Model                                | Data          | Train Size (z,x,y) | Infer Size (z,x,y) | Optimizer | Learning Rate | Scheduler          | Loss     | Valid   | TTA | Public | Private |\n|----|-----|----|---|----|---|---|---|---|---|---|---|\n| 2d-3d U-Net (seresnext26tn_32x4d)   | Denoised only | 128x128x128     | 128x128x128     | AdamW     | 1e-3      | CosineAnnealingLR | DiceLoss | TS_69_2 | 〇  | 0.7134 | 0.7060 |\n| 2d-3d U-Net (coatnet)               | Denoised only | 128x128x128     | 128x128x128     | AdamW     | 1e-3      | CosineAnnealingLR | DiceLoss | TS_69_2 |〇  | 0.7032 | 0.7016 |\n| Ensemble (2d-3d U-Net×2, YOLO)      | -            | -               | -               | -         | -             | -                  | -        | -       | -   | 0.7373 | 0.7264 |\n\n\n# 2. Latesub results\n\nI share the latesub experiments based on **2th and 8th place solutions**.   (Many thanks to the authors of these solutions for their valuable contributions!)  \nMy main focus is **single model results** (without TTA or ensemble).  \n\nThe experiments are categorized into two settings:\n- (A): [8th Place](https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/561515) based U-Net  \n- (B): [2th Place](https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/561568) based U-Net\n\nBelow are the details and results of these experiments.\n\n## **2-1. Dataset and Model Config**\n\n### **Dataset**\n\n- **Data:** Hard label (radius × 0.5), only competition data used.\n- **Valid ID:** TS_69_2\n\n### **Model**\n\n| Category        | Parameter            | (A) [8th Place](https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/561515) based U-Net                      | (B): [2th Place](https://www.kaggle.com/competitions/czii-cryo-et-object-identification/discussion/561568) based U-Net |\n|----------------|----------------------|---------------------------|---------------------------|\n| **Model**      | Architecture         | monai 3dU-Net             | monai 3dU-Net |  \n|                | Channels             | **(32, 64, 128, 256)**    | **(48, 64, 80, 80)** |\n|                | Strides              | (2, 2, 1)                 | (2, 2, 1) |\n|                | Num Residual Units   | 2                         | 2 |\n| **Training**   | Train Size           | **128x128x128**           | **128x384x384** |\n|                | Loss Function        | **DiceLoss**              | **Tversky+CE** |\n|                | Optimizer            | AdamW                     | AdamW |\n|                | Learning Rate        | 1e-3                      | 1e-3 |\n|                | Scheduler            | **CosineAnnealingLR**     | **No** |\n\n## **2-2. Results**\n\n### **(A) 8th Place based U-Net**\n\n| No. | Infer Size  | EMA | Multi-tomo | Public | Private | ΔPublic | ΔPrivate | \n| --- |----------------|:--------:|:-------:|--------|--------|--------|--------| \n|  1  | 128x128x128   | ❌  | ❌  | 0.6810 | 0.6769 |  |  |\n|  2  | 184x512x512   | ❌  | ❌  | 0.6976 | 0.6969 | +0.0166 | +0.0200 | \n|  3  | 128x128x128   | ✅  | ❌  | 0.7136 | 0.7093 | +0.0326 | +0.0323 | \n|  4  | 184x512x512   | ✅  | ❌  | **0.7410** | **0.7357** | **+0.0600** | **+0.0588** | \n|  5  | 184x512x512   | ✅  | ✅  | 0.7324 | 0.7305 | +0.0514 | +0.0536 | \n\n*Multi-tomo: Use denoised, wbp, ctfdeconvolved, isonetcorrected.❌ means only denoised\n\n- Large infer size & EMA improved score by **+0.06**.  \n- The use of multi-tomo types was not effective.  \n\n\n### **(B) 2nd Place based U-Net**\n\n| No. | Infer Size | LR Scheduler | EMA | Multi-tomo | Public | Private | ΔPublic | ΔPrivate |\n| --- |------------|-----------|-----|------------|--------|--------|--------|--------|\n|  1 | 128x384x384 | No  | ❌  | ❌  | 0.7381 | 0.7290 |  |  |\n|  2 | 184x512x512 | No  | ❌  | ❌  | 0.7467 | 0.7423 | +0.0086 | +0.0133 |\n|  3 | 128x384x384 | No  | ✅  | ❌  | 0.7312 | 0.7251 | -0.0069 | -0.0040 |\n|  4 | 128x384x384 | No  | ❌  | ✅  | 0.7479 | 0.7415 | +0.0098 | +0.0125 |\n|  5 | 184x512x512 | No  | ✅  | ❌  | 0.7336 | 0.7308 | -0.0045 | +0.0018 |\n|  6 | 128x384x384| CosineAnnealingLR  | ✅  | ✅  | 0.7228 | 0.7169 | -0.0153 | -0.0121 |\n|  7 | 128x384x384| No   | ✅  | ✅  | 0.7543 | 0.7460 | +0.0315 | +0.0291 |\n|  8 | 184x512x512 | No  | ✅  | ✅  | **0.7642** | **0.7562** | **+0.0261** | **+0.0272** |\n\n*Multi-tomo: Use denoised, wbp, ctfdeconvolved, isonetcorrected.❌ means only denoised\n\n\n- Lightweigh model achieve higher scores.  \n- Large infer size and multi-tomo type was effective.  \n- EMA alone did not work well, but it became effective when used together with multi-tomo type.\n- The model performed better without a scheduler.\n    - Just replacing CosineAnnealingLR with No scheduler improved the score by 0.03.  \n    - Only \"CosineAnnealingLR\" was tested, and other schedulers were not explored. \n\n    | Scheduler |Infer Size  | EMA | Multi-tomo| Public | Private | ΔPublic | ΔPrivate |\n    | --- |--- |--- |--- |--- |--- |--- |--- |\n    | **CosineAnnealingLR** | 128x384x384 | ✅  | ✅  | 0.7228 | 0.7169 |  |  |\n    | **No** | 128x384x384 | ✅  | ✅  | **0.7543** | **0.7460** | **+0.0315** | **+0.0291** |\n\n\n### **Summary**\n\n- **Large inference size consistently contributed to score improvements.**  \n  - Both (A) and (B) showed a performance boost with larger inference sizes.  \n\n- **EMA was not always effective, but generally led to better scores.**  \n  - In (A), EMA alone contributed to performance improvement.  \n  - In (B), EMA alone was ineffective, but combining it with multi-tomo types led to the best results.  \n  - I was unable to make EMA work effectively on 2D-3D U-Net (Backbone: SEResNeXt26TN_32x4d, CoAtNet) during the competition. It might be more suitable for simple models.  \n\n- **Lightweight models performed better.**  \n  - (B), which had a smaller channel size, outperformed (A), suggesting that a more efficient architecture was beneficial.  \n\n- **Multi-tomo type usage was conditionally effective.**  \n  - In (A), adding multi-tomo did not contribute to score improvements.  \n  - In (B), using multi-tomo **with EMA** significantly improved the score.  \n\n#### **Best Performing**\n- **(B) 2nd place-based U-Net**\n- **Infer Size: 184x512x512**\n- **EMA + Multi-Tomo**\n- **Achieved Public: 0.7642 / Private: 0.7562**\n\nFor more detailed experiment logs, you can refer to the **CZII_2024_latesub_results.xlsx**, where all experiments and results are recorded.\n\n### **(Additional) TTA & Ensemble Results**\n\nFor the (B) setting,  \n- Ensembling 2 models with different validation sets (TS_6_4 and TS_69_2) using TTA(90,180,270 rot and xyz flipe) achieved Public: 0.7734 / Private: 0.7632.\n"
  }
}