![\"t-SNE](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2930242%2Fc3849585c995f9ec2e0e597b80cfcfd2%2F00_tsne_patterns.png?generation=1772324957059654&alt=media\")
I ran a few experiments based on my hypothesis, and I'd like to discuss the results.\nMy experimental setup may have some flaws or odd choices, but I hope this can spark a useful discussion.
\nSurveying the top solutions of Vesuvius Challenge 2, we found that all top teams employed 2,000-8,000 epochs of training (1st: 4,000ep, 3rd: 8,000ep, 12th: 2,000-4,000ep). We wanted to understand why such long training was effective.
\nIn our experiments, after training on all data (786 samples), we ran inference on the training set itself and found that predictions were significantly worse on low-quality images (high brightness, blur, low contrast). This led us to investigate the data distribution more closely.
\nHypothesis: Difficult data patterns (Bright, Blur, etc.) converge more slowly, requiring many more epochs to learn adequately.
\nTo investigate the source of the performance gap, we visualized the feature space of all 786 samples. From each 3D volume, we extracted the central 5 slices along XY, XZ, and YZ planes, fed them through an ImageNet-pretrained ConvNeXt-Tiny (timm) to obtain feature vectors, and averaged across the 3 views (768 dimensions). We then applied t-SNE (perplexity=30, PCA initialization) and clustered with KMeans (k=8).
\nThe result showed clear separation by image quality pattern:
\n
High-brightness samples (A:Bright) cluster in the bottom-right, blurry/low-contrast samples (B:Blur, C:LowContrast) in the bottom-left, and Normal data spreads across the upper-center region. This separation aligns strongly with image quality metrics (sharpness, contrast, mean intensity), and samples with lower model performance tend to belong to the peripheral clusters.
\nTo test this hypothesis, we fine-tuned a pre-trained model (Phase 1, 200 epochs) on the full Phase 2 dataset (786 samples) for only 100 epochs, recording per-sample loss at every epoch. The 100-epoch run was designed as a measurement-oriented short run to observe early-stage training dynamics (production training used 500-4,000 epochs).
\nWe used a checkpoint pre-trained for 200 epochs on Phase 1 data (clean labels, no ignore regions). Pre-training accelerates initial convergence.
\n| Setting | \nValue | \n
|---|---|
| Phase 1 Data | \nDataset120_Phase1 | \n
| Phase 1 Epochs | \n200 | \n
| Phase 1 Trainer | \nnnUNetTrainerSkelRecall_200ep | \n
| Phase 1 Loss | \nCE + Dice + SkelRecall | \n
| Phase 1 LR | \n0.005 (cosine annealing) | \n
| Setting | \nValue | \n
|---|---|
| Base Model | \nnnUNetTrainerSkelRecall (CE + Dice + SkelRecall) | \n
| Pre-trained Weights | \nPhase 1 200ep checkpoint_final.pth | \n
| Epochs | \n100 (measurement run; production: 500-4,000ep) | \n
| Fold | \nall (all 786 samples used for training) | \n
| Patch Size | \n128 x 160 x 160 | \n
| Batch Size | \n5 | \n
| LR | \n0.005 (cosine annealing) | \n
| Loss Logging | \nPer-sample loss recorded every epoch | \n
| Clustering | \nt-SNE + KMeans (k=8) | \n
We used nnU-Net defaults plus domain-specific custom augmentations:
\nSpatial transforms:
\nIntensity transforms:
\nCustom transforms (papyrus-specific):
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2930242%2F7588172f89a612044ba9115c0731d4e2%2F01_loss_by_pattern.png?generation=1772325014361578&alt=media\")
| Pattern | \nN | \nInitial Loss | \nFinal Loss | \nImprovement | \n90% Convergence Epoch | \n
|---|---|---|---|---|---|
| Normal | \n545 | \n-0.883 | \n-1.111 | \n0.228 | \n60 | \n
| A:Bright | \n132 | \n-0.407 | \n-0.602 | \n0.195 | \n33 | \n
| B:Blur | \n39 | \n-0.662 | \n-0.815 | \n0.153 | \n54 | \n
| C:LowContrast | \n58 | \n-0.605 | \n-0.769 | \n0.164 | \n96 | \n
| B-:MildBlur | \n12 | \n-0.833 | \n-1.163 | \n0.330 | \n25 | \n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2930242%2F4238e7c0c0f3c354cb489d420a7f30e2%2F02_loss_by_cluster.png?generation=1772325035027860&alt=media\")
| Cluster | \nN | \nInitial Loss | \nFinal Loss | \nImprovement | \nDominant Pattern | \n
|---|---|---|---|---|---|
| C0 (Normal) | \n101 | \n-0.947 | \n-1.118 | \n0.171 | \nNormal:101 | \n
| C1 (Normal) | \n87 | \n-0.793 | \n-1.013 | \n0.220 | \nNormal:86 | \n
| C2 (Normal) | \n85 | \n-0.958 | \n-1.254 | \n0.296 | \nNormal:70, LowContrast:15 | \n
| C3 (Normal) | \n105 | \n-0.814 | \n-1.024 | \n0.210 | \nNormal:93, Blur:8 | \n
| C4 (Normal) | \n97 | \n-0.864 | \n-1.105 | \n0.241 | \nNormal:96 | \n
| C5 (Mixed) | \n100 | \n-0.657 | \n-0.881 | \n0.224 | \nNormal:58, Bright:34 | \n
| C6 (Bright) | \n101 | \n-0.382 | \n-0.586 | \n0.204 | \nA:Bright:97 | \n
| C7 (Mixed) | \n110 | \n-0.783 | \n-0.961 | \n0.178 | \nLowContrast:38, Normal:37, Blur:31 | \n
| Metric | \nValue | \n
|---|---|
| Overall Initial Loss | \n-0.771 | \n
| Overall Final Loss | \n-0.986 | \n
| Overall Improvement | \n0.216 | \n
| Worst Cluster (C6) Final Loss | \n-0.586 | \n
| Best Cluster (C2) Final Loss | \n-1.254 | \n
| Worst-Best Gap | \n0.669 | \n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2930242%2F5608232a866f857d032a51e79f13bfe4%2F04_final_loss_boxplot.png?generation=1772325058699998&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2930242%2F57a7d9a9a05a370203254b9b36baddeb%2F05_improvement_boxplot.png?generation=1772325075898632&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2930242%2F11e9b1a508337154ca886287bc134931%2F06_hardest_samples.png?generation=1772325090638670&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2930242%2F580300c261c928dd6aaeac840cb32df0%2F08_tsne_loss_progression.png?generation=1772325111687075&alt=media\")
The A:Bright cluster (bottom-right) remains red (high loss) even after training. The Normal clusters (upper region) transition to green (low loss) early in training.
\nThe final loss of A:Bright after 100 epochs (-0.602) is 0.281 worse than Normal's initial loss (-0.883). In other words, after 100 epochs of training, Bright samples haven't even reached the level Normal data starts at.
\nThis means learning on Bright data is fundamentally harder, and reaching Normal-level performance requires many more epochs.
\nEpoch at which 90% of total improvement is reached:
\nThe early 90% convergence of A:Bright is misleading --- it reflects a \"plateau at a low level\" rather than fast learning. The total improvement is small, so the 90% threshold is reached early, but the absolute loss remains high.
\nC6 (A:Bright 97/101) has a final loss of -0.586, with a 0.3-0.7 gap from all other clusters. This cluster acts as the bottleneck for overall training efficiency.
\nC:LowContrast requires 96 epochs to reach 90% convergence. At 100 epochs, it has barely made it. Significant further improvement can be expected with 500+ epochs.
\nData diversity: 17% (Bright) + 7% (LowContrast) + 5% (Blur) of the 786 samples are difficult. Roughly 30% of the data has not reached Normal-level convergence at 100 epochs.
Batch size constraint: nnU-Net's batch size is 5 (GPU memory limited). At 250 iterations per epoch = 1,250 patches, only a fraction of the 786 samples are seen each epoch. Difficult samples require many epochs to be sampled sufficiently.
Gradient direction conflicts: Normal and Bright data may require different optimization directions. Short training biases toward the majority Normal pattern, leaving minority difficult patterns under-trained.
2,000-8,000 epochs is justified: A:Bright's loss at 100 epochs (-0.602) hasn't even reached Normal's initial value (-0.883). Closing this gap requires training on the order of thousands of epochs.
This analysis quantitatively confirmed that training convergence speed varies significantly across image quality patterns. A:Bright (high-brightness) data did not reach Normal's initial loss even after 100 epochs, supporting the rationale behind the 2,000-8,000 epoch training adopted by top teams.
\nBased on this hypothesis, we attempted a specialist model approach: separating Blur/Bright/LowContrast samples via rule-based classification and fine-tuning dedicated models for each pattern, with quality-based routing at inference time. However, rule-based domain separation had fundamental limitations --- boundaries between patterns were ambiguous, and borderline samples were difficult to handle correctly.
\nLooking back at the top teams' approaches, the solution to difficult samples was not model separation but sufficient training epochs to lift overall performance. This insight is valuable for training strategies in 3D segmentation tasks with high data diversity.
", "messages": [ { "id": "3415494", "postDate": "03/01/2026 00:34:22", "content": "I ran a few experiments based on my hypothesis, and I'd like to discuss the results.\nMy experimental setup may have some flaws or odd choices, but I hope this can spark a useful discussion.
\nSurveying the top solutions of Vesuvius Challenge 2, we found that all top teams employed 2,000-8,000 epochs of training (1st: 4,000ep, 3rd: 8,000ep, 12th: 2,000-4,000ep). We wanted to understand why such long training was effective.
\nIn our experiments, after training on all data (786 samples), we ran inference on the training set itself and found that predictions were significantly worse on low-quality images (high brightness, blur, low contrast). This led us to investigate the data distribution more closely.
\nHypothesis: Difficult data patterns (Bright, Blur, etc.) converge more slowly, requiring many more epochs to learn adequately.
\nTo investigate the source of the performance gap, we visualized the feature space of all 786 samples. From each 3D volume, we extracted the central 5 slices along XY, XZ, and YZ planes, fed them through an ImageNet-pretrained ConvNeXt-Tiny (timm) to obtain feature vectors, and averaged across the 3 views (768 dimensions). We then applied t-SNE (perplexity=30, PCA initialization) and clustered with KMeans (k=8).
\nThe result showed clear separation by image quality pattern:
\n
High-brightness samples (A:Bright) cluster in the bottom-right, blurry/low-contrast samples (B:Blur, C:LowContrast) in the bottom-left, and Normal data spreads across the upper-center region. This separation aligns strongly with image quality metrics (sharpness, contrast, mean intensity), and samples with lower model performance tend to belong to the peripheral clusters.
\nTo test this hypothesis, we fine-tuned a pre-trained model (Phase 1, 200 epochs) on the full Phase 2 dataset (786 samples) for only 100 epochs, recording per-sample loss at every epoch. The 100-epoch run was designed as a measurement-oriented short run to observe early-stage training dynamics (production training used 500-4,000 epochs).
\nWe used a checkpoint pre-trained for 200 epochs on Phase 1 data (clean labels, no ignore regions). Pre-training accelerates initial convergence.
\n| Setting | \nValue | \n
|---|---|
| Phase 1 Data | \nDataset120_Phase1 | \n
| Phase 1 Epochs | \n200 | \n
| Phase 1 Trainer | \nnnUNetTrainerSkelRecall_200ep | \n
| Phase 1 Loss | \nCE + Dice + SkelRecall | \n
| Phase 1 LR | \n0.005 (cosine annealing) | \n
| Setting | \nValue | \n
|---|---|
| Base Model | \nnnUNetTrainerSkelRecall (CE + Dice + SkelRecall) | \n
| Pre-trained Weights | \nPhase 1 200ep checkpoint_final.pth | \n
| Epochs | \n100 (measurement run; production: 500-4,000ep) | \n
| Fold | \nall (all 786 samples used for training) | \n
| Patch Size | \n128 x 160 x 160 | \n
| Batch Size | \n5 | \n
| LR | \n0.005 (cosine annealing) | \n
| Loss Logging | \nPer-sample loss recorded every epoch | \n
| Clustering | \nt-SNE + KMeans (k=8) | \n
We used nnU-Net defaults plus domain-specific custom augmentations:
\nSpatial transforms:
\nIntensity transforms:
\nCustom transforms (papyrus-specific):
\n
| Pattern | \nN | \nInitial Loss | \nFinal Loss | \nImprovement | \n90% Convergence Epoch | \n
|---|---|---|---|---|---|
| Normal | \n545 | \n-0.883 | \n-1.111 | \n0.228 | \n60 | \n
| A:Bright | \n132 | \n-0.407 | \n-0.602 | \n0.195 | \n33 | \n
| B:Blur | \n39 | \n-0.662 | \n-0.815 | \n0.153 | \n54 | \n
| C:LowContrast | \n58 | \n-0.605 | \n-0.769 | \n0.164 | \n96 | \n
| B-:MildBlur | \n12 | \n-0.833 | \n-1.163 | \n0.330 | \n25 | \n
| Cluster | \nN | \nInitial Loss | \nFinal Loss | \nImprovement | \nDominant Pattern | \n
|---|---|---|---|---|---|
| C0 (Normal) | \n101 | \n-0.947 | \n-1.118 | \n0.171 | \nNormal:101 | \n
| C1 (Normal) | \n87 | \n-0.793 | \n-1.013 | \n0.220 | \nNormal:86 | \n
| C2 (Normal) | \n85 | \n-0.958 | \n-1.254 | \n0.296 | \nNormal:70, LowContrast:15 | \n
| C3 (Normal) | \n105 | \n-0.814 | \n-1.024 | \n0.210 | \nNormal:93, Blur:8 | \n
| C4 (Normal) | \n97 | \n-0.864 | \n-1.105 | \n0.241 | \nNormal:96 | \n
| C5 (Mixed) | \n100 | \n-0.657 | \n-0.881 | \n0.224 | \nNormal:58, Bright:34 | \n
| C6 (Bright) | \n101 | \n-0.382 | \n-0.586 | \n0.204 | \nA:Bright:97 | \n
| C7 (Mixed) | \n110 | \n-0.783 | \n-0.961 | \n0.178 | \nLowContrast:38, Normal:37, Blur:31 | \n
| Metric | \nValue | \n
|---|---|
| Overall Initial Loss | \n-0.771 | \n
| Overall Final Loss | \n-0.986 | \n
| Overall Improvement | \n0.216 | \n
| Worst Cluster (C6) Final Loss | \n-0.586 | \n
| Best Cluster (C2) Final Loss | \n-1.254 | \n
| Worst-Best Gap | \n0.669 | \n
The A:Bright cluster (bottom-right) remains red (high loss) even after training. The Normal clusters (upper region) transition to green (low loss) early in training.
\nThe final loss of A:Bright after 100 epochs (-0.602) is 0.281 worse than Normal's initial loss (-0.883). In other words, after 100 epochs of training, Bright samples haven't even reached the level Normal data starts at.
\nThis means learning on Bright data is fundamentally harder, and reaching Normal-level performance requires many more epochs.
\nEpoch at which 90% of total improvement is reached:
\nThe early 90% convergence of A:Bright is misleading --- it reflects a \"plateau at a low level\" rather than fast learning. The total improvement is small, so the 90% threshold is reached early, but the absolute loss remains high.
\nC6 (A:Bright 97/101) has a final loss of -0.586, with a 0.3-0.7 gap from all other clusters. This cluster acts as the bottleneck for overall training efficiency.
\nC:LowContrast requires 96 epochs to reach 90% convergence. At 100 epochs, it has barely made it. Significant further improvement can be expected with 500+ epochs.
\nData diversity: 17% (Bright) + 7% (LowContrast) + 5% (Blur) of the 786 samples are difficult. Roughly 30% of the data has not reached Normal-level convergence at 100 epochs.
Batch size constraint: nnU-Net's batch size is 5 (GPU memory limited). At 250 iterations per epoch = 1,250 patches, only a fraction of the 786 samples are seen each epoch. Difficult samples require many epochs to be sampled sufficiently.
Gradient direction conflicts: Normal and Bright data may require different optimization directions. Short training biases toward the majority Normal pattern, leaving minority difficult patterns under-trained.
2,000-8,000 epochs is justified: A:Bright's loss at 100 epochs (-0.602) hasn't even reached Normal's initial value (-0.883). Closing this gap requires training on the order of thousands of epochs.
This analysis quantitatively confirmed that training convergence speed varies significantly across image quality patterns. A:Bright (high-brightness) data did not reach Normal's initial loss even after 100 epochs, supporting the rationale behind the 2,000-8,000 epoch training adopted by top teams.
\nBased on this hypothesis, we attempted a specialist model approach: separating Blur/Bright/LowContrast samples via rule-based classification and fine-tuning dedicated models for each pattern, with quality-based routing at inference time. However, rule-based domain separation had fundamental limitations --- boundaries between patterns were ambiguous, and borderline samples were difficult to handle correctly.
\nLooking back at the top teams' approaches, the solution to difficult samples was not model separation but sufficient training epochs to lift overall performance. This insight is valuable for training strategies in 3D segmentation tasks with high data diversity.
", "rawMarkdown": "I ran a few experiments based on my hypothesis, and I'd like to discuss the results.\nMy experimental setup may have some flaws or odd choices, but I hope this can spark a useful discussion.\n\n\n## Background\n\nSurveying the top solutions of Vesuvius Challenge 2, we found that all top teams employed **2,000-8,000 epochs** of training (1st: 4,000ep, 3rd: 8,000ep, 12th: 2,000-4,000ep). We wanted to understand why such long training was effective.\n\nIn our experiments, after training on all data (786 samples), we ran inference on the training set itself and found that predictions were significantly worse on low-quality images (high brightness, blur, low contrast). This led us to investigate the data distribution more closely.\n\n- **Normal**: High contrast, high sharpness (545 samples, 69%)\n- **A:Bright**: High intensity, low contrast (132 samples, 17%)\n- **B:Blur**: Blurry, low sharpness (39 samples, 5%)\n- **C:LowContrast**: Low contrast (58 samples, 7%)\n- **B-:MildBlur**: Mildly blurry (12 samples, 2%)\n\n**Hypothesis**: Difficult data patterns (Bright, Blur, etc.) converge more slowly, requiring many more epochs to learn adequately.\n\n## Preliminary Analysis: t-SNE Data Visualization\n\nTo investigate the source of the performance gap, we visualized the feature space of all 786 samples. From each 3D volume, we extracted the central 5 slices along XY, XZ, and YZ planes, fed them through an ImageNet-pretrained ConvNeXt-Tiny (timm) to obtain feature vectors, and averaged across the 3 views (768 dimensions). We then applied t-SNE (perplexity=30, PCA initialization) and clustered with KMeans (k=8).\n\nThe result showed clear separation by image quality pattern:\n\n\n\nHigh-brightness samples (A:Bright) cluster in the bottom-right, blurry/low-contrast samples (B:Blur, C:LowContrast) in the bottom-left, and Normal data spreads across the upper-center region. This separation aligns strongly with image quality metrics (sharpness, contrast, mean intensity), and samples with lower model performance tend to belong to the peripheral clusters.\n\n## Experimental Setup\n\nTo test this hypothesis, we fine-tuned a pre-trained model (Phase 1, 200 epochs) on the full Phase 2 dataset (786 samples) for only 100 epochs, recording per-sample loss at every epoch. The 100-epoch run was designed as a measurement-oriented short run to observe early-stage training dynamics (production training used 500-4,000 epochs).\n\n### Pre-training\n\nWe used a checkpoint pre-trained for 200 epochs on Phase 1 data (clean labels, no ignore regions). Pre-training accelerates initial convergence.\n\n| Setting | Value |\n|---------|-------|\n| Phase 1 Data | Dataset120_Phase1 |\n| Phase 1 Epochs | 200 |\n| Phase 1 Trainer | nnUNetTrainerSkelRecall_200ep |\n| Phase 1 Loss | CE + Dice + SkelRecall |\n| Phase 1 LR | 0.005 (cosine annealing) |\n\n### Main Experiment (Phase 2 Fine-tune + Loss Tracking)\n\n| Setting | Value |\n|---------|-------|\n| Base Model | nnUNetTrainerSkelRecall (CE + Dice + SkelRecall) |\n| Pre-trained Weights | Phase 1 200ep checkpoint_final.pth |\n| Epochs | 100 (measurement run; production: 500-4,000ep) |\n| Fold | all (all 786 samples used for training) |\n| Patch Size | 128 x 160 x 160 |\n| Batch Size | 5 |\n| LR | 0.005 (cosine annealing) |\n| Loss Logging | Per-sample loss recorded every epoch |\n| Clustering | t-SNE + KMeans (k=8) |\n\n### Data Augmentation\n\nWe used nnU-Net defaults plus domain-specific custom augmentations:\n\n**Spatial transforms:**\n- Elastic deformation (p=0.3, magnitude 10-50 voxels)\n- Rotation (p=0.5, +/-15 degrees per axis)\n- Scaling (p=0.2, 0.7x-1.4x)\n- Mirror (all 3 axes)\n\n**Intensity transforms:**\n- Gaussian Noise (p=0.15, variance 0-0.15)\n- Gaussian Blur (p=0.2, sigma 0.5-1.5)\n- Brightness (p=0.15, 0.5x-1.5x)\n- Contrast (p=0.15, 0.5x-1.5x)\n- Low Resolution Simulation (p=0.25, scale 0.25-1.0)\n- Gamma (p=0.3, gamma 0.7-1.5)\n\n**Custom transforms (papyrus-specific):**\n- BlankRectangle (p=0.4): Blank out rectangular regions to simulate damage\n- Smear (p=0.2): Z-axis ink bleeding simulation\n- InhomogeneousIllumination (p=0.25): Uneven lighting simulation\n\n## Results\n\n### Per-Pattern Loss Curves\n\n\n\n\n| Pattern | N | Initial Loss | Final Loss | Improvement | 90% Convergence Epoch |\n|---------|---:|-----------:|-----------:|----------:|---------------------:|\n| Normal | 545 | -0.883 | -1.111 | 0.228 | 60 |\n| A:Bright | 132 | -0.407 | -0.602 | 0.195 | 33 |\n| B:Blur | 39 | -0.662 | -0.815 | 0.153 | 54 |\n| C:LowContrast | 58 | -0.605 | -0.769 | 0.164 | 96 |\n| B-:MildBlur | 12 | -0.833 | -1.163 | 0.330 | 25 |\n\n### Per-Cluster Loss Curves\n\n\n\n\n| Cluster | N | Initial Loss | Final Loss | Improvement | Dominant Pattern |\n|---------|---:|-----------:|-----------:|----------:|:----------------|\n| C0 (Normal) | 101 | -0.947 | -1.118 | 0.171 | Normal:101 |\n| C1 (Normal) | 87 | -0.793 | -1.013 | 0.220 | Normal:86 |\n| C2 (Normal) | 85 | -0.958 | -1.254 | 0.296 | Normal:70, LowContrast:15 |\n| C3 (Normal) | 105 | -0.814 | -1.024 | 0.210 | Normal:93, Blur:8 |\n| C4 (Normal) | 97 | -0.864 | -1.105 | 0.241 | Normal:96 |\n| C5 (Mixed) | 100 | -0.657 | -0.881 | 0.224 | Normal:58, Bright:34 |\n| C6 (Bright) | 101 | -0.382 | -0.586 | 0.204 | **A:Bright:97** |\n| C7 (Mixed) | 110 | -0.783 | -0.961 | 0.178 | LowContrast:38, Normal:37, Blur:31 |\n\n### Overall Summary\n\n| Metric | Value |\n|--------|------:|\n| Overall Initial Loss | -0.771 |\n| Overall Final Loss | -0.986 |\n| Overall Improvement | 0.216 |\n| Worst Cluster (C6) Final Loss | -0.586 |\n| Best Cluster (C2) Final Loss | -1.254 |\n| Worst-Best Gap | **0.669** |\n\n## Detailed Analysis\n\n### Final Loss Distribution by Cluster\n\n\n\n\n### Improvement Distribution by Cluster\n\n\n\n\n### Top 30 Hardest Samples\n\n\n\n\n### Loss Distribution Change on t-SNE Map\n\n\n\n\nThe A:Bright cluster (bottom-right) remains red (high loss) even after training. The Normal clusters (upper region) transition to green (low loss) early in training.\n\n## Key Findings\n\n### 1. A:Bright Samples Show Severe Convergence Lag\n\nThe final loss of A:Bright after 100 epochs (**-0.602**) is **0.281 worse** than Normal's *initial* loss (**-0.883**). In other words, after 100 epochs of training, Bright samples haven't even reached the level Normal data starts at.\n\nThis means learning on Bright data is fundamentally harder, and reaching Normal-level performance requires **many more epochs**.\n\n### 2. Convergence Speed Varies Dramatically Across Patterns\n\nEpoch at which 90% of total improvement is reached:\n- B-:MildBlur: **25ep** (fastest; small sample count, high variance)\n- A:Bright: **33ep** (quick plateau at a poor absolute level = 100ep is not enough)\n- B:Blur: **54ep**\n- Normal: **60ep**\n- C:LowContrast: **96ep** (slowest; barely converging within 100ep)\n\nThe early 90% convergence of A:Bright is misleading --- it reflects a \"plateau at a low level\" rather than fast learning. The total improvement is small, so the 90% threshold is reached early, but the absolute loss remains high.\n\n### 3. Cluster C6 Is an Outlier in Difficulty\n\nC6 (A:Bright 97/101) has a final loss of -0.586, with a 0.3-0.7 gap from all other clusters. This cluster acts as the bottleneck for overall training efficiency.\n\n### 4. LowContrast Has the Slowest Convergence\n\nC:LowContrast requires **96 epochs** to reach 90% convergence. At 100 epochs, it has barely made it. Significant further improvement can be expected with 500+ epochs.\n\n## Why Long Training Works (Discussion)\n\n1. **Data diversity**: 17% (Bright) + 7% (LowContrast) + 5% (Blur) of the 786 samples are difficult. Roughly 30% of the data has not reached Normal-level convergence at 100 epochs.\n\n2. **Batch size constraint**: nnU-Net's batch size is 5 (GPU memory limited). At 250 iterations per epoch = 1,250 patches, only a fraction of the 786 samples are seen each epoch. Difficult samples require many epochs to be sampled sufficiently.\n\n3. **Gradient direction conflicts**: Normal and Bright data may require different optimization directions. Short training biases toward the majority Normal pattern, leaving minority difficult patterns under-trained.\n\n4. **2,000-8,000 epochs is justified**: A:Bright's loss at 100 epochs (-0.602) hasn't even reached Normal's initial value (-0.883). Closing this gap requires training on the order of thousands of epochs.\n\n## Conclusion\n\nThis analysis quantitatively confirmed that **training convergence speed varies significantly across image quality patterns**. A:Bright (high-brightness) data did not reach Normal's initial loss even after 100 epochs, supporting the rationale behind the 2,000-8,000 epoch training adopted by top teams.\n\nBased on this hypothesis, we attempted a **specialist model** approach: separating Blur/Bright/LowContrast samples via rule-based classification and fine-tuning dedicated models for each pattern, with quality-based routing at inference time. However, rule-based domain separation had fundamental limitations --- boundaries between patterns were ambiguous, and borderline samples were difficult to handle correctly. \n\nLooking back at the top teams' approaches, the solution to difficult samples was not model separation but **sufficient training epochs to lift overall performance**. This insight is valuable for training strategies in 3D segmentation tasks with high data diversity.", "votes": null }, { "id": "3415507", "postDate": "03/01/2026 00:58:53", "content": "Thank you for the excellent writeup! This is something that i've always had questions about. no matter how many epochs i've trained these models for they never seem to quite stop learning, and they have a strange habit of plateauing for hundreds of epochs and suddenly learning again. Maybe this explains it!
\nAs a quick note -- with the exception of the smear transform, which is just something i had chatgpt write up, the custom transforms mentioned seem like they may be from the augmentations used in the villa monorepo (or the vesuvius subpackage), these were themself lifted from a MIC-DKFZ repository here: https://github.com/MIC-DKFZ/MurineAirwaySegmentation , so i want to make sure if you've grabbed them from there the credit goes to the right people!
", "rawMarkdown": "Thank you for the excellent writeup! This is something that i've always had questions about. no matter how many epochs i've trained these models for they never seem to quite stop learning, and they have a strange habit of plateauing for hundreds of epochs and suddenly learning again. Maybe this explains it! \n\nAs a quick note -- with the exception of the smear transform, which is just something i had chatgpt write up, the custom transforms mentioned seem like they may be from the augmentations used in the villa monorepo (or the vesuvius subpackage), these were themself lifted from a MIC-DKFZ repository here: https://github.com/MIC-DKFZ/MurineAirwaySegmentation , so i want to make sure if you've grabbed them from there the credit goes to the right people!", "votes": null }, { "id": "3415540", "postDate": "03/01/2026 01:32:24", "content": "Ideally, the loss should be similar for both Bright and Normal. In practice, however, Bright tends to start with a higher loss and takes longer to converge.
\nWith strong augmentation, those same aggressive transforms are also applied to Bright samples, which can destabilize training and potentially lead to failure. In an ideal world, augmentation could be tailored on a per-sample basis, but that’s difficult in practice—so longer training (more epochs) may be necessary.
\nNormal samples converge faster and appear more often, so they dominate the optimization, and the Bright cases get drowned out.
", "rawMarkdown": "## Additional comment\nIdeally, the loss should be similar for both Bright and Normal. In practice, however, Bright tends to start with a higher loss and takes longer to converge.\n\nWith strong augmentation, those same aggressive transforms are also applied to Bright samples, which can destabilize training and potentially lead to failure. In an ideal world, augmentation could be tailored on a per-sample basis, but that’s difficult in practice—so longer training (more epochs) may be necessary.\n\nNormal samples converge faster and appear more often, so they dominate the optimization, and the Bright cases get drowned out.", "votes": null }, { "id": "3415669", "postDate": "03/01/2026 05:16:15", "content": "I'm also curious about this. My final solution was trained for 1000 epochs. But I am currently retraining my final solution on 4000 epochs. I want to see how much the CV and LB improves.
", "rawMarkdown": "I'm also curious about this. My final solution was trained for 1000 epochs. But I am currently retraining my final solution on 4000 epochs. I want to see how much the CV and LB improves.", "votes": null }, { "id": "3415674", "postDate": "03/01/2026 05:42:39", "content": "it would be helpful if you share your thoughts after it ;)
", "rawMarkdown": "it would be helpful if you share your thoughts after it ;)", "votes": null } ], "comments": [ { "id": 3415507, "author_name": "seanjohnsonsp", "author_url": "", "post_date": "03/01/2026 00:58:53", "content": "Thank you for the excellent writeup! This is something that i've always had questions about. no matter how many epochs i've trained these models for they never seem to quite stop learning, and they have a strange habit of plateauing for hundreds of epochs and suddenly learning again. Maybe this explains it!
\nAs a quick note -- with the exception of the smear transform, which is just something i had chatgpt write up, the custom transforms mentioned seem like they may be from the augmentations used in the villa monorepo (or the vesuvius subpackage), these were themself lifted from a MIC-DKFZ repository here: https://github.com/MIC-DKFZ/MurineAirwaySegmentation , so i want to make sure if you've grabbed them from there the credit goes to the right people!
", "votes": null, "replies": [] }, { "id": 3415540, "author_name": "sugupoko", "author_url": "", "post_date": "03/01/2026 01:32:24", "content": "Ideally, the loss should be similar for both Bright and Normal. In practice, however, Bright tends to start with a higher loss and takes longer to converge.
\nWith strong augmentation, those same aggressive transforms are also applied to Bright samples, which can destabilize training and potentially lead to failure. In an ideal world, augmentation could be tailored on a per-sample basis, but that’s difficult in practice—so longer training (more epochs) may be necessary.
\nNormal samples converge faster and appear more often, so they dominate the optimization, and the Bright cases get drowned out.
", "votes": null, "replies": [] }, { "id": 3415669, "author_name": "cdeotte", "author_url": "", "post_date": "03/01/2026 05:16:15", "content": "I'm also curious about this. My final solution was trained for 1000 epochs. But I am currently retraining my final solution on 4000 epochs. I want to see how much the CV and LB improves.
", "votes": null, "replies": [ { "id": 3415674, "author_name": "ppilania1985", "author_url": "", "post_date": "03/01/2026 05:42:39", "content": "it would be helpful if you share your thoughts after it ;)
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3415494": "I ran a few experiments based on my hypothesis, and I'd like to discuss the results.\nMy experimental setup may have some flaws or odd choices, but I hope this can spark a useful discussion.\n\n\n## Background\n\nSurveying the top solutions of Vesuvius Challenge 2, we found that all top teams employed **2,000-8,000 epochs** of training (1st: 4,000ep, 3rd: 8,000ep, 12th: 2,000-4,000ep). We wanted to understand why such long training was effective.\n\nIn our experiments, after training on all data (786 samples), we ran inference on the training set itself and found that predictions were significantly worse on low-quality images (high brightness, blur, low contrast). This led us to investigate the data distribution more closely.\n\n- **Normal**: High contrast, high sharpness (545 samples, 69%)\n- **A:Bright**: High intensity, low contrast (132 samples, 17%)\n- **B:Blur**: Blurry, low sharpness (39 samples, 5%)\n- **C:LowContrast**: Low contrast (58 samples, 7%)\n- **B-:MildBlur**: Mildly blurry (12 samples, 2%)\n\n**Hypothesis**: Difficult data patterns (Bright, Blur, etc.) converge more slowly, requiring many more epochs to learn adequately.\n\n## Preliminary Analysis: t-SNE Data Visualization\n\nTo investigate the source of the performance gap, we visualized the feature space of all 786 samples. From each 3D volume, we extracted the central 5 slices along XY, XZ, and YZ planes, fed them through an ImageNet-pretrained ConvNeXt-Tiny (timm) to obtain feature vectors, and averaged across the 3 views (768 dimensions). We then applied t-SNE (perplexity=30, PCA initialization) and clustered with KMeans (k=8).\n\nThe result showed clear separation by image quality pattern:\n\n\n\nHigh-brightness samples (A:Bright) cluster in the bottom-right, blurry/low-contrast samples (B:Blur, C:LowContrast) in the bottom-left, and Normal data spreads across the upper-center region. This separation aligns strongly with image quality metrics (sharpness, contrast, mean intensity), and samples with lower model performance tend to belong to the peripheral clusters.\n\n## Experimental Setup\n\nTo test this hypothesis, we fine-tuned a pre-trained model (Phase 1, 200 epochs) on the full Phase 2 dataset (786 samples) for only 100 epochs, recording per-sample loss at every epoch. The 100-epoch run was designed as a measurement-oriented short run to observe early-stage training dynamics (production training used 500-4,000 epochs).\n\n### Pre-training\n\nWe used a checkpoint pre-trained for 200 epochs on Phase 1 data (clean labels, no ignore regions). Pre-training accelerates initial convergence.\n\n| Setting | Value |\n|---------|-------|\n| Phase 1 Data | Dataset120_Phase1 |\n| Phase 1 Epochs | 200 |\n| Phase 1 Trainer | nnUNetTrainerSkelRecall_200ep |\n| Phase 1 Loss | CE + Dice + SkelRecall |\n| Phase 1 LR | 0.005 (cosine annealing) |\n\n### Main Experiment (Phase 2 Fine-tune + Loss Tracking)\n\n| Setting | Value |\n|---------|-------|\n| Base Model | nnUNetTrainerSkelRecall (CE + Dice + SkelRecall) |\n| Pre-trained Weights | Phase 1 200ep checkpoint_final.pth |\n| Epochs | 100 (measurement run; production: 500-4,000ep) |\n| Fold | all (all 786 samples used for training) |\n| Patch Size | 128 x 160 x 160 |\n| Batch Size | 5 |\n| LR | 0.005 (cosine annealing) |\n| Loss Logging | Per-sample loss recorded every epoch |\n| Clustering | t-SNE + KMeans (k=8) |\n\n### Data Augmentation\n\nWe used nnU-Net defaults plus domain-specific custom augmentations:\n\n**Spatial transforms:**\n- Elastic deformation (p=0.3, magnitude 10-50 voxels)\n- Rotation (p=0.5, +/-15 degrees per axis)\n- Scaling (p=0.2, 0.7x-1.4x)\n- Mirror (all 3 axes)\n\n**Intensity transforms:**\n- Gaussian Noise (p=0.15, variance 0-0.15)\n- Gaussian Blur (p=0.2, sigma 0.5-1.5)\n- Brightness (p=0.15, 0.5x-1.5x)\n- Contrast (p=0.15, 0.5x-1.5x)\n- Low Resolution Simulation (p=0.25, scale 0.25-1.0)\n- Gamma (p=0.3, gamma 0.7-1.5)\n\n**Custom transforms (papyrus-specific):**\n- BlankRectangle (p=0.4): Blank out rectangular regions to simulate damage\n- Smear (p=0.2): Z-axis ink bleeding simulation\n- InhomogeneousIllumination (p=0.25): Uneven lighting simulation\n\n## Results\n\n### Per-Pattern Loss Curves\n\n\n\n\n| Pattern | N | Initial Loss | Final Loss | Improvement | 90% Convergence Epoch |\n|---------|---:|-----------:|-----------:|----------:|---------------------:|\n| Normal | 545 | -0.883 | -1.111 | 0.228 | 60 |\n| A:Bright | 132 | -0.407 | -0.602 | 0.195 | 33 |\n| B:Blur | 39 | -0.662 | -0.815 | 0.153 | 54 |\n| C:LowContrast | 58 | -0.605 | -0.769 | 0.164 | 96 |\n| B-:MildBlur | 12 | -0.833 | -1.163 | 0.330 | 25 |\n\n### Per-Cluster Loss Curves\n\n\n\n\n| Cluster | N | Initial Loss | Final Loss | Improvement | Dominant Pattern |\n|---------|---:|-----------:|-----------:|----------:|:----------------|\n| C0 (Normal) | 101 | -0.947 | -1.118 | 0.171 | Normal:101 |\n| C1 (Normal) | 87 | -0.793 | -1.013 | 0.220 | Normal:86 |\n| C2 (Normal) | 85 | -0.958 | -1.254 | 0.296 | Normal:70, LowContrast:15 |\n| C3 (Normal) | 105 | -0.814 | -1.024 | 0.210 | Normal:93, Blur:8 |\n| C4 (Normal) | 97 | -0.864 | -1.105 | 0.241 | Normal:96 |\n| C5 (Mixed) | 100 | -0.657 | -0.881 | 0.224 | Normal:58, Bright:34 |\n| C6 (Bright) | 101 | -0.382 | -0.586 | 0.204 | **A:Bright:97** |\n| C7 (Mixed) | 110 | -0.783 | -0.961 | 0.178 | LowContrast:38, Normal:37, Blur:31 |\n\n### Overall Summary\n\n| Metric | Value |\n|--------|------:|\n| Overall Initial Loss | -0.771 |\n| Overall Final Loss | -0.986 |\n| Overall Improvement | 0.216 |\n| Worst Cluster (C6) Final Loss | -0.586 |\n| Best Cluster (C2) Final Loss | -1.254 |\n| Worst-Best Gap | **0.669** |\n\n## Detailed Analysis\n\n### Final Loss Distribution by Cluster\n\n\n\n\n### Improvement Distribution by Cluster\n\n\n\n\n### Top 30 Hardest Samples\n\n\n\n\n### Loss Distribution Change on t-SNE Map\n\n\n\n\nThe A:Bright cluster (bottom-right) remains red (high loss) even after training. The Normal clusters (upper region) transition to green (low loss) early in training.\n\n## Key Findings\n\n### 1. A:Bright Samples Show Severe Convergence Lag\n\nThe final loss of A:Bright after 100 epochs (**-0.602**) is **0.281 worse** than Normal's *initial* loss (**-0.883**). In other words, after 100 epochs of training, Bright samples haven't even reached the level Normal data starts at.\n\nThis means learning on Bright data is fundamentally harder, and reaching Normal-level performance requires **many more epochs**.\n\n### 2. Convergence Speed Varies Dramatically Across Patterns\n\nEpoch at which 90% of total improvement is reached:\n- B-:MildBlur: **25ep** (fastest; small sample count, high variance)\n- A:Bright: **33ep** (quick plateau at a poor absolute level = 100ep is not enough)\n- B:Blur: **54ep**\n- Normal: **60ep**\n- C:LowContrast: **96ep** (slowest; barely converging within 100ep)\n\nThe early 90% convergence of A:Bright is misleading --- it reflects a \"plateau at a low level\" rather than fast learning. The total improvement is small, so the 90% threshold is reached early, but the absolute loss remains high.\n\n### 3. Cluster C6 Is an Outlier in Difficulty\n\nC6 (A:Bright 97/101) has a final loss of -0.586, with a 0.3-0.7 gap from all other clusters. This cluster acts as the bottleneck for overall training efficiency.\n\n### 4. LowContrast Has the Slowest Convergence\n\nC:LowContrast requires **96 epochs** to reach 90% convergence. At 100 epochs, it has barely made it. Significant further improvement can be expected with 500+ epochs.\n\n## Why Long Training Works (Discussion)\n\n1. **Data diversity**: 17% (Bright) + 7% (LowContrast) + 5% (Blur) of the 786 samples are difficult. Roughly 30% of the data has not reached Normal-level convergence at 100 epochs.\n\n2. **Batch size constraint**: nnU-Net's batch size is 5 (GPU memory limited). At 250 iterations per epoch = 1,250 patches, only a fraction of the 786 samples are seen each epoch. Difficult samples require many epochs to be sampled sufficiently.\n\n3. **Gradient direction conflicts**: Normal and Bright data may require different optimization directions. Short training biases toward the majority Normal pattern, leaving minority difficult patterns under-trained.\n\n4. **2,000-8,000 epochs is justified**: A:Bright's loss at 100 epochs (-0.602) hasn't even reached Normal's initial value (-0.883). Closing this gap requires training on the order of thousands of epochs.\n\n## Conclusion\n\nThis analysis quantitatively confirmed that **training convergence speed varies significantly across image quality patterns**. A:Bright (high-brightness) data did not reach Normal's initial loss even after 100 epochs, supporting the rationale behind the 2,000-8,000 epoch training adopted by top teams.\n\nBased on this hypothesis, we attempted a **specialist model** approach: separating Blur/Bright/LowContrast samples via rule-based classification and fine-tuning dedicated models for each pattern, with quality-based routing at inference time. However, rule-based domain separation had fundamental limitations --- boundaries between patterns were ambiguous, and borderline samples were difficult to handle correctly. \n\nLooking back at the top teams' approaches, the solution to difficult samples was not model separation but **sufficient training epochs to lift overall performance**. This insight is valuable for training strategies in 3D segmentation tasks with high data diversity.", "3415507": "Thank you for the excellent writeup! This is something that i've always had questions about. no matter how many epochs i've trained these models for they never seem to quite stop learning, and they have a strange habit of plateauing for hundreds of epochs and suddenly learning again. Maybe this explains it! \n\nAs a quick note -- with the exception of the smear transform, which is just something i had chatgpt write up, the custom transforms mentioned seem like they may be from the augmentations used in the villa monorepo (or the vesuvius subpackage), these were themself lifted from a MIC-DKFZ repository here: https://github.com/MIC-DKFZ/MurineAirwaySegmentation , so i want to make sure if you've grabbed them from there the credit goes to the right people!", "3415540": "## Additional comment\nIdeally, the loss should be similar for both Bright and Normal. In practice, however, Bright tends to start with a higher loss and takes longer to converge.\n\nWith strong augmentation, those same aggressive transforms are also applied to Bright samples, which can destabilize training and potentially lead to failure. In an ideal world, augmentation could be tailored on a per-sample basis, but that’s difficult in practice—so longer training (more epochs) may be necessary.\n\nNormal samples converge faster and appear more often, so they dominate the optimization, and the Bright cases get drowned out.", "3415669": "I'm also curious about this. My final solution was trained for 1000 epochs. But I am currently retraining my final solution on 4000 epochs. I want to see how much the CV and LB improves.", "3415674": "it would be helpful if you share your thoughts after it ;)" }, "source": "meta" }