First of all, I would like to express my deep gratitude to the organizers Yale, UNC-CH, Kaggle. This competition was interesting and educational, and I'm glad I participated.
\nI would also like to deeply thank @brendanartley for providing the public code. My solution is developed based on Bartley's public code.
Table 1 – Single-model performance (lower is better)
\n| Backbone | \nEpochs | \nCV MAE | \nPublic MAE | \nNotes | \n
|---|---|---|---|---|
| ConvNeXt-Base | \n200 | \n23.87 | \n25.7 | \nBest single model | \n
| CAFormer-m36 | \n180 | \n24.08 | \n26.0 | \nStarReLU | \n
| CAFormer-b36 | \n180 | \n24.96 | \n29.5 | \nStarReLU | \n
| ConvNeXt-Base | \n150 | \n25.58 | \n26.7 | \nCheckpoint fine-tuned twice | \n
| ConvNeXt-Large | \n150 | \n26.08 | \n27.8 | \n— | \n
| Flash InternImage-b | \n50 | \n34.60 | \n37.6 | \nFP32, ≈ 210 min / epoch | \n
| ConvNeXt-Base (full data) | \n218 | \n— | \n27.7 | \nTrained on 100 % data | \n
| ConvNeXt-Base (full data) | \n243 | \n— | \n26.7 | \nTrained on 100 % data | \n
| ConvNeXt-Base (full data) | \n258 | \n— | \n26.6 | \nTrained on 100 % data | \n
Hyper-parameters tuned: weight_decay, dropout_rate, decoder_channels, clip_grad_norm
\nLR restarts: resumed from the previous terminal LR (1 e-5) with CosineAnnealingLR(T_max = 50), which proved more stable than warm restarts.
Table 2 – Ensemble performance (Public MAE)
\n| Method | \nSubmissions | \nPublic MAE | \nΔ vs. Best Single | \n
|---|---|---|---|
| Simple median | \n9 | \n23.4 | \n−2.3 | \n
| Weighted median (final) | \n9 | \n23.2 | \n−2.5 | \n
clip(1500, 4500) (additional –0.002 CV MAE).| Category | \nKey settings | \n
|---|---|
| Compiler / Optimizer | \ntorch.compile(mode=\"max-autotune\"), Fused AdamW | \n
| Mixed Precision | \nAMP (bfloat16) for all ConvNeXt / CAFormer runs | \n
| DataLoader | \nnum_workers tuned, prefetch_factor tuned, pin_memory=True, persistent_workers=True | \n
| Hardware | \nExperiments run on RTX 3090 & A100 Flash InternImage remained FP32 due to DCNv4 → ≈ 210 min / epoch (A100) | \n
“num_workers was experimentally tuned—the value is a sweet spot: too many or too few threads slowed training, so we benchmark-benchmarked different settings and used the fastest one for all runs.”
\n| What failed | \nLikely cause / lesson | \n
|---|---|
| Type-wise weights, heavy post-processing | \nMedian already robust; extra weighting brought no gain | \n
| Flash InternImage-b | \nAMP unstable; FP32 made training prohibitively slow, limiting epochs | \n
| EVA family | \nMishandled window / resize parameters – never converged | \n
| Full-data training (100 %) | \nSlightly worse Public MAE vs. 88 % split, much longer runtime | \n
First of all, I would like to express my deep gratitude to the organizers Yale, UNC-CH, Kaggle. This competition was interesting and educational, and I'm glad I participated.
\nI would also like to deeply thank @brendanartley for providing the public code. My solution is developed based on Bartley's public code.
Table 1 – Single-model performance (lower is better)
\n| Backbone | \nEpochs | \nCV MAE | \nPublic MAE | \nNotes | \n
|---|---|---|---|---|
| ConvNeXt-Base | \n200 | \n23.87 | \n25.7 | \nBest single model | \n
| CAFormer-m36 | \n180 | \n24.08 | \n26.0 | \nStarReLU | \n
| CAFormer-b36 | \n180 | \n24.96 | \n29.5 | \nStarReLU | \n
| ConvNeXt-Base | \n150 | \n25.58 | \n26.7 | \nCheckpoint fine-tuned twice | \n
| ConvNeXt-Large | \n150 | \n26.08 | \n27.8 | \n— | \n
| Flash InternImage-b | \n50 | \n34.60 | \n37.6 | \nFP32, ≈ 210 min / epoch | \n
| ConvNeXt-Base (full data) | \n218 | \n— | \n27.7 | \nTrained on 100 % data | \n
| ConvNeXt-Base (full data) | \n243 | \n— | \n26.7 | \nTrained on 100 % data | \n
| ConvNeXt-Base (full data) | \n258 | \n— | \n26.6 | \nTrained on 100 % data | \n
Hyper-parameters tuned: weight_decay, dropout_rate, decoder_channels, clip_grad_norm
\nLR restarts: resumed from the previous terminal LR (1 e-5) with CosineAnnealingLR(T_max = 50), which proved more stable than warm restarts.
Table 2 – Ensemble performance (Public MAE)
\n| Method | \nSubmissions | \nPublic MAE | \nΔ vs. Best Single | \n
|---|---|---|---|
| Simple median | \n9 | \n23.4 | \n−2.3 | \n
| Weighted median (final) | \n9 | \n23.2 | \n−2.5 | \n
clip(1500, 4500) (additional –0.002 CV MAE).| Category | \nKey settings | \n
|---|---|
| Compiler / Optimizer | \ntorch.compile(mode=\"max-autotune\"), Fused AdamW | \n
| Mixed Precision | \nAMP (bfloat16) for all ConvNeXt / CAFormer runs | \n
| DataLoader | \nnum_workers tuned, prefetch_factor tuned, pin_memory=True, persistent_workers=True | \n
| Hardware | \nExperiments run on RTX 3090 & A100 Flash InternImage remained FP32 due to DCNv4 → ≈ 210 min / epoch (A100) | \n
“num_workers was experimentally tuned—the value is a sweet spot: too many or too few threads slowed training, so we benchmark-benchmarked different settings and used the fastest one for all runs.”
\n| What failed | \nLikely cause / lesson | \n
|---|---|
| Type-wise weights, heavy post-processing | \nMedian already robust; extra weighting brought no gain | \n
| Flash InternImage-b | \nAMP unstable; FP32 made training prohibitively slow, limiting epochs | \n
| EVA family | \nMishandled window / resize parameters – never converged | \n
| Full-data training (100 %) | \nSlightly worse Public MAE vs. 88 % split, much longer runtime | \n
Thanks for many experimental results! They are very useful, but some results are not obvious. Do you have some idea why?
\nWeighted median is great! I only thought about median and weighted mean. Is it implemented in well known libraries or in public notebook?
", "rawMarkdown": "Thanks for many experimental results! They are very useful, but some results are not obvious. Do you have some idea why?\n\n* Why ConvNeXt-Base (full data) runs are worse than the first ConvNeXt-Base although there ~10% more data? Difference in learning rates?\n* Why ConvNeXt-Large is worse than Base for same 150 epochs? Do you see larger train-val loss gap (overfit)? Some people said larger models were always better in this competition\n\nWeighted median is great! I only thought about median and weighted mean. Is it implemented in well known libraries or in public notebook?", "votes": null }, { "id": "3240376", "postDate": "07/03/2025 18:23:58", "content": "Thank you for your comment!
\n\n\nWhy ConvNeXt-Base (full data) runs are worse than the first ConvNeXt-Base although there ~10% more data? Difference in learning rates?
\n
I realize my earlier conclusion that \"full-data didn't help\" was premature because the two runs weren't strictly comparable.
\nThe key differences we've identified are:
\n100% DATA: batch_size = 64, epochs = 300, drop_last_train = True
\n88% DATA: batch_size = 32, epochs = 200, drop_last_train = False
Because the CosineAnnealingLR schedule was tied to epochs, the LR in the 100 % run fell much sooner (in step terms) than in the 88 % run—likely the key factor. The bigger batch and drop_last_train=True probably played a role as well.
\n\nWhy ConvNeXt-Large is worse than Base for same 150 epochs? Do you see larger train-val loss gap (overfit)? Some people said larger models were always better in this competition
\n
I’m not sure of the reason, and I wonder why the Large model stagnated earlier than expected.
\nIn my experiments, the Large tends to overfit the training data more than the Base, and its validation MAE stops improving at an earlier stage. To counter this, I adjusted the hyper-parameters for the Large model as shown below, but even with these tweaks the validation MAE still plateaued sooner.
| Model | \nTrain MAE | \nValidation MAE | \nKey Hyper-parameters | \n
|---|---|---|---|
| ConvNeXt-Base | \n19.83 | \n25.58 | \ndrop_path_rate = 0.2, weight_decay = 0.03, layer_decay = 1.0 (disabled) | \n
| ConvNeXt-Large | \n18.62 | \n26.08 (validation MAE plateaus earlier than Base) | \ndrop_path_rate = 0.4, weight_decay = 0.05, layer_decay = 0.7 | \n
I intend to study other participants’ solutions to better understand how to improve the Large model.
\nIf you have any ideas, I’d like to hear them.
Thanks for more investigation!
\nI see, I know some strong Kagglers use full-data training, but I was worrying training without val score; e.g. should we change batch size to have same number steps or keep batch size and change learning rate? Your result confirms that full-data training can be difficult and requires experience. In this competition, we have lots of data, so I keep one file (500 data) per family as a \"test set,\" in addition to K-fold validation set. This helped me having \"test score\" in the almost full-data training.
", "rawMarkdown": "Thanks for more investigation!\n\nI see, I know some strong Kagglers use full-data training, but I was worrying training without val score; e.g. should we change batch size to have same number steps or keep batch size and change learning rate? Your result confirms that full-data training can be difficult and requires experience. In this competition, we have lots of data, so I keep one file (500 data) per family as a \"test set,\" in addition to K-fold validation set. This helped me having \"test score\" in the almost full-data training.", "votes": null }, { "id": "3241187", "postDate": "07/04/2025 16:06:51", "content": "I think your point is spot-on. In fact, bumping the training set from 88% to 95% would probably be enough to see whether the extra data boosts score. Putting too much faith in the Public Leaderboard is risky in many competitions—although in this one we were lucky that the public and private scores were closely aligned.
\n\n\nWeighted median is great! I only thought about median and weighted mean. Is it implemented in well known libraries or in public notebook?
\n
I missed your last question. Since Polars doesn't have a built-in weighted median, I implemented my own. As far as I know, this approach hasn't been widely used for ensembling in Kaggle notebooks. Median was already giving a much better score than the mean, so I experimented with variations on the median, and the weighted median happened to work well in this case.
", "rawMarkdown": "I think your point is spot-on. In fact, bumping the training set from 88% to 95% would probably be enough to see whether the extra data boosts score. Putting too much faith in the Public Leaderboard is risky in many competitions—although in this one we were lucky that the public and private scores were closely aligned.\n\n>Weighted median is great! I only thought about median and weighted mean. Is it implemented in well known libraries or in public notebook?\n\nI missed your last question. Since Polars doesn't have a built-in weighted median, I implemented my own. As far as I know, this approach hasn't been widely used for ensembling in Kaggle notebooks. Median was already giving a much better score than the mean, so I experimented with variations on the median, and the weighted median happened to work well in this case.", "votes": null } ], "comments": [ { "id": 3239435, "author_name": "junkoda", "author_url": "", "post_date": "07/02/2025 20:18:10", "content": "Thanks for many experimental results! They are very useful, but some results are not obvious. Do you have some idea why?
\nWeighted median is great! I only thought about median and weighted mean. Is it implemented in well known libraries or in public notebook?
", "votes": null, "replies": [ { "id": 3240376, "author_name": "iwaiwa", "author_url": "", "post_date": "07/03/2025 18:23:58", "content": "Thank you for your comment!
\n\n\nWhy ConvNeXt-Base (full data) runs are worse than the first ConvNeXt-Base although there ~10% more data? Difference in learning rates?
\n
I realize my earlier conclusion that \"full-data didn't help\" was premature because the two runs weren't strictly comparable.
\nThe key differences we've identified are:
\n100% DATA: batch_size = 64, epochs = 300, drop_last_train = True
\n88% DATA: batch_size = 32, epochs = 200, drop_last_train = False
Because the CosineAnnealingLR schedule was tied to epochs, the LR in the 100 % run fell much sooner (in step terms) than in the 88 % run—likely the key factor. The bigger batch and drop_last_train=True probably played a role as well.
\n\nWhy ConvNeXt-Large is worse than Base for same 150 epochs? Do you see larger train-val loss gap (overfit)? Some people said larger models were always better in this competition
\n
I’m not sure of the reason, and I wonder why the Large model stagnated earlier than expected.
\nIn my experiments, the Large tends to overfit the training data more than the Base, and its validation MAE stops improving at an earlier stage. To counter this, I adjusted the hyper-parameters for the Large model as shown below, but even with these tweaks the validation MAE still plateaued sooner.
| Model | \nTrain MAE | \nValidation MAE | \nKey Hyper-parameters | \n
|---|---|---|---|
| ConvNeXt-Base | \n19.83 | \n25.58 | \ndrop_path_rate = 0.2, weight_decay = 0.03, layer_decay = 1.0 (disabled) | \n
| ConvNeXt-Large | \n18.62 | \n26.08 (validation MAE plateaus earlier than Base) | \ndrop_path_rate = 0.4, weight_decay = 0.05, layer_decay = 0.7 | \n
I intend to study other participants’ solutions to better understand how to improve the Large model.
\nIf you have any ideas, I’d like to hear them.
Thanks for more investigation!
\nI see, I know some strong Kagglers use full-data training, but I was worrying training without val score; e.g. should we change batch size to have same number steps or keep batch size and change learning rate? Your result confirms that full-data training can be difficult and requires experience. In this competition, we have lots of data, so I keep one file (500 data) per family as a \"test set,\" in addition to K-fold validation set. This helped me having \"test score\" in the almost full-data training.
", "votes": null, "replies": [ { "id": 3241187, "author_name": "iwaiwa", "author_url": "", "post_date": "07/04/2025 16:06:51", "content": "I think your point is spot-on. In fact, bumping the training set from 88% to 95% would probably be enough to see whether the extra data boosts score. Putting too much faith in the Public Leaderboard is risky in many competitions—although in this one we were lucky that the public and private scores were closely aligned.
\n\n\nWeighted median is great! I only thought about median and weighted mean. Is it implemented in well known libraries or in public notebook?
\n
I missed your last question. Since Polars doesn't have a built-in weighted median, I implemented my own. As far as I know, this approach hasn't been widely used for ensembling in Kaggle notebooks. Median was already giving a much better score than the mean, so I experimented with variations on the median, and the weighted median happened to work well in this case.
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3238099": "## Introduction\nFirst of all, I would like to express my deep gratitude to the organizers Yale, UNC-CH, Kaggle. This competition was interesting and educational, and I'm glad I participated.\nI would also like to deeply thank @brendanartley for providing the public code. My solution is developed based on Bartley's public code.\n\n---\n## 1.Overview\n- Trained several models using ConvNeXt and CAFormer models with different model-size configurations\n- Best single model was ConvNeXt-Base (CV=23.87, LB=25.7)\n- Final submission was a weighted median ensemble of 9 submissions achieving Public MAE = 23.2, Private MAE = 23.1\n- Due to the large dataset size, cross-validation was challenging, so I adopted an extended hold-out approach, which maintained high correlation with Public LB\n\n## 2. Dataset\n- **OpenFWI (all data)** only \n- No resizing, no augmentation, no synthetic generation \n\n## 3. Validation Strategy\n- **Expanded hold-out** split: **Train ≈ 88 % / Val ≈ 12 %** from the public code. \n\n## 4. Single-Model Results \n\n**Table 1 – Single-model performance (lower is better)** \n\n| Backbone | Epochs | CV MAE | Public MAE | Notes |\n|----------|-------:|-------:|-----------:|-------|\n| ConvNeXt-Base | 200 | **23.87** | **25.7** | Best single model |\n| CAFormer-m36 | 180 | 24.08 | 26.0 | StarReLU |\n| CAFormer-b36 | 180 | 24.96 | 29.5 | StarReLU |\n| ConvNeXt-Base | 150 | 25.58 | 26.7 | Checkpoint fine-tuned twice |\n| ConvNeXt-Large | 150 | 26.08 | 27.8 | — |\n| Flash InternImage-b | 50 | 34.60 | 37.6 | FP32, ≈ 210 min / epoch |\n| ConvNeXt-Base (full data) | 218 | — | 27.7 | Trained on 100 % data |\n| ConvNeXt-Base (full data) | 243 | — | 26.7 | Trained on 100 % data |\n| ConvNeXt-Base (full data) | 258 | — | 26.6 | Trained on 100 % data |\n\nHyper-parameters tuned: `weight_decay`, `dropout_rate`, `decoder_channels`, `clip_grad_norm` \nLR restarts: resumed from the previous terminal LR (`1 e-5`) with `CosineAnnealingLR(T_max = 50)`, which proved more stable than warm restarts.\n\n## 5. Ensembling & Post-processing \n\n**Table 2 – Ensemble performance (Public MAE)** \n\n| Method | Submissions | Public MAE | Δ vs. Best Single |\n|--------|-----------:|----:|------------------:|\n| Simple median | 9 | 23.4 | −2.3 |\n| **Weighted median (final)** | **9** | **23.2** | **−2.5** |\n\n- Median-based methods clearly outperform means for MAE. \n- Weighted median improved MAE by **0.2** over the simple median. \n- Type-wise weighted means (after predicting sample type with 99.5 % accuracy) did **not** beat median. \n- Post-processing is limited to **`clip(1500, 4500)`** (additional –0.002 CV MAE).\n\n## 6. Speed-up Techniques\n| Category | Key settings |\n|----------|--------------|\n| **Compiler / Optimizer** | `torch.compile(mode=\"max-autotune\")`, **Fused AdamW** |\n| **Mixed Precision** | AMP (**bfloat16**) for all ConvNeXt / CAFormer runs |\n| **DataLoader** | `num_workers` tuned, `prefetch_factor` tuned, `pin_memory=True`, `persistent_workers=True` |\n| **Hardware** | Experiments run on **RTX 3090** & **A100**