![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F44a7cfd8c762cf2da6b321f4a1c1686f%2FSelection_999(8225).png?generation=1749052179015479&alt=media\")
Happy to share an improved full resolution model. This model uses a CAFormer encoder with an improved decoder (pixel shuffle, SCSE, intermediate convolutions, etc.).
\n\n+--------------+----------+\n|||\n+--------------+----------+\n|||\n|||\n|||\n|||\n|||\n|||\n|||\n|||\n|||\n|||\n+--------------+----------+\n|||\n+--------------+----------+\nThis will be my last notebook in this competition, but I am hopeful that we can push this architecture further as a community!
\nHappy Kaggling š
", "messages": [ { "id": 3215873, "postDate": "2025-06-02T18:38:52.270Z", "content": "Happy to share an improved full resolution model. This model uses a CAFormer encoder with an improved decoder (pixel shuffle, SCSE, intermediate convolutions, etc.).
\n\n+--------------+----------+\n|||\n+--------------+----------+\n|||\n|||\n|||\n|||\n|||\n|||\n|||\n|||\n|||\n|||\n+--------------+----------+\n|||\n+--------------+----------+\nThis will be my last notebook in this competition, but I am hopeful that we can push this architecture further as a community!
\nHappy Kaggling š
", "rawMarkdown": "Happy to share an improved full resolution model. This model uses a CAFormer encoder with an improved decoder (pixel shuffle, SCSE, intermediate convolutions, etc.).\n\nNotebook [here](https://www.kaggle.com/code/brendanartley/caformer-full-resolution-improved)\nDataset [here](https://www.kaggle.com/datasets/brendanartley/openfwi-preprocessed-72x72)\n\n```\n+--------------+----------+\n| Dataset | Caformer |\n+--------------+----------+\n| CurveFault_A | 4.00 |\n| CurveFault_B | 71.06 |\n| CurveVel_A | 9.19 |\n| CurveVel_B | 38.60 |\n| FlatFault_A | 2.58 |\n| FlatFault_B | 24.29 |\n| FlatVel_A | 1.31 |\n| FlatVel_B | 7.20 |\n| Style_A | 34.34 |\n| Style_B | 48.90 |\n+--------------+----------+\n| Overall | 24.15 |\n+--------------+----------+\n```\n\nThis will be my last notebook in this competition, but I am hopeful that we can push this architecture further as a community!\n\nHappy Kaggling š", "votes": 77 }, { "id": 3217147, "postDate": "2025-06-04T15:15:24.837Z", "content": "\"this pipeline takes far too long to converge.\"
\nhttps://arxiv.org/pdf/2412.19510v1
\nParameter Efficient Fine-Tuning for Deep Learning-Based Full-Waveform Inversion
try : Full fine-tuning(FFT-PFM) and LoRA-PFM ?
\n
From a quick glance, the results of that paper look subpar to any of the current LB models. Seems convoluted and unintuitive to me. At least do a proper MOE modelā¦ Happy to be disproved
", "rawMarkdown": "From a quick glance, the results of that paper look subpar to any of the current LB models. Seems convoluted and unintuitive to me. At least do a proper MOE model... Happy to be disproved", "votes": 2 }, { "id": 3219834, "postDate": "2025-06-08T11:13:00.057Z", "content": "\n\nFrom a quick glance, the results of that paper look subpar to any of the current LB models.
\n
Well, if we can't get better results than non-Kagglers, what are we even doing here š
", "rawMarkdown": "> From a quick glance, the results of that paper look subpar to any of the current LB models. \n\nWell, if we can't get better results than non-Kagglers, what are we even doing here š", "votes": 4 }, { "id": 3220620, "postDate": "2025-06-09T15:50:30.773Z", "content": "But it's pretty rare to see a Kaggle competition where methods from papers in the past few years can't even beat some brute-force training of a public timm model. Kinda makes you wonder if it's even worth continuing to research thisā¦
", "rawMarkdown": "But it's pretty rare to see a Kaggle competition where methods from papers in the past few years can't even beat some brute-force training of a public timm model. Kinda makes you wonder if it's even worth continuing to research this..." }, { "id": 3220658, "postDate": "2025-06-09T17:14:03.743Z", "content": "I always wonder - is this frowned upon in research to use pretrained model ? Or is it because the goal of research is to build networks from scratch ? (I kaggle as hobby and not from ML/Data Science background) so it is a genuine question ā¦
", "rawMarkdown": "I always wonder - is this frowned upon in research to use pretrained model ? Or is it because the goal of research is to build networks from scratch ? (I kaggle as hobby and not from ML/Data Science background) so it is a genuine question ..." }, { "id": 3220677, "postDate": "2025-06-09T17:35:34.087Z", "content": "\n\nKinda makes you wonder if it's even worth continuing to research thisā¦
\n
Why wouldn't it? The novelty is in defining the problem, creating data, applying, verifying etc. I dare to say, few researchers can get competitive results against top Kagglers, on almost any dataset that is not trivial (i.e. small/simple). But we still need researchers š
\n\n\nBut it's pretty rare to see a Kaggle competition where methods from papers in the past few years can't even beat some brute-force training of a public timm model
\n
It's far less trivial than what you imply. Without a 'certain' Kaggler, most people here would probably still be stuck somewhere at ~LB50+
", "rawMarkdown": "> Kinda makes you wonder if it's even worth continuing to research thisā¦\n\nWhy wouldn't it? The novelty is in defining the problem, creating data, applying, verifying etc. I dare to say, few researchers can get competitive results against top Kagglers, on almost any dataset that is not trivial (i.e. small/simple). But we still need researchers š\n\n>But it's pretty rare to see a Kaggle competition where methods from papers in the past few years can't even beat some brute-force training of a public timm model \n\nIt's far less trivial than what you imply. Without a 'certain' Kaggler, most people here would probably still be stuck somewhere at ~LB50+", "votes": 1 }, { "id": 3220712, "postDate": "2025-06-09T18:42:49.967Z", "content": "\n\nsome brute-force training of a public timm model.
\n
If this is what you think @brendanartley notebooks are about then you miss a lot. I suggest reading his code. You'll see that he did a bit more than using a timm model.
", "rawMarkdown": "> some brute-force training of a public timm model.\n\nIf this is what you think @brendanartley notebooks are about then you miss a lot. I suggest reading his code. You'll see that he did a bit more than using a timm model.", "votes": 4 }, { "id": 3220721, "postDate": "2025-06-09T19:15:26.503Z", "content": "\n\nwonder if it's even worth continuing to research
\n
Alex Krizhevsky is probably more a \"researcher\" than a \"Kaggler\". But he basically kicked everyone's ass when he did a competition in 2012.
\nIt's not an exaggeration to say that because of that event, Alex brought neural network back to life after decades of it being shunned away, and the rest was history.
\nOr let's take the researcher Tianqi Chen, who developed practical gradient boosting, which I don't think any Kaggler today can live without.
\nI think it's not only worth it but in fact essential to keep doing research.
", "rawMarkdown": ">wonder if it's even worth continuing to research\n\nAlex Krizhevsky is probably more a \"researcher\" than a \"Kaggler\". But he basically kicked everyone's ass when he did a competition in 2012.\n\nIt's not an exaggeration to say that because of that event, Alex brought neural network back to life after decades of it being shunned away, and the rest was history.\n\nOr let's take the researcher Tianqi Chen, who developed practical gradient boosting, which I don't think any Kaggler today can live without.\n\nI think it's not only worth it but in fact *essential* to keep doing research.", "votes": 3 }, { "id": 3220778, "postDate": "2025-06-10T00:35:19.477Z", "content": "This topic can easily spiral into a much broader discussionāno one would dare deny the importance of research in general. Thatās why I originally limited the scope of my comment to āpapers from recent years related to this specific competition.ā
\n@brendanartley did a great job with the task, but I wouldnāt consider what he did to be traditional research in this specific field. Itās more like applying a general-purpose method to this domain.
\nBut if I were a researcher in this field and saw these results, Iād probably cough up blood and die.
", "rawMarkdown": "This topic can easily spiral into a much broader discussionāno one would dare deny the importance of research in general. Thatās why I originally limited the scope of my comment to āpapers from recent years related to this specific competition.ā\n\n@brendanartley did a great job with the task, but I wouldnāt consider what he did to be traditional research in this specific field. Itās more like applying a general-purpose method to this domain.\n\nBut if I were a researcher in this field and saw these results, Iād probably cough up blood and die.", "votes": 2 } ] } ] }, { "id": 3217142, "postDate": "2025-06-04T15:12:18.253Z", "content": "a trick is use fp32 in input and model for inference. you will get something like +0.0025 in lb (?)
", "rawMarkdown": "a trick is use fp32 in input and model for inference. you will get something like +0.0025 in lb (?)", "votes": 8, "replies": [ { "id": 3217254, "postDate": "2025-06-04T18:02:46.680Z", "content": "@hengck23
\nhow you do that? I tried use model = model.to(torch.float) and test input with fp32 in /kaggle/input/waveform-inversion/test but seem nothing improved.
Disable auto cast. Lb score is the same (due to decimal improvement only)but better ranking
", "rawMarkdown": "Disable auto cast. Lb score is the same (due to decimal improvement only)but better ranking", "votes": 5 }, { "id": 3217273, "postDate": "2025-06-04T18:50:48.220Z", "content": "I will try it. Thanks.
", "rawMarkdown": "I will try it. Thanks.", "votes": 1 } ] }, { "id": 3220406, "postDate": "2025-06-09T08:55:08.300Z", "content": "Did the score improve during your local testing?
", "rawMarkdown": "Did the score improve during your local testing?" } ] }, { "id": 3216127, "postDate": "2025-06-03T06:36:51.947Z", "content": "@brendanartley Have you tried re-training convnext large with the same settings as caformer i.e. 150 epochs + improved decoder + cosine scheduler ?
\nCaformer training time was 2x longer than convnext large so I'm not sure it's worth the extra cost.
Good question @andy2709.
\nI have not tried convnext large due to compute limits, but 2x faster training sounds nice!
this is segmentation head in your code:
\n (nn.Module):\n ():\n ().__init__()\n .conv = nn.Conv2d(\n in_channels, out_channels, kernel_size=kernel_size,\n padding=kernel_size // \n )\n .upsample = UpSample(\n spatial_dims=,\n in_channels=out_channels,\n out_channels=out_channels,\n scale_factor=scale_factor,\n mode=mode,\n )\n\n ():\n x = .conv(x)\n x = .upsample(x)\n x\ni haven't done in-depth experiment yet, but below are from my experiences:
\n```
\ne.g. something like
last_feature = lastdecoder(input = from previous decoder, skip = original input)
\npredicted_velocity = prediction(last_feature)
where prediction= nn.Seqential(
\nā¦.
\nnn.Conv2d( 1x1 kernel)
\n)
\n``
You are bound by upscaling error only if you upsample with something like bilinear. If you upsample with ConvTranspose2d there is no bound (although it might be harder to learn than upscaling to full resolution before final layer, but then it's trade-off against compute and time, no bound).
\nTheoretically we can compress all the seis into a single vector and then predict from this vector each pixel - there is nothing imposing any bound.
@brendanartley
\nWhen turning a Train, the IO seems to be a problem(?). What is your countermeasure for this?
To be precise, the speed decreases around the 800-step mark.![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3790528%2F06453f06b67c9a8dede330ce4e0d66b2%2F.png?generation=1748964167653260&alt=media\")
@brendanartley Thanks for sharing one more milestone very hard to break struggle to reach <100 --> 69.7 --> 33.2 --> 28.8, could you share CV scores of training 20-25 epochs of CurveFault-B ( your cv and training logs if possible )
", "rawMarkdown": "@brendanartley Thanks for sharing one more milestone very hard to break **struggle to reach <100** --> **69.7** --> **33.2** --> **28.8**, could you share [CV scores](https://www.kaggle.com/code/brendanartley/convnext-full-resolution-baseline/comments#3206500) of training 20-25 epochs of CurveFault-B ( your cv and training logs if possible )", "votes": 4 }, { "id": 3224682, "postDate": "2025-06-15T10:12:01.490Z", "content": "This is very interesting
", "rawMarkdown": "This is very interesting", "votes": 1 }, { "id": 3223522, "postDate": "2025-06-13T12:44:36.783Z", "content": "whats the upsample mode , pixel shuffle or deconv ??
", "rawMarkdown": "whats the upsample mode , pixel shuffle or deconv ??", "votes": 1, "replies": [ { "id": 3224846, "postDate": "2025-06-15T15:02:57.960Z", "content": "Read the code :D
", "rawMarkdown": "Read the code :D", "votes": 2 } ] }, { "id": 3217724, "postDate": "2025-06-05T10:59:35.717Z", "content": "When you're testing model performance on a small order of magnitude, do you use CurveFault_B, and if so, what the results are, and then you use them for training on all datasets?
", "rawMarkdown": "When you're testing model performance on a small order of magnitude, do you use CurveFault_B, and if so, what the results are, and then you use them for training on all datasets?", "votes": 1, "replies": [ { "id": 3219155, "postDate": "2025-06-07T08:13:20.073Z", "content": "https://www.kaggle.com/code/brendanartley/convnext-full-resolution-baseline/comments#3206500
", "rawMarkdown": "https://www.kaggle.com/code/brendanartley/convnext-full-resolution-baseline/comments#3206500", "votes": 2 } ] }, { "id": 3217157, "postDate": "2025-06-04T15:33:31.020Z", "content": "Thanks for sharing best model with full trainingā¦.
", "rawMarkdown": "Thanks for sharing best model with full training....", "votes": 1 }, { "id": 3216520, "postDate": "2025-06-03T17:32:56.413Z", "content": "\nOk - maybe its a lack of coffee or the fact that I am an old fart - but I think I read all the other comments several times and failed to find the string that leads us to the code for training this latest nice bit of work. I tried re-using the previous train but that failed.
\nIs it your intent to not share the training code for your last model?
\n\n", "rawMarkdown": "[Bartley](https://www.kaggle.com/brendanartley)\n\nOk - maybe its a lack of coffee or the fact that I am an old fart - but I think I read all the other comments several times and failed to find the string that leads us to the code for training this latest nice bit of work. I tried re-using the previous train but that failed.\n\nIs it your intent to not share the training code for your last model?\n\n>Hi @billfan88, I appreciate your questions but please read other comments first. Cheers š", "votes": 1, "replies": [ { "id": 3216559, "postDate": "2025-06-03T18:31:55.850Z", "content": "Hi @billfan88, I appreciate your questions but please read other comments first. Cheers š
\n
Hi @pcjimmmy, @ttyn4519, thanks for your comments.
\nI did not include training code in this notebook because the model is trained outside of the Kaggle environment. I do not see issues when running locally.
", "rawMarkdown": "Hi @pcjimmmy, @ttyn4519, thanks for your comments.\n\nI did not include training code in this notebook because the model is trained outside of the Kaggle environment. I do not see issues when running locally.", "votes": 1, "replies": [ { "id": 3216567, "postDate": "2025-06-03T18:37:43.663Z", "content": "Yes - and I fork it outside of kaggle :) That's the only place that multiple days of running are possible. Your previous ran 10 days on one of my local machines.
\nAre you telling me that you can re-use the previous train.py, util.py and the full cfg and it runs OK?
", "rawMarkdown": "Yes - and I fork it outside of kaggle :) That's the only place that multiple days of running are possible. Your previous ran 10 days on one of my local machines.\n\nAre you telling me that you can re-use the previous train.py, util.py and the full cfg and it runs OK? ", "votes": 1 }, { "id": 3216578, "postDate": "2025-06-03T18:45:13.180Z", "content": "I use a slightly different training script, but the cfg is the same. It's hard to predict how the pipeline will act across different GPUs, CUDA versions, and environments, but it works for me š¤·āāļø
", "rawMarkdown": "I use a slightly different training script, but the cfg is the same. It's hard to predict how the pipeline will act across different GPUs, CUDA versions, and environments, but it works for me š¤·āāļø", "votes": 2 }, { "id": 3216633, "postDate": "2025-06-03T21:37:16.347Z", "content": "So- in summary - your not sharing the full code for this last model?
", "rawMarkdown": "So- in summary - your not sharing the full code for this last model?", "votes": -1 }, { "id": 3220810, "postDate": "2025-06-10T02:24:29.347Z", "rawMarkdown": "", "isDeleted": true } ] } ] }, { "id": 3216040, "postDate": "2025-06-03T03:24:04.390Z", "content": "Thanks for sharing best model with full training - one more break through notebook and solution.
\nCould you share your training logs of these models after 50 epochs?
@waterjoe @seshurajup here are the loss values. After 50 epochs training loss is ~21.5 and validation loss is 29.3.
\n![\"Loss](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5570735%2F1900a9c815836e8b891baeed2ed69461%2FScreen%20Shot%202025-06-02%20at%209.16.55%20PM.png?generation=1748924280482744&alt=media\")
Was the model trained as in previous baselines?
", "rawMarkdown": "Was the model trained as in previous baselines?", "votes": 1, "replies": [ { "id": 3215926, "postDate": "2025-06-02T21:28:33.530Z", "content": "Hi @xbar19 and @billfan88, thanks for the comments.
\nThe training is very similar. The differences are that this model is trained for 150 epochs and uses a custom learning rate scheduler shown here
", "rawMarkdown": "Hi @xbar19 and @billfan88, thanks for the comments.\n \nThe training is very similar. The differences are that this model is trained for 150 epochs and uses a custom learning rate scheduler shown [here](https://www.kaggle.com/code/brendanartley/caformer-full-resolution-improved?scriptVersionId=243291347&cellId=4)", "votes": 2, "replies": [ { "id": 3215938, "postDate": "2025-06-02T21:48:06.603Z", "content": "Thank you for your reply and your work!
", "rawMarkdown": "Thank you for your reply and your work!", "votes": 1 }, { "id": 3215946, "postDate": "2025-06-02T22:21:46.487Z", "content": "How much time it took to train on 150 epochs? Damn this is one long training
", "rawMarkdown": "How much time it took to train on 150 epochs? Damn this is one long training", "votes": 4 }, { "id": 3215958, "postDate": "2025-06-02T23:25:38.563Z", "content": "@shlomoron, it took 8 days (~192 hours) on a single 4090.
\nYou make a good point ā this pipeline takes far too long to converge. I am hoping people can share ideas on how we can decrease this time!
", "rawMarkdown": "@shlomoron, it took 8 days (~192 hours) on a single 4090.\n\nYou make a good point ā this pipeline takes far too long to converge. I am hoping people can share ideas on how we can decrease this time!", "votes": 5 }, { "id": 3215959, "postDate": "2025-06-02T23:29:23.150Z", "content": "Holy * shit! 8 daysā¦
\nFull 8*24 days? Damn
\n\nthis pipeline takes far too long to converge.
\n
MAE is always slow to converge. Alas.
", "rawMarkdown": "> this pipeline takes far too long to converge. \n\nMAE is always slow to converge. Alas.", "votes": 1 }, { "id": 3217497, "postDate": "2025-06-05T04:30:26.740Z", "content": "It would take me ~8 days on a single 4090 to hit my current LB (17.1), the trainings take a long time to converge
", "rawMarkdown": "It would take me ~8 days on a single 4090 to hit my current LB (17.1), the trainings take a long time to converge ", "votes": 4 }, { "id": 3217741, "postDate": "2025-06-05T11:18:10.863Z", "content": "Magic of L1 --> Run Long - Thanks for sharing @brendanartley @harshitsheoran
\n\n", "rawMarkdown": "**Magic of L1 --> Run Long** - Thanks for sharing @brendanartley @harshitsheoran \n- it took 8 days (~192 hours) on a single 4090 -- @brendanartley \n- It would take me ~8 days on a single 4090 to hit my current LB (17.1), the trainings take a long time to converge @harshitsheoran \n\n--- \n\n> -- How to decided when to stop, most of epochs getting negligible loss improvement (small improvement - my experiments based on only CurveFault_B)\n -- some batches are too high even using clip the grads\n", "votes": 1 }, { "id": 3217866, "postDate": "2025-06-05T13:15:42.077Z", "content": "-- How to decided when to stop, most of epochs getting negligible loss improvement (small improvement - my experiments based on only CurveFault_B)
\n
\n -- some batches are too high even using clip the grads
\n
Since there is a lot of information on 4090, I will share the results of V100x8 DDP.
\nTo reproduce this old notebook's convnext[CV 31.9 LB 36.4] locally, it took 2 days for 150 epochs of training, and I was able to obtain the result of [CV30.07, LB35.1].
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F761268%2Fc49c05fe040a84a4cb8bd2e1b3e1644e%2FScreenshot%202025-06-05%20at%204.46.22PM.png?generation=1749122195332342&alt=media\")
=========================\nCurveFault_A 6.15\nCurveFault_B 86.72\nCurveVel_A 13.69\nCurveVel_B 48.81\nFlatFault_A 4.34\nFlatFault_B 33.73\nFlatVel_A 2.92\nFlatVel_B 12.11\nStyle_A 37.07\nStyle_B 55.18\n=========================\n\nCould you please tell me whether the data you used was the original FP32 data? When I tried to reproduce Caformerās training results using Kaggleās FP16 open dataset, the cross-validation converged around 34, and Iām not sure if this is a dataset issue.
", "rawMarkdown": "Could you please tell me whether the data you used was the original FP32 data? When I tried to reproduce Caformerās training results using Kaggleās FP16 open dataset, the cross-validation converged around 34, and Iām not sure if this is a dataset issue.", "votes": 1 }, { "id": 3220288, "postDate": "2025-06-09T05:39:21.670Z", "content": "Hi @oigckko
\nIf you have any questions regarding my above post \"Val MAE: 30.07\", I hope this helps.
The training dataset remains unchanged from the public notebook.
\ndf= pd()\n...\np1 = os(, row)\n p2 = os(, row(), , row())\n p3 = os(, row)\n p4 = os(, row(), , row())\nTrain section Same as starter notebook?
", "rawMarkdown": "Train section Same as starter notebook?", "votes": 2, "replies": [ { "id": 3216322, "postDate": "2025-06-03T12:18:06.103Z", "content": "Hi @billfan88, I appreciate your questions but please read other comments first. Cheers š
", "rawMarkdown": "Hi @billfan88, I appreciate your questions but please read other comments first. Cheers š" } ] }, { "id": 3215917, "postDate": "2025-06-02T20:57:34.400Z", "content": "Awesome!
\nI wonder how many backbones you tried. You obviously have a good methodology to pick the ones you test.
", "rawMarkdown": "Awesome!\n\nI wonder how many backbones you tried. You obviously have a good methodology to pick the ones you test.", "votes": 2, "replies": [ { "id": 3215933, "postDate": "2025-06-02T21:42:40.240Z", "content": "Hi @cpmpml, I have tried Convnext, Caformer, Hgnet, NextVIT and Hornet so far.
\nMy methodology is quite simple. First I try and modify the backbone for 1000 x 70 in as few steps as possible. Then, I follow this validation scheme. It may not be the most accurate, but it seems to work well given limited compute!
", "rawMarkdown": "Hi @cpmpml, I have tried Convnext, Caformer, Hgnet, NextVIT and Hornet so far.\n\nMy methodology is quite simple. First I try and modify the backbone for 1000 x 70 in as few steps as possible. Then, I follow [this](https://www.kaggle.com/code/brendanartley/convnext-full-resolution-baseline/comments#3206500) validation scheme. It may not be the most accurate, but it seems to work well given limited compute!\n", "votes": 7, "replies": [ { "id": 3215939, "postDate": "2025-06-02T21:49:12.983Z", "content": "Yes, I noticed you use a subset to validate your choice. This is a good idea.
", "rawMarkdown": "Yes, I noticed you use a subset to validate your choice. This is a good idea.", "votes": 4 } ] } ] }, { "id": 3221540, "postDate": "2025-06-11T06:45:49.307Z", "content": "Why is my train loss always higher than vaild loss during training, is this normal?
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F25710556%2F53e5d71b04b7954200dfa184996221ba%2F2025-06-11%20144329.png?generation=1749624341620051&alt=media\")
@oliver34 its same for me too, but it got switch after few epochs when i tested on CurveFault_B dataset experiments.
\n-- How many epochs you are running of CAFormer? Did you run from backbone or from Bartley weights?
", "rawMarkdown": "@oliver34 its same for me too, but it got switch after few epochs when i tested on [CurveFault_B](https://www.kaggle.com/competitions/waveform-inversion/discussion/580892#3221434) dataset experiments.\n\n-- How many epochs you are running of CAFormer? Did you run from backbone or from Bartley weights?", "votes": 1, "replies": [ { "id": 3221557, "postDate": "2025-06-11T07:11:46.630Z", "content": "44epoch,I only used the weights of the backbone
", "rawMarkdown": "44epoch,I only used the weights of the backbone", "votes": 2 }, { "id": 3221558, "postDate": "2025-06-11T07:15:54.113Z", "content": "@oliver34 will you share your train/val loss for epochs 5, 10, 20, 30 ? -- which gpu - training finishing in 1h 25mins for epoch?
", "rawMarkdown": "@oliver34 will you share your train/val loss for epochs 5, 10, 20, 30 ? -- which gpu - training finishing in 1h 25mins for epoch?", "votes": 1 }, { "id": 3221562, "postDate": "2025-06-11T07:23:45.047Z", "content": "They are 47.70, 40.59, 31.86, 28.70. I also encountered the situation you mentioned at CurveFault_B. --I'm using two 4090s and it seems like it takes 9 days to complete a 150epochš„
", "rawMarkdown": "They are 47.70, 40.59, 31.86, 28.70. I also encountered the situation you mentioned at CurveFault_B. --I'm using two 4090s and it seems like it takes 9 days to complete a 150epochš„", "votes": 1 }, { "id": 3221639, "postDate": "2025-06-11T09:46:46.287Z", "content": "Hello, may I ask if you are using the fp32 data or the fp16 dataset processed by Kaggle?
", "rawMarkdown": "Hello, may I ask if you are using the fp32 data or the fp16 dataset processed by Kaggle?" }, { "id": 3221666, "postDate": "2025-06-11T10:30:15.037Z", "content": "the fp16 dataset processed by Kaggle
", "rawMarkdown": "the fp16 dataset processed by Kaggle", "votes": 1 }, { "id": 3224699, "postDate": "2025-06-15T10:45:50.537Z", "content": "@oliver34 could you share cv scores of 40, 60, 80, 100?
", "rawMarkdown": "@oliver34 could you share cv scores of 40, 60, 80, 100? ", "votes": 1 }, { "id": 3224761, "postDate": "2025-06-15T12:45:32.147Z", "content": "I would love to share them, 26.88, 24.58, 23.08, 22.11.
", "rawMarkdown": "I would love to share them, 26.88, 24.58, 23.08, 22.11.", "votes": 1 }, { "id": 3225362, "postDate": "2025-06-16T09:50:55.937Z", "content": "With the \"Tips to speed up training\" optimizations, training can be completed within 2 days using 2Ć4090
", "rawMarkdown": "With the \"[Tips to speed up training](https://www.kaggle.com/competitions/waveform-inversion/discussion/583896)\" optimizations, training can be completed within 2 days using 2Ć4090 ", "votes": 1 }, { "id": 3225364, "postDate": "2025-06-16T09:53:50.547Z", "content": "I donāt think 4090 can train caformer for 150 epoch for 2 days
", "rawMarkdown": "I donāt think 4090 can train caformer for 150 epoch for 2 days", "votes": 1 } ] }, { "id": 3222445, "postDate": "2025-06-12T07:46:55.827Z", "content": "batch-wise average vs overall average
", "rawMarkdown": "batch-wise average vs overall average", "votes": 1 }, { "id": 3223545, "postDate": "2025-06-13T13:28:12.613Z", "content": "\n\nmy train loss always higher than vaild loss during training
\n
I also observe this. I haven't observed a single epoch in which my val loss is worse than my train loss. I'm still relatively early in my training epochs, though. Glad to know I'm not the only one. š
", "rawMarkdown": ">my train loss always higher than vaild loss during training\n\nI also observe this. I haven't observed a single epoch in which my val loss is worse than my train loss. I'm still relatively early in my training epochs, though. Glad to know I'm not the only one. š" } ] }, { "id": 3221539, "postDate": "2025-06-11T06:45:11.713Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3225006, "postDate": "2025-06-15T19:37:03.183Z", "content": "Great work!
", "rawMarkdown": "Great work!", "votes": 1 }, { "id": 3223364, "postDate": "2025-06-13T07:42:11.890Z", "content": "Thanks for the information
", "rawMarkdown": "Thanks for the information" } ], "comments": [ { "id": 3217147, "author_name": "hengck23", "author_url": "", "post_date": "2025-06-04T15:15:24.837000", "content": "\"this pipeline takes far too long to converge.\"
\nhttps://arxiv.org/pdf/2412.19510v1
\nParameter Efficient Fine-Tuning for Deep Learning-Based Full-Waveform Inversion
try : Full fine-tuning(FFT-PFM) and LoRA-PFM ?
\n
From a quick glance, the results of that paper look subpar to any of the current LB models. Seems convoluted and unintuitive to me. At least do a proper MOE modelā¦ Happy to be disproved
", "votes": 2, "replies": [] }, { "id": 3219834, "author_name": "greySnow", "author_url": "", "post_date": "2025-06-08T11:13:00.057000", "content": "\n\nFrom a quick glance, the results of that paper look subpar to any of the current LB models.
\n
Well, if we can't get better results than non-Kagglers, what are we even doing here š
", "votes": 4, "replies": [] }, { "id": 3220620, "author_name": "atom1231", "author_url": "", "post_date": "2025-06-09T15:50:30.773000", "content": "But it's pretty rare to see a Kaggle competition where methods from papers in the past few years can't even beat some brute-force training of a public timm model. Kinda makes you wonder if it's even worth continuing to research thisā¦
", "votes": 0, "replies": [] }, { "id": 3220658, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2025-06-09T17:14:03.743000", "content": "I always wonder - is this frowned upon in research to use pretrained model ? Or is it because the goal of research is to build networks from scratch ? (I kaggle as hobby and not from ML/Data Science background) so it is a genuine question ā¦
", "votes": 0, "replies": [] }, { "id": 3220677, "author_name": "greySnow", "author_url": "", "post_date": "2025-06-09T17:35:34.087000", "content": "\n\nKinda makes you wonder if it's even worth continuing to research thisā¦
\n
Why wouldn't it? The novelty is in defining the problem, creating data, applying, verifying etc. I dare to say, few researchers can get competitive results against top Kagglers, on almost any dataset that is not trivial (i.e. small/simple). But we still need researchers š
\n\n\nBut it's pretty rare to see a Kaggle competition where methods from papers in the past few years can't even beat some brute-force training of a public timm model
\n
It's far less trivial than what you imply. Without a 'certain' Kaggler, most people here would probably still be stuck somewhere at ~LB50+
", "votes": 1, "replies": [] }, { "id": 3220712, "author_name": "CPMP", "author_url": "", "post_date": "2025-06-09T18:42:49.967000", "content": "\n\nsome brute-force training of a public timm model.
\n
If this is what you think @brendanartley notebooks are about then you miss a lot. I suggest reading his code. You'll see that he did a bit more than using a timm model.
", "votes": 4, "replies": [] }, { "id": 3220721, "author_name": "Truth Seeker", "author_url": "", "post_date": "2025-06-09T19:15:26.503000", "content": "\n\nwonder if it's even worth continuing to research
\n
Alex Krizhevsky is probably more a \"researcher\" than a \"Kaggler\". But he basically kicked everyone's ass when he did a competition in 2012.
\nIt's not an exaggeration to say that because of that event, Alex brought neural network back to life after decades of it being shunned away, and the rest was history.
\nOr let's take the researcher Tianqi Chen, who developed practical gradient boosting, which I don't think any Kaggler today can live without.
\nI think it's not only worth it but in fact essential to keep doing research.
", "votes": 3, "replies": [] }, { "id": 3220778, "author_name": "atom1231", "author_url": "", "post_date": "2025-06-10T00:35:19.477000", "content": "This topic can easily spiral into a much broader discussionāno one would dare deny the importance of research in general. Thatās why I originally limited the scope of my comment to āpapers from recent years related to this specific competition.ā
\n@brendanartley did a great job with the task, but I wouldnāt consider what he did to be traditional research in this specific field. Itās more like applying a general-purpose method to this domain.
\nBut if I were a researcher in this field and saw these results, Iād probably cough up blood and die.
", "votes": 2, "replies": [] } ] } ] }, { "id": 3217142, "author_name": "hengck23", "author_url": "", "post_date": "2025-06-04T15:12:18.253000", "content": "a trick is use fp32 in input and model for inference. you will get something like +0.0025 in lb (?)
", "votes": 8, "replies": [ { "id": 3217254, "author_name": "Kawa", "author_url": "", "post_date": "2025-06-04T18:02:46.680000", "content": "@hengck23
\nhow you do that? I tried use model = model.to(torch.float) and test input with fp32 in /kaggle/input/waveform-inversion/test but seem nothing improved.
Disable auto cast. Lb score is the same (due to decimal improvement only)but better ranking
", "votes": 5, "replies": [] }, { "id": 3217273, "author_name": "Kawa", "author_url": "", "post_date": "2025-06-04T18:50:48.220000", "content": "I will try it. Thanks.
", "votes": 1, "replies": [] } ] }, { "id": 3220406, "author_name": "Siyuan Yin", "author_url": "", "post_date": "2025-06-09T08:55:08.300000", "content": "Did the score improve during your local testing?
", "votes": 0, "replies": [] } ] }, { "id": 3216127, "author_name": "NguyenThanhNhan", "author_url": "", "post_date": "2025-06-03T06:36:51.947000", "content": "@brendanartley Have you tried re-training convnext large with the same settings as caformer i.e. 150 epochs + improved decoder + cosine scheduler ?
\nCaformer training time was 2x longer than convnext large so I'm not sure it's worth the extra cost.
Good question @andy2709.
\nI have not tried convnext large due to compute limits, but 2x faster training sounds nice!
this is segmentation head in your code:
\n (nn.Module):\n ():\n ().__init__()\n .conv = nn.Conv2d(\n in_channels, out_channels, kernel_size=kernel_size,\n padding=kernel_size // \n )\n .upsample = UpSample(\n spatial_dims=,\n in_channels=out_channels,\n out_channels=out_channels,\n scale_factor=scale_factor,\n mode=mode,\n )\n\n ():\n x = .conv(x)\n x = .upsample(x)\n x\ni haven't done in-depth experiment yet, but below are from my experiences:
\n```
\ne.g. something like
last_feature = lastdecoder(input = from previous decoder, skip = original input)
\npredicted_velocity = prediction(last_feature)
where prediction= nn.Seqential(
\nā¦.
\nnn.Conv2d( 1x1 kernel)
\n)
\n``
You are bound by upscaling error only if you upsample with something like bilinear. If you upsample with ConvTranspose2d there is no bound (although it might be harder to learn than upscaling to full resolution before final layer, but then it's trade-off against compute and time, no bound).
\nTheoretically we can compress all the seis into a single vector and then predict from this vector each pixel - there is nothing imposing any bound.
@brendanartley
\nWhen turning a Train, the IO seems to be a problem(?). What is your countermeasure for this?
To be precise, the speed decreases around the 800-step mark.
@brendanartley Thanks for sharing one more milestone very hard to break struggle to reach <100 --> 69.7 --> 33.2 --> 28.8, could you share CV scores of training 20-25 epochs of CurveFault-B ( your cv and training logs if possible )
", "votes": 4, "replies": [] }, { "id": 3224682, "author_name": "Tony Yang", "author_url": "", "post_date": "2025-06-15T10:12:01.490000", "content": "This is very interesting
", "votes": 1, "replies": [] }, { "id": 3223522, "author_name": "Surya_trainer", "author_url": "", "post_date": "2025-06-13T12:44:36.783000", "content": "whats the upsample mode , pixel shuffle or deconv ??
", "votes": 1, "replies": [ { "id": 3224846, "author_name": "CPMP", "author_url": "", "post_date": "2025-06-15T15:02:57.960000", "content": "Read the code :D
", "votes": 2, "replies": [] } ] }, { "id": 3217724, "author_name": "Flowers", "author_url": "", "post_date": "2025-06-05T10:59:35.717000", "content": "When you're testing model performance on a small order of magnitude, do you use CurveFault_B, and if so, what the results are, and then you use them for training on all datasets?
", "votes": 1, "replies": [ { "id": 3219155, "author_name": "water joe", "author_url": "", "post_date": "2025-06-07T08:13:20.073000", "content": "https://www.kaggle.com/code/brendanartley/convnext-full-resolution-baseline/comments#3206500
", "votes": 2, "replies": [] } ] }, { "id": 3217157, "author_name": "Sarah Arshad", "author_url": "", "post_date": "2025-06-04T15:33:31.020000", "content": "Thanks for sharing best model with full trainingā¦.
", "votes": 1, "replies": [] }, { "id": 3216520, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2025-06-03T17:32:56.413000", "content": "\nOk - maybe its a lack of coffee or the fact that I am an old fart - but I think I read all the other comments several times and failed to find the string that leads us to the code for training this latest nice bit of work. I tried re-using the previous train but that failed.
\nIs it your intent to not share the training code for your last model?
\n\n", "votes": 1, "replies": [ { "id": 3216559, "author_name": "Bartley", "author_url": "", "post_date": "2025-06-03T18:31:55.850000", "content": "Hi @billfan88, I appreciate your questions but please read other comments first. Cheers š
\n
Hi @pcjimmmy, @ttyn4519, thanks for your comments.
\nI did not include training code in this notebook because the model is trained outside of the Kaggle environment. I do not see issues when running locally.
", "votes": 1, "replies": [ { "id": 3216567, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2025-06-03T18:37:43.663000", "content": "Yes - and I fork it outside of kaggle :) That's the only place that multiple days of running are possible. Your previous ran 10 days on one of my local machines.
\nAre you telling me that you can re-use the previous train.py, util.py and the full cfg and it runs OK?
", "votes": 1, "replies": [] }, { "id": 3216578, "author_name": "Bartley", "author_url": "", "post_date": "2025-06-03T18:45:13.180000", "content": "I use a slightly different training script, but the cfg is the same. It's hard to predict how the pipeline will act across different GPUs, CUDA versions, and environments, but it works for me š¤·āāļø
", "votes": 2, "replies": [] }, { "id": 3216633, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2025-06-03T21:37:16.347000", "content": "So- in summary - your not sharing the full code for this last model?
", "votes": -1, "replies": [] }, { "id": 3220810, "author_name": "", "author_url": "", "post_date": "2025-06-10T02:24:29.347000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 3216040, "author_name": "water joe", "author_url": "", "post_date": "2025-06-03T03:24:04.390000", "content": "Thanks for sharing best model with full training - one more break through notebook and solution.
\nCould you share your training logs of these models after 50 epochs?
@waterjoe @seshurajup here are the loss values. After 50 epochs training loss is ~21.5 and validation loss is 29.3.
\n
Was the model trained as in previous baselines?
", "votes": 1, "replies": [ { "id": 3215926, "author_name": "Bartley", "author_url": "", "post_date": "2025-06-02T21:28:33.530000", "content": "Hi @xbar19 and @billfan88, thanks for the comments.
\nThe training is very similar. The differences are that this model is trained for 150 epochs and uses a custom learning rate scheduler shown here
", "votes": 2, "replies": [ { "id": 3215938, "author_name": "", "author_url": "", "post_date": "2025-06-02T21:48:06.603000", "content": "Thank you for your reply and your work!
", "votes": 1, "replies": [] }, { "id": 3215946, "author_name": "greySnow", "author_url": "", "post_date": "2025-06-02T22:21:46.487000", "content": "How much time it took to train on 150 epochs? Damn this is one long training
", "votes": 4, "replies": [] }, { "id": 3215958, "author_name": "Bartley", "author_url": "", "post_date": "2025-06-02T23:25:38.563000", "content": "@shlomoron, it took 8 days (~192 hours) on a single 4090.
\nYou make a good point ā this pipeline takes far too long to converge. I am hoping people can share ideas on how we can decrease this time!
", "votes": 5, "replies": [] }, { "id": 3215959, "author_name": "greySnow", "author_url": "", "post_date": "2025-06-02T23:29:23.150000", "content": "Holy * shit! 8 daysā¦
\nFull 8*24 days? Damn
\n\nthis pipeline takes far too long to converge.
\n
MAE is always slow to converge. Alas.
", "votes": 1, "replies": [] }, { "id": 3217497, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2025-06-05T04:30:26.740000", "content": "It would take me ~8 days on a single 4090 to hit my current LB (17.1), the trainings take a long time to converge
", "votes": 4, "replies": [] }, { "id": 3217741, "author_name": "SeshuRaju š§āāļø", "author_url": "", "post_date": "2025-06-05T11:18:10.863000", "content": "Magic of L1 --> Run Long - Thanks for sharing @brendanartley @harshitsheoran
\n\n", "votes": 1, "replies": [] }, { "id": 3217866, "author_name": "yukiZ", "author_url": "", "post_date": "2025-06-05T13:15:42.077000", "content": "-- How to decided when to stop, most of epochs getting negligible loss improvement (small improvement - my experiments based on only CurveFault_B)
\n
\n -- some batches are too high even using clip the grads
\n
Since there is a lot of information on 4090, I will share the results of V100x8 DDP.
\nTo reproduce this old notebook's convnext[CV 31.9 LB 36.4] locally, it took 2 days for 150 epochs of training, and I was able to obtain the result of [CV30.07, LB35.1].
\n=========================\nCurveFault_A 6.15\nCurveFault_B 86.72\nCurveVel_A 13.69\nCurveVel_B 48.81\nFlatFault_A 4.34\nFlatFault_B 33.73\nFlatVel_A 2.92\nFlatVel_B 12.11\nStyle_A 37.07\nStyle_B 55.18\n=========================\n\nCould you please tell me whether the data you used was the original FP32 data? When I tried to reproduce Caformerās training results using Kaggleās FP16 open dataset, the cross-validation converged around 34, and Iām not sure if this is a dataset issue.
", "votes": 1, "replies": [] }, { "id": 3220288, "author_name": "yukiZ", "author_url": "", "post_date": "2025-06-09T05:39:21.670000", "content": "Hi @oigckko
\nIf you have any questions regarding my above post \"Val MAE: 30.07\", I hope this helps.
The training dataset remains unchanged from the public notebook.
\ndf= pd()\n...\np1 = os(, row)\n p2 = os(, row(), , row())\n p3 = os(, row)\n p4 = os(, row(), , row())\nTrain section Same as starter notebook?
", "votes": 2, "replies": [ { "id": 3216322, "author_name": "Bartley", "author_url": "", "post_date": "2025-06-03T12:18:06.103000", "content": "Hi @billfan88, I appreciate your questions but please read other comments first. Cheers š
", "votes": 0, "replies": [] } ] }, { "id": 3215917, "author_name": "CPMP", "author_url": "", "post_date": "2025-06-02T20:57:34.400000", "content": "Awesome!
\nI wonder how many backbones you tried. You obviously have a good methodology to pick the ones you test.
", "votes": 2, "replies": [ { "id": 3215933, "author_name": "Bartley", "author_url": "", "post_date": "2025-06-02T21:42:40.240000", "content": "Hi @cpmpml, I have tried Convnext, Caformer, Hgnet, NextVIT and Hornet so far.
\nMy methodology is quite simple. First I try and modify the backbone for 1000 x 70 in as few steps as possible. Then, I follow this validation scheme. It may not be the most accurate, but it seems to work well given limited compute!
", "votes": 7, "replies": [ { "id": 3215939, "author_name": "CPMP", "author_url": "", "post_date": "2025-06-02T21:49:12.983000", "content": "Yes, I noticed you use a subset to validate your choice. This is a good idea.
", "votes": 4, "replies": [] } ] } ] }, { "id": 3221540, "author_name": "Flowers", "author_url": "", "post_date": "2025-06-11T06:45:49.307000", "content": "Why is my train loss always higher than vaild loss during training, is this normal?
\n
@oliver34 its same for me too, but it got switch after few epochs when i tested on CurveFault_B dataset experiments.
\n-- How many epochs you are running of CAFormer? Did you run from backbone or from Bartley weights?
", "votes": 1, "replies": [ { "id": 3221557, "author_name": "Flowers", "author_url": "", "post_date": "2025-06-11T07:11:46.630000", "content": "44epoch,I only used the weights of the backbone
", "votes": 2, "replies": [] }, { "id": 3221558, "author_name": "SeshuRaju š§āāļø", "author_url": "", "post_date": "2025-06-11T07:15:54.113000", "content": "@oliver34 will you share your train/val loss for epochs 5, 10, 20, 30 ? -- which gpu - training finishing in 1h 25mins for epoch?
", "votes": 1, "replies": [] }, { "id": 3221562, "author_name": "Flowers", "author_url": "", "post_date": "2025-06-11T07:23:45.047000", "content": "They are 47.70, 40.59, 31.86, 28.70. I also encountered the situation you mentioned at CurveFault_B. --I'm using two 4090s and it seems like it takes 9 days to complete a 150epochš„
", "votes": 1, "replies": [] }, { "id": 3221639, "author_name": "lin-kukuoreoa", "author_url": "", "post_date": "2025-06-11T09:46:46.287000", "content": "Hello, may I ask if you are using the fp32 data or the fp16 dataset processed by Kaggle?
", "votes": 0, "replies": [] }, { "id": 3221666, "author_name": "Flowers", "author_url": "", "post_date": "2025-06-11T10:30:15.037000", "content": "the fp16 dataset processed by Kaggle
", "votes": 1, "replies": [] }, { "id": 3224699, "author_name": "SeshuRaju š§āāļø", "author_url": "", "post_date": "2025-06-15T10:45:50.537000", "content": "@oliver34 could you share cv scores of 40, 60, 80, 100?
", "votes": 1, "replies": [] }, { "id": 3224761, "author_name": "Flowers", "author_url": "", "post_date": "2025-06-15T12:45:32.147000", "content": "I would love to share them, 26.88, 24.58, 23.08, 22.11.
", "votes": 1, "replies": [] }, { "id": 3225362, "author_name": "DJ_Xia", "author_url": "", "post_date": "2025-06-16T09:50:55.937000", "content": "With the \"Tips to speed up training\" optimizations, training can be completed within 2 days using 2Ć4090
", "votes": 1, "replies": [] }, { "id": 3225364, "author_name": "hengck23", "author_url": "", "post_date": "2025-06-16T09:53:50.547000", "content": "I donāt think 4090 can train caformer for 150 epoch for 2 days
", "votes": 1, "replies": [] } ] }, { "id": 3222445, "author_name": "M Sato", "author_url": "", "post_date": "2025-06-12T07:46:55.827000", "content": "batch-wise average vs overall average
", "votes": 1, "replies": [] }, { "id": 3223545, "author_name": "Truth Seeker", "author_url": "", "post_date": "2025-06-13T13:28:12.613000", "content": "\n\nmy train loss always higher than vaild loss during training
\n
I also observe this. I haven't observed a single epoch in which my val loss is worse than my train loss. I'm still relatively early in my training epochs, though. Glad to know I'm not the only one. š
", "votes": 0, "replies": [] } ] }, { "id": 3221539, "author_name": "", "author_url": "", "post_date": "2025-06-11T06:45:11.713000", "content": "", "votes": 0, "replies": [] }, { "id": 3225006, "author_name": "Iqra Rafique", "author_url": "", "post_date": "2025-06-15T19:37:03.183000", "content": "Great work!
", "votes": 1, "replies": [] }, { "id": 3223364, "author_name": "Du Qinglin", "author_url": "", "post_date": "2025-06-13T07:42:11.890000", "content": "Thanks for the information
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3215873": "Happy to share an improved full resolution model. This model uses a CAFormer encoder with an improved decoder (pixel shuffle, SCSE, intermediate convolutions, etc.).\n\nNotebook [here](https://www.kaggle.com/code/brendanartley/caformer-full-resolution-improved)\nDataset [here](https://www.kaggle.com/datasets/brendanartley/openfwi-preprocessed-72x72)\n\n```\n+--------------+----------+\n| Dataset | Caformer |\n+--------------+----------+\n| CurveFault_A | 4.00 |\n| CurveFault_B | 71.06 |\n| CurveVel_A | 9.19 |\n| CurveVel_B | 38.60 |\n| FlatFault_A | 2.58 |\n| FlatFault_B | 24.29 |\n| FlatVel_A | 1.31 |\n| FlatVel_B | 7.20 |\n| Style_A | 34.34 |\n| Style_B | 48.90 |\n+--------------+----------+\n| Overall | 24.15 |\n+--------------+----------+\n```\n\nThis will be my last notebook in this competition, but I am hopeful that we can push this architecture further as a community!\n\nHappy Kaggling š", "3217147": "\"this pipeline takes far too long to converge.\"\n- you can classify the input type ( 3 familiesāVel, Fault, and Style) for the test with a classifier (on the predicted velocity)\n- then you can train seperate models for each type (this happens for most paper)\n- if you have many machines, you can train in parallel", "3217142": "a trick is use fp32 in input and model for inference. you will get something like +0.0025 in lb (?)", "3216127": "@brendanartley Have you tried re-training convnext large with the same settings as caformer i.e. 150 epochs + improved decoder + cosine scheduler ? \nCaformer training time was 2x longer than convnext large so I'm not sure it's worth the extra cost.", "3219000": "this is segmentation head in your code:\n\n```\n\n\nclass SegmentationHead2d(nn.Module):\n def __init__(\n self,\n in_channels,\n out_channels,\n scale_factor: tuple[int] = (2, 2),\n kernel_size: int = 3,\n mode: str = \"nontrainable\",\n ):\n super().__init__()\n self.conv = nn.Conv2d(\n in_channels, out_channels, kernel_size=kernel_size,\n padding=kernel_size // 2\n )\n self.upsample = UpSample(\n spatial_dims=2,\n in_channels=out_channels,\n out_channels=out_channels,\n scale_factor=scale_factor,\n mode=mode,\n )\n\n def forward(self, x):\n x = self.conv(x)\n x = self.upsample(x)\n return x\n\n```\n\ni haven't done in-depth experiment yet, but below are from my experiences:\n- this is regression problem, so value of the prediction is important\n- do not use upsample as last layer. you will be bounded by up scaling error.\n- use original input data resolution (or 72x72 sampled version) in final decoder layers\n- the last prediction module should be 1x1 conv (not even 3x3)\n\n```\ne.g. something like\n\nlast_feature = lastdecoder(input = from previous decoder, skip = original input)\npredicted_velocity = prediction(last_feature)\n\nwhere prediction= nn.Seqential(\n....\nnn.Conv2d( 1x1 kernel)\n)\n``", "3216409": "@brendanartley \nWhen turning a Train, the IO seems to be a problem(?). What is your countermeasure for this?", "3216093": "@brendanartley Thanks for sharing one more milestone very hard to break **struggle to reach <100** --> **69.7** --> **33.2** --> **28.8**, could you share [CV scores](https://www.kaggle.com/code/brendanartley/convnext-full-resolution-baseline/comments#3206500) of training 20-25 epochs of CurveFault-B ( your cv and training logs if possible )", "3224682": "This is very interesting", "3223522": "whats the upsample mode , pixel shuffle or deconv ??", "3217724": "When you're testing model performance on a small order of magnitude, do you use CurveFault_B, and if so, what the results are, and then you use them for training on all datasets?", "3217157": "Thanks for sharing best model with full training....", "3216520": "[Bartley](https://www.kaggle.com/brendanartley)\n\nOk - maybe its a lack of coffee or the fact that I am an old fart - but I think I read all the other comments several times and failed to find the string that leads us to the code for training this latest nice bit of work. I tried re-using the previous train but that failed.\n\nIs it your intent to not share the training code for your last model?\n\n>Hi @billfan88, I appreciate your questions but please read other comments first. Cheers š", "3216040": "Thanks for sharing best model with full training - one more break through notebook and solution.\nCould you share your training logs of these models after 50 epochs?\n", "3215914": "Was the model trained as in previous baselines?", "3215969": "Train section Same as starter notebook?", "3215917": "Awesome!\n\nI wonder how many backbones you tried. You obviously have a good methodology to pick the ones you test.", "3221540": "Why is my train loss always higher than vaild loss during training, is this normal?\n", "3221539": "", "3225006": "Great work!", "3223364": "Thanks for the information" } }