\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F291846%2Fbbda54814df51697ff7445fdb8482044%2FBartley-LB.png?generation=1747831623968838&alt=media\")
Here is another baseline trained on the full-resolution data. Like my previous notebook, I provide a few trained checkpoints and leave lots of room for customization.
\n\nHere are the CV scores for this approach. Cheers!
\n+--------------+--------+\n| Dataset | Score |\n+--------------+--------+\n| CurveFault_A | |\n| CurveFault_B | |\n| CurveVel_A | |\n| CurveVel_B | |\n| FlatFault_A | |\n| FlatFault_B | |\n| FlatVel_A | |\n| FlatVel_B | |\n| Style_A | |\n| Style_B | |\n+--------------+--------+\nHere is another baseline trained on the full-resolution data. Like my previous notebook, I provide a few trained checkpoints and leave lots of room for customization.
\n\nHere are the CV scores for this approach. Cheers!
\n+--------------+--------+\n| Dataset | Score |\n+--------------+--------+\n| CurveFault_A | |\n| CurveFault_B | |\n| CurveVel_A | |\n| CurveVel_B | |\n| FlatFault_A | |\n| FlatFault_B | |\n| FlatVel_A | |\n| FlatVel_B | |\n| Style_A | |\n| Style_B | |\n+--------------+--------+\nA simple regression model shows that the LB score of the next \"baseline\" Bartley model, due out next week, is less than 20. Tremble, leaders!
\n
Huge Respect!!
", "rawMarkdown": "Huge Respect!!", "votes": 2 } ] }, { "id": 3206037, "postDate": "2025-05-20T18:54:33.627Z", "content": "Loss curves below. Each model takes ~10hrs for 50 epochs on 1×4090 with optimizations (e.g., torch.compile(), fused optimizer, increased num_workers, mixed precision, etc.).
\n![\"WandbLogs\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5570735%2F893d5096069da688ce1249ab739ef7de%2Flogs_wandb.JPG?generation=1747767254477184&alt=media\")
@brendanartley are you evaluated every 20% of epoch?
\n\n", "rawMarkdown": "@brendanartley are you evaluated every 20% of epoch?\n> These charts helps to cross check the stability of the local codes, Thanks you.", "votes": 1, "replies": [ { "id": 3206132, "postDate": "2025-05-20T22:04:42.027Z", "content": "These charts helps to cross check the stability of the local codes, Thanks you.
\n
Hi @seshurajup, evaluation happens once each epoch.
", "rawMarkdown": "Hi @seshurajup, evaluation happens once each epoch.", "votes": 2 } ] }, { "id": 3206948, "postDate": "2025-05-22T02:34:35.700Z", "content": "@brendanartley Thank you for sharing your approach in this competition—it’s impressive. On my end, even with multiple GPUs（float16 full resolution), a single epoch still takes around 30 minutes.
\nyou’re able to run 50 epochs on a single RTX 4090 in roughly the same time. What optimizations or tricks are you using to achieve that speed?
When using multiple gpu's, are you properly sharding the dataset? Otherwise, 30m per epoch is not too bad either.
", "rawMarkdown": "When using multiple gpu's, are you properly sharding the dataset? Otherwise, 30m per epoch is not too bad either.", "votes": 3 }, { "id": 3207213, "postDate": "2025-05-22T12:11:35.550Z", "content": "Hi @koukinn, its hard to give tips as each machine could have different bottlenecks.
\nFor me, the biggest speedups came from using AdamW(..., fused=True) and torch.compile(..., mode=\"max-autotune\"). These two changes made my pipeline ~2x faster.
Since you said RTX 4090 I assume you ran it outside of Kaggle?
\nIf yes, how did you do so please? (overall as I am struggling to do so; training on kaggle takes2 hours per episode)
Thank you in advance
", "rawMarkdown": "Since you said RTX 4090 I assume you ran it outside of Kaggle? \nIf yes, how did you do so please? (overall as I am struggling to do so; training on kaggle takes2 hours per episode)\n\nThank you in advance", "votes": 1 }, { "id": 3207705, "postDate": "2025-05-23T06:42:28.770Z", "content": "Thank you so much
", "rawMarkdown": "Thank you so much", "votes": 1 }, { "id": 3209113, "postDate": "2025-05-25T08:49:29.897Z", "content": "For my model, the difference between max-autotune and reduce-overhead is minor, but max-autotune takes a lot (!) longer for initial optimization. Like 2m vs 30m+. Might be worth checking out as well.
", "rawMarkdown": "For my model, the difference between max-autotune and reduce-overhead is minor, but max-autotune takes a lot (!) longer for initial optimization. Like 2m vs 30m+. Might be worth checking out as well.", "votes": 1 } ] }, { "id": 3214227, "postDate": "2025-05-31T06:07:54.490Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 3207307, "postDate": "2025-05-22T15:04:10.927Z", "content": "\nAwesome Open!, I am sharing a little bit more finetuned models of yours
\nhttps://www.kaggle.com/datasets/harshitsheoran/simple-further-finetuned-bartley-open-models
\nYou can add them to your notebook if you want ( don't want the credit, the hard work was yours ), I have shared you the notebook that would use these weights, the link is at
\nhttps://www.kaggle.com/code/harshitsheoran/convnext-baseline/
As long as there is no leak that is happening in my finetuning, it's CV 0.289 :)
\nIf you do choose to use these models, ofc, I will say what the finetuning was :D
\nUpdate:
\nHere is my exact code that I used to finetune the models:
https://www.kaggle.com/code/harshitsheoran/recipe-simple-further-finetuned-bartley-models
\nHave a good day!
", "rawMarkdown": "Hi @brendanartley \n\nAwesome Open!, I am sharing a little bit more finetuned models of yours\n\nhttps://www.kaggle.com/datasets/harshitsheoran/simple-further-finetuned-bartley-open-models\n\nYou can add them to your notebook if you want ( don't want the credit, the hard work was yours ), I have shared you the notebook that would use these weights, the link is at\nhttps://www.kaggle.com/code/harshitsheoran/convnext-baseline/\n\nAs long as there is no leak that is happening in my finetuning, it's CV 0.289 :)\n\nIf you do choose to use these models, ofc, I will say what the finetuning was :D\n\nUpdate:\nHere is my exact code that I used to finetune the models:\n\nhttps://www.kaggle.com/code/harshitsheoran/recipe-simple-further-finetuned-bartley-models\n\nHave a good day!", "votes": 13, "replies": [ { "id": 3207338, "postDate": "2025-05-22T15:43:37.623Z", "content": "Amazing @harshitsheoran, you deserve the credit on this one! Just updated the public notebook with your weights 😄
\nCV: 31.9 -> 28.9
\nLB: 36.4 -> 33.2
As you have seen in the code, Finetuning had no change in model or data, the only difference was that I copied your model code and placed it my notebook, which had changes in:
\nNot to be a party spoiler , but @harshitsheoran if you have shared the code with only @brendanartley , then would this not be Private sharing ? since you both are not part of the same team . Thanks for sharing the weights and giving hints . Be careful man ..
", "rawMarkdown": "Not to be a party spoiler , but @harshitsheoran if you have shared the code with only @brendanartley , then would this not be Private sharing ? since you both are not part of the same team . Thanks for sharing the weights and giving hints . Be careful man ..", "votes": 7 }, { "id": 3207755, "postDate": "2025-05-23T07:35:02.947Z", "content": "@phoenix9032 I will open the code to everyone, I had the intention that I am opening the weights to everyone and the method to use those weights is very simple, as it's just plug and play, but I wanted a secure line where if I say that weights can just be used as is, someone else might copy his work before him, ofc, now it's not needed and I have opened the notebook so we can see that there is nothing I have shared that isn't the same code as his original notebook haha :)
\nThanks for advice, I should have thought this through
", "rawMarkdown": "@phoenix9032 I will open the code to everyone, I had the intention that I am opening the weights to everyone and the method to use those weights is very simple, as it's just plug and play, but I wanted a secure line where if I say that weights can just be used as is, someone else might copy his work before him, ofc, now it's not needed and I have opened the notebook so we can see that there is nothing I have shared that isn't the same code as his original notebook haha :)\n\nThanks for advice, I should have thought this through", "votes": 4 }, { "id": 3207801, "postDate": "2025-05-23T08:36:01.603Z", "content": "So what is The way EMA is applied mean?
", "rawMarkdown": "So what is The way EMA is applied mean?", "votes": 2 }, { "id": 3207803, "postDate": "2025-05-23T08:41:49.667Z", "content": "I don't think EMA is any different, I am just using a different library to apply EMA which I have been using for an year, impact should be negligible regardless, my guess is that score comes from just having a larger global batch size and a slight decay in learning rate
", "rawMarkdown": "I don't think EMA is any different, I am just using a different library to apply EMA which I have been using for an year, impact should be negligible regardless, my guess is that score comes from just having a larger global batch size and a slight decay in learning rate", "votes": 4 }, { "id": 3207807, "postDate": "2025-05-23T08:52:22.077Z", "content": "No worries … both of you have been doing great work in this competition and putting a lot of effort , so just wanted to caution …
", "rawMarkdown": "No worries ... both of you have been doing great work in this competition and putting a lot of effort , so just wanted to caution ... ", "votes": 3 }, { "id": 3214068, "postDate": "2025-05-30T22:03:38.980Z", "content": "@harshitsheoran You have not shared publicly the code you commented as follows:
\n\n\nAs you have seen in the code, Finetuning had no change in model or data, the only difference was that I copied your model code and placed it my notebook, which had changes in:
\nLearning Rate and Global Batch Size
\n
\n The way EMA is applied
\n The finetuning was short, like 10-20 epochs, then the score was converging slowly so I stopped the training
\n and
\n I am just using a different library to apply EMA which I have been using for an year,
You shared a copy of @brendanartley notebook but it is not the one you used to finetune the models.
\n\nI react because it is not fine to claim you shared your code publicly. Please don't lie to us.
\nI don't want to shadow that you shared your improved weights. This is great, thanks for that.
\nEdit: I update this because there was no private sharing.
", "rawMarkdown": "@harshitsheoran You have not shared publicly the code you commented as follows:\n\n> As you have seen in the code, Finetuning had no change in model or data, the only difference was that I copied your model code and placed it my notebook, which had changes in:\n>\n > Learning Rate and Global Batch Size\n> The way EMA is applied\n > The finetuning was short, like 10-20 epochs, then the score was converging slowly so I stopped the training\nand\n> I am just using a different library to apply EMA which I have been using for an year,\n\nYou shared a copy of @brendanartley notebook but it is not the one you used to finetune the models.\n\n~~It is not fine to privately share code but that's Kaggle business to deal with it, not mine.~~\n\nI react because it is not fine to claim you shared your code publicly. Please don't lie to us.\n\nI don't want to shadow that you shared your improved weights. This is great, thanks for that.\n\nEdit: I update this because there was no private sharing.\n\n", "votes": -1 }, { "id": 3214089, "postDate": "2025-05-30T23:18:06.680Z", "content": "Totally agree @cpmpml. The right way is to make the notebook public before sharing - though I cant control what is shared with me. The only code I have seen is here
\n@harshitsheoran Would be great if you could give more details, but understand if you would rather not 🙂
", "rawMarkdown": "Totally agree @cpmpml. The right way is to make the notebook public before sharing - though I cant control what is shared with me. The only code I have seen is [here](https://www.kaggle.com/code/harshitsheoran/convnext-baseline/)\n\n@harshitsheoran Would be great if you could give more details, but understand if you would rather not 🙂", "votes": 2 }, { "id": 3214112, "postDate": "2025-05-31T00:58:07.477Z", "content": "Great, then there is no private sharing. I'll edit my previous reply then.
", "rawMarkdown": "Great, then there is no private sharing. I'll edit my previous reply then.", "votes": 2 }, { "id": 3214140, "postDate": "2025-05-31T02:26:57.960Z", "content": "Hi @cpmpml !
\nI am sorry but I think I am misunderstanding the quote
\n\n\nIt is not fine to claim that you shared your code publicly
\n
Are we talking about the code that was shared first only to bartley? Yes, that was an unintentional mistake I made as there was no new code in that notebook so I failed to think it through before sharing it and I opened that code to everyone as soon as that mistake was pointed out to show that there was no new code in that notebook.
\nSorry, I re-read your message from the start:
\nTalking about the code that was used to finetune (it was finetuned on local machine), where I said \"I will say what the finetuning was\", although I meant it literally, I have no issues making the code public because there is nothing good in that, I believe that this great work can be pushed much further than my dumb finetune.
By \"As you have seen in the code\" I meant that as everyone has noticed in the inference code that I did not change any data or model so inference remains the same
\nEdit: Updated the original comment with the link to the code that was used for finetuning
", "rawMarkdown": "Hi @cpmpml !\n\nI am sorry but I think I am misunderstanding the quote\n>It is not fine to claim that you shared your code publicly\n\nAre we talking about the code that was shared first only to bartley? Yes, that was an unintentional mistake I made as there was no new code in that notebook so I failed to think it through before sharing it and I opened that code to everyone as soon as that mistake was pointed out to show that there was no new code in that notebook.\n\nSorry, I re-read your message from the start:\nTalking about the code that was used to finetune (it was finetuned on local machine), where I said \"I will say what the finetuning was\", although I meant it literally, I have no issues making the code public because there is nothing good in that, I believe that this great work can be pushed much further than my dumb finetune.\n\nBy \"As you have seen in the code\" I meant that as everyone has noticed in the inference code that I did not change any data or model so inference remains the same\n\nEdit: Updated the original comment with the link to the code that was used for finetuning", "votes": 4 }, { "id": 3214297, "postDate": "2025-05-31T08:25:39.283Z", "content": "Awesome, thanks.
", "rawMarkdown": "Awesome, thanks." } ] } ] }, { "id": 3214069, "postDate": "2025-05-30T22:06:38.087Z", "content": "There is a lot to learn from your notebook, thanks for sharing.
\nI never used mmap in numpy, this is really neat. Not sure who first share dit here, but kudos to whom did it.
\nI still have to dig into how you modified convnext!
", "rawMarkdown": "There is a lot to learn from your notebook, thanks for sharing.\n\nI never used mmap in numpy, this is really neat. Not sure who first share dit here, but kudos to whom did it.\n\nI still have to dig into how you modified convnext!", "votes": 5 }, { "id": 3212157, "postDate": "2025-05-29T11:54:01.607Z", "content": "Based on past few days experimentation, I realized how strong the baseline @brendanartley has provided. There are lot of tricks that are implemented in the network, vanilla network would not perform this well even with out of the box encoder and decoder ! Great work!
", "rawMarkdown": "Based on past few days experimentation, I realized how strong the baseline @brendanartley has provided. There are lot of tricks that are implemented in the network, vanilla network would not perform this well even with out of the box encoder and decoder ! Great work!", "votes": 5 }, { "id": 3206983, "postDate": "2025-05-22T04:10:19.450Z", "content": "Huge respect! Although it is honestly kinda depressing to see my ranking drop from 14 -> 150+ in two days.😂
", "rawMarkdown": "Huge respect! Although it is honestly kinda depressing to see my ranking drop from 14 -> 150+ in two days.😂", "votes": 5 }, { "id": 3206224, "postDate": "2025-05-21T03:58:34.537Z", "content": "Show biggest respect to you!
", "rawMarkdown": "Show biggest respect to you!", "votes": 6 }, { "id": 3214010, "postDate": "2025-05-30T18:36:35.933Z", "content": "Excellent notebook, thanks for sharing!
\nJust 1 question, how did you train on \"1x4090\" GPU?
\nThe code seems to require \"Requires >= 2 GPUs with CUDA enabled.\"
", "rawMarkdown": "Excellent notebook, thanks for sharing!\n\nJust 1 question, how did you train on \"1x4090\" GPU?\n\nThe code seems to require \"Requires >= 2 GPUs with CUDA enabled.\"", "votes": 1, "replies": [ { "id": 3214086, "postDate": "2025-05-30T23:03:48.723Z", "content": "Hi @williamwu88, good point. You will have to modify the training loop for the script to work with 1 GPU.
", "rawMarkdown": "Hi @williamwu88, good point. You will have to modify the training loop for the script to work with 1 GPU.", "votes": 1 } ] }, { "id": 3212174, "postDate": "2025-05-29T12:31:38.427Z", "content": "Can someone please explain why was the stride custom changed in this case?
", "rawMarkdown": "Can someone please explain why was the stride custom changed in this case? ", "votes": 1, "replies": [ { "id": 3212189, "postDate": "2025-05-29T13:00:33.073Z", "content": "Hi @meddhifallah, does this comment help?
", "rawMarkdown": "Hi @meddhifallah, does [this comment](https://www.kaggle.com/code/brendanartley/hgnet-v2-starter/comments#3200035) help?", "votes": 2 } ] }, { "id": 3210285, "postDate": "2025-05-27T02:09:52.153Z", "content": "That's truly amazing. I'm learning a lot.
", "rawMarkdown": "That's truly amazing. I'm learning a lot.", "votes": 1 }, { "id": 3210078, "postDate": "2025-05-26T17:31:44.220Z", "content": "Looks truly awesome
", "rawMarkdown": "Looks truly awesome ", "votes": 1 }, { "id": 3208519, "postDate": "2025-05-24T09:22:36.323Z", "content": "That's really great!!
", "rawMarkdown": "That's really great!!", "votes": 1 }, { "id": 3207893, "postDate": "2025-05-23T11:42:16.180Z", "content": "it will be very helpful in nova days!
", "rawMarkdown": "it will be very helpful in nova days!", "votes": 1 }, { "id": 3207856, "postDate": "2025-05-23T10:32:20.167Z", "content": "hi @brendanartley couple of quick thoughts/questions, and I'd love to hear your perspective if you have time:
\nResolution scaling: Did you observe any significant performance changes or convergence issues when training on the full resolution compared to down sampled versions? Given the computational cost, I just wanted to understanding the trade-off
\nLoss function Insights: What loss function did you primarily use for this ConvNeXt model? Did you explore any alternatives, perhaps something more tailored to the characteristics of geophysical waveforms or a hybrid loss?
\nFuture Directions: Based on these results, where do you see the most immediate opportunities for improvement? Perhaps exploring different attention mechanisms, a deeper understanding of the \"B\" datasets, or specialized data augmentation techniques?
Hi @bruhjjwal, thanks for the comment.
\nUsing the full resolution data helped significantly.
L1 loss (have not tried other loss functions yet)
Its hard to tell where to focus efforts at the moment. Currently I am playing with batch size / learning rate combinations. In addition, I would like to have a fully converged model before analyzing out of fold predictions.
Would recommend reading this thread. @hoffmanns makes some good points!
", "rawMarkdown": "Hi @bruhjjwal, thanks for the comment.\n\n1. Using the full resolution data helped significantly.\n\n2. L1 loss (have not tried other loss functions yet)\n\n3. Its hard to tell where to focus efforts at the moment. Currently I am playing with batch size / learning rate combinations. In addition, I would like to have a fully converged model before analyzing out of fold predictions. \n\nWould recommend reading [this](https://www.kaggle.com/code/tpmeli/beyond-mae-depth-curves-spectra-and-residuals/comments#3203025) thread. @hoffmanns makes some good points!", "votes": 1 } ] }, { "id": 3207848, "postDate": "2025-05-23T10:16:42.233Z", "content": "What batch size do you train the model with? 256 or 16?
", "rawMarkdown": "What batch size do you train the model with? 256 or 16?\n", "votes": 1, "replies": [ { "id": 3207913, "postDate": "2025-05-23T12:23:37.177Z", "content": "Hi @vonmainstein, I used a batch size of 16 for this model.
", "rawMarkdown": "Hi @vonmainstein, I used a batch size of 16 for this model.", "votes": 1 } ] }, { "id": 3207633, "postDate": "2025-05-23T04:36:22.677Z", "content": "Huge Respect!
", "rawMarkdown": "Huge Respect!", "votes": 1 }, { "id": 3206499, "postDate": "2025-05-21T11:58:55Z", "content": "HUGE RESPECT!
", "rawMarkdown": "HUGE RESPECT!", "votes": 1 }, { "id": 3206193, "postDate": "2025-05-21T02:42:39.023Z", "content": "Is the main improvement due to the use of full-resolution data?🥰
", "rawMarkdown": "Is the main improvement due to the use of full-resolution data?🥰", "votes": 1, "replies": [ { "id": 3206195, "postDate": "2025-05-21T02:51:56.513Z", "content": "Hi @guoooooooss, I think so. Though this was my first experiment using the full-resolution data.
", "rawMarkdown": "Hi @guoooooooss, I think so. Though this was my first experiment using the full-resolution data.", "votes": 3, "replies": [ { "id": 3206200, "postDate": "2025-05-21T02:57:29.917Z", "content": "thank you very much!🥰
", "rawMarkdown": "thank you very much!🥰", "votes": 1 } ] } ] }, { "id": 3207695, "postDate": "2025-05-23T06:22:24.473Z", "content": "Nice work! Appreciate so much
", "rawMarkdown": "Nice work! Appreciate so much\n", "votes": 2 }, { "id": 3206070, "postDate": "2025-05-20T19:28:02.483Z", "content": "Bro casually messing up the LB with \"Baseline\" and \"Starter\" notebooks :)
\nJokes aside, I love seeing your notebooks popping up.
", "rawMarkdown": "Bro casually messing up the LB with \"Baseline\" and \"Starter\" notebooks :)\n\nJokes aside, I love seeing your notebooks popping up.", "votes": 2 }, { "id": 3206580, "postDate": "2025-05-21T14:26:12.983Z", "content": "nice sharing 👍
\nDoes pretrained encoder help?
Beautiful LB.
\nIn this match, we achieved communism.
\nHuge respect to Bartley. Love you bro!
Great work :) :)
", "rawMarkdown": "Great work :) :)", "votes": 1 }, { "id": 3206771, "postDate": "2025-05-21T18:43:15.097Z", "content": "Thanks for sharing !
", "rawMarkdown": "Thanks for sharing !\n", "votes": 1 } ], "comments": [ { "id": 3206525, "author_name": "ln", "author_url": "", "post_date": "2025-05-21T12:47:47.330000", "content": "A simple regression model shows that the LB score of the next \"baseline\" Bartley model, due out next week, is less than 20. Tremble, leaders!
\n
Huge Respect!!
", "votes": 2, "replies": [] } ] }, { "id": 3206037, "author_name": "Bartley", "author_url": "", "post_date": "2025-05-20T18:54:33.627000", "content": "Loss curves below. Each model takes ~10hrs for 50 epochs on 1×4090 with optimizations (e.g., torch.compile(), fused optimizer, increased num_workers, mixed precision, etc.).
\n
@brendanartley are you evaluated every 20% of epoch?
\n\n", "votes": 1, "replies": [ { "id": 3206132, "author_name": "Bartley", "author_url": "", "post_date": "2025-05-20T22:04:42.027000", "content": "These charts helps to cross check the stability of the local codes, Thanks you.
\n
Hi @seshurajup, evaluation happens once each epoch.
", "votes": 2, "replies": [] } ] }, { "id": 3206948, "author_name": "Gao Xin", "author_url": "", "post_date": "2025-05-22T02:34:35.700000", "content": "@brendanartley Thank you for sharing your approach in this competition—it’s impressive. On my end, even with multiple GPUs（float16 full resolution), a single epoch still takes around 30 minutes.
\nyou’re able to run 50 epochs on a single RTX 4090 in roughly the same time. What optimizations or tricks are you using to achieve that speed?
When using multiple gpu's, are you properly sharding the dataset? Otherwise, 30m per epoch is not too bad either.
", "votes": 3, "replies": [] }, { "id": 3207213, "author_name": "Bartley", "author_url": "", "post_date": "2025-05-22T12:11:35.550000", "content": "Hi @koukinn, its hard to give tips as each machine could have different bottlenecks.
\nFor me, the biggest speedups came from using AdamW(..., fused=True) and torch.compile(..., mode=\"max-autotune\"). These two changes made my pipeline ~2x faster.
Since you said RTX 4090 I assume you ran it outside of Kaggle?
\nIf yes, how did you do so please? (overall as I am struggling to do so; training on kaggle takes2 hours per episode)
Thank you in advance
", "votes": 1, "replies": [] }, { "id": 3207705, "author_name": "Gao Xin", "author_url": "", "post_date": "2025-05-23T06:42:28.770000", "content": "Thank you so much
", "votes": 1, "replies": [] }, { "id": 3209113, "author_name": "Sebastian Hoffmann", "author_url": "", "post_date": "2025-05-25T08:49:29.897000", "content": "For my model, the difference between max-autotune and reduce-overhead is minor, but max-autotune takes a lot (!) longer for initial optimization. Like 2m vs 30m+. Might be worth checking out as well.
", "votes": 1, "replies": [] } ] }, { "id": 3214227, "author_name": "", "author_url": "", "post_date": "2025-05-31T06:07:54.490000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 3207307, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2025-05-22T15:04:10.927000", "content": "\nAwesome Open!, I am sharing a little bit more finetuned models of yours
\nhttps://www.kaggle.com/datasets/harshitsheoran/simple-further-finetuned-bartley-open-models
\nYou can add them to your notebook if you want ( don't want the credit, the hard work was yours ), I have shared you the notebook that would use these weights, the link is at
\nhttps://www.kaggle.com/code/harshitsheoran/convnext-baseline/
As long as there is no leak that is happening in my finetuning, it's CV 0.289 :)
\nIf you do choose to use these models, ofc, I will say what the finetuning was :D
\nUpdate:
\nHere is my exact code that I used to finetune the models:
https://www.kaggle.com/code/harshitsheoran/recipe-simple-further-finetuned-bartley-models
\nHave a good day!
", "votes": 13, "replies": [ { "id": 3207338, "author_name": "Bartley", "author_url": "", "post_date": "2025-05-22T15:43:37.623000", "content": "Amazing @harshitsheoran, you deserve the credit on this one! Just updated the public notebook with your weights 😄
\nCV: 31.9 -> 28.9
\nLB: 36.4 -> 33.2
As you have seen in the code, Finetuning had no change in model or data, the only difference was that I copied your model code and placed it my notebook, which had changes in:
\nNot to be a party spoiler , but @harshitsheoran if you have shared the code with only @brendanartley , then would this not be Private sharing ? since you both are not part of the same team . Thanks for sharing the weights and giving hints . Be careful man ..
", "votes": 7, "replies": [] }, { "id": 3207755, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2025-05-23T07:35:02.947000", "content": "@phoenix9032 I will open the code to everyone, I had the intention that I am opening the weights to everyone and the method to use those weights is very simple, as it's just plug and play, but I wanted a secure line where if I say that weights can just be used as is, someone else might copy his work before him, ofc, now it's not needed and I have opened the notebook so we can see that there is nothing I have shared that isn't the same code as his original notebook haha :)
\nThanks for advice, I should have thought this through
", "votes": 4, "replies": [] }, { "id": 3207801, "author_name": "guo dashuai", "author_url": "", "post_date": "2025-05-23T08:36:01.603000", "content": "So what is The way EMA is applied mean?
", "votes": 2, "replies": [] }, { "id": 3207803, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2025-05-23T08:41:49.667000", "content": "I don't think EMA is any different, I am just using a different library to apply EMA which I have been using for an year, impact should be negligible regardless, my guess is that score comes from just having a larger global batch size and a slight decay in learning rate
", "votes": 4, "replies": [] }, { "id": 3207807, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2025-05-23T08:52:22.077000", "content": "No worries … both of you have been doing great work in this competition and putting a lot of effort , so just wanted to caution …
", "votes": 3, "replies": [] }, { "id": 3214068, "author_name": "CPMP", "author_url": "", "post_date": "2025-05-30T22:03:38.980000", "content": "@harshitsheoran You have not shared publicly the code you commented as follows:
\n\n\nAs you have seen in the code, Finetuning had no change in model or data, the only difference was that I copied your model code and placed it my notebook, which had changes in:
\nLearning Rate and Global Batch Size
\n
\n The way EMA is applied
\n The finetuning was short, like 10-20 epochs, then the score was converging slowly so I stopped the training
\n and
\n I am just using a different library to apply EMA which I have been using for an year,
You shared a copy of @brendanartley notebook but it is not the one you used to finetune the models.
\n\nI react because it is not fine to claim you shared your code publicly. Please don't lie to us.
\nI don't want to shadow that you shared your improved weights. This is great, thanks for that.
\nEdit: I update this because there was no private sharing.
", "votes": -1, "replies": [] }, { "id": 3214089, "author_name": "Bartley", "author_url": "", "post_date": "2025-05-30T23:18:06.680000", "content": "Totally agree @cpmpml. The right way is to make the notebook public before sharing - though I cant control what is shared with me. The only code I have seen is here
\n@harshitsheoran Would be great if you could give more details, but understand if you would rather not 🙂
", "votes": 2, "replies": [] }, { "id": 3214112, "author_name": "CPMP", "author_url": "", "post_date": "2025-05-31T00:58:07.477000", "content": "Great, then there is no private sharing. I'll edit my previous reply then.
", "votes": 2, "replies": [] }, { "id": 3214140, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2025-05-31T02:26:57.960000", "content": "Hi @cpmpml !
\nI am sorry but I think I am misunderstanding the quote
\n\n\nIt is not fine to claim that you shared your code publicly
\n
Are we talking about the code that was shared first only to bartley? Yes, that was an unintentional mistake I made as there was no new code in that notebook so I failed to think it through before sharing it and I opened that code to everyone as soon as that mistake was pointed out to show that there was no new code in that notebook.
\nSorry, I re-read your message from the start:
\nTalking about the code that was used to finetune (it was finetuned on local machine), where I said \"I will say what the finetuning was\", although I meant it literally, I have no issues making the code public because there is nothing good in that, I believe that this great work can be pushed much further than my dumb finetune.
By \"As you have seen in the code\" I meant that as everyone has noticed in the inference code that I did not change any data or model so inference remains the same
\nEdit: Updated the original comment with the link to the code that was used for finetuning
", "votes": 4, "replies": [] }, { "id": 3214297, "author_name": "CPMP", "author_url": "", "post_date": "2025-05-31T08:25:39.283000", "content": "Awesome, thanks.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3214069, "author_name": "CPMP", "author_url": "", "post_date": "2025-05-30T22:06:38.087000", "content": "There is a lot to learn from your notebook, thanks for sharing.
\nI never used mmap in numpy, this is really neat. Not sure who first share dit here, but kudos to whom did it.
\nI still have to dig into how you modified convnext!
", "votes": 5, "replies": [] }, { "id": 3212157, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2025-05-29T11:54:01.607000", "content": "Based on past few days experimentation, I realized how strong the baseline @brendanartley has provided. There are lot of tricks that are implemented in the network, vanilla network would not perform this well even with out of the box encoder and decoder ! Great work!
", "votes": 5, "replies": [] }, { "id": 3206983, "author_name": "Siming Shan", "author_url": "", "post_date": "2025-05-22T04:10:19.450000", "content": "Huge respect! Although it is honestly kinda depressing to see my ranking drop from 14 -> 150+ in two days.😂
", "votes": 5, "replies": [] }, { "id": 3206224, "author_name": "hyd", "author_url": "", "post_date": "2025-05-21T03:58:34.537000", "content": "Show biggest respect to you!
", "votes": 6, "replies": [] }, { "id": 3214010, "author_name": "William Wu Chengyuan", "author_url": "", "post_date": "2025-05-30T18:36:35.933000", "content": "Excellent notebook, thanks for sharing!
\nJust 1 question, how did you train on \"1x4090\" GPU?
\nThe code seems to require \"Requires >= 2 GPUs with CUDA enabled.\"
", "votes": 1, "replies": [ { "id": 3214086, "author_name": "Bartley", "author_url": "", "post_date": "2025-05-30T23:03:48.723000", "content": "Hi @williamwu88, good point. You will have to modify the training loop for the script to work with 1 GPU.
", "votes": 1, "replies": [] } ] }, { "id": 3212174, "author_name": "Med Dhifallah", "author_url": "", "post_date": "2025-05-29T12:31:38.427000", "content": "Can someone please explain why was the stride custom changed in this case?
", "votes": 1, "replies": [ { "id": 3212189, "author_name": "Bartley", "author_url": "", "post_date": "2025-05-29T13:00:33.073000", "content": "Hi @meddhifallah, does this comment help?
", "votes": 2, "replies": [] } ] }, { "id": 3210285, "author_name": "habibi9901", "author_url": "", "post_date": "2025-05-27T02:09:52.153000", "content": "That's truly amazing. I'm learning a lot.
", "votes": 1, "replies": [] }, { "id": 3210078, "author_name": "Stanislav Sobolev", "author_url": "", "post_date": "2025-05-26T17:31:44.220000", "content": "Looks truly awesome
", "votes": 1, "replies": [] }, { "id": 3208519, "author_name": "Nihant At", "author_url": "", "post_date": "2025-05-24T09:22:36.323000", "content": "That's really great!!
", "votes": 1, "replies": [] }, { "id": 3207893, "author_name": "kasey644", "author_url": "", "post_date": "2025-05-23T11:42:16.180000", "content": "it will be very helpful in nova days!
", "votes": 1, "replies": [] }, { "id": 3207856, "author_name": "Prajjwal Pratap Singh", "author_url": "", "post_date": "2025-05-23T10:32:20.167000", "content": "hi @brendanartley couple of quick thoughts/questions, and I'd love to hear your perspective if you have time:
\nResolution scaling: Did you observe any significant performance changes or convergence issues when training on the full resolution compared to down sampled versions? Given the computational cost, I just wanted to understanding the trade-off
\nLoss function Insights: What loss function did you primarily use for this ConvNeXt model? Did you explore any alternatives, perhaps something more tailored to the characteristics of geophysical waveforms or a hybrid loss?
\nFuture Directions: Based on these results, where do you see the most immediate opportunities for improvement? Perhaps exploring different attention mechanisms, a deeper understanding of the \"B\" datasets, or specialized data augmentation techniques?
Hi @bruhjjwal, thanks for the comment.
\nUsing the full resolution data helped significantly.
L1 loss (have not tried other loss functions yet)
Its hard to tell where to focus efforts at the moment. Currently I am playing with batch size / learning rate combinations. In addition, I would like to have a fully converged model before analyzing out of fold predictions.
Would recommend reading this thread. @hoffmanns makes some good points!
", "votes": 1, "replies": [] } ] }, { "id": 3207848, "author_name": "Erich von Mainstein", "author_url": "", "post_date": "2025-05-23T10:16:42.233000", "content": "What batch size do you train the model with? 256 or 16?
", "votes": 1, "replies": [ { "id": 3207913, "author_name": "Bartley", "author_url": "", "post_date": "2025-05-23T12:23:37.177000", "content": "Hi @vonmainstein, I used a batch size of 16 for this model.
", "votes": 1, "replies": [] } ] }, { "id": 3207633, "author_name": "Hima Bindu Marpini", "author_url": "", "post_date": "2025-05-23T04:36:22.677000", "content": "Huge Respect!
", "votes": 1, "replies": [] }, { "id": 3206499, "author_name": "Med Dhifallah", "author_url": "", "post_date": "2025-05-21T11:58:55", "content": "HUGE RESPECT!
", "votes": 1, "replies": [] }, { "id": 3206193, "author_name": "guo dashuai", "author_url": "", "post_date": "2025-05-21T02:42:39.023000", "content": "Is the main improvement due to the use of full-resolution data?🥰
", "votes": 1, "replies": [ { "id": 3206195, "author_name": "Bartley", "author_url": "", "post_date": "2025-05-21T02:51:56.513000", "content": "Hi @guoooooooss, I think so. Though this was my first experiment using the full-resolution data.
", "votes": 3, "replies": [ { "id": 3206200, "author_name": "guo dashuai", "author_url": "", "post_date": "2025-05-21T02:57:29.917000", "content": "thank you very much!🥰
", "votes": 1, "replies": [] } ] } ] }, { "id": 3207695, "author_name": "", "author_url": "", "post_date": "2025-05-23T06:22:24.473000", "content": "Nice work! Appreciate so much
", "votes": 2, "replies": [] }, { "id": 3206070, "author_name": "MGöksu", "author_url": "", "post_date": "2025-05-20T19:28:02.483000", "content": "Bro casually messing up the LB with \"Baseline\" and \"Starter\" notebooks :)
\nJokes aside, I love seeing your notebooks popping up.
", "votes": 2, "replies": [] }, { "id": 3206580, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2025-05-21T14:26:12.983000", "content": "nice sharing 👍
\nDoes pretrained encoder help?
Beautiful LB.
\nIn this match, we achieved communism.
\nHuge respect to Bartley. Love you bro!
Great work :) :)
", "votes": 1, "replies": [] }, { "id": 3206771, "author_name": "Taha Saleem", "author_url": "", "post_date": "2025-05-21T18:43:15.097000", "content": "Thanks for sharing !
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "3206031": "Here is another baseline trained on the full-resolution data. Like my previous notebook, I provide a few trained checkpoints and leave lots of room for customization.\n\nNotebook [here](https://www.kaggle.com/code/brendanartley/convnext-baseline)\nWeights [here](https://www.kaggle.com/datasets/brendanartley/openfwi-preprocessed-72x72)\n\nHere are the CV scores for this approach. Cheers!\n\n```python\n+--------------+--------+\n| Dataset | Score |\n+--------------+--------+\n| CurveFault_A | 6.07 |\n| CurveFault_B | 92.54 |\n| CurveVel_A | 15.07 |\n| CurveVel_B | 53.67 |\n| FlatFault_A | 4.32 |\n| FlatFault_B | 37.60 |\n| FlatVel_A | 2.62 |\n| FlatVel_B | 12.89 |\n| Style_A | 37.38 |\n| Style_B | 57.70 |\n+--------------+--------+\n```", "3206525": "A simple regression model shows that the LB score of the next \"baseline\" Bartley model, due out next week, is less than 20. Tremble, leaders!\n", "3206037": "Loss curves below. Each model takes ~10hrs for 50 epochs on 1×4090 with optimizations (e.g., torch.compile(), fused optimizer, increased num_workers, mixed precision, etc.).\n\n", "3207307": "Hi @brendanartley \n\nAwesome Open!, I am sharing a little bit more finetuned models of yours\n\nhttps://www.kaggle.com/datasets/harshitsheoran/simple-further-finetuned-bartley-open-models\n\nYou can add them to your notebook if you want ( don't want the credit, the hard work was yours ), I have shared you the notebook that would use these weights, the link is at\nhttps://www.kaggle.com/code/harshitsheoran/convnext-baseline/\n\nAs long as there is no leak that is happening in my finetuning, it's CV 0.289 :)\n\nIf you do choose to use these models, ofc, I will say what the finetuning was :D\n\nUpdate:\nHere is my exact code that I used to finetune the models:\n\nhttps://www.kaggle.com/code/harshitsheoran/recipe-simple-further-finetuned-bartley-models\n\nHave a good day!", "3214069": "There is a lot to learn from your notebook, thanks for sharing.\n\nI never used mmap in numpy, this is really neat. Not sure who first share dit here, but kudos to whom did it.\n\nI still have to dig into how you modified convnext!", "3212157": "Based on past few days experimentation, I realized how strong the baseline @brendanartley has provided. There are lot of tricks that are implemented in the network, vanilla network would not perform this well even with out of the box encoder and decoder ! Great work!", "3206983": "Huge respect! Although it is honestly kinda depressing to see my ranking drop from 14 -> 150+ in two days.😂", "3206224": "Show biggest respect to you!", "3214010": "Excellent notebook, thanks for sharing!\n\nJust 1 question, how did you train on \"1x4090\" GPU?\n\nThe code seems to require \"Requires >= 2 GPUs with CUDA enabled.\"", "3212174": "Can someone please explain why was the stride custom changed in this case? ", "3210285": "That's truly amazing. I'm learning a lot.", "3210078": "Looks truly awesome ", "3208519": "That's really great!!", "3207893": "it will be very helpful in nova days!", "3207856": "hi @brendanartley couple of quick thoughts/questions, and I'd love to hear your perspective if you have time:\n\nResolution scaling: Did you observe any significant performance changes or convergence issues when training on the full resolution compared to down sampled versions? Given the computational cost, I just wanted to understanding the trade-off\nLoss function Insights: What loss function did you primarily use for this ConvNeXt model? Did you explore any alternatives, perhaps something more tailored to the characteristics of geophysical waveforms or a hybrid loss?\nFuture Directions: Based on these results, where do you see the most immediate opportunities for improvement? Perhaps exploring different attention mechanisms, a deeper understanding of the \"B\" datasets, or specialized data augmentation techniques?", "3207848": "What batch size do you train the model with? 256 or 16?\n", "3207633": "Huge Respect!", "3206499": "HUGE RESPECT!", "3206193": "Is the main improvement due to the use of full-resolution data?🥰", "3207695": "Nice work! Appreciate so much\n", "3206070": "Bro casually messing up the LB with \"Baseline\" and \"Starter\" notebooks :)\n\nJokes aside, I love seeing your notebooks popping up.", "3206580": "nice sharing 👍\nDoes pretrained encoder help?", "3207057": "Beautiful LB.\nIn this match, we achieved communism.\nHuge respect to Bartley. Love you bro!", "3207055": "", "3206865": "Great work :) :)", "3206771": "Thanks for sharing !\n" } }