![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fa1044b1c3294f19fde8c19bb5af5d27f%2FSelection_133.png?generation=1751329401848736&alt=media\")
Hi everyone,
\nFirst of all, thank you so much to the organizers for such a fun and truly deep learning competition! Lots of high quality data, no shenanigans with the train - test split, was such a joy to participate! π
\nAnd thank you so much to @brendanartley for the amazing shares! Made for such a great time learning and trying out novel approaches π
\nThe question I would like to ask is this: did you manage to train a model like in Bartley's notebooks (could be with modifications) beyond the 22 - 24 CV range?
\nThe reason I am asking is that I made some modifications to the training to optimize for speed / theoretically help with convergence, but in the end I didn't manage to break the ~22 - 24 CV barrier.
\nThe changes I made:
\nconvnext_large.fb_in22k_ft_in1k for the resnet versionIf you could please share your experience, that would be very helpful π I would like to \"debug\" my approach in the hopes of learning how I could improve.
\nI plan to keep my single GPU at home busy over the next couple of weeks, attempting to get to the root of the issue, but if you could please share your experience, that would be greatly appreciated!
\nSeems this competition has definitely raised my appetite for more after a couple of years' hiatus π Quite looking forward to the future proper deep learning ones!
", "messages": [ { "id": "3237127", "postDate": "07/01/2025 00:04:28", "content": "Hi everyone,
\nFirst of all, thank you so much to the organizers for such a fun and truly deep learning competition! Lots of high quality data, no shenanigans with the train - test split, was such a joy to participate! π
\nAnd thank you so much to @brendanartley for the amazing shares! Made for such a great time learning and trying out novel approaches π
\nThe question I would like to ask is this: did you manage to train a model like in Bartley's notebooks (could be with modifications) beyond the 22 - 24 CV range?
\nThe reason I am asking is that I made some modifications to the training to optimize for speed / theoretically help with convergence, but in the end I didn't manage to break the ~22 - 24 CV barrier.
\nThe changes I made:
\nconvnext_large.fb_in22k_ft_in1k for the resnet versionIf you could please share your experience, that would be very helpful π I would like to \"debug\" my approach in the hopes of learning how I could improve.
\nI plan to keep my single GPU at home busy over the next couple of weeks, attempting to get to the root of the issue, but if you could please share your experience, that would be greatly appreciated!
\nSeems this competition has definitely raised my appetite for more after a couple of years' hiatus π Quite looking forward to the future proper deep learning ones!
", "rawMarkdown": "Hi everyone,\n\nFirst of all, thank you so much to the organizers for such a fun and truly deep learning competition! Lots of high quality data, no shenanigans with the train - test split, was such a joy to participate! π\n\nAnd thank you so much to @brendanartley for the amazing shares! Made for such a great time learning and trying out novel approaches π\n\nThe question I would like to ask is this: did you manage to train a model like in Bartley's notebooks (could be with modifications) beyond the 22 - 24 CV range?\n\nThe reason I am asking is that I made some modifications to the training to optimize for speed / theoretically help with convergence, but in the end I didn't manage to break the ~22 - 24 CV barrier.\n\nThe changes I made:\n- training on normalized data (mean and std normalization)\n- creating custom train - val split with possibly better shuffling\n- training in bfloat16\n- switching to `convnext_large.fb_in22k_ft_in1k` for the resnet version\n- increasing the number of channels in the decoder\n- removing the EMA model due to how long the weights update takes\n\nIf you could please share your experience, that would be very helpful π I would like to \"debug\" my approach in the hopes of learning how I could improve.\n\nI plan to keep my single GPU at home busy over the next couple of weeks, attempting to get to the root of the issue, but if you could please share your experience, that would be greatly appreciated!\n\nSeems this competition has definitely raised my appetite for more after a couple of years' hiatus π Quite looking forward to the future proper deep learning ones!", "votes": null }, { "id": "3237132", "postDate": "07/01/2025 00:07:42", "content": "add augmentation and you can get lb20.
\naugmentation:\n) mix of same family : aug = (-mask)*v1 + mask*v2\n)aug seismic = FMW( aug )\n\nmask half mask, etc\nalternatively, use gan to generate aug velocity
", "rawMarkdown": "add augmentation and you can get lb20.\n```\naugmentation:\n1) mix 2 velocity of same family : aug velocity = (1-mask)*v1 + mask*v2\n2)aug seismic = FMW( aug velocity )\n\nmask random half mask, etc\n```\n\nalternatively, use gan to generate aug velocity", "votes": null }, { "id": "3237149", "postDate": "07/01/2025 00:19:55", "content": "my score (13.4) is only using Bartley caformer. I don't know anything in computer vision, hence focused on how to best train the best public model.
\nThe only change I made (very proud it was also used by @harshitsheoran ) was to also train it with the 5 seismic planes side by side, i.e. a single channel 1000 x 350 image.
", "rawMarkdown": "my score (13.4) is only using Bartley caformer. I don't know anything in computer vision, hence focused on how to best train the best public model.\n\nThe only change I made (very proud it was also used by @harshitsheoran ) was to also train it with the 5 seismic planes side by side, i.e. a single channel 1000 x 350 image.", "votes": null }, { "id": "3237151", "postDate": "07/01/2025 00:20:29", "content": "Thank you so much for your reply @hengck23!
\nSo essentially, it seems there might have not been anything wrong with my setup, there might have been a hard ceiling for these models at around ~22lb - ~24lb without other tricks (like data aug)?
\nThat would be a great relief π
", "rawMarkdown": "Thank you so much for your reply @hengck23!\n\nSo essentially, it seems there might have not been anything wrong with my setup, there might have been a hard ceiling for these models at around ~22lb - ~24lb without other tricks (like data aug)?\n\nThat would be a great relief π ", "votes": null }, { "id": "3237157", "postDate": "07/01/2025 00:25:13", "content": "example of augmention
\n
using mix mask
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F539f32dc55b130cf92fac2b47d93c5f7%2FSelection_197.png?generation=1751329511441305&alt=media\")
here is the breakdown:
\n28.8 : bartely's caformer
\n25.+ : ensemble of (bartely's caformers)
\n20.+ : augmentation
\n15.+ : add test samples into the learning set (predicted velocity, simulated seismic)
\nsome more gain if you fine-tune for each family
by adding 2m generated samples (from our own velocity simulation code + forward modeling), long training (150-300 epochs) and larger convnext backbones (large, xxlarge or hugev2), we can achieve 10.x - 11.x scores with the exactly same Bartley type's model.
\nWe didn't experiment with anything related to architecture (collapsing the time dimension, resizing images to a square or larger resolution), which turned out to be a mistake. We got stuck after reaching 11 for quite a while :D.
", "rawMarkdown": "by adding 2m generated samples (from our own velocity simulation code + forward modeling), long training (150-300 epochs) and larger convnext backbones (large, xxlarge or hugev2), we can achieve 10.x - 11.x scores with the exactly same Bartley type's model.\n\nWe didn't experiment with anything related to architecture (collapsing the time dimension, resizing images to a square or larger resolution), which turned out to be a mistake. We got stuck after reaching 11 for quite a while :D.", "votes": null }, { "id": "3237177", "postDate": "07/01/2025 00:40:25", "content": "so it proves that with 10x data can approximate the physics inverse model
", "rawMarkdown": "so it proves that with 10x data can approximate the physics inverse model", "votes": null }, { "id": "3237183", "postDate": "07/01/2025 00:44:20", "content": "Thank you so much for your reply, @cpmpml! ππ»
\nCould I please ask you if you trained in float32? Technically, bfloat16 shouldnβt make a difference, as far as I understand, but that is one thing about my approach I didnβt verifyβ¦
", "rawMarkdown": "Thank you so much for your reply, @cpmpml! ππ»\n\nCould I please ask you if you trained in float32? Technically, bfloat16 shouldnβt make a difference, as far as I understand, but that is one thing about my approach I didnβt verifyβ¦", "votes": null }, { "id": "3237187", "postDate": "07/01/2025 00:46:18", "content": "Thank you for sharing, @andy2709, super helpful!
\nCould I please ask you if you trained in float32?
", "rawMarkdown": "Thank you for sharing, @andy2709, super helpful!\n\nCould I please ask you if you trained in float32?", "votes": null }, { "id": "3237188", "postDate": "07/01/2025 00:47:46", "content": "Thank you, @hengck23, extremely helpful ππ»π
", "rawMarkdown": "Thank you, @hengck23, extremely helpful ππ»π", "votes": null }, { "id": "3237193", "postDate": "07/01/2025 00:51:43", "content": "no @radek1, we trained all models in bfloat16.
", "rawMarkdown": "no @radek1, we trained all models in bfloat16.", "votes": null }, { "id": "3237195", "postDate": "07/01/2025 00:51:50", "content": "I trained with bf16.
", "rawMarkdown": "I trained with bf16.", "votes": null }, { "id": "3237196", "postDate": "07/01/2025 00:51:53", "content": "And @hengck23, if I could please ask you, did you train in float32?
\nIβm curious if that could be one of the things I missedβ¦
", "rawMarkdown": "And @hengck23, if I could please ask you, did you train in float32?\n\nIβm curious if that could be one of the things I missedβ¦", "votes": null }, { "id": "3237223", "postDate": "07/01/2025 01:13:33", "content": "I only use bartley's caformer and finetune it with pseudo labels, and then I can get 24.3.
", "rawMarkdown": "I only use bartley's caformer and finetune it with pseudo labels, and then I can get 24.3.", "votes": null }, { "id": "3237228", "postDate": "07/01/2025 01:22:20", "content": "Bfloat16 is good enough
", "rawMarkdown": "Bfloat16 is good enough", "votes": null }, { "id": "3237256", "postDate": "07/01/2025 02:15:05", "content": "Glad to hear you are coming back. Your and Chris Dβs efforts in LLM science exam really changed my entire career trajectory! :)
", "rawMarkdown": "Glad to hear you are coming back. Your and Chris Dβs efforts in LLM science exam really changed my entire career trajectory! :)", "votes": null }, { "id": "3237270", "postDate": "07/01/2025 02:29:48", "content": "Wow, that is terrific to hear! π π
", "rawMarkdown": "Wow, that is terrific to hear! π π", "votes": null }, { "id": "3237503", "postDate": "07/01/2025 06:17:00", "content": "May i ask how do you finetune with pseudo labels?
", "rawMarkdown": "May i ask how do you finetune with pseudo labels?", "votes": null }, { "id": "3237652", "postDate": "07/01/2025 08:03:47", "content": "Just want to add that this competition should be put in ML competition textbook.
\nIt is actually is very good for tutorial.
\nit has all aspects and tricks : novel network design, synthetic data generation, physics modeling, multi-gpu training , fast cuda coding β¦
1) Predict given test seis data to get vel data y_hat
\n2) Use Forward Modeling to generate new seis data x_hat with predicted vel data y_hat
\n3) Finetune model with (x_hat, y_hat)
The following is forward modeling.
\nAm i correct?
https://www.kaggle.com/code/manatoyo/improved-vel-to-seis
", "rawMarkdown": "The following is forward modeling. \nAm i correct?\n\nhttps://www.kaggle.com/code/manatoyo/improved-vel-to-seis", "votes": null }, { "id": "3237717", "postDate": "07/01/2025 09:02:27", "content": "Yes, but I use this code: https://www.kaggle.com/competitions/waveform-inversion/discussion/585236
", "rawMarkdown": "Yes, but I use this code: https://www.kaggle.com/competitions/waveform-inversion/discussion/585236", "votes": null }, { "id": "3238557", "postDate": "07/02/2025 01:04:30", "content": "Yes, 100%! π Also the data is great and there is a lot of data to go by, what an awesome competition this was!!!!
", "rawMarkdown": "Yes, 100%! π Also the data is great and there is a lot of data to go by, what an awesome competition this was!!!!", "votes": null }, { "id": "3238635", "postDate": "07/02/2025 04:01:40", "content": "i have a suggestino for you
\n1) start a community competition, entitled:
\n\"post competition study of Yale/UNC-CH - Geophysicale β¦\"
\nobjective:
\nafter reading the top solution, you may think \"hey, the top solution is smiliar to mine. why his results is much better when i did what he did?\" This study aim to algorithm debug. Let's find out what is done wrong and help to do better in future competition.
dataset:
\nscaled version of full data (e.g. just choose curvefault^ ,β¦.
plan:
\nlayout what you want to do:
duration:
\ni would say 4 weeks is a good timeline.
note that is a \"group-study competition\".everyone can share code and discuss. the leaderboard is just a check to see if everything was done correctly: you should get same score as others have
\nwhy this effort?
\nThat is a great idea! I have been actually thinking of doing something along these lines, but using the late submission to this competition π
\nLet me work on this a bit more and will share π But thank you so much for all your thoughts, @hengck23, really appreciated π
", "rawMarkdown": "That is a great idea! I have been actually thinking of doing something along these lines, but using the late submission to this competition π\n\nLet me work on this a bit more and will share π But thank you so much for all your thoughts, @hengck23, really appreciated π", "votes": null }, { "id": "3238912", "postDate": "07/02/2025 10:08:06", "content": "\n\n1) Predict given test seis data to get vel data y_hat
\n
\n 2) Use Forward Modeling to generate new seis data x_hat with predicted vel data y_hat
\n 3) Finetune model with (x_hat, y_hat)
I got caformer to 13.4 with this method, see my writeup.
", "rawMarkdown": "> 1) Predict given test seis data to get vel data y_hat\n> 2) Use Forward Modeling to generate new seis data x_hat with predicted vel data y_hat\n> 3) Finetune model with (x_hat, y_hat)\n\nI got caformer to 13.4 with this method, see my writeup.", "votes": null } ], "comments": [ { "id": 3237132, "author_name": "hengck23", "author_url": "", "post_date": "07/01/2025 00:07:42", "content": "add augmentation and you can get lb20.
\naugmentation:\n) mix of same family : aug = (-mask)*v1 + mask*v2\n)aug seismic = FMW( aug )\n\nmask half mask, etc\nalternatively, use gan to generate aug velocity
", "votes": null, "replies": [ { "id": 3237151, "author_name": "radek1", "author_url": "", "post_date": "07/01/2025 00:20:29", "content": "Thank you so much for your reply @hengck23!
\nSo essentially, it seems there might have not been anything wrong with my setup, there might have been a hard ceiling for these models at around ~22lb - ~24lb without other tricks (like data aug)?
\nThat would be a great relief π
", "votes": null, "replies": [ { "id": 3237166, "author_name": "hengck23", "author_url": "", "post_date": "07/01/2025 00:28:55", "content": "here is the breakdown:
\n28.8 : bartely's caformer
\n25.+ : ensemble of (bartely's caformers)
\n20.+ : augmentation
\n15.+ : add test samples into the learning set (predicted velocity, simulated seismic)
\nsome more gain if you fine-tune for each family
example of augmention
\n
using mix mask
\n
Thank you, @hengck23, extremely helpful ππ»π
", "votes": null, "replies": [ { "id": 3237196, "author_name": "radek1", "author_url": "", "post_date": "07/01/2025 00:51:53", "content": "And @hengck23, if I could please ask you, did you train in float32?
\nIβm curious if that could be one of the things I missedβ¦
", "votes": null, "replies": [ { "id": 3237228, "author_name": "hengck23", "author_url": "", "post_date": "07/01/2025 01:22:20", "content": "Bfloat16 is good enough
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 3237149, "author_name": "cpmpml", "author_url": "", "post_date": "07/01/2025 00:19:55", "content": "my score (13.4) is only using Bartley caformer. I don't know anything in computer vision, hence focused on how to best train the best public model.
\nThe only change I made (very proud it was also used by @harshitsheoran ) was to also train it with the 5 seismic planes side by side, i.e. a single channel 1000 x 350 image.
", "votes": null, "replies": [ { "id": 3237183, "author_name": "radek1", "author_url": "", "post_date": "07/01/2025 00:44:20", "content": "Thank you so much for your reply, @cpmpml! ππ»
\nCould I please ask you if you trained in float32? Technically, bfloat16 shouldnβt make a difference, as far as I understand, but that is one thing about my approach I didnβt verifyβ¦
", "votes": null, "replies": [ { "id": 3237195, "author_name": "cpmpml", "author_url": "", "post_date": "07/01/2025 00:51:50", "content": "I trained with bf16.
", "votes": null, "replies": [] } ] } ] }, { "id": 3237172, "author_name": "andy2709", "author_url": "", "post_date": "07/01/2025 00:33:35", "content": "by adding 2m generated samples (from our own velocity simulation code + forward modeling), long training (150-300 epochs) and larger convnext backbones (large, xxlarge or hugev2), we can achieve 10.x - 11.x scores with the exactly same Bartley type's model.
\nWe didn't experiment with anything related to architecture (collapsing the time dimension, resizing images to a square or larger resolution), which turned out to be a mistake. We got stuck after reaching 11 for quite a while :D.
", "votes": null, "replies": [ { "id": 3237177, "author_name": "hengck23", "author_url": "", "post_date": "07/01/2025 00:40:25", "content": "so it proves that with 10x data can approximate the physics inverse model
", "votes": null, "replies": [] }, { "id": 3237187, "author_name": "radek1", "author_url": "", "post_date": "07/01/2025 00:46:18", "content": "Thank you for sharing, @andy2709, super helpful!
\nCould I please ask you if you trained in float32?
", "votes": null, "replies": [ { "id": 3237193, "author_name": "andy2709", "author_url": "", "post_date": "07/01/2025 00:51:43", "content": "no @radek1, we trained all models in bfloat16.
", "votes": null, "replies": [] } ] } ] }, { "id": 3237223, "author_name": "i2nfinit3y", "author_url": "", "post_date": "07/01/2025 01:13:33", "content": "I only use bartley's caformer and finetune it with pseudo labels, and then I can get 24.3.
", "votes": null, "replies": [ { "id": 3237503, "author_name": "overvalueawareness", "author_url": "", "post_date": "07/01/2025 06:17:00", "content": "May i ask how do you finetune with pseudo labels?
", "votes": null, "replies": [ { "id": 3237657, "author_name": "i2nfinit3y", "author_url": "", "post_date": "07/01/2025 08:05:57", "content": "1) Predict given test seis data to get vel data y_hat
\n2) Use Forward Modeling to generate new seis data x_hat with predicted vel data y_hat
\n3) Finetune model with (x_hat, y_hat)
The following is forward modeling.
\nAm i correct?
https://www.kaggle.com/code/manatoyo/improved-vel-to-seis
", "votes": null, "replies": [ { "id": 3237717, "author_name": "i2nfinit3y", "author_url": "", "post_date": "07/01/2025 09:02:27", "content": "Yes, but I use this code: https://www.kaggle.com/competitions/waveform-inversion/discussion/585236
", "votes": null, "replies": [ { "id": 3238912, "author_name": "cpmpml", "author_url": "", "post_date": "07/02/2025 10:08:06", "content": "\n\n1) Predict given test seis data to get vel data y_hat
\n
\n 2) Use Forward Modeling to generate new seis data x_hat with predicted vel data y_hat
\n 3) Finetune model with (x_hat, y_hat)
I got caformer to 13.4 with this method, see my writeup.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] }, { "id": 3237256, "author_name": "cody11null", "author_url": "", "post_date": "07/01/2025 02:15:05", "content": "Glad to hear you are coming back. Your and Chris Dβs efforts in LLM science exam really changed my entire career trajectory! :)
", "votes": null, "replies": [ { "id": 3237270, "author_name": "radek1", "author_url": "", "post_date": "07/01/2025 02:29:48", "content": "Wow, that is terrific to hear! π π
", "votes": null, "replies": [] } ] }, { "id": 3237652, "author_name": "hengck23", "author_url": "", "post_date": "07/01/2025 08:03:47", "content": "Just want to add that this competition should be put in ML competition textbook.
\nIt is actually is very good for tutorial.
\nit has all aspects and tricks : novel network design, synthetic data generation, physics modeling, multi-gpu training , fast cuda coding β¦
Yes, 100%! π Also the data is great and there is a lot of data to go by, what an awesome competition this was!!!!
", "votes": null, "replies": [ { "id": 3238635, "author_name": "hengck23", "author_url": "", "post_date": "07/02/2025 04:01:40", "content": "i have a suggestino for you
\n1) start a community competition, entitled:
\n\"post competition study of Yale/UNC-CH - Geophysicale β¦\"
\nobjective:
\nafter reading the top solution, you may think \"hey, the top solution is smiliar to mine. why his results is much better when i did what he did?\" This study aim to algorithm debug. Let's find out what is done wrong and help to do better in future competition.
dataset:
\nscaled version of full data (e.g. just choose curvefault^ ,β¦.
plan:
\nlayout what you want to do:
duration:
\ni would say 4 weeks is a good timeline.
note that is a \"group-study competition\".everyone can share code and discuss. the leaderboard is just a check to see if everything was done correctly: you should get same score as others have
\nwhy this effort?
\nThat is a great idea! I have been actually thinking of doing something along these lines, but using the late submission to this competition π
\nLet me work on this a bit more and will share π But thank you so much for all your thoughts, @hengck23, really appreciated π
", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "3237127": "Hi everyone,\n\nFirst of all, thank you so much to the organizers for such a fun and truly deep learning competition! Lots of high quality data, no shenanigans with the train - test split, was such a joy to participate! π\n\nAnd thank you so much to @brendanartley for the amazing shares! Made for such a great time learning and trying out novel approaches π\n\nThe question I would like to ask is this: did you manage to train a model like in Bartley's notebooks (could be with modifications) beyond the 22 - 24 CV range?\n\nThe reason I am asking is that I made some modifications to the training to optimize for speed / theoretically help with convergence, but in the end I didn't manage to break the ~22 - 24 CV barrier.\n\nThe changes I made:\n- training on normalized data (mean and std normalization)\n- creating custom train - val split with possibly better shuffling\n- training in bfloat16\n- switching to `convnext_large.fb_in22k_ft_in1k` for the resnet version\n- increasing the number of channels in the decoder\n- removing the EMA model due to how long the weights update takes\n\nIf you could please share your experience, that would be very helpful π I would like to \"debug\" my approach in the hopes of learning how I could improve.\n\nI plan to keep my single GPU at home busy over the next couple of weeks, attempting to get to the root of the issue, but if you could please share your experience, that would be greatly appreciated!\n\nSeems this competition has definitely raised my appetite for more after a couple of years' hiatus π Quite looking forward to the future proper deep learning ones!", "3237132": "add augmentation and you can get lb20.\n```\naugmentation:\n1) mix 2 velocity of same family : aug velocity = (1-mask)*v1 + mask*v2\n2)aug seismic = FMW( aug velocity )\n\nmask random half mask, etc\n```\n\nalternatively, use gan to generate aug velocity", "3237149": "my score (13.4) is only using Bartley caformer. I don't know anything in computer vision, hence focused on how to best train the best public model.\n\nThe only change I made (very proud it was also used by @harshitsheoran ) was to also train it with the 5 seismic planes side by side, i.e. a single channel 1000 x 350 image.", "3237151": "Thank you so much for your reply @hengck23!\n\nSo essentially, it seems there might have not been anything wrong with my setup, there might have been a hard ceiling for these models at around ~22lb - ~24lb without other tricks (like data aug)?\n\nThat would be a great relief π ", "3237157": "example of augmention\n\n\n\n\nusing mix mask\n", "3237166": "here is the breakdown:\n28.8 : bartely's caformer\n25.+ : ensemble of (bartely's caformers)\n20.+ : augmentation\n15.+ : add test samples into the learning set (predicted velocity, simulated seismic)\nsome more gain if you fine-tune for each family", "3237172": "by adding 2m generated samples (from our own velocity simulation code + forward modeling), long training (150-300 epochs) and larger convnext backbones (large, xxlarge or hugev2), we can achieve 10.x - 11.x scores with the exactly same Bartley type's model.\n\nWe didn't experiment with anything related to architecture (collapsing the time dimension, resizing images to a square or larger resolution), which turned out to be a mistake. We got stuck after reaching 11 for quite a while :D.", "3237177": "so it proves that with 10x data can approximate the physics inverse model", "3237183": "Thank you so much for your reply, @cpmpml! ππ»\n\nCould I please ask you if you trained in float32? Technically, bfloat16 shouldnβt make a difference, as far as I understand, but that is one thing about my approach I didnβt verifyβ¦", "3237187": "Thank you for sharing, @andy2709, super helpful!\n\nCould I please ask you if you trained in float32?", "3237188": "Thank you, @hengck23, extremely helpful ππ»π", "3237193": "no @radek1, we trained all models in bfloat16.", "3237195": "I trained with bf16.", "3237196": "And @hengck23, if I could please ask you, did you train in float32?\n\nIβm curious if that could be one of the things I missedβ¦", "3237223": "I only use bartley's caformer and finetune it with pseudo labels, and then I can get 24.3.", "3237228": "Bfloat16 is good enough", "3237256": "Glad to hear you are coming back. Your and Chris Dβs efforts in LLM science exam really changed my entire career trajectory! :)", "3237270": "Wow, that is terrific to hear! π π", "3237503": "May i ask how do you finetune with pseudo labels?", "3237652": "Just want to add that this competition should be put in ML competition textbook.\nIt is actually is very good for tutorial.\nit has all aspects and tricks : novel network design, synthetic data generation, physics modeling, multi-gpu training , fast cuda coding ...", "3237657": "1) Predict given test seis data to get vel data **y_hat**\n2) Use Forward Modeling to generate new seis data **x_hat** with predicted vel data **y_hat**\n3) Finetune model with (**x_hat**, **y_hat**)", "3237689": "The following is forward modeling. \nAm i correct?\n\nhttps://www.kaggle.com/code/manatoyo/improved-vel-to-seis", "3237717": "Yes, but I use this code: https://www.kaggle.com/competitions/waveform-inversion/discussion/585236", "3238557": "Yes, 100%! π Also the data is great and there is a lot of data to go by, what an awesome competition this was!!!!", "3238635": "i have a suggestino for you\n\n1) start a community competition, entitled:\n\"post competition study of Yale/UNC-CH - Geophysicale ...\"\nobjective:\nafter reading the top solution, you may think \"hey, the top solution is smiliar to mine. why his results is much better when i did what he did?\" This study aim to algorithm debug. Let's find out what is done wrong and help to do better in future competition.\n\ndataset:\nscaled version of full data (e.g. just choose curvefault^ ,....\n\nplan:\nlayout what you want to do:\n- design new network (that is better than what you have used in the current competitioon)\n- understand the physics of wave modeling and implement cuda code\n- ... repeat winning solution ...\n\nduration:\ni would say 4 weeks is a good timeline.\n- consistently published your code notebook for the plan above\n- start a discord group, people can join and debug each other code,algorithm\n\nnote that is a \"group-study competition\".everyone can share code and discuss. the leaderboard is just a check to see if everything was done correctly: you should get same score as others have\n\n---\n\nwhy this effort?\n- my experience is that a (large) group of people trying to debug and improve your code/agorithm is more efficient then doing it yourself.\nother may give you unexpected improvement", "3238831": "That is a great idea! I have been actually thinking of doing something along these lines, but using the late submission to this competition π\n\nLet me work on this a bit more and will share π But thank you so much for all your thoughts, @hengck23, really appreciated π", "3238912": "> 1) Predict given test seis data to get vel data y_hat\n> 2) Use Forward Modeling to generate new seis data x_hat with predicted vel data y_hat\n> 3) Finetune model with (x_hat, y_hat)\n\nI got caformer to 13.4 with this method, see my writeup." }, "source": "meta" }