![\"LossImages\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5570735%2F4ae987ace74101119d0c0ec8de06971f%2Fimage.jpg?generation=1747189628091945&alt=media\")
Excited to share a notebook that builds on the great starter notebook from @egortrushin. The updated notebook showcases training on 2x T4 GPUs to maximize our GPU quota!
\n\nI also provide a few trained checkpoints for a HGNet-V2-Unet model, flip augmentations, EMA, and more. Here are the CV scores by dataset type. Experiment away! 😀
\n+--------------+--------+\n| Dataset | Score |\n+--------------+--------+\n| CurveFault_A | |\n| CurveFault_B | |\n| CurveVel_A | |\n| CurveVel_B | |\n| FlatFault_A | |\n| FlatFault_B | |\n| FlatVel_A | |\n| FlatVel_B | |\n| Style_A | |\n| Style_B | |\n+--------------+--------+\nExcited to share a notebook that builds on the great starter notebook from @egortrushin. The updated notebook showcases training on 2x T4 GPUs to maximize our GPU quota!
\n\nI also provide a few trained checkpoints for a HGNet-V2-Unet model, flip augmentations, EMA, and more. Here are the CV scores by dataset type. Experiment away! 😀
\n+--------------+--------+\n| Dataset | Score |\n+--------------+--------+\n| CurveFault_A | |\n| CurveFault_B | |\n| CurveVel_A | |\n| CurveVel_B | |\n| FlatFault_A | |\n| FlatFault_B | |\n| FlatVel_A | |\n| FlatVel_B | |\n| Style_A | |\n| Style_B | |\n+--------------+--------+\n@brendanartley
\nThank you for sharing your notebook.
\nIt seems that the CV scores differ significantly between the different datasets, do you have any new insights or advice on the differences?For example, should we focus on a particular dataset to address it, etc.
\nBest regards.
Hi @ov104ss,
\nA 2-stage approach could work well. For example, you could train a model to predict which class each data point belongs to. Then, you could train a family specific model for the velocity-map reconstruction step. It may be that some model architectures work well for some families but not for others. Definitely worth trying!
\nMaybe others can chime in if they have already compared 1-stage vs 2-stage approaches?
", "rawMarkdown": "Hi @ov104ss,\n\nA 2-stage approach could work well. For example, you could train a model to predict which class each data point belongs to. Then, you could train a family specific model for the velocity-map reconstruction step. It may be that some model architectures work well for some families but not for others. Definitely worth trying!\n\nMaybe others can chime in if they have already compared 1-stage vs 2-stage approaches?", "votes": null }, { "id": "3199157", "postDate": "05/10/2025 14:35:45", "content": "@brendanartley
\nThank you for your reply. That idea is worth a try. I was thinking that it would be difficult to cover all families with a single model, so I will consider it. Thank you very much.
DDP does appear that its the way of the future for running multiple GPU's, but I am finding it pretty difficult to debug errors when making changes to the code. Any suggestions (besides lots of print code) to ease this issue using Visual Code on local machine?
\nAnother thing I knew but forgot and relearned the hard way, its going to write over any existing config, utils.py, etc when you run it local. If your like me and playing with lots of the shared notebooks in the competition your going to be unhappy if you don't get into a habit of separate folders per shared notebook on your local machine.
", "rawMarkdown": "DDP does appear that its the way of the future for running multiple GPU's, but I am finding it pretty difficult to debug errors when making changes to the code. Any suggestions (besides lots of print code) to ease this issue using Visual Code on local machine?\n\nAnother thing I knew but forgot and relearned the hard way, its going to write over any existing config, utils.py, etc when you run it local. If your like me and playing with lots of the shared notebooks in the competition your going to be unhappy if you don't get into a habit of separate folders per shared notebook on your local machine.", "votes": null }, { "id": "3199327", "postDate": "05/10/2025 19:54:24", "content": "I wonder if a MoE type of approach would work for it ?, Cause the task is to recreate the velocity maps in all the types, but the scores vary, as far as I'm aware this indicates a change in distribution, a router based mixture of experts type of thing would work pretty well in my opinion.
", "rawMarkdown": "I wonder if a MoE type of approach would work for it ?, Cause the task is to recreate the velocity maps in all the types, but the scores vary, as far as I'm aware this indicates a change in distribution, a router based mixture of experts type of thing would work pretty well in my opinion.", "votes": null }, { "id": "3199388", "postDate": "05/10/2025 21:57:38", "content": "Hi @pcjimmmy, thanks for the comment.
\nOne thing you could do is extract pieces of this notebook and insert them into an existing pipeline that works for you (models, augs, snippets, etc). This might help pinpoint issues in your environment. Hope this helps!
", "rawMarkdown": "Hi @pcjimmmy, thanks for the comment.\n\nOne thing you could do is extract pieces of this notebook and insert them into an existing pipeline that works for you (models, augs, snippets, etc). This might help pinpoint issues in your environment. Hope this helps!", "votes": null }, { "id": "3201496", "postDate": "05/14/2025 00:38:44", "content": "Update 13/5/2025
\nCV: 65.2 -> 55.6
\nLB: 69.7 -> 60.9
Thanks everyone for waiting, the training script now works on Kaggle. I also released 3x stronger pretrained checkpoints (trained for 150 epochs), and added more customizable decoder. Cheers!
", "rawMarkdown": "**Update 13/5/2025**\n\nCV: 65.2 -> 55.6\nLB: 69.7 -> 60.9\n\nThanks everyone for waiting, the training script now works on Kaggle. I also released 3x stronger pretrained checkpoints (trained for 150 epochs), and added more customizable decoder. Cheers!", "votes": null }, { "id": "3201517", "postDate": "05/14/2025 01:49:22", "content": "May I ask what's your training set MAE after 50/150 epochs? In my case train/CV/LB : 25/36/36 after 40 epochs, I am considering whether it has fully converged.
", "rawMarkdown": "May I ask what's your training set MAE after 50/150 epochs? In my case train/CV/LB : 25/36/36 after 40 epochs, I am considering whether it has fully converged.", "votes": null }, { "id": "3201530", "postDate": "05/14/2025 02:29:01", "content": "Sure, here are the loss curves for a 150 epoch training run.
\n
Crazy epochs! Out of curiosity, how long did the 250 epochs take?
", "rawMarkdown": "Crazy epochs! Out of curiosity, how long did the 250 epochs take?", "votes": null }, { "id": "3201538", "postDate": "05/14/2025 03:00:12", "content": "Thanks for sharing! The improvement is mainly due to the changes in pretrained model, right? And may I ask about your training data for pretrained model, with all OpenFWI dataset?
", "rawMarkdown": "Thanks for sharing! The improvement is mainly due to the changes in pretrained model, right? And may I ask about your training data for pretrained model, with all OpenFWI dataset?", "votes": null }, { "id": "3201548", "postDate": "05/14/2025 03:14:10", "content": "Hi @chenxin1991, thanks for the comment.
\nThe pre-trained model is actually the same from V1 -> V4, there is just more customizability now. The models are trained on the same train/valid set as the public notebook (~98% of all OpenFWI), and the CV/LB improvement came purely from increasing the training epochs!
", "rawMarkdown": "Hi @chenxin1991, thanks for the comment.\n\nThe pre-trained model is actually the same from V1 -> V4, there is just more customizability now. The models are trained on the same train/valid set as the public notebook (~98% of all OpenFWI), and the CV/LB improvement came purely from increasing the training epochs!", "votes": null }, { "id": "3201550", "postDate": "05/14/2025 03:31:06", "content": "Hello, what are the differences between these three models?
", "rawMarkdown": "Hello, what are the differences between these three models?", "votes": null }, { "id": "3201573", "postDate": "05/14/2025 04:30:54", "content": "It takes ~24hrs on 2x T4 GPUs for 150 epochs, though it is much faster on more modern GPUs.
", "rawMarkdown": "It takes ~24hrs on 2x T4 GPUs for 150 epochs, though it is much faster on more modern GPUs.", "votes": null }, { "id": "3201600", "postDate": "05/14/2025 05:57:39", "content": "Your dataset is excellent—it eliminates the bottleneck caused by disk loading.
", "rawMarkdown": "Your dataset is excellent—it eliminates the bottleneck caused by disk loading.", "votes": null }, { "id": "3201686", "postDate": "05/14/2025 08:54:23", "content": "May I ask what's the difference among checkpoint best0~2 during training? Since they can all be loaded by the same model structure, it suggests the architecture is identical. Is the difference due to 3-fold cross-validation?
", "rawMarkdown": "May I ask what's the difference among checkpoint best0~2 during training? Since they can all be loaded by the same model structure, it suggests the architecture is identical. Is the difference due to 3-fold cross-validation?", "votes": null }, { "id": "3201822", "postDate": "05/14/2025 12:46:23", "content": "@cqrcqr, @wayne127 thanks for your comments.
\nThe only difference between the three models is their seed. They were trained and evaluated on the same fold as in the public notebook.
", "rawMarkdown": "cqrcqr, @wayne127 thanks for your comments.\n\nThe only difference between the three models is their seed. They were trained and evaluated on the same fold as in the public notebook.", "votes": null }, { "id": "3202350", "postDate": "05/15/2025 09:01:25", "content": "Thank you for sharing. I noticed that you used data augmentation. May I ask why the width of the flip speed graph corresponds to the flipping of num_Sources and num_deceivers? I don't quite understand the relationship between them. I hope you can help me🥹
", "rawMarkdown": "Thank you for sharing. I noticed that you used data augmentation. May I ask why the width of the flip speed graph corresponds to the flipping of num_Sources and num_deceivers? I don't quite understand the relationship between them. I hope you can help me🥹", "votes": null }, { "id": "3202468", "postDate": "05/15/2025 12:24:11", "content": "https://www.kaggle.com/code/brendanartley/hgnet-v2-starter/comments#3200039 🌚
", "rawMarkdown": "https://www.kaggle.com/code/brendanartley/hgnet-v2-starter/comments#3200039 🌚", "votes": null }, { "id": "3202524", "postDate": "05/15/2025 14:14:52", "content": "Bartley, thank you for sharing your work. I'm learning a lot from it.
\nYour network seems to be composed of many different blocks. Is there a methodology to the way you pick the building blocks? Or is it experience and/or intuition?
", "rawMarkdown": "Bartley, thank you for sharing your work. I'm learning a lot from it. \n\nYour network seems to be composed of many different blocks. Is there a methodology to the way you pick the building blocks? Or is it experience and/or intuition?", "votes": null }, { "id": "3202551", "postDate": "05/15/2025 14:44:12", "content": "Hi @mgoksu, most of the blocks are there for customizability.
\nFor me, choosing which blocks to use is a mix of intuition and cross validation (mostly the latter). It is common for my intuition to be wrong, but a robust cross validation scheme rarely lies.
", "rawMarkdown": "Hi @mgoksu, most of the blocks are there for customizability. \n\nFor me, choosing which blocks to use is a mix of intuition and cross validation (mostly the latter). It is common for my intuition to be wrong, but a robust cross validation scheme rarely lies.", "votes": null }, { "id": "3202696", "postDate": "05/15/2025 19:12:56", "content": "thanks 嘴爷😍
", "rawMarkdown": "thanks 嘴爷😍", "votes": null } ], "comments": [ { "id": 3199046, "author_name": "ov104ss", "author_url": "", "post_date": "05/10/2025 11:29:17", "content": "@brendanartley
\nThank you for sharing your notebook.
\nIt seems that the CV scores differ significantly between the different datasets, do you have any new insights or advice on the differences?For example, should we focus on a particular dataset to address it, etc.
\nBest regards.
Hi @ov104ss,
\nA 2-stage approach could work well. For example, you could train a model to predict which class each data point belongs to. Then, you could train a family specific model for the velocity-map reconstruction step. It may be that some model architectures work well for some families but not for others. Definitely worth trying!
\nMaybe others can chime in if they have already compared 1-stage vs 2-stage approaches?
", "votes": null, "replies": [ { "id": 3199157, "author_name": "ov104ss", "author_url": "", "post_date": "05/10/2025 14:35:45", "content": "@brendanartley
\nThank you for your reply. That idea is worth a try. I was thinking that it would be difficult to cover all families with a single model, so I will consider it. Thank you very much.
DDP does appear that its the way of the future for running multiple GPU's, but I am finding it pretty difficult to debug errors when making changes to the code. Any suggestions (besides lots of print code) to ease this issue using Visual Code on local machine?
\nAnother thing I knew but forgot and relearned the hard way, its going to write over any existing config, utils.py, etc when you run it local. If your like me and playing with lots of the shared notebooks in the competition your going to be unhappy if you don't get into a habit of separate folders per shared notebook on your local machine.
", "votes": null, "replies": [ { "id": 3199388, "author_name": "brendanartley", "author_url": "", "post_date": "05/10/2025 21:57:38", "content": "Hi @pcjimmmy, thanks for the comment.
\nOne thing you could do is extract pieces of this notebook and insert them into an existing pipeline that works for you (models, augs, snippets, etc). This might help pinpoint issues in your environment. Hope this helps!
", "votes": null, "replies": [] } ] }, { "id": 3199327, "author_name": "pshikk", "author_url": "", "post_date": "05/10/2025 19:54:24", "content": "I wonder if a MoE type of approach would work for it ?, Cause the task is to recreate the velocity maps in all the types, but the scores vary, as far as I'm aware this indicates a change in distribution, a router based mixture of experts type of thing would work pretty well in my opinion.
", "votes": null, "replies": [] }, { "id": 3201496, "author_name": "brendanartley", "author_url": "", "post_date": "05/14/2025 00:38:44", "content": "Update 13/5/2025
\nCV: 65.2 -> 55.6
\nLB: 69.7 -> 60.9
Thanks everyone for waiting, the training script now works on Kaggle. I also released 3x stronger pretrained checkpoints (trained for 150 epochs), and added more customizable decoder. Cheers!
", "votes": null, "replies": [ { "id": 3201517, "author_name": "w5833946", "author_url": "", "post_date": "05/14/2025 01:49:22", "content": "May I ask what's your training set MAE after 50/150 epochs? In my case train/CV/LB : 25/36/36 after 40 epochs, I am considering whether it has fully converged.
", "votes": null, "replies": [ { "id": 3201530, "author_name": "brendanartley", "author_url": "", "post_date": "05/14/2025 02:29:01", "content": "Sure, here are the loss curves for a 150 epoch training run.
\n
Crazy epochs! Out of curiosity, how long did the 250 epochs take?
", "votes": null, "replies": [ { "id": 3201573, "author_name": "brendanartley", "author_url": "", "post_date": "05/14/2025 04:30:54", "content": "It takes ~24hrs on 2x T4 GPUs for 150 epochs, though it is much faster on more modern GPUs.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3201538, "author_name": "chenxin1991", "author_url": "", "post_date": "05/14/2025 03:00:12", "content": "Thanks for sharing! The improvement is mainly due to the changes in pretrained model, right? And may I ask about your training data for pretrained model, with all OpenFWI dataset?
", "votes": null, "replies": [ { "id": 3201548, "author_name": "brendanartley", "author_url": "", "post_date": "05/14/2025 03:14:10", "content": "Hi @chenxin1991, thanks for the comment.
\nThe pre-trained model is actually the same from V1 -> V4, there is just more customizability now. The models are trained on the same train/valid set as the public notebook (~98% of all OpenFWI), and the CV/LB improvement came purely from increasing the training epochs!
", "votes": null, "replies": [] } ] }, { "id": 3201686, "author_name": "wayne127", "author_url": "", "post_date": "05/14/2025 08:54:23", "content": "May I ask what's the difference among checkpoint best0~2 during training? Since they can all be loaded by the same model structure, it suggests the architecture is identical. Is the difference due to 3-fold cross-validation?
", "votes": null, "replies": [] } ] }, { "id": 3201550, "author_name": "cqrcqr", "author_url": "", "post_date": "05/14/2025 03:31:06", "content": "Hello, what are the differences between these three models?
", "votes": null, "replies": [ { "id": 3201822, "author_name": "brendanartley", "author_url": "", "post_date": "05/14/2025 12:46:23", "content": "@cqrcqr, @wayne127 thanks for your comments.
\nThe only difference between the three models is their seed. They were trained and evaluated on the same fold as in the public notebook.
", "votes": null, "replies": [] } ] }, { "id": 3201600, "author_name": "bibanh", "author_url": "", "post_date": "05/14/2025 05:57:39", "content": "Your dataset is excellent—it eliminates the bottleneck caused by disk loading.
", "votes": null, "replies": [] }, { "id": 3202350, "author_name": "peilwang", "author_url": "", "post_date": "05/15/2025 09:01:25", "content": "Thank you for sharing. I noticed that you used data augmentation. May I ask why the width of the flip speed graph corresponds to the flipping of num_Sources and num_deceivers? I don't quite understand the relationship between them. I hope you can help me🥹
", "votes": null, "replies": [ { "id": 3202468, "author_name": "zui0711", "author_url": "", "post_date": "05/15/2025 12:24:11", "content": "https://www.kaggle.com/code/brendanartley/hgnet-v2-starter/comments#3200039 🌚
", "votes": null, "replies": [ { "id": 3202696, "author_name": "peilwang", "author_url": "", "post_date": "05/15/2025 19:12:56", "content": "thanks 嘴爷😍
", "votes": null, "replies": [] } ] } ] }, { "id": 3202524, "author_name": "mgoksu", "author_url": "", "post_date": "05/15/2025 14:14:52", "content": "Bartley, thank you for sharing your work. I'm learning a lot from it.
\nYour network seems to be composed of many different blocks. Is there a methodology to the way you pick the building blocks? Or is it experience and/or intuition?
", "votes": null, "replies": [ { "id": 3202551, "author_name": "brendanartley", "author_url": "", "post_date": "05/15/2025 14:44:12", "content": "Hi @mgoksu, most of the blocks are there for customizability.
\nFor me, choosing which blocks to use is a mix of intuition and cross validation (mostly the latter). It is common for my intuition to be wrong, but a robust cross validation scheme rarely lies.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3198769": "Excited to share a notebook that builds on the great starter notebook from @egortrushin. The updated notebook showcases training on 2x T4 GPUs to maximize our GPU quota!\n\nNotebook [here](https://www.kaggle.com/code/brendanartley/hgnet-v2-starter-cv-65-2-lb-69-7)\nDataset [here](https://www.kaggle.com/datasets/brendanartley/openfwi-preprocessed-72x72)\n\nI also provide a few trained checkpoints for a HGNet-V2-Unet model, flip augmentations, EMA, and more. Here are the CV scores by dataset type. Experiment away! 😀\n\n```python\n+--------------+--------+\n| Dataset | Score |\n+--------------+--------+\n| CurveFault_A | 15.47 |\n| CurveFault_B | 132.66 |\n| CurveVel_A | 37.10 |\n| CurveVel_B | 96.18 |\n| FlatFault_A | 14.07 |\n| FlatFault_B | 71.20 |\n| FlatVel_A | 19.20 |\n| FlatVel_B | 48.95 |\n| Style_A | 51.15 |\n| Style_B | 70.72 |\n+--------------+--------+\n\n```", "3199046": "brendanartley \nThank you for sharing your notebook.\nIt seems that the CV scores differ significantly between the different datasets, do you have any new insights or advice on the differences?For example, should we focus on a particular dataset to address it, etc.\nBest regards.", "3199141": "Hi @ov104ss,\n\nA 2-stage approach could work well. For example, you could train a model to predict which class each data point belongs to. Then, you could train a family specific model for the velocity-map reconstruction step. It may be that some model architectures work well for some families but not for others. Definitely worth trying!\n\nMaybe others can chime in if they have already compared 1-stage vs 2-stage approaches?", "3199157": "brendanartley \nThank you for your reply. That idea is worth a try. I was thinking that it would be difficult to cover all families with a single model, so I will consider it. Thank you very much.", "3199190": "DDP does appear that its the way of the future for running multiple GPU's, but I am finding it pretty difficult to debug errors when making changes to the code. Any suggestions (besides lots of print code) to ease this issue using Visual Code on local machine?\n\nAnother thing I knew but forgot and relearned the hard way, its going to write over any existing config, utils.py, etc when you run it local. If your like me and playing with lots of the shared notebooks in the competition your going to be unhappy if you don't get into a habit of separate folders per shared notebook on your local machine.", "3199327": "I wonder if a MoE type of approach would work for it ?, Cause the task is to recreate the velocity maps in all the types, but the scores vary, as far as I'm aware this indicates a change in distribution, a router based mixture of experts type of thing would work pretty well in my opinion.", "3199388": "Hi @pcjimmmy, thanks for the comment.\n\nOne thing you could do is extract pieces of this notebook and insert them into an existing pipeline that works for you (models, augs, snippets, etc). This might help pinpoint issues in your environment. Hope this helps!", "3201496": "**Update 13/5/2025**\n\nCV: 65.2 -> 55.6\nLB: 69.7 -> 60.9\n\nThanks everyone for waiting, the training script now works on Kaggle. I also released 3x stronger pretrained checkpoints (trained for 150 epochs), and added more customizable decoder. Cheers!", "3201517": "May I ask what's your training set MAE after 50/150 epochs? In my case train/CV/LB : 25/36/36 after 40 epochs, I am considering whether it has fully converged.", "3201530": "Sure, here are the loss curves for a 150 epoch training run.\n\n", "3201534": "Crazy epochs! Out of curiosity, how long did the 250 epochs take?", "3201538": "Thanks for sharing! The improvement is mainly due to the changes in pretrained model, right? And may I ask about your training data for pretrained model, with all OpenFWI dataset?", "3201548": "Hi @chenxin1991, thanks for the comment.\n\nThe pre-trained model is actually the same from V1 -> V4, there is just more customizability now. The models are trained on the same train/valid set as the public notebook (~98% of all OpenFWI), and the CV/LB improvement came purely from increasing the training epochs!", "3201550": "Hello, what are the differences between these three models?", "3201573": "It takes ~24hrs on 2x T4 GPUs for 150 epochs, though it is much faster on more modern GPUs.", "3201600": "Your dataset is excellent—it eliminates the bottleneck caused by disk loading.", "3201686": "May I ask what's the difference among checkpoint best0~2 during training? Since they can all be loaded by the same model structure, it suggests the architecture is identical. Is the difference due to 3-fold cross-validation?", "3201822": "cqrcqr, @wayne127 thanks for your comments.\n\nThe only difference between the three models is their seed. They were trained and evaluated on the same fold as in the public notebook.", "3202350": "Thank you for sharing. I noticed that you used data augmentation. May I ask why the width of the flip speed graph corresponds to the flipping of num_Sources and num_deceivers? I don't quite understand the relationship between them. I hope you can help me🥹", "3202468": "https://www.kaggle.com/code/brendanartley/hgnet-v2-starter/comments#3200039 🌚", "3202524": "Bartley, thank you for sharing your work. I'm learning a lot from it. \n\nYour network seems to be composed of many different blocks. Is there a methodology to the way you pick the building blocks? Or is it experience and/or intuition?", "3202551": "Hi @mgoksu, most of the blocks are there for customizability. \n\nFor me, choosing which blocks to use is a mix of intuition and cross validation (mostly the latter). It is common for my intuition to be wrong, but a robust cross validation scheme rarely lies.", "3202696": "thanks 嘴爷😍" }, "source": "meta" }