![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F8a05eebaa2abfdbbc4903c0972ff0ba7%2FSelection_252.png?generation=1750036637016034&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F4ec93d83b8121bf5e188b3b901555fc2%2FSelection_261.png?generation=1750068630074827&alt=media\")
\ncode: https://www.kaggle.com/datasets/hengck23/kaggle-competition-waveform-inversion-discussion-0
\n
\ncode: https://www.kaggle.com/datasets/hengck23/kaggle-competition-waveform-inversion-discussion-0
gpu = 2x RTX 6000 (close to 4090)
\nvalid samples = 20 np arrays = 2 npy from each family (1,10 npy, following Bartley's fold0)
\ntrain samples = rest of arrays
\n(no data subsampling used)
at most 3 days for 150 epoch(?)
\nbasically Bartley'smodel with my modifications
\ntrain hyper-parameters:
\nfor other configurations, please refer to code and log in the public dataset link
\n\now
", "messages": [ { "id": 3225091, "postDate": "2025-06-16T01:30:25.183Z", "content": "
\n
\ncode: https://www.kaggle.com/datasets/hengck23/kaggle-competition-waveform-inversion-discussion-0
gpu = 2x RTX 6000 (close to 4090)
\nvalid samples = 20 np arrays = 2 npy from each family (1,10 npy, following Bartley's fold0)
\ntrain samples = rest of arrays
\n(no data subsampling used)
at most 3 days for 150 epoch(?)
\nbasically Bartley'smodel with my modifications
\ntrain hyper-parameters:
\nfor other configurations, please refer to code and log in the public dataset link
\n\now
", "rawMarkdown": "\n\ncode: https://www.kaggle.com/datasets/hengck23/kaggle-competition-waveform-inversion-discussion-0\n\n---\n\ngpu = 2x RTX 6000 (close to 4090)\nvalid samples = 20 np arrays = 2 npy from each family (1,10 npy, following Bartley's fold0) \ntrain samples = rest of arrays\n(no data subsampling used)\n\n##training speed:\n- convnext base at 30min per epoch\n- convnext small at 24min\n- covnnext large at 40min\n\nat most 3 days for 150 epoch(?)\n\n##implementation:\n- basically Bartley'smodel with my modifications\n - my own stem\n - batch norm in unet\n - instance-norm2d #1 in convnext block but remove contiguous() in forward\n\n- train hyper-parameters:\n - I wrote my own trainer, the train MAE shown is moving average over the last 100 batches. validation MAE exclude flip TTA\n - batch size = 64 for each GPU (faster if I use batch size =128, which I intend to use at later epoch of fine tunning with smaller lr). i did not use sync batch norm for multi-gpu training, so batch size has to be large\n - bfloat16, hence autocast scaler is not required\n - Bartley's EMA #1\n - I cannot use optimizer fuse and torch compile. i have errors below.\n - I cannot use FP8 training (torchAO and FSDP2) for now\n\n\nfor other configurations, please refer to code and log in the public dataset link\n\n```\n#1 : I find this to have great effect in accuracy\n```ow", "votes": 16 }, { "id": 3227610, "postDate": "2025-06-19T05:54:55.447Z", "content": "how to do experiment fast. Bartley's caformer use more complicated decoder with attention and intermediate layers. i want to see if decoder is important in this competition. hence:
\nI can further unfreeze encoder and train and train all after initial warmup.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F044a2cbc3848434184e879595c187149%2FSelection_093.png?generation=1750312490647785&alt=media\")
other worth trying:
\ndensenetersion of covnext:
\nhttps://huggingface.co/naver-ai/rdnet_base.nv_in1k
if strong encoder doesn't help, try strong decoder (e.g. attention layer, skip layer as in Bartley's caformer; unet++ or multi connect unet)
", "rawMarkdown": "other worth trying:\n- densenetersion of covnext:\nhttps://huggingface.co/naver-ai/rdnet_base.nv_in1k\n\n- https://github.com/AILab-CVC/UniRepLKNet\n- https://github.com/LMMMEng/OverLoCK\n\nif strong encoder doesn't help, try strong decoder (e.g. attention layer, skip layer as in Bartley's caformer; unet++ or multi connect unet)", "votes": 1 }, { "id": 3225630, "postDate": "2025-06-16T17:01:53.410Z", "content": "a stronger and fast backbone (? or/and !)
\nhttps://huggingface.co/timm/inception_next_base.sail_in1k
This is another metaformer like caformer. Both are from the same lab SAIL.
\nNOTE: inception-next is batchnorm based, so I used \"net = torch.nn.SyncBatchNorm.convert_sync_batchnorm(net) \"
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fc7232f69b59601495251b2062df593d4%2FSelection_264.png?generation=1750104201525874&alt=media\")
@hengck23, when do you decided to stop training and try next experiment?
\nAs i tested multiple experiments on CurveFault_B ( with your setup - 1e-3 and with 1e-4 ) --> 1e-4 got better improvement --> not tested on all dataset. as we train more - gap between training loss improving 3x and validation improving with 0.75x.
\nwill you test to scale the velocity / 100 and see how your training converging ? Nice to see you in this compatition @hengck23
", "rawMarkdown": "@hengck23, when do you decided to stop training and try next experiment?\n\nAs i tested multiple experiments on CurveFault_B ( with your setup - 1e-3 and with 1e-4 ) --> 1e-4 got better improvement --> not tested on all dataset. as we train more - gap between training loss improving 3x and validation improving with 0.75x.\n\nwill you test to scale the velocity / 100 and see how your training converging ? Nice to see you in this compatition @hengck23 ", "votes": 1, "replies": [ { "id": 3225671, "postDate": "2025-06-16T17:44:16.307Z", "content": "i first train a fast model:
\n1e-3 for 35 epoch, 1e-4 for next 15 epoch, 1e-5 for next 5 epoch.
that confirms:
\n1e-4 at at most improve mae by 3 to4 , 1e-5 can almost improve 0.5 to 1
also, from your validation, you can estimate if current mae loss = xxx,
\nto reduce 1 mae loss, you need e.g. 1 epoch
\nreduce next 1 mae, you need 2
\nreduce next 1 mae, you need 4
\nreduce next 1 mae, you need 8
\nreduce next 1 mae, you need 12 ….
so you can estimate the final performance at the middle of training if you only have 150 or 200 epoch
\nyou should confirm this with public models (eg, small convnext) where other already train for 50, 100 and 150 epoches.
\nlastly assume model A has validation =40 at epoch 30,
\nfor model B to achieve +5 better at 150, B must be better than A by eg +6 at epoch 30.
so you can try different backbone till epoch 30.
\n(30,5,6 are all exmpale numbers, you need to do experiments to find out)
", "rawMarkdown": "i first train a fast model:\n1e-3 for 35 epoch, 1e-4 for next 15 epoch, 1e-5 for next 5 epoch.\n\nthat confirms:\n1e-4 at at most improve mae by 3 to4 , 1e-5 can almost improve 0.5 to 1\n\nalso, from your validation, you can estimate if current mae loss = xxx,\nto reduce 1 mae loss, you need e.g. 1 epoch\nreduce next 1 mae, you need 2\nreduce next 1 mae, you need 4\nreduce next 1 mae, you need 8\nreduce next 1 mae, you need 12 ....\n\nso you can estimate the final performance at the middle of training if you only have 150 or 200 epoch\n\nyou should confirm this with public models (eg, small convnext) where other already train for 50, 100 and 150 epoches.\n\n\n---\n\nlastly assume model A has validation =40 at epoch 30,\nfor model B to achieve +5 better at 150, B must be better than A by eg +6 at epoch 30.\n\nso you can try different backbone till epoch 30.\n\n(30,5,6 are all exmpale numbers, you need to do experiments to find out)", "votes": 2 } ] } ] }, { "id": 3225615, "postDate": "2025-06-16T16:40:15.917Z", "content": "torch.compile(… mode='max) is very helpful.
", "rawMarkdown": "torch.compile(... mode='max) is very helpful.", "votes": 1 }, { "id": 3227013, "postDate": "2025-06-18T11:26:27.640Z", "content": "covnext-large with data sampling.
\nCV after training for 24hr:
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F09632d6ed0ddcf89643fc6e3a1b6db9a%2FSelection_089.png?generation=1750269732895607&alt=media\")
Pardon me , but why does epoch has values after decimal??
\nLike are you considering batches done also??
update!!!
\ni change machine.
\nOLD: 2xRTX A6000
\nNEW: 2xRTX 6000 ada, shift all data to SSD
previous: convnext base at 30min per epoch
\nnew: convnext base at 21 min per epoch !!!
i think a lot is due to ssd
", "rawMarkdown": "update!!!\n\ni change machine.\nOLD: 2xRTX A6000\nNEW: 2xRTX 6000 ada, shift all data to SSD\n\nprevious: convnext base at 30min per epoch\nnew: convnext base at 21 min per epoch !!!\n\ni think a lot is due to ssd", "votes": 2, "replies": [ { "id": 3226103, "postDate": "2025-06-17T08:51:39.017Z", "content": "did you add weighted random sampler during train ? that would probably be a big gain.
", "rawMarkdown": "did you add weighted random sampler during train ? that would probably be a big gain. ", "votes": 2, "replies": [ { "id": 3226110, "postDate": "2025-06-17T08:58:25.767Z", "content": "yes,i note that for different epoch the mae loss for simple family doesn't change. the imporvment comes for difficult family.
\nnow i am testing different backbone(upt o 30 epoch) and ifferent train config (eg fp8), so i use all train samples first.
thanks @darraghdog for the hint. effects of sampling difficult train data:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F5873887d0fa2dd34ba9f1becfa1c5d74%2FSelection_084.png?generation=1750186244010579&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F5eab51a72bb9196a19538ef380402df0%2FSelection_085.png?generation=1750186258173534&alt=media\")
class MyFWIDataset(Dataset):\n def __init__(\n self,\n df,\n cfg,\n mode='train', \n ):\n ...\n\n def __getitem__(self, ):\n\n .mode == :\n family = .family[np..choice(\n np.arange(len(.family)),\n p=.family_p\n )]\n df = ..get_group(family)\n i = np..choice(df.)\n j = np..choice()\n :\n i = // \n j = % \n\n seismic = .seismic[i][j]\n velocity = .velocity[i][j]\nbut there is a catch: you are using error of validation IN training, i.e. you are overfitting validation data.
\nbut is this a problem and do we need or need not solve it? This is homework for kagglers to think about.
hint: it is all about if public test is in-distribution or out-distribution
", "rawMarkdown": "thanks @darraghdog for the hint. effects of sampling difficult train data:\n\n\n\n\n\n\n```\nclass MyFWIDataset(Dataset):\n def __init__(\n self,\n df,\n cfg,\n mode='train', \n ):\n ...\n\n def __getitem__(self, index):\n\n if self.mode == 'train':\n family = self.family[np.random.choice(\n np.arange(len(self.family)),\n p=self.family_p\n )]\n df = self.group.get_group(family)\n i = np.random.choice(df.index)\n j = np.random.choice(LENGTH)\n else:\n i = index // LENGTH\n j = index % LENGTH\n\n seismic = self.seismic[i][j]\n velocity = self.velocity[i][j]\n\n```\n\nbut there is a catch: you are using error of validation IN training, i.e. you are overfitting validation data.\nbut is this a problem and do we need or need not solve it? This is homework for kagglers to think about.\n\nhint: it is all about if public test is in-distribution or out-distribution" }, { "id": 3226567, "postDate": "2025-06-17T20:22:18.710Z", "content": "@hengck23 is the family_p approach as similar to torch.utils.data.WeightedRandomSampler?
", "rawMarkdown": "@hengck23 is the family_p approach as similar to torch.utils.data.WeightedRandomSampler?" }, { "id": 3226570, "postDate": "2025-06-17T20:26:49.227Z", "content": "probabilty the same, but i need to use distributed parallel sampler for DDP multi gpu training
", "rawMarkdown": "probabilty the same, but i need to use distributed parallel sampler for DDP multi gpu training", "votes": 1 }, { "id": 3226717, "postDate": "2025-06-18T03:30:09.400Z", "content": "in my experiment, it overfitting the training dataset well but not well for validation @hengck23 ->
\n| epoch | \nCV | \nTrain | \nGap | \n
|---|---|---|---|
| \n | \n | \n | \n |
| 1 | \n84.8 | \n37.7 | \n47.1 | \n
| 2 | \n73.6 | \n29.1 | \n44.5 | \n
| 3 | \n67.7 | \n25.6 | \n42.1 | \n
| 4 | \n63.7 | \n23.8 | \n38.1 | \n
| 5 | \n57.8 | \n19.8 | \n38 | \n
| 6 | \n55.7 | \n18.4 | \n37.3 | \n
| 7 | \n55.6 | \n18.8 | \n36.8 | \n
| 8 | \n55.0 | \n18.6 | \n36.4 | \n
| 9 | \n51.8 | \n15.9 | \n35. 9 | \n
something is very wrong, without code or train parameters/train log, i cannot diagnose the error.
\nplease post them if you can.
i can only say that:
\n1) assume that you are using my fold.csv
\n2) then, training at constant 1e-3, validation loss is always better than train loss(moving average) until they converge.
they converge at:
\nmae loss = 35 for covnext small,
\n33 for base
\n30 for large
(this is also the approximate LB score)
\ni think there is a bug in your code. it is easy to check
\ndid you subsample more difficult samples or mistaken sample the easier ones?
\n@hengck23 - its because since you are sample more difficult sample, train loss is higher than the unsampled version (btw: high loss means greater gradient backprop, so faster converge)
", "rawMarkdown": "@hengck23 - its because **since you are sample more difficult sample, train loss is higher than the unsampled version (btw: high loss means greater gradient backprop, so faster converge)**" }, { "id": 3227061, "postDate": "2025-06-18T12:39:24.733Z", "content": "it is either you oversample (too many difficult and too few simple) or wrong/under sampling (too many simple and too few difficult).
\nassuming there is no other bug …
", "rawMarkdown": "it is either you oversample (too many difficult and too few simple) or wrong/under sampling (too many simple and too few difficult).\n \nassuming there is no other bug ...", "votes": 1 }, { "id": 3227430, "postDate": "2025-06-19T02:35:03.770Z", "content": "Thanks for your implementation WeightedSampler for DDP training. Besides, could you please share your experiences of setting self.family_p? Are they all equal to 0.1 ?
my solution may not be the best:
\n ():\n ():\n\n sampling_alpha=\n .cfg = cfg\n .mode = mode\n .df = df\n .seismic, .velocity = load_data(df)\n .length = ((.df)*LENGTH*subsample_ratio)\n .subsample_ratio=subsample_ratio\n\n \n .group = df.groupby()\n .mapping={\n : ,\n : ,\n : ,\n : ,\n : ,\n : ,\n : ,\n : ,\n : ,\n : ,\n }\n \n\n .family = (.mapping.keys())\n .family_p = np.array((.mapping.values()))\n .family_p = (.family_p/.family_p.()) ** sampling_alpha \n .family_p = .family_p/.family_p.()\nDoes your college give these resources, or do you source them yourself?
", "rawMarkdown": "Does your college give these resources, or do you source them yourself?", "votes": 1 } ] }, { "id": 3225094, "postDate": "2025-06-16T01:33:48.533Z", "content": "i wonder did anyone have similiar error:
\ntorch compile:
\n \nfuse = True in Adam optimizer
\n[]: \n[]: \n[]: \nSame here. It seems that torch.compile can not be applied with fuse optimizer in distributing training
", "rawMarkdown": "Same here. It seems that torch.compile can not be applied with fuse optimizer in distributing training", "votes": 1, "replies": [ { "id": 3225123, "postDate": "2025-06-16T02:58:03.673Z", "content": "No. It can be done for if i use encoder only.
", "rawMarkdown": "No. It can be done for if i use encoder only.", "votes": 1 }, { "id": 3228015, "postDate": "2025-06-19T15:11:12.990Z", "content": "UPDATE:
\n The reason maybe I delete the \".contiguous()\" in the convnext_block forward function. In my experiment, when I add \".contiguous()\" into forward function, model can work with fuse optimizer.
thanks, it tried that. but fused optimizer doesn't seems to improve speed for my case.
\ncontiguous() seems to be slow too. maybe it is only for my case.
i hope chatgpt is not giving me hallucinations …
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Ff70e64cd4ee9ba207b21c0ddfe712f81%2FSelection_999(8255).png?generation=1750480646134888&alt=media\")
speedup trick
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F8f97fcddb20b3029863fb3e0605b3493%2FSelection_999(8256).png?generation=1750480660497901&alt=media\")
for those using my code please change upsampling in decoder layer from nearest neighbour to deconv (I,e, transpose conv)
\nI make a mistake. we are not doing segmentation or pixel based context.for each pixel location in the target velocity map, the context can be anywhere in the semsic data (since wave can be reflected etc). more importantly the axis in semsic data is time.
\nso we need feature to be propagated in scaling up. (i.e. we need the scaling-up weights that is dependent on feature value)
\ninterploation uses constant weight, which is not suitable for our task
on a side note large conv/deconv kernel in decoder may work better. but I haven't try yet. e.g. convnext decoder
\nsegmentation is image to image translation problem.
\nhere are are doing wave (time aixs) to velocity. it is not \"same coordinate\" translation problem. rather it is \"converting from one modality to another, sharing one common axis(x) and yet differ in (z and time)\".
\nit is more like encoder to semsic latent, then semsic latent to velocity latent, then decoder to upscale .
", "rawMarkdown": "##update!!!\n\nfor those using my code please change upsampling in decoder layer from nearest neighbour to deconv (I,e, transpose conv)\n\nI make a mistake. we are not doing segmentation or pixel based context.for each pixel location in the target velocity map, the context can be anywhere in the semsic data (since wave can be reflected etc). more importantly the axis in semsic data is time.\n\nso we need feature to be propagated in scaling up. (i.e. we need the scaling-up weights that is dependent on feature value)\ninterploation uses constant weight, which is not suitable for our task\n\n---\n\non a side note large conv/deconv kernel in decoder may work better. but I haven't try yet. e.g. convnext decoder\n\n---\n\nsegmentation is image to image translation problem.\n\nhere are are doing wave (time aixs) to velocity. it is not \"same coordinate\" translation problem. rather it is \"converting from one modality to another, sharing one common axis(x) and yet differ in (z and time)\".\n\nit is more like encoder to semsic latent, then semsic latent to velocity latent, then decoder to upscale .", "replies": [ { "id": 3229317, "postDate": "2025-06-21T09:58:50.523Z", "content": "deconv is worse than nearest neighbour in my experiment. So weird.
", "rawMarkdown": "deconv is worse than nearest neighbour in my experiment. So weird." } ] }, { "id": 3228826, "postDate": "2025-06-20T14:59:24.087Z", "content": "i think my idea work (?) @shlomoron here are some initial experiments
\nhere are the results
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fc873cd4fb1906a15b3bbcae57c41b204%2FSelection_284.png?generation=1750431510814355&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F3fc9a283130634a1258827c80c6a161e%2FSelection_287.png?generation=1750434856419984&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F1167c056e31da787c2833aa1b63b8b52%2FSelection_289.png?generation=1750446143887763&alt=media\")
training in progress … update later …
", "rawMarkdown": "i think my idea work (?) @shlomoron here are some initial experiments\n\n1. given a model m\n2. do validation on set v (mae 30). obtain p = model(v)\n3. do forward wave modeling r = fwi(p)\n4. create a set (p,r) = (p, fwi(p. this set has mae loss = 18.\n4. create new train set = old train set + (p,r)\n\nhere are the results\n\n---\n\n###fine-tune: model initialisation=m, traindset = old train set\n\n\n---\n###fine-tune: model initialisation=m, traindset = old train set + (p,r)\n\n\n\n\n---\n###fine-tune: model initialisation=m, traindset = old train set + (p,r) + second round (p,r)\n\n\n\ntraining in progress ... update later ...", "replies": [ { "id": 3228834, "postDate": "2025-06-20T15:07:18.347Z", "content": "related paper (?)
\nchatgpt is really helpful to find paper
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Ffe0a723baac66d3be6682d240fdf42fb%2FSelection_286.png?generation=1750432036254103&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F7b81e6295e829140d29f062ad89ce39d%2FSelection_288.png?generation=1750435433963643&alt=media\")
error of different models are show above
\nleft : m
\nright :fine-tune: model initialisation=m, traindset = old train set + (p,r)
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fc9235a28c9a1912684b7bd6c40ceff97%2FSelection_094.png?generation=1750415260721070&alt=media\")
with the torch version of forward code, I wonder if direct route 1 is possible?
", "rawMarkdown": "\n\nwith the torch version of forward code, I wonder if direct route 1 is possible?", "replies": [ { "id": 3228595, "postDate": "2025-06-20T10:40:15.550Z", "content": "navie method (just refinement)
\n\n():\n .velocity = nn.parameter( from unet ....)\n\n():\n just run forward wave modeling code ....\n /Didn't you continue to try using the forward modeling loss function?
", "rawMarkdown": "Didn't you continue to try using the forward modeling loss function?", "replies": [ { "id": 3227181, "postDate": "2025-06-18T16:20:54.037Z", "content": "No. I am trying few things now, including physics solution. U cannot win by just unet. This thread is more for efficient implementation
", "rawMarkdown": "No. I am trying few things now, including physics solution. U cannot win by just unet. This thread is more for efficient implementation ", "votes": 1, "replies": [ { "id": 3227186, "postDate": "2025-06-18T16:23:38.973Z", "content": "Yes, I think so too now.
", "rawMarkdown": "Yes, I think so too now." }, { "id": 3227456, "postDate": "2025-06-19T02:58:17.093Z", "content": "I would love to see you post about the physics loss , i had trouble doing forward modeling….
", "rawMarkdown": "I would love to see you post about the physics loss , i had trouble doing forward modeling...." } ] } ] }, { "id": 3227458, "postDate": "2025-06-19T03:00:14.493Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 3227610, "author_name": "hengck23", "author_url": "", "post_date": "2025-06-19T05:54:55.447000", "content": "how to do experiment fast. Bartley's caformer use more complicated decoder with attention and intermediate layers. i want to see if decoder is important in this competition. hence:
\nI can further unfreeze encoder and train and train all after initial warmup.
\n
other worth trying:
\ndensenetersion of covnext:
\nhttps://huggingface.co/naver-ai/rdnet_base.nv_in1k
if strong encoder doesn't help, try strong decoder (e.g. attention layer, skip layer as in Bartley's caformer; unet++ or multi connect unet)
", "votes": 1, "replies": [] }, { "id": 3225630, "author_name": "hengck23", "author_url": "", "post_date": "2025-06-16T17:01:53.410000", "content": "a stronger and fast backbone (? or/and !)
\nhttps://huggingface.co/timm/inception_next_base.sail_in1k
This is another metaformer like caformer. Both are from the same lab SAIL.
\nNOTE: inception-next is batchnorm based, so I used \"net = torch.nn.SyncBatchNorm.convert_sync_batchnorm(net) \"
@hengck23, when do you decided to stop training and try next experiment?
\nAs i tested multiple experiments on CurveFault_B ( with your setup - 1e-3 and with 1e-4 ) --> 1e-4 got better improvement --> not tested on all dataset. as we train more - gap between training loss improving 3x and validation improving with 0.75x.
\nwill you test to scale the velocity / 100 and see how your training converging ? Nice to see you in this compatition @hengck23
", "votes": 1, "replies": [ { "id": 3225671, "author_name": "hengck23", "author_url": "", "post_date": "2025-06-16T17:44:16.307000", "content": "i first train a fast model:
\n1e-3 for 35 epoch, 1e-4 for next 15 epoch, 1e-5 for next 5 epoch.
that confirms:
\n1e-4 at at most improve mae by 3 to4 , 1e-5 can almost improve 0.5 to 1
also, from your validation, you can estimate if current mae loss = xxx,
\nto reduce 1 mae loss, you need e.g. 1 epoch
\nreduce next 1 mae, you need 2
\nreduce next 1 mae, you need 4
\nreduce next 1 mae, you need 8
\nreduce next 1 mae, you need 12 ….
so you can estimate the final performance at the middle of training if you only have 150 or 200 epoch
\nyou should confirm this with public models (eg, small convnext) where other already train for 50, 100 and 150 epoches.
\nlastly assume model A has validation =40 at epoch 30,
\nfor model B to achieve +5 better at 150, B must be better than A by eg +6 at epoch 30.
so you can try different backbone till epoch 30.
\n(30,5,6 are all exmpale numbers, you need to do experiments to find out)
", "votes": 2, "replies": [] } ] } ] }, { "id": 3225615, "author_name": "Wu Qiuyi", "author_url": "", "post_date": "2025-06-16T16:40:15.917000", "content": "torch.compile(… mode='max) is very helpful.
", "votes": 1, "replies": [] }, { "id": 3227013, "author_name": "hengck23", "author_url": "", "post_date": "2025-06-18T11:26:27.640000", "content": "covnext-large with data sampling.
\nCV after training for 24hr:
Pardon me , but why does epoch has values after decimal??
\nLike are you considering batches done also??
update!!!
\ni change machine.
\nOLD: 2xRTX A6000
\nNEW: 2xRTX 6000 ada, shift all data to SSD
previous: convnext base at 30min per epoch
\nnew: convnext base at 21 min per epoch !!!
i think a lot is due to ssd
", "votes": 2, "replies": [ { "id": 3226103, "author_name": "Darragh", "author_url": "", "post_date": "2025-06-17T08:51:39.017000", "content": "did you add weighted random sampler during train ? that would probably be a big gain.
", "votes": 2, "replies": [ { "id": 3226110, "author_name": "hengck23", "author_url": "", "post_date": "2025-06-17T08:58:25.767000", "content": "yes,i note that for different epoch the mae loss for simple family doesn't change. the imporvment comes for difficult family.
\nnow i am testing different backbone(upt o 30 epoch) and ifferent train config (eg fp8), so i use all train samples first.
thanks @darraghdog for the hint. effects of sampling difficult train data:
\n
class MyFWIDataset(Dataset):\n def __init__(\n self,\n df,\n cfg,\n mode='train', \n ):\n ...\n\n def __getitem__(self, ):\n\n .mode == :\n family = .family[np..choice(\n np.arange(len(.family)),\n p=.family_p\n )]\n df = ..get_group(family)\n i = np..choice(df.)\n j = np..choice()\n :\n i = // \n j = % \n\n seismic = .seismic[i][j]\n velocity = .velocity[i][j]\nbut there is a catch: you are using error of validation IN training, i.e. you are overfitting validation data.
\nbut is this a problem and do we need or need not solve it? This is homework for kagglers to think about.
hint: it is all about if public test is in-distribution or out-distribution
", "votes": 0, "replies": [] }, { "id": 3226567, "author_name": "SeshuRaju 🧘♂️", "author_url": "", "post_date": "2025-06-17T20:22:18.710000", "content": "@hengck23 is the family_p approach as similar to torch.utils.data.WeightedRandomSampler?
", "votes": 0, "replies": [] }, { "id": 3226570, "author_name": "hengck23", "author_url": "", "post_date": "2025-06-17T20:26:49.227000", "content": "probabilty the same, but i need to use distributed parallel sampler for DDP multi gpu training
", "votes": 1, "replies": [] }, { "id": 3226717, "author_name": "SeshuRaju 🧘♂️", "author_url": "", "post_date": "2025-06-18T03:30:09.400000", "content": "in my experiment, it overfitting the training dataset well but not well for validation @hengck23 ->
\n| epoch | \nCV | \nTrain | \nGap | \n
|---|---|---|---|
| \n | \n | \n | \n |
| 1 | \n84.8 | \n37.7 | \n47.1 | \n
| 2 | \n73.6 | \n29.1 | \n44.5 | \n
| 3 | \n67.7 | \n25.6 | \n42.1 | \n
| 4 | \n63.7 | \n23.8 | \n38.1 | \n
| 5 | \n57.8 | \n19.8 | \n38 | \n
| 6 | \n55.7 | \n18.4 | \n37.3 | \n
| 7 | \n55.6 | \n18.8 | \n36.8 | \n
| 8 | \n55.0 | \n18.6 | \n36.4 | \n
| 9 | \n51.8 | \n15.9 | \n35. 9 | \n
something is very wrong, without code or train parameters/train log, i cannot diagnose the error.
\nplease post them if you can.
i can only say that:
\n1) assume that you are using my fold.csv
\n2) then, training at constant 1e-3, validation loss is always better than train loss(moving average) until they converge.
they converge at:
\nmae loss = 35 for covnext small,
\n33 for base
\n30 for large
(this is also the approximate LB score)
\ni think there is a bug in your code. it is easy to check
\ndid you subsample more difficult samples or mistaken sample the easier ones?
\n@hengck23 - its because since you are sample more difficult sample, train loss is higher than the unsampled version (btw: high loss means greater gradient backprop, so faster converge)
", "votes": 0, "replies": [] }, { "id": 3227061, "author_name": "hengck23", "author_url": "", "post_date": "2025-06-18T12:39:24.733000", "content": "it is either you oversample (too many difficult and too few simple) or wrong/under sampling (too many simple and too few difficult).
\nassuming there is no other bug …
", "votes": 1, "replies": [] }, { "id": 3227430, "author_name": "AnnieGo", "author_url": "", "post_date": "2025-06-19T02:35:03.770000", "content": "Thanks for your implementation WeightedSampler for DDP training. Besides, could you please share your experiences of setting self.family_p? Are they all equal to 0.1 ?
my solution may not be the best:
\n ():\n ():\n\n sampling_alpha=\n .cfg = cfg\n .mode = mode\n .df = df\n .seismic, .velocity = load_data(df)\n .length = ((.df)*LENGTH*subsample_ratio)\n .subsample_ratio=subsample_ratio\n\n \n .group = df.groupby()\n .mapping={\n : ,\n : ,\n : ,\n : ,\n : ,\n : ,\n : ,\n : ,\n : ,\n : ,\n }\n \n\n .family = (.mapping.keys())\n .family_p = np.array((.mapping.values()))\n .family_p = (.family_p/.family_p.()) ** sampling_alpha \n .family_p = .family_p/.family_p.()\nDoes your college give these resources, or do you source them yourself?
", "votes": 1, "replies": [] } ] }, { "id": 3225094, "author_name": "hengck23", "author_url": "", "post_date": "2025-06-16T01:33:48.533000", "content": "i wonder did anyone have similiar error:
\ntorch compile:
\n \nfuse = True in Adam optimizer
\n[]: \n[]: \n[]: \nSame here. It seems that torch.compile can not be applied with fuse optimizer in distributing training
", "votes": 1, "replies": [ { "id": 3225123, "author_name": "hengck23", "author_url": "", "post_date": "2025-06-16T02:58:03.673000", "content": "No. It can be done for if i use encoder only.
", "votes": 1, "replies": [] }, { "id": 3228015, "author_name": "I2nfinit3y", "author_url": "", "post_date": "2025-06-19T15:11:12.990000", "content": "UPDATE:
\n The reason maybe I delete the \".contiguous()\" in the convnext_block forward function. In my experiment, when I add \".contiguous()\" into forward function, model can work with fuse optimizer.
thanks, it tried that. but fused optimizer doesn't seems to improve speed for my case.
\ncontiguous() seems to be slow too. maybe it is only for my case.
i hope chatgpt is not giving me hallucinations …
\n
speedup trick
\n
for those using my code please change upsampling in decoder layer from nearest neighbour to deconv (I,e, transpose conv)
\nI make a mistake. we are not doing segmentation or pixel based context.for each pixel location in the target velocity map, the context can be anywhere in the semsic data (since wave can be reflected etc). more importantly the axis in semsic data is time.
\nso we need feature to be propagated in scaling up. (i.e. we need the scaling-up weights that is dependent on feature value)
\ninterploation uses constant weight, which is not suitable for our task
on a side note large conv/deconv kernel in decoder may work better. but I haven't try yet. e.g. convnext decoder
\nsegmentation is image to image translation problem.
\nhere are are doing wave (time aixs) to velocity. it is not \"same coordinate\" translation problem. rather it is \"converting from one modality to another, sharing one common axis(x) and yet differ in (z and time)\".
\nit is more like encoder to semsic latent, then semsic latent to velocity latent, then decoder to upscale .
", "votes": 0, "replies": [ { "id": 3229317, "author_name": "I2nfinit3y", "author_url": "", "post_date": "2025-06-21T09:58:50.523000", "content": "deconv is worse than nearest neighbour in my experiment. So weird.
", "votes": 0, "replies": [] } ] }, { "id": 3228826, "author_name": "hengck23", "author_url": "", "post_date": "2025-06-20T14:59:24.087000", "content": "i think my idea work (?) @shlomoron here are some initial experiments
\nhere are the results
\n
training in progress … update later …
", "votes": 0, "replies": [ { "id": 3228834, "author_name": "hengck23", "author_url": "", "post_date": "2025-06-20T15:07:18.347000", "content": "related paper (?)
\nchatgpt is really helpful to find paper
\n
error of different models are show above
\nleft : m
\nright :fine-tune: model initialisation=m, traindset = old train set + (p,r)
with the torch version of forward code, I wonder if direct route 1 is possible?
", "votes": 0, "replies": [ { "id": 3228595, "author_name": "hengck23", "author_url": "", "post_date": "2025-06-20T10:40:15.550000", "content": "navie method (just refinement)
\n\n():\n .velocity = nn.parameter( from unet ....)\n\n():\n just run forward wave modeling code ....\n /Didn't you continue to try using the forward modeling loss function?
", "votes": 0, "replies": [ { "id": 3227181, "author_name": "hengck23", "author_url": "", "post_date": "2025-06-18T16:20:54.037000", "content": "No. I am trying few things now, including physics solution. U cannot win by just unet. This thread is more for efficient implementation
", "votes": 1, "replies": [ { "id": 3227186, "author_name": "guo dashuai", "author_url": "", "post_date": "2025-06-18T16:23:38.973000", "content": "Yes, I think so too now.
", "votes": 0, "replies": [] }, { "id": 3227456, "author_name": "Suryansh Mishra ", "author_url": "", "post_date": "2025-06-19T02:58:17.093000", "content": "I would love to see you post about the physics loss , i had trouble doing forward modeling….
", "votes": 0, "replies": [] } ] } ] }, { "id": 3227458, "author_name": "", "author_url": "", "post_date": "2025-06-19T03:00:14.493000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "3225091": "\n\ncode: https://www.kaggle.com/datasets/hengck23/kaggle-competition-waveform-inversion-discussion-0\n\n---\n\ngpu = 2x RTX 6000 (close to 4090)\nvalid samples = 20 np arrays = 2 npy from each family (1,10 npy, following Bartley's fold0) \ntrain samples = rest of arrays\n(no data subsampling used)\n\n##training speed:\n- convnext base at 30min per epoch\n- convnext small at 24min\n- covnnext large at 40min\n\nat most 3 days for 150 epoch(?)\n\n##implementation:\n- basically Bartley'smodel with my modifications\n - my own stem\n - batch norm in unet\n - instance-norm2d #1 in convnext block but remove contiguous() in forward\n\n- train hyper-parameters:\n - I wrote my own trainer, the train MAE shown is moving average over the last 100 batches. validation MAE exclude flip TTA\n - batch size = 64 for each GPU (faster if I use batch size =128, which I intend to use at later epoch of fine tunning with smaller lr). i did not use sync batch norm for multi-gpu training, so batch size has to be large\n - bfloat16, hence autocast scaler is not required\n - Bartley's EMA #1\n - I cannot use optimizer fuse and torch compile. i have errors below.\n - I cannot use FP8 training (torchAO and FSDP2) for now\n\n\nfor other configurations, please refer to code and log in the public dataset link\n\n```\n#1 : I find this to have great effect in accuracy\n```ow", "3227610": "how to do experiment fast. Bartley's caformer use more complicated decoder with attention and intermediate layers. i want to see if decoder is important in this competition. hence:\n1. I change the decoder to Bartley's\n2. load weights of previously trained encoder and freeze them\n3. I train with new decoder only\nnow I can compare if Bartley's is better:\n\nI can further unfreeze encoder and train and train all after initial warmup.\n\n\n", "3225771": "other worth trying:\n- densenetersion of covnext:\nhttps://huggingface.co/naver-ai/rdnet_base.nv_in1k\n\n- https://github.com/AILab-CVC/UniRepLKNet\n- https://github.com/LMMMEng/OverLoCK\n\nif strong encoder doesn't help, try strong decoder (e.g. attention layer, skip layer as in Bartley's caformer; unet++ or multi connect unet)", "3225630": "a stronger and fast backbone (? or/and !)\nhttps://huggingface.co/timm/inception_next_base.sail_in1k\n\nThis is another metaformer like caformer. Both are from the same lab SAIL.\nNOTE: inception-next is batchnorm based, so I used \"net = torch.nn.SyncBatchNorm.convert_sync_batchnorm(net) \"\n\n", "3225615": "torch.compile(... mode='max) is very helpful.", "3227013": "covnext-large with data sampling.\nCV after training for 24hr:\n\n", "3226100": "update!!!\n\ni change machine.\nOLD: 2xRTX A6000\nNEW: 2xRTX 6000 ada, shift all data to SSD\n\nprevious: convnext base at 30min per epoch\nnew: convnext base at 21 min per epoch !!!\n\ni think a lot is due to ssd", "3225094": "i wonder did anyone have similiar error:\n\ntorch compile:\n```\n... torch/fx/experimental/symbolic_shapes.py:4449] [0/1] xindex is not in var_ranges, defaulting to unknown range.\n```\n\nfuse = True in Adam optimizer\n```\n[rank0]: File \"/home/hp/app/anaconda3.11-vision/lib/python3.11/site-packages/torch/optim/adam.py\", line 676, in _fused_adam\n[rank0]: torch._fused_adam_(\n[rank0]: RuntimeError: params, grads, exp_avgs, and exp_avg_sqs must have same dtype, device, and layout\n```\n\n", "3229148": "i hope chatgpt is not giving me hallucinations ...\n\n.png?generation=1750480646134888&alt=media)\n\nspeedup trick\n.png?generation=1750480660497901&alt=media)", "3228844": "##update!!!\n\nfor those using my code please change upsampling in decoder layer from nearest neighbour to deconv (I,e, transpose conv)\n\nI make a mistake. we are not doing segmentation or pixel based context.for each pixel location in the target velocity map, the context can be anywhere in the semsic data (since wave can be reflected etc). more importantly the axis in semsic data is time.\n\nso we need feature to be propagated in scaling up. (i.e. we need the scaling-up weights that is dependent on feature value)\ninterploation uses constant weight, which is not suitable for our task\n\n---\n\non a side note large conv/deconv kernel in decoder may work better. but I haven't try yet. e.g. convnext decoder\n\n---\n\nsegmentation is image to image translation problem.\n\nhere are are doing wave (time aixs) to velocity. it is not \"same coordinate\" translation problem. rather it is \"converting from one modality to another, sharing one common axis(x) and yet differ in (z and time)\".\n\nit is more like encoder to semsic latent, then semsic latent to velocity latent, then decoder to upscale .", "3228826": "i think my idea work (?) @shlomoron here are some initial experiments\n\n1. given a model m\n2. do validation on set v (mae 30). obtain p = model(v)\n3. do forward wave modeling r = fwi(p)\n4. create a set (p,r) = (p, fwi(p. this set has mae loss = 18.\n4. create new train set = old train set + (p,r)\n\nhere are the results\n\n---\n\n###fine-tune: model initialisation=m, traindset = old train set\n\n\n---\n###fine-tune: model initialisation=m, traindset = old train set + (p,r)\n\n\n\n\n---\n###fine-tune: model initialisation=m, traindset = old train set + (p,r) + second round (p,r)\n\n\n\ntraining in progress ... update later ...", "3228586": "\n\nwith the torch version of forward code, I wonder if direct route 1 is possible?", "3227180": "Didn't you continue to try using the forward modeling loss function?", "3227458": "" } }