![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F745525%2Fe4a93277d74573f40280e27ecbfcabf5%2Fimage.png?generation=1751342036295658&alt=media\")
I discovered this competition after the BYU competition had ended. I found it extremely interesting and regretted having spent so much time on the BYU competition instead.
\nWe would like to express our gratitude to the competition host and the Kaggle staff for organizing this outstanding competition. Below, we introduce the yu4u's part of the solution of Team monnu and yu4u. Please refer to awesome monnu’s solution here.
\nThe input tensor with shape 5 × 1000 × 70 is resized to 224 × 224 and fed into the backbone; a forward pass produces a 7 × 7 output, which is then expanded to 70 × 70 using pixel shuffle. With this approach, no dedicated decoder is required.
\nDuring the forward pass in the backbone, the five channels are processed individually, and at the outputs of stages 1 and 2, the channels are aggregated from 5 to 3, and 3 to 1 using average pooling respectively.
\nThe implementation of this 2.5D model is based on the version I employed in the CZII competition.
The initial model training was conducted on the entire OpenFWI dataset with the following settings: AdamW optimizer, 64 epochs, batch size = 8, learning rate = 2e-4, weight decay = 5e-2, and an exponential moving average (EMA) decay of 0.999. Data augmentation was limited to horizontal flipping only.
\nAfter the initial training phase, the inference outputs for the test and validation data were converted back to the input format (seismic data) via seismic forward modeling, and those pairs were used to finetune the pretrained model.
\nDuring finetuning, the learning rate was reduced to 1e-5 and learning-rate warm-up was also applied.
Using a back-propagation-capable forward-modeling implementation, each test sample was processed individually: the pretrained model’s predictions were transformed back into seismic data, and the model was iteratively optimized to minimize the reconstruction error.
\nAlthough this procedure does not directly optimize the velocity map, we empirically found that the mean absolute error (MAE) improved as the number of iterations increased.
A simplified implementation example is shown below:
\n = MyModel()\n x test_xs:\n .load_state_dict(torch.load()) \n optimizer = AdamW(...)\n .train()\n i range():\n optimizer.zero_grad()\n pred_y = (x)\n reconst_x = FWM(pred_y)\n loss = criterion(ori_x, reconst_x)\n loss.backward()\n optimizer.()\n\n| Stage | \nCV | \nPrivate LB | \n
|---|---|---|
| Initial Training | \n19.50 | \n21.6 | \n
| Finetuning | \n16.96 | \n17.5 | \n
| Iterative Optimization | \nN/A | \n13.7 | \n
| Finetuning | \n13.36 | \n12.8 | \n
| Finetuning | \n13.16 | \n12.5 | \n
I discovered this competition after the BYU competition had ended. I found it extremely interesting and regretted having spent so much time on the BYU competition instead.
\nWe would like to express our gratitude to the competition host and the Kaggle staff for organizing this outstanding competition. Below, we introduce the yu4u's part of the solution of Team monnu and yu4u. Please refer to awesome monnu’s solution here.
\nThe input tensor with shape 5 × 1000 × 70 is resized to 224 × 224 and fed into the backbone; a forward pass produces a 7 × 7 output, which is then expanded to 70 × 70 using pixel shuffle. With this approach, no dedicated decoder is required.
\nDuring the forward pass in the backbone, the five channels are processed individually, and at the outputs of stages 1 and 2, the channels are aggregated from 5 to 3, and 3 to 1 using average pooling respectively.
\nThe implementation of this 2.5D model is based on the version I employed in the CZII competition.
The initial model training was conducted on the entire OpenFWI dataset with the following settings: AdamW optimizer, 64 epochs, batch size = 8, learning rate = 2e-4, weight decay = 5e-2, and an exponential moving average (EMA) decay of 0.999. Data augmentation was limited to horizontal flipping only.
\nAfter the initial training phase, the inference outputs for the test and validation data were converted back to the input format (seismic data) via seismic forward modeling, and those pairs were used to finetune the pretrained model.
\nDuring finetuning, the learning rate was reduced to 1e-5 and learning-rate warm-up was also applied.
Using a back-propagation-capable forward-modeling implementation, each test sample was processed individually: the pretrained model’s predictions were transformed back into seismic data, and the model was iteratively optimized to minimize the reconstruction error.
\nAlthough this procedure does not directly optimize the velocity map, we empirically found that the mean absolute error (MAE) improved as the number of iterations increased.
A simplified implementation example is shown below:
\n = MyModel()\n x test_xs:\n .load_state_dict(torch.load()) \n optimizer = AdamW(...)\n .train()\n i range():\n optimizer.zero_grad()\n pred_y = (x)\n reconst_x = FWM(pred_y)\n loss = criterion(ori_x, reconst_x)\n loss.backward()\n optimizer.()\n\n| Stage | \nCV | \nPrivate LB | \n
|---|---|---|
| Initial Training | \n19.50 | \n21.6 | \n
| Finetuning | \n16.96 | \n17.5 | \n
| Iterative Optimization | \nN/A | \n13.7 | \n
| Finetuning | \n13.36 | \n12.8 | \n
| Finetuning | \n13.16 | \n12.5 | \n
Thanks for the writeup.
\n\"we empirically found that the mean absolute error (MAE) improved as the number of iterations increased.\"
in my experiment, MAE will go down then it will go up if the iterations is run long enough.
\n\"loss = criterion(ori_x, reconst_x)\" always decreases.
\ndo you observe similarly?
In my experiments, the optimization converged within around 100 steps, but I did not observe any degradation in performance even when the number of iterations was increased beyond that. I confirmed that the loss consistently decreased. However, slight oscillations were observed during the optimization process.
\nMy teammate @fuumin621 conducted an excellent experiment using the sampled validation data.
\nThis figure shows the gain in MAE as a function of the number of iterations performed and the percentage of data optimized, assuming optimization is applied in descending order of reconstruction error.
Thanks for the writeup.
\n\"we empirically found that the mean absolute error (MAE) improved as the number of iterations increased.\"
in my experiment, MAE will go down then it will go up if the iterations is run long enough.
\n\"loss = criterion(ori_x, reconst_x)\" always decreases.
\ndo you observe similarly?
In my experiments, the optimization converged within around 100 steps, but I did not observe any degradation in performance even when the number of iterations was increased beyond that. I confirmed that the loss consistently decreased. However, slight oscillations were observed during the optimization process.
\nMy teammate @fuumin621 conducted an excellent experiment using the sampled validation data.
\nThis figure shows the gain in MAE as a function of the number of iterations performed and the percentage of data optimized, assuming optimization is applied in descending order of reconstruction error.