![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Ffa6e55c3a97072379442a35f867a51ab%2FScreenshot%202025-07-01%20at%2011.20.48.png?generation=1751336509771951&alt=media\")
Congratulations to everyone who was able to make progress beyond Bartley’s strong model(s).
\nI could not make further progress myself, so I am sharing some findings from my experiments on backpropagating through forward models.
\nThis technique is mainly used in generative image models [1, 2]; for example, obtaining input image which closely relate to the prompt text with CLIP embedding space.
The experiment was as follows:
\nResults:
\nThis approach can indeed improve the metric. However, I found it impractical due to computational efficiency: it took 150 seconds to optimize a single velocity map over 400 epochs.
\nThis means optimizing 65,000 images would require 113 days on a single RTX 4090, which is clearly unrealistic.
Perhaps a more efficient forward model could produce results in a reasonable timeframe.
\nDetails of the experimental setup:
\nTips on Efficiency
\nDid not work:
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F21c7f0c6f65a52c834c22dacf83b809f%2FScreenshot%202025-07-01%20at%2011.20.58.png?generation=1751336521559770&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Fac3399e4495e6487fe91069d114c3afa%2FScreenshot%202025-07-01%20at%2011.23.02.png?generation=1751336595264759&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F584e44a9f69905efd39ac58eea70bb80%2FScreenshot%202025-07-01%20at%2011.21.08.png?generation=1751336533273340&alt=media\")
Congratulations to everyone who was able to make progress beyond Bartley’s strong model(s).
\nI could not make further progress myself, so I am sharing some findings from my experiments on backpropagating through forward models.
\nThis technique is mainly used in generative image models [1, 2]; for example, obtaining input image which closely relate to the prompt text with CLIP embedding space.
The experiment was as follows:
\nResults:
\nThis approach can indeed improve the metric. However, I found it impractical due to computational efficiency: it took 150 seconds to optimize a single velocity map over 400 epochs.
\nThis means optimizing 65,000 images would require 113 days on a single RTX 4090, which is clearly unrealistic.
Perhaps a more efficient forward model could produce results in a reasonable timeframe.
\nDetails of the experimental setup:
\nTips on Efficiency
\nDid not work:
\n
This is essentially what I did. Full details in the weekend, but some specific things responding to your suggestions:
\nTo be continued!
", "rawMarkdown": "This is essentially what I did. Full details in the weekend, but some specific things responding to your suggestions:\n\n- Using priors is critical (i.e. an approximation of the distribution from which the velocity map is taken). The pure inversion problem is far too ill-posed to be solved without one.\n- Extremely efficient forward model and backpropagation is critical.\n- Gradient descent with ADAM is fine for neural networks, but smaller scale problems like this need more dedicated solvers. In my case I used BFGS and Gauss-Newton.\n- It is indeed computationally very expensive.\n\nTo be continued!", "votes": null }, { "id": "3237495", "postDate": "07/01/2025 06:13:17", "content": "\n\nIt is indeed computationally very expensive.
\n
I can't agree more. Back propagation for x1000 times 5x5 convolution is surely compute intensive.
\nI consulted Chat-GPT to further drastically reduced the computation amount, but I found it seems we can't reduce them under $O(n_t * n_z * n_x)$.
\n\nUsing priors is critical
\n
Yes. I also tested for initializing with a gradation of 1500-4500 m/s vel maps, and it only resulted in ~MAE=300 with this prior after 400 epochs.
\n(Note that this implementation of total variation is wrong, so we can observe artifact of horizontal lines.)
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F226255ecea4cc582f39977be84c91749%2FScreenshot%202025-07-01%20at%2015.10.52.png?generation=1751350287562241&alt=media\")
\n\nGradient descent with ADAM is fine for neural networks, but smaller scale problems like this need more dedicated solvers. In my case I used BFGS and Gauss-Newton.
\n
Interesting. Thank you for sharing!
", "rawMarkdown": "> It is indeed computationally very expensive.\n\nI can't agree more. Back propagation for x1000 times 5x5 convolution is surely compute intensive.\nI consulted Chat-GPT to further drastically reduced the computation amount, but I found it seems we can't reduce them under $O(n_t * n_z * n_x)$. \n\n> Using priors is critical\n\nYes. I also tested for initializing with a gradation of 1500-4500 m/s vel maps, and it only resulted in ~MAE=300 with this prior after 400 epochs.\n(Note that this implementation of total variation is wrong, so we can observe artifact of horizontal lines.)\n\n\n\n> Gradient descent with ADAM is fine for neural networks, but smaller scale problems like this need more dedicated solvers. In my case I used BFGS and Gauss-Newton.\n\nInteresting. Thank you for sharing!", "votes": null }, { "id": "3237640", "postDate": "07/01/2025 07:52:09", "content": "@jeroencottaar
\nFWI:
Result
\n| Optimization | \nEpochs | \nTime [sec] | \nMAE [m/s] | \n
|---|---|---|---|
| Initial | \n0 | \n0 | \n405.00 | \n
| Adam | \n400 | \n161 | \n255.17 | \n
| Adam | \n800 | \n323 | \n224.62 | \n
| LBFGS | \n80 | \n160 | \n348.62 | \n
| LBFGS | \n160 | \n312 | \n327.36 | \n
Hyper Parameter:
\nopt = optim.LBFGS(\n [vel_param], lr=, history_size=, max_iter=, line_search_fn=\n)\nInitial velocity map (gradation map over 1500-4000 [m/s]):
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Fcb0b78e5b7cb234c4d3307a6d84da155%2FScreenshot%202025-07-01%20at%2016.44.36.png?generation=1751355894668105&alt=media\")
Adam (800 epoch)
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F08af2f683b884692cc573e3c553ff700%2FScreenshot%202025-07-01%20at%2016.44.23.png?generation=1751355910999978&alt=media\")
LBFGS (160epoch)
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F82bbd3067c784a4b607b090e5efd1f8f%2FScreenshot%202025-07-01%20at%2016.44.29.png?generation=1751355922783371&alt=media\")
Let's continue this when I've written up my full solution (after some work deadlines).
", "rawMarkdown": "Let's continue this when I've written up my full solution (after some work deadlines).", "votes": null } ], "comments": [ { "id": 3237479, "author_name": "jeroencottaar", "author_url": "", "post_date": "07/01/2025 06:00:55", "content": "This is essentially what I did. Full details in the weekend, but some specific things responding to your suggestions:
\nTo be continued!
", "votes": null, "replies": [ { "id": 3237495, "author_name": "tatamikenn", "author_url": "", "post_date": "07/01/2025 06:13:17", "content": "\n\nIt is indeed computationally very expensive.
\n
I can't agree more. Back propagation for x1000 times 5x5 convolution is surely compute intensive.
\nI consulted Chat-GPT to further drastically reduced the computation amount, but I found it seems we can't reduce them under $O(n_t * n_z * n_x)$.
\n\nUsing priors is critical
\n
Yes. I also tested for initializing with a gradation of 1500-4500 m/s vel maps, and it only resulted in ~MAE=300 with this prior after 400 epochs.
\n(Note that this implementation of total variation is wrong, so we can observe artifact of horizontal lines.)
\n\nGradient descent with ADAM is fine for neural networks, but smaller scale problems like this need more dedicated solvers. In my case I used BFGS and Gauss-Newton.
\n
Interesting. Thank you for sharing!
", "votes": null, "replies": [] } ] }, { "id": 3237640, "author_name": "tatamikenn", "author_url": "", "post_date": "07/01/2025 07:52:09", "content": "@jeroencottaar
\nFWI:
Result
\n| Optimization | \nEpochs | \nTime [sec] | \nMAE [m/s] | \n
|---|---|---|---|
| Initial | \n0 | \n0 | \n405.00 | \n
| Adam | \n400 | \n161 | \n255.17 | \n
| Adam | \n800 | \n323 | \n224.62 | \n
| LBFGS | \n80 | \n160 | \n348.62 | \n
| LBFGS | \n160 | \n312 | \n327.36 | \n
Hyper Parameter:
\nopt = optim.LBFGS(\n [vel_param], lr=, history_size=, max_iter=, line_search_fn=\n)\nInitial velocity map (gradation map over 1500-4000 [m/s]):
\n
Adam (800 epoch)
\n
LBFGS (160epoch)
\n
Let's continue this when I've written up my full solution (after some work deadlines).
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3237291": "Congratulations to everyone who was able to make progress beyond Bartley’s strong model(s).\n\nI could not make further progress myself, so I am sharing some findings from my experiments on backpropagating through forward models.\nThis technique is mainly used in generative image models \\[1, 2]; for example, obtaining input image which closely relate to the prompt text with CLIP embedding space.\n\n* \\[1] [https://arxiv.org/abs/2502.07753](https://arxiv.org/abs/2502.07753)\n* \\[2] [https://arxiv.org/abs/2208.01618](https://arxiv.org/abs/2208.01618)\n\nThe experiment was as follows:\n\n* I initialized the input image using Bartley’s model predictions (for example, an image from CurveFault-B with MAE=44.18).\n* I optimized the input image to make the seismogram closer to the “ground truth” (see the 4th picture below).\n\n**Results:**\n\n* Initial image (MAE=44.18)\n* After 400 epochs (MAE=29.92)\n* After 800 epochs (MAE=27.53)\n\nThis approach can indeed improve the metric. However, I found it impractical due to computational efficiency: it took 150 seconds to optimize a single velocity map over 400 epochs.\nThis means optimizing 65,000 images would require **113 days** on a single RTX 4090, which is clearly unrealistic.\n\nPerhaps a more efficient forward model could produce results in a reasonable timeframe.\n\n**Details of the experimental setup:**\n\n* Learning rate: 20.0 (eta_min=0.05)\n* Optimizer: Adam (betas=(0.9, 0.99))\n* Schedule: constant LR\n* Criterion: Huber Loss (delta=1.0) + 0.01 * Total Variation\n* Using mu-law converted seismograms instead of the original ones\n\n**Tips on Efficiency**\n\n* Converting rolling operations to 2D convolutions\n* Converting 2D convolutions to FFT: we can perform FFT + element-wise multiplication + inverse FFT, which is equivalent to a Conv2D with cyclic padding (**1.3~1.5 speedup**)\n\n**Did not work:**\n\n* Perceptual loss\n\n\n\n\n\n\n\n", "3237479": "This is essentially what I did. Full details in the weekend, but some specific things responding to your suggestions:\n\n- Using priors is critical (i.e. an approximation of the distribution from which the velocity map is taken). The pure inversion problem is far too ill-posed to be solved without one.\n- Extremely efficient forward model and backpropagation is critical.\n- Gradient descent with ADAM is fine for neural networks, but smaller scale problems like this need more dedicated solvers. In my case I used BFGS and Gauss-Newton.\n- It is indeed computationally very expensive.\n\nTo be continued!", "3237495": "> It is indeed computationally very expensive.\n\nI can't agree more. Back propagation for x1000 times 5x5 convolution is surely compute intensive.\nI consulted Chat-GPT to further drastically reduced the computation amount, but I found it seems we can't reduce them under $O(n_t * n_z * n_x)$. \n\n> Using priors is critical\n\nYes. I also tested for initializing with a gradation of 1500-4500 m/s vel maps, and it only resulted in ~MAE=300 with this prior after 400 epochs.\n(Note that this implementation of total variation is wrong, so we can observe artifact of horizontal lines.)\n\n\n\n> Gradient descent with ADAM is fine for neural networks, but smaller scale problems like this need more dedicated solvers. In my case I used BFGS and Gauss-Newton.\n\nInteresting. Thank you for sharing!", "3237640": "jeroencottaar \nFWI: \n- the result of gradation velocity prior seems wrong; the same setup than the latest setup generates better result.\n- I also tested LBFGS. It performs better per step, but worse result per actual computation time compared to Adam (more parameter tuning might improve the result)\n\n**Result**\n\n| Optimization | Epochs | Time [sec] | MAE [m/s] |\n|--------------|--------|------------|-----------|\n| Initial | 0 | 0 | 405.00 |\n| Adam | 400 | 161 | 255.17 |\n| Adam | 800 | 323 | 224.62 |\n| LBFGS | 80 | 160 | 348.62 |\n| LBFGS | 160 | 312 | 327.36 |\n\n**Hyper Parameter**:\n\n```python\nopt = optim.LBFGS(\n [vel_param], lr=5.0, history_size=10, max_iter=4, line_search_fn=\"strong_wolfe\"\n)\n```\n\n**Initial velocity map (gradation map over 1500-4000 [m/s])**:\n\n\n**Adam (800 epoch)**\n\n\n**LBFGS (160epoch)**\n", "3237641": "Let's continue this when I've written up my full solution (after some work deadlines)." }, "source": "meta" }