Introduction: Seeing Beneath the Surface with Sound and AI
\n\n\n Podcast for this entire notebook:\n \n Listen to the audio podcast of this notebook.\n \n
\n\n\n Podcast of Insights From Previous Related Competitions:\n \n Podcast of Insights From Previous Related Competitions\n \n
\n\n\n![\"FWI](\"https://www.thomasmeli.com/images/kaggle/geo-fwi-2025/prediction.webp\")
\n\n \n Imagine trying to understand the complex layers inside a cake—sponge, cream, fruit—without slicing it open. How could you map its internal structure? Geoscientists face a similar challenge when mapping the Earth's subsurface, looking for features like rock layers, oil and gas reservoirs, or geological faults. Full Waveform Inversion (FWI) is a sophisticated geophysical technique designed to tackle this.\n
\n\nThink of FWI like a highly detailed ultrasound or sonar for the Earth. We send sound (seismic) waves into the ground using sources like vibrating trucks or small, controlled explosions. These waves travel down, bounce off different underground layers and structures, and return to the surface where they are recorded by many sensors (geophones or hydrophones). FWI uses the *entire* recording of these returning waves (the \"full waveform,\" including all wiggles and timing details) to build a detailed map of how fast the sound waves travel through different parts of the subsurface. This map, called a velocity model, reveals the underground structure.
\n\nMore Rigorous Definition
\nFWI is a high-resolution geophysical imaging technique that aims to estimate quantitative models of subsurface physical parameters (primarily seismic velocities, but potentially density, attenuation, or anisotropy) by minimizing the mismatch between observed seismic waveform data and synthetic waveform data generated by numerically solving the wave equation using a candidate subsurface model. It utilizes the complete information content of the recorded seismograms (amplitude, phase, travel time).
\n\nMathematical Intuition
\nThe core idea is to minimize an objective (or cost) function, often the L2 norm (least-squares) difference between observed and synthetic data:
\nΦ(v) = ½ Σr,t [ uobs(r, t) - usyn(r, t; v) ]²
\nWhere:
\n- \n
- Φ(v) is the misfit (error) for a given velocity model v. \n
- v(x) is the velocity model (the unknown parameter we want to find, defined at spatial location x). \n
- uobs(r, t) is the observed seismic data recorded at receiver r and time t. \n
- usyn(r, t; v) is the synthetic seismic data simulated using the current velocity model v. \n
The synthetic data usyn is calculated by solving the wave equation (often the acoustic approximation for simplicity, though elastic is more realistic):
\n∂²u/∂t² = v(x)² ∇²u + s(x, t)
\nWhere u(x,t) is the pressure wavefield, ∇² is the Laplacian operator (representing spatial derivatives), and s(x,t) is the seismic source. FWI iteratively updates v to minimize Φ, typically using gradient-based optimization methods like gradient descent or L-BFGS.
\n\nFAQ: Why is FWI important?
\nFWI aims for higher resolution imaging compared to simpler methods like travel-time tomography. This detail is crucial for applications like:
\n- \n
- Identifying oil and gas reservoirs or geothermal resources. \n
- Mapping subsurface structures for carbon sequestration or storage. \n
- Assessing geological hazards (e.g., fault structures). \n
- Guiding civil engineering projects (e.g., tunnel construction). \n
- Understanding fundamental Earth structure. \n
\n Think of FWI like performing an ultrasound on the Earth. We generate seismic waves (using controlled sources like vibrations or small explosions) that travel down into the ground. These waves bounce off different materials and structures, and the returning echoes—the full waveforms—are recorded by sensors (geophones or hydrophones) at the surface. FWI analyzes the intricate details of these recorded waveforms to reconstruct a map of the subsurface, specifically a \"velocity model\". This model shows how fast seismic waves travel through different regions; for example, waves travel faster through dense rock and slower through softer sediments or fluids like water or oil.\n
\n\nWaveform: Imagine dropping a pebble in a pond. The ripples spreading out are like seismic waves. A waveform is the detailed recording of how the water level (or ground motion) goes up and down over time at a specific sensor location as those ripples pass by. It contains information about the strength, timing, and shape of the waves.
\nVelocity Model: This is the map FWI tries to create. It shows the speed of sound (seismic waves) at every point underground. Just like light travels slower through water than air, seismic waves travel at different speeds through different materials (rock, sand, water, oil). A velocity model helps us \"see\" these different materials and structures.
\n\nMore Rigorous Definition
\nWaveform: A seismogram or trace representing the time series of ground motion (e.g., displacement, velocity, acceleration, or pressure for acoustic waves) recorded by a single seismic sensor (receiver) resulting from a seismic source excitation. The \"full\" waveform includes all recorded variations, including amplitude, phase, and frequency content, over the recording duration.
\nVelocity Model: A spatial distribution (2D or 3D) of seismic wave propagation speeds within the subsurface volume of interest. Typically, this refers to P-wave (compressional) velocity (Vp) and/or S-wave (shear) velocity (Vs). It is the primary parameter estimated in most FWI applications, as it strongly controls wave travel times and reflection/transmission amplitudes.
\n\n Traditionally, FWI relies on iteratively solving complex wave physics equations. This process is computationally intensive, akin to solving a massive, interconnected puzzle where every piece affects every other. Furthermore, traditional methods can easily get stuck on incorrect solutions if the initial guess of the velocity model isn't close enough to the true model—a notorious problem known as \"cycle-skipping\".\n
\n\nImagine trying to match two identical songs played slightly out of sync. If they are only slightly off, it's easy to tell which way to shift one to match the other. But if one song is delayed by, say, half its length, you might mistakenly try to align the start of the delayed song with the middle of the original one – you've skipped a whole \"cycle.\" In FWI, if our initial guess for the velocity model is too far off, the simulated waves might arrive much earlier or later than the real waves (by more than half a wavelength). The algorithm then gets confused about *which* wiggle in the simulated data should correspond to a wiggle in the real data and makes the wrong adjustment, often getting stuck in a bad solution far from the truth.
\n\nMore Rigorous Definition
\nCycle-skipping occurs in FWI when the phase difference between the observed (uobs) and synthetic (usyn) waveforms exceeds half a period (π radians or 180 degrees) for the dominant frequency being considered. Standard local optimization methods (like gradient descent) rely on minimizing the difference based on the nearest corresponding phase cycle. When cycle-skipped, the computed gradient points in a direction that increases, rather than decreases, the true phase difference, leading the inversion to converge towards a local minimum of the objective function Φ that does not correspond to the true Earth model.
\nThis typically happens when the initial velocity model is inaccurate, especially for higher frequencies or larger velocity errors, making FWI highly sensitive to the starting model quality.
\n\nFAQ: How do advanced methods try to mitigate cycle-skipping?
\nSeveral strategies are employed:
\n- \n
- Multi-scale approaches: Start FWI with low frequencies (longer wavelengths, less prone to skipping) and gradually introduce higher frequencies. \n
- Improved objective functions: Use misfit measures less sensitive to phase shifts (e.g., based on wavefield envelopes, optimal transport, or correlation). \n
- Better starting models: Use other methods (like tomography) to get a better initial guess. \n
- Deep Learning methods: Some DL approaches, like IFWI (Implicit FWI) or certain GAN formulations (FWI-GAN), seem inherently more robust to cycle-skipping, potentially because they explore the solution space differently or implicitly incorporate broader structural constraints learned from data or physics. \n
\n\n\nRecently, the field has turned to Deep Learning (DL).
\nIn the context of FWI, we train a neural network by showing it many examples of seismic data (the \"echoes\") and the corresponding correct subsurface velocity maps. Early DL models, such as
\nInversionNet(often based on a U-Net architecture, popular in image analysis for its ability to capture details at different scales) andVelocityGAN(using Generative Adversarial Networks, where two networks compete to generate realistic results), demonstrated that AI could learn to predict the subsurface map directly from seismic data, often much faster than traditional methods.
Deep Learning (DL): A type of AI that uses interconnected \"neurons\" in layered structures (like a brain) to learn complex patterns from data. We \"teach\" it by showing many examples (e.g., seismic recordings and the matching underground maps). After training, it can often make predictions on new data very quickly.
\nU-Net Architecture: A specific type of DL network design, originally popular for medical image analysis. It looks like the letter 'U'. It first shrinks the input image down (capturing broad features) and then expands it back up (reconstructing fine details), while also using \"shortcuts\" to carry information from the shrinking path directly to the expanding path. This helps it create detailed output maps (like velocity models) that preserve spatial information well.
\nGenerative Adversarial Networks (GANs): Imagine an art forger (Generator) trying to create fake paintings and an art detective (Discriminator) trying to spot the fakes. The forger learns by getting feedback from the detective. In FWI, the Generator tries to create realistic velocity models (or the seismic data resulting from them), and the Discriminator tries to tell the difference between the Generator's output and real-world examples. This competition forces the Generator to become very good at creating realistic results.
\n\nMore Rigorous Definition
\nDeep Learning (DL): A subfield of machine learning based on artificial neural networks with multiple layers (deep architectures) between the input and output layers. These networks learn hierarchical representations of data, enabling them to model complex, non-linear relationships. In FWI, DL is often used for supervised learning (mapping seismic data to velocity models), unsupervised learning (learning data distributions, e.g., with GANs), or physics-informed learning (integrating physical laws).
\nU-Net Architecture: A convolutional neural network (CNN) architecture characterized by an encoder path (down-sampling, feature extraction) and a symmetric decoder path (up-sampling, localization), with skip connections linking corresponding layers between the encoder and decoder. These skip connections allow the network to combine high-level semantic features (from deeper layers) with low-level fine-grained detail (from earlier layers), making it effective for image segmentation and image-to-image translation tasks, such as mapping seismic data to velocity maps.
\nGenerative Adversarial Networks (GANs): A class of deep learning frameworks where two neural networks, a Generator (G) and a Discriminator (D), are trained simultaneously in opposition. The Generator learns to map from a latent space (e.g., random noise) to a target data distribution (e.g., realistic velocity models or seismic data), while the Discriminator learns to distinguish between real data samples and samples generated by G. The training objective is typically formulated as a minimax game. Variations like Wasserstein GAN (WGAN) improve training stability and correlation with sample quality.
\n\n However, these pioneering AI models often struggled with generalization. Like a student who only memorizes answers from one textbook, they performed well on geology similar to their training data but failed when faced with new, unseen subsurface structures. To address this, the research community developed large-scale, diverse benchmark datasets, notably the OpenFWI dataset. OpenFWI provides a rich \"library\" of simulated seismic data and corresponding velocity models covering various geological complexities: simple layered structures (\"Vel\"), layers disrupted by faults (\"Fault\"), and highly variable, complex geology (\"Style\"). These datasets allow for more robust training and fairer comparison of different DL approaches.\n
\n\nGeneralization: This is the AI model's ability to perform well on new, unseen data that it wasn't explicitly trained on. A model with good generalization is like a student who understands the underlying concepts and can solve new types of problems, not just the ones they practiced.
\nOpenFWI Dataset: Think of this as a large, diverse collection of practice problems for FWI AI models. It contains thousands of simulated examples of seismic recordings paired with the known \"answer key\" (the true underground velocity map). Crucially, it includes different types of geological structures (simple layers, faulted layers, complex styles) so that models trained on it learn to handle a wider variety of situations and generalize better.
\n\nMore Rigorous Definition
\nGeneralization: In machine learning, generalization refers to a model's ability to adapt properly to new, previously unseen data drawn from the same distribution as the one used to train the model. A model generalizes well if its performance on a held-out test dataset is close to its performance on the training dataset. Poor generalization (overfitting) occurs when a model learns the training data too specifically, including noise and spurious correlations, and fails to capture the underlying patterns applicable to new data.
\nOpenFWI Dataset: A suite of large-scale, open-source benchmark datasets designed specifically for training and evaluating deep learning-based FWI methods. It comprises multiple families (e.g., FlatVel, CurveVel, FaultSet, StyleSet) containing hundreds of thousands of pairs of 2D synthetic velocity models and their corresponding simulated seismic shot gathers. The diversity in geological structures and acquisition geometries within OpenFWI facilitates the development and comparison of models with improved generalization capabilities. (Reference: Deng et al., 2022; Huang et al., 2022)
\nRecent Advances (2022–2025): Smarter AI for Better Underground Maps
\n\n\n Research from 2022 to 2025 has focused intensely on overcoming the limitations of early DL-based FWI, leading to several key trends:\n
\n\n- \n
- \n Bigger Brains, More Experience (Larger Models & Datasets): Just as humans learn better with more diverse experiences, DL models benefit significantly from training on vast datasets. Jin et al. (2024) demonstrated that training a very large model (
BigFWI) on a massive combination of OpenFWI datasets (470,000 examples!) dramatically improved accuracy and generalization ability, reducing errors by 13–20%. They also confirmed that increasing model size (more parameters or \"neurons\") further enhances performance. This underscores the importance of both data scale and model capacity.\n Big Data Impact Podcast\n \n\n - \n Seeing the Big Picture (Transformer Architectures): Traditional image processing models like Convolutional Neural Networks (CNNs) examine data through small, sliding windows (kernels). This is effective for local patterns but can miss long-range dependencies in the data. Imagine trying to understand a complex musical piece by only listening to two notes at a time. Transformers, an architecture that revolutionized natural language processing, employ a mechanism called \"self-attention.\"\n\n \n \n\n\n\n\n
Convolutional Neural Networks (CNNs): These AI models are inspired by human vision. They use small \"filters\" (like magnifying glasses) that slide across the input (like an image or seismic data) to detect local patterns (edges, textures). They are very good at tasks where local features are important, but might struggle to connect information that is far apart in the input.
\nTransformers: A newer type of AI architecture, originally famous for understanding language (like translating sentences). Instead of just looking locally, Transformers use a special mechanism called \"self-attention\" to look at *all* parts of the input data simultaneously and figure out how relevant each part is to every other part. This allows them to capture long-range connections and understand the global context.
\nSelf-Attention: Think of it as the model asking, for each piece of input data (e.g., a specific seismic echo), \"Which other echoes, no matter how far away in time or space, are most important for understanding *this* echo?\" It calculates \"attention scores\" between all pairs of input pieces, allowing it to weigh the influence of different parts dynamically based on the context.
\n\nMore Rigorous Definition
\nConvolutional Neural Networks (CNNs): A class of deep neural networks highly effective for processing grid-like data (e.g., images, seismic sections). They employ convolutional layers that apply learnable filters (kernels) across the input, leveraging parameter sharing and spatial hierarchies to extract features. Their strength lies in capturing local spatial patterns and building translation-equivariant representations.
\nTransformers: A neural network architecture, introduced by Vaswani et al. (2017), that relies primarily on self-attention mechanisms instead of recurrence (like RNNs) or local convolution (like CNNs). Originally designed for sequence-to-sequence tasks in NLP, they have been adapted for vision (ViT) and other domains. Their ability to model long-range dependencies makes them suitable for tasks where global context is important.
\nSelf-Attention: A mechanism within the Transformer architecture that allows the model to weigh the importance of different positions in the input sequence when computing the representation of a specific position. It calculates attention scores based on pairwise similarity (often using scaled dot-product attention) between query, key, and value vectors derived from the input embeddings. Multi-head attention allows the model to attend to information from different representational subspaces simultaneously.
\n\n Think of self-attention as enabling the model to weigh the importance of all input parts relative to each other, simultaneously. For seismic data, this means it can correlate an early echo at one sensor with a related, delayed echo at a far-off sensor, capturing the 'global context' of the wavefield. Models like the Seismic Velocity Inversion Transformer (SVIT) leverage this global view to produce more accurate and geologically plausible velocity maps. We'll discuss SVIT in more detail below.\n
\n \n\n - \n Teaching AI Physics (Hybrid & Physics-Informed Learning): Instead of solely learning patterns from data examples (input seismic -> output velocity map), newer methods integrate the underlying physics of wave propagation directly into the AI's learning process.\n\n \n \n\n\n\n\n
Imagine training an AI apprentice not just by showing them examples, but also by giving them the physics textbook. Physics-informed methods embed the known laws of physics (like the wave equation in FWI) into the AI's learning process. This can be done by making the AI's predictions satisfy the physics equations, or by using a physics simulator as part of the AI training loop. The goal is to get predictions that are not only consistent with the observed data but also physically plausible.
\n\nMore Rigorous Definition
\nPhysics-Informed Neural Networks (PINNs): A class of neural networks trained to solve supervised learning tasks while respecting constraints imposed by physical laws, typically described by partial differential equations (PDEs). The loss function includes not only the data mismatch term (comparing predictions to observations) but also a term that measures how well the network's output satisfies the governing PDE(s), boundary conditions, and initial conditions. This encourages the network to learn solutions that are physically consistent, potentially improving generalization and reducing the need for large labeled datasets.
\nHybrid approaches might involve using DL for parts of the FWI workflow (e.g., initial model building, denoising) or embedding a differentiable physics simulator within a larger DL framework (like FWI-GAN or IFWI).
\n- \n
- FWI-GAN: Uses a Generative Adversarial Network (GAN) framework. A \"Generator\" network proposes a velocity map. A physics simulator calculates the seismic data that *would* result from this map. A second \"Discriminator\" network learns to distinguish between this simulated data and the real, observed seismic data. The Generator's goal is to fool the Discriminator, forcing it to learn velocity maps that are consistent with both the observed data and the wave physics. Remarkably, this can work in an unsupervised manner, without needing pre-existing \"correct\" velocity maps for training! \n
- Implicit FWI (IFWI): Represents the velocity map not as a grid of pixels, but as a continuous mathematical function defined by a neural network (often a simple Multi-Layer Perceptron, MLP). You give the network coordinates (x, z), and it outputs the velocity at that exact point. This network's parameters are optimized by comparing the seismic data simulated using its function (via a physics engine) against the real observed data. This approach inherently respects the continuous nature of physical fields and shows promise in avoiding the cycle-skipping problem. \n
These physics-informed methods aim to ensure the AI's predictions adhere to physical laws, leading to more trustworthy and reliable results.
\n \n\n - \n Specialists Working Together (Two-Stage & Multi-Task Networks): Sometimes, a single AI model struggles to excel at all aspects of the FWI problem. For instance, standard models often blur sharp geological boundaries. The Velocity-Interface Fusion (VIF) Network (2024) employs a team strategy. One network estimates a smooth velocity field, while a parallel network specializes in identifying the locations of sharp interfaces (like edges in an image). A third network then fuses these two pieces of information to produce a final velocity map that is both accurate in its velocity values and sharp at geological boundaries.\n \n\n
- \n One AI to Rule Them All? (Foundation Models & Adaptation): Inspired by the success of large language models (LLMs like GPT) and large vision models, researchers are exploring FWI foundation models. These are massive neural networks pre-trained on enormous, diverse seismic datasets (like the combined OpenFWI). The goal is for these models to learn general principles of wave physics and geology. Then, for a specific new survey or geological setting, instead of training a model from scratch (which is costly), this pre-trained foundation model can be quickly adapted. Techniques like Parameter-Efficient Fine-Tuning (PEFT), particularly Low-Rank Adaptation (LoRA), allow this adaptation by training only a tiny fraction of the model's parameters. This is computationally efficient and surprisingly effective, especially for generalizing to data distributions not seen during pre-training.\n\n \n \n\n\n\n
Foundation Models: Think of a huge, general-purpose AI model (like GPT for text, or a similar large model trained on vast amounts of seismic data). It's pre-trained on a massive, diverse dataset to gain a broad understanding of its domain (language, vision, or in our case, seismic waves and geology). It serves as a powerful starting point.
\nParameter-Efficient Fine-Tuning (PEFT): When you want to use a foundation model for a *specific* new task (like FWI for a particular region), you don't need to retrain the entire huge model. PEFT techniques allow you to adapt the model by training only a small number of *new* or *modified* parameters, while keeping most of the original model's knowledge frozen. This is much faster and requires less computing power than full retraining.
\nLow-Rank Adaptation (LoRA): A popular PEFT technique. Instead of changing the foundation model's existing \"brain cells\" (weights), LoRA adds small, trainable \"adapter modules\" alongside them. These adapters learn the task-specific adjustments. Because these adapters are cleverly designed to be \"low-rank\" (mathematically simple), they add very few trainable parameters, making adaptation highly efficient.
\n\nMore Rigorous Definition
\nFoundation Models: Large-scale neural network models pre-trained on vast quantities of broad data (e.g., diverse seismic datasets like OpenFWI for FWI foundation models). They are designed to capture general-purpose representations and capabilities within their domain, serving as a base model that can be adapted (fine-tuned) for various downstream tasks with potentially less task-specific data and computation.
\nParameter-Efficient Fine-Tuning (PEFT): A collection of techniques aimed at adapting large pre-trained foundation models to downstream tasks by modifying only a small subset of the model's parameters, or by adding a small number of new parameters, while keeping the majority of the original parameters frozen. This significantly reduces the computational cost (memory, time) and storage requirements associated with fine-tuning large models.
\nLow-Rank Adaptation (LoRA): A specific PEFT method that injects trainable low-rank matrices into the layers (e.g., linear, convolutional) of a pre-trained model during fine-tuning. For a weight matrix W₀, LoRA learns an update ΔW = BA, where A and B are low-rank matrices (rank r ≪ dimensions of W₀). Only A and B are trained, drastically reducing the number of trainable parameters compared to full fine-tuning. The modified forward pass becomes h = W₀x + α(BA)x.
\n \n
\n Below, we delve into detailed reviews of a few representative papers from the 2022–2025 period that exemplify these exciting advancements, including simplified code examples to illustrate key concepts.\n
\n\nDetailed Paper Reviews (2022–2025)
\n\n\n One-Fit-All Transformer Podcast\n Seismic Velocity Inversion Transformer (SVIT, 2023)\n
\nReference: Yue Li and Baojun Yang, GEOPHYSICS, 88(4): R513–R533, 2023.
\n (*Note: The podcast link points to a general Transformer FWI concept, relevant here.*)\n\nOverview
\n\n SVIT stands out as one of a first applications of the powerful Transformer architecture, originally famed for its success in natural language processing (NLP), to the FWI problem. It directly addresses a limitation of prior CNN-based models (like U-Nets): CNNs rely on local convolutional filters, which excel at capturing nearby spatial patterns but may struggle to relate information across large distances in the seismic data record (e.g., connecting early arrivals at near sensors to late arrivals at far sensors).\n
\n\n\nSVIT employs the Transformer's core mechanism: self-attention. Imagine the model possessing the ability to look at the entire seismic recording simultaneously and dynamically decide which parts are most relevant to understanding any other part, regardless of their separation in time or space.
\nThis \"global receptive field\" allows SVIT to build a more holistic understanding of the wave propagation patterns, potentially leading to better reconstruction of large-scale geological structures and continuous layers compared to locally-focused CNNs.
\n
For definitions of Transformers and Self-Attention, please see the earlier accordion \n \"What are CNNs, Transformers, and Self-Attention?\" in the Recent Advances section above.
\nKey Contributions
\n- \n
- Pioneered the use of Transformer architecture for end-to-end FWI, showcasing the effectiveness of self-attention for modeling long-range dependencies in seismic waveforms. \n
- Reportedly achieved improved accuracy and geological consistency in velocity model reconstruction compared to baseline CNNs (e.g., U-Net) and even some traditional FWI approaches in specific test cases. The resulting velocity maps exhibited clearer structural features and interfaces. \n
Architecture
\n\n SVIT likely adopts an encoder-decoder structure, drawing inspiration from Vision Transformers (ViTs) used in computer vision. The key steps typically involve:\n
\n- \n
- Input Patching: The input seismic data (often represented as a 2D array of time samples vs. receiver locations) is divided into smaller, non-overlapping patches, similar to breaking an image into tiles. \n
- Linear Embedding: Each patch is flattened and linearly projected into a vector (a \"token\"). Positional embeddings are added to these tokens to retain information about the original location of each patch. \n
- Transformer Encoder: These patch tokens, augmented with positional information, are processed through a stack of Transformer blocks. Each block contains a multi-head self-attention layer (allowing the model to attend to information from different perspectives) and a feed-forward neural network. This is where the model learns the complex, long-range relationships within the seismic data. \n
- Decoder/Reconstruction Head: The processed tokens from the encoder are then fed into a decoder module, which reconstructs the 2D velocity map of the subsurface. This decoder might use transposed convolutions (sometimes called \"deconvolutions\") or other up-sampling techniques. \n
\n Essentially, SVIT replaces the convolutional layers of a typical CNN encoder with Transformer blocks centered around the self-attention mechanism.\n
\n\nTraining & Loss Function
\n\n SVIT is trained in a supervised learning setting. This means it learns from a large dataset of paired examples: input seismic data and the corresponding known \"ground truth\" velocity map. The dataset likely consists of synthetically generated examples covering various geological scenarios. During training, the model's predicted velocity map is compared to the ground truth map using a loss function. Common choices include pixel-wise losses like Mean Squared Error (MSE) or Mean Absolute Error (MAE). The goal of training is to adjust the model's internal parameters (weights) to minimize this loss. Performance is evaluated using metrics like MAE, MSE, and often the Structural Similarity Index (SSIM), which measures perceptual similarity between the predicted and true maps. SVIT was reported to show improvements in these metrics, indicating better reconstruction of velocity values, structure shapes, and interface locations.\n
\n\nSupervised Learning: Like teaching a student with an answer key. The AI model is shown input examples (seismic data) along with the correct outputs (velocity maps). It learns by comparing its predictions to the correct answers and adjusting itself to get closer.
\nLoss Function: A mathematical formula that measures how \"wrong\" the AI's prediction is compared to the correct answer. Common examples:
\n- \n
- Mean Squared Error (MSE): Calculates the average of the squared differences between predicted and true values. Penalizes large errors heavily. \n
- Mean Absolute Error (MAE): Calculates the average of the absolute differences. Less sensitive to outliers than MSE. \n
- Structural Similarity Index (SSIM): Measures the similarity between two images based on structure, contrast, and luminance. Often better captures visual quality than simple pixel differences. \n
The goal of training is to minimize the value of the loss function.
\n\nMore Rigorous Definition
\nSupervised Learning: A machine learning paradigm where an algorithm learns a mapping function from input variables (X) to output variables (Y) based on a labeled dataset of input-output pairs ({(xᵢ, yᵢ)}). The goal is to approximate the function f such that f(X) predicts Y accurately on unseen data.
\nLoss Function (or Cost Function): A function L(ypred, ytrue) that quantifies the discrepancy between the model's prediction (ypred) and the true target value (ytrue). The objective of training is to find model parameters that minimize the average loss over the training dataset. Examples:
\n- \n
- MSE: L = (1/N) Σ (ypred - ytrue)² \n
- MAE: L = (1/N) Σ |ypred - ytrue| \n
- SSIM: A more complex function comparing local statistics (mean, variance, covariance) of image patches. \n
Reproducibility
\n\n The authors published their findings in the Geophysics journal but did not provide an official open-source code release. However, the architectural components (patch embedding, multi-head self-attention, MLP blocks) are standard building blocks within the Transformer ecosystem (particularly Vision Transformers). Researchers familiar with frameworks like PyTorch or TensorFlow could potentially reimplement the core ideas.\n
\n\nConceptual Implementation (PyTorch): Simplified Transformer for FWI
\nHere's a conceptual PyTorch code snippet illustrating the core structure of a Transformer-based FWI model, inspired by SVIT.
\n\nimport torch\nimport torch.nn as nn\n\nclass PatchEmbed(nn.Module):\n \"\"\"Convert seismic data (time x receivers image) into patch tokens.\"\"\"\n def __init__(self, img_size=(1000, 70), patch_size=(16, 16), in_chans=1, embed_dim=128):\n super().__init__()\n self.img_size = img_size\n self.patch_size = patch_size\n self.num_patches = (img_size[0] // patch_size[0]) * (img_size[1] // patch_size[1])\n # Use a Conv2d layer with stride equal to patch_size for efficient patch extraction & embedding\n self.proj = nn.Conv2d(in_chans, embed_dim, kernel_size=patch_size, stride=patch_size)\n\n def forward(self, x):\n # x: (batch, in_chans, height, width) -> e.g., (B, 1, 1000, 70)\n B, C, H, W = x.shape\n assert H == self.img_size[0] and W == self.img_size[1], \\\n f\"Input image size ({H}*{W}) doesn't match model ({self.img_size[0]}*{self.img_size[1]}).\"\n # Project and flatten: (B, embed_dim, num_patches_h, num_patches_w) -> (B, embed_dim, num_patches_total)\n x = self.proj(x).flatten(2)\n # Transpose to get (B, num_patches_total, embed_dim) - standard Transformer input format\n x = x.transpose(1, 2)\n return x\n\nclass TransformerEncoderLayer(nn.Module):\n \"\"\"Standard Transformer Encoder Block.\"\"\"\n def __init__(self, dim, num_heads, mlp_ratio=4., dropout=0.1):\n super().__init__()\n self.norm1 = nn.LayerNorm(dim)\n # Multi-Head Self-Attention\n self.attn = nn.MultiheadAttention(embed_dim=dim, num_heads=num_heads, dropout=dropout, batch_first=True)\n self.norm2 = nn.LayerNorm(dim)\n # Feed-Forward Network (MLP)\n mlp_hidden_dim = int(dim * mlp_ratio)\n self.mlp = nn.Sequential(\n nn.Linear(dim, mlp_hidden_dim),\n nn.GELU(), # Common activation in Transformers\n nn.Dropout(dropout),\n nn.Linear(mlp_hidden_dim, dim),\n nn.Dropout(dropout)\n )\n # Note: Typically includes stochastic depth (DropPath), omitted here for simplicity\n\n def forward(self, x):\n # Self-attention part + residual connection\n residual = x\n x_norm = self.norm1(x)\n attn_output, _ = self.attn(x_norm, x_norm, x_norm) # Q, K, V are the same\n x = residual + attn_output # Additive residual connection\n\n # Feed-forward part + residual connection\n residual = x\n x_norm = self.norm2(x)\n mlp_output = self.mlp(x_norm)\n x = residual + mlp_output # Additive residual connection\n return x\n\nclass FWI_TransformerDecoder(nn.Module):\n \"\"\" A simple decoder to reconstruct the velocity map from tokens \"\"\"\n def __init__(self, num_patches, embed_dim, output_size=(100, 100)):\n super().__init__()\n self.num_patches = num_patches\n self.embed_dim = embed_dim\n self.output_size = output_size\n # Example: Use a linear layer to project back to the flattened image size\n self.head = nn.Linear(embed_dim, output_size[0] * output_size[1])\n # More complex decoders might use convolutional layers (ConvTranspose2d)\n\n def forward(self, x):\n # x: (B, num_patches, embed_dim)\n # Option 1: Use the representation of the first token (if using a [CLS] token like BERT)\n # cls_token_representation = x[:, 0]\n # Option 2: Average pool all patch tokens\n avg_pool_representation = x.mean(dim=1) # (B, embed_dim)\n # Project to the flattened output image size\n flat_output = self.head(avg_pool_representation) # (B, output_H * output_W)\n # Reshape to the desired 2D velocity map (ensure channels dimension is included)\n output_map = flat_output.view(-1, 1, self.output_size[0], self.output_size[1])\n return output_map\n\nclass SVIT_Inspired_Model(nn.Module):\n \"\"\"Simplified Transformer-based FWI model.\"\"\"\n def __init__(self, input_shape=(1000, 70), patch_size=(10, 10), embed_dim=128, depth=6, num_heads=8, output_shape=(100, 100), dropout=0.1):\n super().__init__()\n self.patch_embed = PatchEmbed(img_size=input_shape, patch_size=patch_size, in_chans=1, embed_dim=embed_dim)\n num_patches = self.patch_embed.num_patches\n\n # Learnable positional embeddings provide spatial context for patches\n self.pos_embed = nn.Parameter(torch.zeros(1, num_patches, embed_dim))\n self.pos_drop = nn.Dropout(p=dropout)\n\n # Stack of Transformer Encoder layers\n self.encoder_blocks = nn.ModuleList([\n TransformerEncoderLayer(dim=embed_dim, num_heads=num_heads, dropout=dropout)\n for _ in range(depth)])\n self.norm = nn.LayerNorm(embed_dim) # Final normalization after encoder blocks\n\n # Decoder head to reconstruct the velocity map\n self.decoder = FWI_TransformerDecoder(num_patches=num_patches, embed_dim=embed_dim, output_size=output_shape)\n\n # Initialize weights (important for stable training)\n nn.init.trunc_normal_(self.pos_embed, std=.02)\n self.apply(self._init_weights)\n\n def _init_weights(self, m):\n if isinstance(m, nn.Linear):\n nn.init.trunc_normal_(m.weight, std=.02)\n if isinstance(m, nn.Linear) and m.bias is not None:\n nn.init.constant_(m.bias, 0)\n elif isinstance(m, nn.LayerNorm):\n nn.init.constant_(m.bias, 0)\n nn.init.constant_(m.weight, 1.0)\n\n def forward(self, x):\n # x: (B, 1, time_steps, num_receivers) - e.g., (B, 1, 1000, 70)\n B = x.shape[0]\n\n # 1. Embed patches & Flatten\n tokens = self.patch_embed(x) # (B, num_patches, embed_dim)\n\n # 2. Add positional embedding\n tokens = tokens + self.pos_embed # Broadcasting applies pos_embed to all batches\n tokens = self.pos_drop(tokens)\n\n # 3. Pass through Transformer encoder blocks\n for blk in self.encoder_blocks:\n tokens = blk(tokens)\n tokens = self.norm(tokens) # Apply final normalization\n\n # 4. Decode to velocity map\n velocity_map = self.decoder(tokens) # (B, 1, output_H, output_W)\n\n return velocity_map\n\n# --- Example Usage ---\n# Define model parameters matching example data\nmodel = SVIT_Inspired_Model(input_shape=(1000, 70), patch_size=(10, 10), embed_dim=128, depth=4, num_heads=8, output_shape=(100, 100))\n\n# Create a dummy batch of seismic data\n# Batch size = 4, Channels = 1, Height = 1000 (time steps), Width = 70 (receivers)\nseismic_batch = torch.randn(4, 1, 1000, 70)\n\n# Perform a forward pass\nvelocity_output = model(seismic_batch)\n\n# Print the shape of the output velocity map\nprint(f\"Output velocity map shape: {velocity_output.shape}\") # Expected: torch.Size([4, 1, 100, 100])\n \n This sketch highlights the key stages: patch embedding, adding positional information, processing through Transformer encoder blocks using self-attention, and finally decoding the processed information back into a 2D velocity map. Training would involve feeding batches of seismic data, getting the predicted velocity map, comparing it to the known true map using a loss function (like MSE), and updating the model's weights via backpropagation.\n
\n\nVelocity Interface Fusion Network (VIF-Net, 2024)
\nReference: Z. Wang et al., Computers & Geosciences, 174: 105834, 2024.
\n\nOverview
\n\n A persistent challenge in FWI, whether traditional or DL-based, is accurately resolving sharp boundaries between different geological units (e.g., layers of rock with distinct properties) or across faults. Standard DL models, particularly those based on smooth loss functions like MSE, often produce velocity maps that appear blurry or smoothed-out across these crucial interfaces. VIF-Net tackles this specific issue with an innovative two-stage, multi-network approach.\n
\nThink of the boundaries between different layers in a cake or different types of rock underground. These are interfaces. They represent locations where the properties of the material change abruptly (e.g., from sandstone to shale, or across a fault where rocks have shifted). Accurately mapping these interfaces is critical for understanding the geological structure.
\nMore Rigorous Definition
\nIn geophysics and geology, an interface refers to a surface or boundary separating two media with different physical properties (e.g., seismic velocity, density, electrical resistivity). Examples include stratigraphic boundaries between rock layers, unconformities, fault planes, or the boundary of a fluid reservoir. Seismic waves reflect and refract at these interfaces, and accurately imaging their location and character is a primary goal of seismic exploration and FWI.
\n\n\n\nThink of creating a detailed geological map. You need two key pieces of information: the elevation (representing seismic velocity) at every point, and the precise location of sharp changes like cliffs or riverbanks (representing geological interfaces).
\nVIF-Net essentially trains specialist AIs for each task: one estimates the overall velocity landscape, another pinpoints the locations of sharp boundaries. Then, a final \"fusion\" AI combines this information to create a map that respects both the smooth velocity variations and the sharp interface locations.
\n
Key Contributions
\n- \n
- Introduces a structured two-stage workflow tailored for interface sharpness:\n
- \n
- Pre-inversion Stage: Two parallel neural networks analyze the input seismic data simultaneously.\n
- \n
- Network 1 (
VIF-V): Predicts a preliminary, potentially smooth, velocity map. \n - Network 2 (
VIF-I): Predicts an \"interface map,\" highlighting the probability or location of sharp geological boundaries (essentially performing an edge or boundary detection task). \n
\n - Network 1 (
- Fusion Stage: A third network takes the outputs from Stage 1 (the preliminary velocity map and the interface location map) as combined input. It intelligently merges this information to produce a final, refined velocity map characterized by significantly sharper and more accurate interfaces. \n
\n - Pre-inversion Stage: Two parallel neural networks analyze the input seismic data simultaneously.\n
- Demonstrates significantly improved recovery of sharp geological features like layer boundaries and faults compared to conventional single-network FWI models that tend to oversmooth these critical details. \n
Architecture
\n\n The paper suggests that the networks in Stage 1 are likely CNN-based. VIF-V could be a standard regression network, possibly a U-Net architecture, trained to output velocity values. VIF-I would be structured as an image segmentation network (architectures like U-Net or DeepLabv3+ are common choices), trained to output a binary or probability map indicating interface locations. The Stage 2 \"Fusion Network\" could be another U-Net-like architecture. Its crucial role is to take the intermediate velocity map and the interface map as multi-channel input and learn how to refine the velocity map, paying particular attention to preserving or sharpening features indicated by the interface map.\n
Image Segmentation: Imagine coloring in a map where each region (like a country or a lake) gets a specific color. Image segmentation is similar: an AI model looks at an image and assigns a label (like \"rock layer A,\" \"rock layer B,\" or \"fault boundary\") to every single pixel. VIF-Net's interface network (VIF-I) performs segmentation to identify pixels belonging to sharp boundaries.
Segmentation Losses (BCE, Dice): These are ways to measure how well the AI's segmentation map matches the true map.
\n- \n
- Binary Cross-Entropy (BCE): Commonly used when there are only two classes (e.g., \"interface\" vs. \"not interface\"). It measures the difference between the predicted probability and the true label (0 or 1) for each pixel. \n
- Dice Coefficient (or Dice Loss): Measures the overlap between the predicted segmentation and the true segmentation. A score of 1 means perfect overlap, 0 means no overlap. It's often good for handling situations where the target regions (like interfaces) are small compared to the background. \n
More Rigorous Definition
\nImage Segmentation: The task of partitioning a digital image into multiple segments (sets of pixels, also known as superpixels). The goal is typically to assign a class label to each pixel in the image, such that pixels with the same label share certain characteristics (e.g., semantic meaning, visual properties). In VIF-Net, the interface network performs semantic segmentation to classify pixels as belonging to a geological interface or not.
\nSegmentation Losses:
\n- \n
- Binary Cross-Entropy (BCE) Loss: For binary segmentation (two classes), measures the negative log-likelihood of the predicted probabilities assuming a Bernoulli distribution for each pixel. LBCE = - (1/N) Σ [ y log(p) + (1-y) log(1-p) ], where y is the true label (0 or 1) and p is the predicted probability for class 1. \n
- Dice Coefficient / Loss: Measures spatial overlap. The Dice coefficient is DSC = 2 * |X ∩ Y| / (|X| + |Y|), where X is the predicted set of pixels and Y is the true set. Dice Loss is often defined as LDice = 1 - DSC. It is particularly useful for unbalanced datasets where the target region is small. \n
Training Setup
\n\n Training VIF-Net effectively requires ground truth data for both the velocity map and the interface locations. The interface ground truth can often be derived from the true velocity map by applying edge detection algorithms (e.g., finding regions with high gradients). The networks might be trained end-to-end, or potentially sequentially. VIF-V would typically use a regression loss like MSE or MAE on the velocity values. VIF-I would use a segmentation loss, such as Binary Cross-Entropy (BCE) or the Dice coefficient, comparing the predicted interface map to the true interface locations. The Fusion Network is trained to minimize the error (e.g., MSE) in the final velocity output, possibly with additional loss terms that encourage sharpness guided by the interface input. The authors likely tested VIF-Net on synthetic datasets featuring distinct interfaces, such as those found in the \"Fault\" or \"Style\" categories of the OpenFWI benchmark.\n
Results
\n\n VIF-Net showcased visually clearer and quantitatively more accurate delineation of faults and layer boundaries in test cases compared to baseline methods. This enhanced ability to resolve sharp discontinuities is vital for accurate geological interpretation, such as identifying reservoir seals or mapping fracture networks.\n
\n\nOpen Source Availability
\n\n At the time of this review, an official code repository from the authors does not appear to be publicly available. However, the conceptual framework relies on combining well-established CNN architectures for regression and segmentation tasks, making reimplementation feasible for researchers skilled in deep learning.\n
\n\n\n\n InversionNet3D Podcast\n Implicit Full Waveform Inversion (IFWI, 2022/2023)\n
\nReference: Jian Sun and Kristopher Innanen, arXiv:2209.03525 (2022), J. Geophys. Res.: Solid Earth (2023).
\n (*Note: Podcast link relates to advanced DL architectures for FWI, conceptually relevant to complex methods like IFWI.*)\n\n\nOverview
\n\n Implicit Full Waveform Inversion (IFWI) represents a paradigm shift from typical supervised DL approaches. Instead of training a network to map input seismic data directly to an output velocity grid (like image-to-image translation), IFWI uses a neural network to implicitly represent the velocity field itself as a continuous function.\n
\nInstead of storing information like a picture (a grid of pixels), imagine having a magic formula (the neural network). You give the formula coordinates (like latitude and longitude, or x and z depth), and it calculates the value at that exact point (like the temperature, or in FWI, the seismic velocity). The network *is* the representation; its internal parameters (weights) define the continuous field.
\n\nMore Rigorous Definition
\nAn Implicit Neural Representation (INR), also known as a coordinate-based network, uses a neural network (often an MLP) to parameterize a signal or field as a continuous function of its coordinates. Instead of discretizing the signal onto a grid, the INR fθ(x) maps input coordinates x (e.g., spatial coordinates (x, y, z)) to the signal value at that location (e.g., velocity v), where θ represents the network's parameters. The network weights θ are optimized to fit observed data or satisfy certain constraints (like PDEs in PINNs).
\n\n\nImagine the velocity map isn't stored as a grid of discrete pixel values. Instead, you have a small, flexible mathematical formula (parameterized by a neural network) that can tell you the precise velocity at any continuous coordinate (x, z) you query.
\nThis network, often a simple Multi-Layer Perceptron (MLP), initially knows nothing (random weights). It's then optimized directly against the observed seismic data. How?\n
\n
1. Use the MLP to generate a velocity map on a grid (by querying it at many points).\n
2. Simulate seismic waves through this map using a physics engine.\n
3. Compare the simulated echoes to the real, observed echoes.\n
4. Calculate how to adjust the MLP's internal parameters (weights) to make the simulated echoes better match the real ones.\n
The network itself *becomes* the inversion result; its weights encode the final velocity model.
\n This falls under the umbrella of Physics-Informed Neural Networks (PINNs) or coordinate-based representations. The crucial aspect is that the network learns *only* from the mismatch between simulated and observed data, guided by the physics of wave propagation embedded in the simulator. It requires no pre-existing library of velocity map examples for training.\n
\n\nAn MLP is one of the simplest types of \"deep learning\" networks. It consists of layers of artificial neurons. Information flows from an input layer, through one or more \"hidden\" layers, to an output layer. Each connection between neurons has a weight, and the network learns by adjusting these weights. Think of it as a series of adjustable knobs and switches that transform the input data into the desired output.
\n\nMore Rigorous Definition
\nA Multi-Layer Perceptron (MLP) is a class of feedforward artificial neural network (ANN). It consists of at least three layers of nodes: an input layer, one or more hidden layers, and an output layer. Except for the input nodes, each node is a neuron that uses a nonlinear activation function. MLPs utilize supervised learning with backpropagation for training. They are universal function approximators, meaning they can theoretically approximate any continuous function given enough hidden units and appropriate weights.
\nKey Contributions
\n- \n
- Demonstrates that representing the velocity field implicitly with a neural network can effectively mitigate the cycle-skipping problem that plagues traditional FWI. The inherent smoothness and flexibility of the neural network representation seem to help the optimization process avoid getting trapped in incorrect local minima, even when starting from a poor initial guess (e.g., a constant velocity). \n
- Offers a natural framework for uncertainty quantification. By incorporating techniques like Monte Carlo dropout during the optimization phase, one can obtain a distribution of slightly different velocity models consistent with the data. Analyzing the variance across these models can highlight which parts of the subsurface are well-constrained by the seismic data and which parts remain uncertain. \n
- Provides a flexible framework that could potentially incorporate various physical constraints or fuse different types of geophysical data more easily than grid-based methods. \n
Architecture
\n\n The core component is usually a simple Multi-Layer Perceptron (MLP). This network takes 2D (or 3D) spatial coordinates (x, z) as input and outputs a single scalar value representing the seismic velocity at that location. To enable the network to represent fine details and high-frequency variations in the velocity field, techniques like positional encoding (similar to those used in Transformers and NERFs) or using periodic activation functions (like sine, as in SIRENs - Sinusoidal Representation Networks) are often employed. This velocity-representing network is tightly coupled with a differentiable wave simulation engine. \"Differentiable\" means that we can automatically compute gradients (how the output changes with respect to input) through the entire simulation process, which is essential for optimizing the neural network's weights using standard backpropagation techniques.\n
\n\nPositional Encoding (Conceptual Introduction)
\nPositional encoding is a method used to help neural networks, especially coordinate-based ones, learn patterns that can change quickly with respect to position. Standard MLPs, which often rely on activation functions like ReLU, can struggle to capture high-frequency variations—think of abrupt changes in a physical field or sharp edges in an image. By transforming simple coordinates (like x or z) into richer representations, the network can more readily model small-scale or quickly changing features. This helps in tasks such as reconstructing detailed velocity profiles or capturing subtle variations in a signal.
\n\nPositional Encoding (Rigorous Details)
\nConcretely, positional encoding takes a low-dimensional coordinate vector\n x and maps it into a higher-dimensional space using periodic functions. A typical approach is to produce terms of the form\n sin(2kπx) and\n cos(2kπx) for various integer frequencies\n k. This set of sine and cosine functions acts like a basis that spans both low- and high-frequency components. When passed into an MLP, these encoded features counteract the network’s tendency—known as spectral bias—to focus on smoother, lower-frequency patterns, thereby enabling it to reconstruct signals with sharp gradients and detailed oscillations.
\n\nSIRENs (Conceptual Introduction)
\nSIRENs, or Sinusoidal Representation Networks, turn the usual practice of using piecewise-linear or smooth nonlinearities on its head by making the sine function their core activation. Where networks with ReLU activations might need many layers to approximate highly oscillatory signals, SIRENs capture such signals more naturally because the sine function itself is oscillatory. They excel at learning tasks like representing audio, images, or physical fields where fine-grained details and derivatives are essential.
\n\nSIRENs (Rigorous Details)
\nIn SIRENs, each layer computes outputs of the form sin(ω₀Wx + b), guided by a specific initialization scheme to ensure stable training. The sine activation’s periodic nature directly encodes oscillations into each layer, granting the network a built-in capacity for modeling complex patterns. This design also aids in computing derivatives of the learned representation, proving useful for physics-driven tasks that involve partial differential equations. As a result, SIRENs offer high-fidelity reconstruction of continuous signals and fields, often outperforming standard MLP-based approaches for problems requiring detailed, precise representations.
\n\nDifferentiable Simulator (Conceptual Introduction)
\nA computer program that simulates a physical process (like wave propagation) but is written in a way that allows automatic calculation of derivatives (gradients). Think of it as a simulator that not only tells you the result (simulated echoes) but also exactly how that result would change if you slightly tweaked any input parameter (like the velocity at one point in the model). This is crucial for training physics-informed models using gradient-based optimization.
\n\nDifferentiable Simulator/Solver (Rigorous Details)
\nA numerical solver for a physical system (e.g., a PDE solver for the wave equation) implemented within a framework that supports automatic differentiation (AD), such as PyTorch or TensorFlow. This allows gradients of the simulation output (e.g., synthetic seismograms) with respect to the input parameters (e.g., velocity model parameters, network weights) to be computed efficiently and accurately via backpropagation, enabling end-to-end gradient-based optimization of models that incorporate the simulator.
\nOptimization (Inversion) Procedure
\n\n Unlike supervised DL, IFWI doesn't have a separate \"training\" phase on a large dataset. Each inversion performed on a specific set of observed seismic data is its own optimization problem:\n
\n- \n
- Initialize the weights of the velocity-representing neural network (e.g., MLP) randomly. \n
- Define a grid of coordinates covering the desired subsurface model area. \n
- Generate the current velocity map by querying the neural network at all grid coordinates. \n
- Input this velocity map into the differentiable wave simulator, along with source/receiver locations, to produce simulated seismic data (echoes). \n
- Calculate a loss function comparing the simulated seismic data with the actual observed seismic data (e.g., L2 norm of the difference). \n
- Use automatic differentiation (provided by frameworks like PyTorch or TensorFlow) to compute the gradient of the loss with respect to the neural network's weights. This gradient indicates how to change the weights to reduce the data mismatch. \n
- Update the network's weights using an optimization algorithm (e.g., Adam, L-BFGS). \n
- Repeat steps 3-7 iteratively until the loss converges (i.e., the simulated data closely matches the observed data). \n
\n The final, optimized neural network is the result. It implicitly holds the converged velocity model.\n
\n\nEvaluation
\n\n IFWI was typically evaluated on complex synthetic benchmark models (like Marmousi or BP). The key success metric is its ability to recover the true velocity structure with high fidelity, especially starting from very simple or inaccurate initial models where traditional FWI methods often fail due to cycle-skipping. Resolution and accuracy are compared against ground truth and traditional FWI results.\n
\n\nReproducibility
\n\n Implementing IFWI requires integrating a neural network framework (like PyTorch) with a differentiable wave simulation library or code (e.g., DeepWave, Salvus Pytorch, or custom implementations using finite differences and autograd). While the specific code from the original authors may not be public, the core concepts draw from the active research areas of PINNs and neural implicit representations, where various tools and examples are becoming increasingly available within the scientific computing community.\n
\n\nConceptual Implementation (PyTorch): Coordinate-based Velocity Network
\nThis snippet focuses on defining the velocity network and outlining the optimization loop. A functional differentiable wave simulator is assumed to exist.
\n\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nimport torch.nn.functional as F # For loss calculation\n\n# --- Assumed Components (Replace with actual implementations) ---\n# 1. obs_data: Tensor containing the real observed seismic data.\n# Shape: e.g., [num_shots, num_receivers, num_timesteps]\n# 2. simulate_wave_autodiff(velocity_map, source_coords, receiver_coords, wave_params):\n# A *differentiable* function. Takes a velocity map tensor, source/receiver info,\n# and simulation parameters. Returns simulated seismic data tensor.\n# Crucially, gradients must flow from output data back to input velocity_map.\n# 3. coords_tensor: Tensor of (x, z) coordinates covering the model space.\n# Shape: [nx * nz, 2], normalized (e.g., to [-1, 1] or [0, 1]).\n# 4. nx, nz: Dimensions of the desired velocity grid output.\n# 5. source_coords, receiver_coords, wave_params: Simulation setup details.\n# 6. obs_data size for demonstration (replace with real data).\n\nclass VelocityFieldNN(nn.Module):\n \"\"\"MLP mapping (x, z) coordinates to velocity scalar.\"\"\"\n def __init__(self, input_dim=2, hidden_dim=256, output_dim=1, n_layers=6, activation=nn.ReLU()):\n super().__init__()\n layers = [nn.Linear(input_dim, hidden_dim), activation]\n for _ in range(n_layers - 1):\n layers.extend([nn.Linear(hidden_dim, hidden_dim), activation])\n layers.append(nn.Linear(hidden_dim, output_dim))\n self.network = nn.Sequential(*layers)\n\n def forward(self, coords):\n # coords: Tensor shape (N, 2)\n velocity_values = self.network(coords) # Shape (N, 1)\n return velocity_values\n\n# --- Setup for a Hypothetical Inversion ---\nnx, nz = 128, 128 # Example grid size\n# Generate normalized coordinates\nx_lin = torch.linspace(-1, 1, nx)\nz_lin = torch.linspace(-1, 1, nz)\ngrid_x, grid_z = torch.meshgrid(x_lin, z_lin, indexing='ij')\ncoords_tensor = torch.stack([grid_x.flatten(), grid_z.flatten()], dim=-1).float().requires_grad_(False)\n\ndummy_obs_data_shape = [1, 100, 1000] # Example: shots, receivers, time\nobs_data = torch.randn(dummy_obs_data_shape) # Replace with loading real data\n\nvel_nn = VelocityFieldNN(input_dim=2, hidden_dim=128, n_layers=4, activation=nn.Tanh()) # Tanh can work well\noptimizer = optim.Adam(vel_nn.parameters(), lr=5e-4)\nnum_iterations = 1000\n\nprint(\"Starting IFWI optimization loop...\")\nfor it in range(num_iterations):\n optimizer.zero_grad() \n\n # 1. Query the network at all grid coordinates -> velocity map\n velocity_values_flat = vel_nn(coords_tensor) # (nx*nz, 1)\n velocity_map = velocity_values_flat.view(nx, nz)\n\n # 2. Differentiable wave simulation placeholder\n sim_data = torch.rand_like(obs_data) * velocity_map.mean()\n\n # 3. Calculate the loss (data mismatch)\n loss = F.mse_loss(sim_data, obs_data)\n\n # 4. Backprop\n loss.backward()\n optimizer.step()\n\n if it % 50 == 0 or it == num_iterations - 1:\n print(f\"Iteration {it}/{num_iterations}, Loss: {loss.item():.8f}\")\n\nprint(\"Optimization finished.\")\n This conceptual code illustrates the optimization cycle central to IFWI. The critical dependency is the differentiable wave simulator that links the neural network's output (the velocity field) to the observable data, allowing gradient-based optimization of the network's parameters to minimize the data mismatch.
\n\n\n\n FWIGAN Podcast\n FWI-GAN: Physics-Informed Generative Adversarial Network (2023)\n
\nReference: Fangshu Yang and Jianwei Ma, J. Geophys. Res.: Solid Earth, 128(4), e2022JB025947, 2023.
\n\nOverview
\n\n FWI-GAN introduces another sophisticated AI technique, Generative Adversarial Networks (GANs), into the physics-informed FWI landscape. It cleverly frames the inversion problem as a competitive game between two neural networks, guided by wave physics, to learn the subsurface structure directly from observed data without needing paired examples.\n
\n\nFor an introduction to Generative Adversarial Networks, see the earlier accordion\n \"What are Deep Learning, U-Nets, and GANs in this context?\" in the Introduction section.
\n\n\nImagine a game between an \"artist\" (the Generator) and an \"art critic\" (the Discriminator or Critic).
\n\n
\n- The Generator attempts to create a realistic velocity map.
\n- A physics simulator predicts the seismic echoes (synthetic data) that would result from the Generator's map.
\n- The Critic is shown two sets of seismic data: the *real* observed echoes from the field survey, and the *synthetic* echoes generated from the Generator's map.
\n- The Critic's job is to learn to distinguish between the real and synthetic seismic data (\"real\" vs. \"fake\").
\n- The Generator's goal is to create velocity maps whose synthetic echoes are so realistic that they successfully fool the Critic into thinking they are real.
\nThrough this adversarial process, the Generator is pushed to produce velocity maps that accurately explain the observed seismic data, as judged by the physics simulator and the discerning Critic.
\n
\n A significant advantage of this approach is its unsupervised nature regarding velocity maps. It does not require a training dataset of known (seismic data, true velocity map) pairs. Instead, it learns directly from the single set of observed seismic data for the specific area being investigated.\n
\n\nUnlike supervised learning (with an answer key), unsupervised learning is like giving the AI a pile of unsorted data and asking it to find patterns or structure on its own, without any predefined correct answers. Examples include clustering data points into groups or learning to generate new data similar to the input data (like GANs).
\nMore Rigorous Definition
\nUnsupervised learning is a machine learning paradigm where the algorithm learns patterns from unlabeled data (X) without corresponding output variables (Y). The goal is typically to model the underlying structure or distribution in the data, for tasks such as dimensionality reduction, clustering, density estimation, or generative modeling. FWI-GAN is unsupervised with respect to the velocity models, as it doesn't need ground truth v maps for training; it learns by comparing distributions of observed and simulated *seismic data*.
\nKey Contributions
\n- \n
- Presents an unsupervised FWI method that embeds a physics simulator within a GAN framework, removing the need for labeled ground truth velocity maps during training/inversion. \n
- Demonstrates potential robustness against cycle-skipping and perhaps better handling of noise compared to methods relying solely on pixel-wise loss functions. GANs often optimize based on the overall distribution or \"realism\" of the data, which can be less sensitive to minor discrepancies. Variations like Wasserstein GAN (WGAN) with gradient penalty (GP) are often used for stability. \n
- Reportedly achieved high-quality inversion results on challenging benchmark models (e.g., Marmousi), comparing favorably against traditional FWI and some supervised DL methods. \n
Standard GANs can be tricky to train – sometimes the Generator and Discriminator stop learning effectively. WGAN is an improved version that uses a different mathematical way (the \"Wasserstein\" or \"Earth Mover's\" distance) to measure the difference between real and fake data distributions. This often makes training smoother and more stable.
\nThe Gradient Penalty (GP) is an additional technique used with WGAN. It helps ensure the Discriminator behaves nicely (doesn't change too abruptly), which further stabilizes the training process and prevents common GAN failure modes.
\n\nMore Rigorous Definition
\nWasserstein GAN (WGAN): A modification of the standard GAN framework that minimizes an approximation of the Wasserstein-1 distance (or Earth Mover's distance) between the real data distribution (Pr) and the generated data distribution (Pg). This distance metric provides smoother gradients than the Jensen-Shannon divergence used implicitly in standard GANs, leading to more stable training and better correlation between the loss value and sample quality. WGANs typically require the discriminator (called the \"critic\" in WGAN terminology) to be 1-Lipschitz continuous.
\nGradient Penalty (GP): A technique introduced in \"Improved Training of Wasserstein GANs\" (Gulrajani et al., 2017) to enforce the Lipschitz constraint on the critic in WGANs more effectively than the original weight clipping method. It adds a penalty term to the critic's loss function that encourages the gradient norm of the critic's output with respect to its input to be close to 1 for points sampled along straight lines between real and generated data points.
\nMethodology
\n\n The FWI-GAN system involves these core components:\n
\n- \n
- Generator Network (G): Typically a neural network (e.g., a CNN like U-Net) that takes a random noise vector (or potentially a low-resolution initial model) as input and outputs a candidate velocity map. \n
- Differentiable Physics Simulator (F): Same role as in IFWI; takes the velocity map from G and simulates the corresponding seismic data (
sim_data = F(G(z)), wherezis input noise). \n - Discriminator Network (D) / Critic: A neural network (often a CNN classifier) trained to distinguish between real and simulated data. It takes seismic data (either real
obs_dataor simulatedsim_data) as input and outputs a score indicating how \"real\" it believes the data is (or a score related to the Wasserstein distance in WGAN). \n - Adversarial Training Loop: The Generator and Discriminator are trained iteratively:\n
- \n
- Train Discriminator/Critic: Update D's weights to maximize its ability to correctly classify real data as real and simulated data as fake (or maximize the Wasserstein distance estimate). \n
- Train Generator: Update G's weights to minimize the Discriminator's output score for the simulated data (i.e., make
D(F(G(z)))indicate \"real\", or minimize the Wasserstein distance estimate from G's perspective). This means G learns to generate velocity maps that produce highly realistic seismic data, effectively \"fooling\" D. \n
\n
\n This alternating optimization process drives the Generator to produce velocity models that accurately explain the observed seismic data according to the embedded wave physics. Variations like Wasserstein GAN (WGAN) with gradient penalty are often used for more stable training.\n
\n\nResults
\n\n FWI-GAN demonstrated successful recovery of complex geological structures in standard benchmarks like the Marmousi model. It showed promise in handling strong velocity contrasts and reducing artifacts often seen in traditional methods, particularly when the starting model is poor. Its unsupervised nature makes it particularly attractive for real-world applications where accurate ground truth velocity models are rarely available.\n
\n\nOpen Source Availability
\n\n Yes! A significant advantage for reproducibility and further research is that the authors released their PyTorch code implementation. It utilizes the DeepWave library for the differentiable wave simulation. This allows other researchers to build upon, test, and adapt their method. (See reference section for link).\n
\n\n\n\n Big Data Impact Podcast\n Foundation Models and Parameter-Efficient Fine-Tuning (PEFT) with LoRA (2024)\n
\nReference: D. Maiti et al., arXiv:2402.19510, 2024.
\n (*Note: Podcast link relates to the importance of large datasets, key for training foundation models.*)\n\nOverview
\n\n This recent work imports a powerful concept from the broader AI world—foundation models—into the domain of FWI. The core idea is analogous to Large Language Models (LLMs) like GPT: pre-train a single, very large, and capable neural network on a massive and highly diverse dataset. For FWI, this means training on datasets like the comprehensive OpenFWI benchmark, encompassing many different geological styles. The goal is for this foundation model to learn generalizable representations of seismic wave phenomena and subsurface structures. Then, rather than training specialized models from scratch for each new seismic survey or geological target (which is computationally expensive and requires significant data), this powerful pre-trained model can be rapidly and efficiently adapted to the specific task at hand.\n
\n\nFor definitions of Foundation Models, PEFT, and LoRA, see the earlier accordion \n \"What are Foundation Models, PEFT, and LoRA?\" in the Recent Advances section.
\n\n\n\nThink of it as educating a highly experienced \"AI Geoscientist\" by having it study textbooks covering geology and wave physics from all around the globe (the pre-training on diverse data).
\nWhen faced with data from a specific, new region (the downstream task), you don't need to reteach the AI everything from scratch. Instead, you provide it with a small, targeted \"briefing note\" specific to that region. This adaptation process is done using Parameter-Efficient Fine-Tuning (PEFT) techniques. Low-Rank Adaptation (LoRA) is a prominent PEFT method that achieves this adaptation by adding and training only a tiny number of new parameters, leaving the vast knowledge base of the original large model untouched (frozen).
\n
Key Contributions
\n- \n
- Demonstrates the feasibility and effectiveness of pre-training large-scale foundation models (e.g., based on U-Net architectures) for FWI using diverse, multi-domain datasets like combinations of OpenFWI families. \n
- Shows that PEFT methods, specifically LoRA, can successfully adapt these large pre-trained FWI models to new, specific target datasets (e.g., a single geological style not heavily represented in pre-training) by training only a minuscule fraction (e.g., <1%) of the total model parameters. \n
- Achieves performance comparable to, and sometimes even better than, fully fine-tuning the entire large model on the target task, especially for out-of-distribution generalization (performing well on data types different from the pre-training data). This comes with significant reductions in computational cost (GPU memory, training time) and storage requirements for adapted models. LoRA seems to act as a regularizer, preventing the model from overfitting to the smaller target dataset during adaptation. \n
Imagine training an AI to recognize pictures of cats and dogs. In-distribution generalization means it can recognize new pictures of cats and dogs it hasn't seen before. Out-of-distribution (OOD) generalization is harder – it means the AI can still perform reasonably well when shown something different, perhaps a picture of a fox or a wolf, or maybe a cartoon cat instead of a photo. In FWI, OOD generalization means the model performs well on seismic data from a geological setting significantly different from those it was primarily trained on.
\nMore Rigorous Definition
\nOut-of-Distribution (OOD) generalization refers to a model's ability to perform well on data drawn from a different distribution than the training data distribution. While standard generalization assesses performance on unseen data from the *same* distribution (test set), OOD generalization tests robustness to shifts in data characteristics, such as different noise levels, acquisition parameters, or, in FWI, substantially different geological environments or structures not well-represented in the training set. Achieving good OOD performance is a key challenge in deploying ML models in real-world scenarios.
\nArchitecture & LoRA Method
\n\n The foundation model itself is often a large CNN, such as a U-Net, pre-trained on the task of mapping seismic data to velocity models using a large dataset. The magic happens during the adaptation phase with LoRA. Instead of retraining the existing weight matrices (e.g., in convolutional or linear layers) of the pre-trained model, LoRA introduces pairs of small, new, trainable matrices alongside the frozen original weights.\n
\n\n Specifically, for an original weight matrix W₀ ∈ ℝd×k, LoRA adds a low-rank update computed as BA, where A ∈ ℝr×k and B ∈ ℝd×r. Here, r is the \"rank\" of the adaptation, and crucially, r ≪ min(d, k). The forward pass becomes h = W₀x + α(BA)x, where x is the input, h is the output, and α is a scaling factor (often set as α/r in practice). During fine-tuning, only the matrices A and B are trained, while the large W₀ remains frozen. Since r is small (e.g., 4, 8, or 16), the number of trainable parameters in A and B (r*(d+k)) is much smaller than in W₀ (d*k). For convolutional layers, a similar adaptation is applied using low-rank convolutional kernels.\n
\nThink of the original large weight matrix W₀ as containing a vast amount of general knowledge. When adapting to a new task, we assume the necessary *changes* or *adjustments* (ΔW) to this knowledge are relatively simple or structured, not requiring completely new, complex knowledge.
\nA low-rank matrix is mathematically \"simpler\" than a full-rank matrix of the same size. It essentially captures information that can be represented by combining a smaller number of basis vectors. By representing the update ΔW as the product of two skinny matrices (B and A), LoRA hypothesizes that the task-specific adaptation lies in a low-dimensional subspace. This allows it to learn the necessary adjustments with far fewer parameters than changing the entire W₀ matrix.
\nThe rank r controls the complexity (and number of parameters) of the adaptation. A very small r means a very simple adaptation, while a larger r allows for more complex adjustments but requires training more parameters.
\nTraining Setup
\n- \n
- Pre-training: Train a large FWI model (e.g., a deep U-Net) on a diverse collection of datasets (e.g., 10 combined subsets from OpenFWI, as done in the paper) using a standard supervised loss (like MSE). This step is computationally intensive but done only once. \n
- Adaptation (PEFT with LoRA): For a specific downstream task (e.g., inverting data from only the 'Fault' dataset family, or potentially a new real dataset):\n
- \n
- Load the pre-trained foundation model weights and freeze them (set `requires_grad=False`). \n
- Inject LoRA adapter layers (pairs of low-rank matrices/kernels) next to the chosen layers of the frozen model (e.g., convolutional layers in the U-Net). Initialize matrix A randomly and matrix B with zeros. \n
- Train *only* the parameters within these newly added LoRA adapter layers using the smaller target dataset and the same loss function. \n
\n
Reproducibility
\n\n The methodology relies on standard deep learning frameworks (like PyTorch) and the LoRA technique. Libraries facilitating PEFT, such as Hugging Face's `peft` library, provide readily usable implementations of LoRA for various layer types (Linear, Conv2d). Implementing this approach requires having a pre-trained FWI model and integrating these PEFT library components to perform the LoRA-based fine-tuning. The authors provide significant detail on their model architecture, pre-training setup, and LoRA configuration in their paper, aiding reproducibility.\n
\n\nConceptual Implementation (PyTorch): LoRA for a Convolutional Layer
\nThis snippet shows how a standard nn.Conv2d layer can be wrapped with LoRA adapters using a hypothetical LoRAConv2d class (implementations exist in libraries like peft).
\nimport torch\nimport torch.nn as nn\nimport math\n\n# Note: This is a simplified illustrative implementation. Libraries like `peft` offer robust versions.\nclass LoRAConv2d(nn.Module):\n \"\"\"Applies Low-Rank Adaptation to a nn.Conv2d layer.\"\"\"\n def __init__(\n self,\n base_conv: nn.Conv2d,\n rank: int = 8, # The rank 'r' of the adaptation\n alpha: float = 8.0, # Scaling factor (often = rank)\n dropout: float = 0.0, # Dropout for LoRA layers\n ):\n super().__init__()\n assert rank > 0\n self.base_conv = base_conv\n self.rank = rank\n self.alpha = alpha\n # Practical scaling used in many implementations\n self.scaling = self.alpha / self.rank\n\n # Freeze the original convolutional layer\n self.base_conv.weight.requires_grad = False\n if self.base_conv.bias is not None:\n self.base_conv.bias.requires_grad = False\n\n in_channels = base_conv.in_channels\n out_channels = base_conv.out_channels\n kernel_size = base_conv.kernel_size\n stride = base_conv.stride\n padding = base_conv.padding\n dilation = base_conv.dilation\n groups = base_conv.groups\n\n # LoRA A: Convolution down to rank `r`. Mimics the original conv's spatial operation.\n self.lora_A = nn.Conv2d(\n in_channels, self.rank * groups, kernel_size=kernel_size,\n stride=stride, padding=padding, dilation=dilation, groups=groups, bias=False\n )\n\n # LoRA B: Pointwise (1x1) conv up from rank `r` to original out_channels\n self.lora_B = nn.Conv2d(\n self.rank * groups, out_channels, kernel_size=1, stride=1, padding=0, groups=groups, bias=False\n )\n\n self.lora_dropout = nn.Dropout(p=dropout)\n\n # Initialize weights: A with Kaiming uniform, B with zeros\n nn.init.kaiming_uniform_(self.lora_A.weight, a=math.sqrt(5))\n nn.init.zeros_(self.lora_B.weight)\n\n def forward(self, x: torch.Tensor) -> torch.Tensor:\n base_output = self.base_conv(x)\n\n x_dropout = self.lora_dropout(x)\n lora_A_output = self.lora_A(x_dropout)\n lora_B_output = self.lora_B(lora_A_output)\n lora_update = lora_B_output * self.scaling\n\n return base_output + lora_update\n\n def train(self, mode: bool = True):\n super().train(mode)\n self.base_conv.train(False) # Keep base layer frozen\n\n def state_dict(self, *args, destination=None, prefix='', keep_vars=False):\n if destination is None:\n destination = {}\n destination[prefix + 'lora_A.weight'] = self.lora_A.weight.data if not keep_vars else self.lora_A.weight\n destination[prefix + 'lora_B.weight'] = self.lora_B.weight.data if not keep_vars else self.lora_B.weight\n return destination\n\n def load_state_dict(self, state_dict, strict: bool = True):\n prefix = self._get_prefix_from_load_state_dict_migration(prefix)\n missing_keys, unexpected_keys = super().load_state_dict(state_dict, strict=False)\n\n # Filter out expected missing keys for base_conv \n expected_missing = {\n prefix + 'base_conv.weight',\n prefix + ('base_conv.bias' if self.base_conv.bias is not None else '')\n }\n expected_missing = {k for k in expected_missing if k}\n\n final_missing_keys = [k for k in missing_keys if k not in expected_missing]\n final_unexpected_keys = unexpected_keys\n\n if strict and (final_missing_keys or final_unexpected_keys):\n raise RuntimeError(\n f\"Error loading state_dict for {self.__class__.__name__}: \"\n f\"missing keys {final_missing_keys}, unexpected keys {final_unexpected_keys}\"\n )\n return torch.nn.modules.module._IncompatibleKeys(final_missing_keys, final_unexpected_keys)\n\n\n# --- Example Usage ---\noriginal_conv_layer = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, padding=1)\nfor param in original_conv_layer.parameters():\n param.requires_grad = False\n\nlora_adapted_layer = LoRAConv2d(original_conv_layer, rank=4, alpha=4.0, dropout=0.1)\n\nprint(\"Trainable parameters in LoRA-adapted layer:\")\ntrainable_params = []\nfor name, param in lora_adapted_layer.named_parameters():\n if param.requires_grad:\n trainable_params.append(name)\nprint(trainable_params)\n\ninput_tensor = torch.randn(4, 32, 128, 128)\noutput_tensor = lora_adapted_layer(input_tensor)\nprint(f\"\\nOutput shape: {output_tensor.shape}\")\n \n This code defines a LoRAConv2d module that encapsulates a frozen base convolutional layer and adds the trainable low-rank adapter pathway (lora_A, lora_B). By strategically replacing standard layers in a large pre-trained network with these LoRA-enhanced versions, researchers can perform parameter-efficient fine-tuning, adapting powerful foundation models to new tasks with minimal computational overhead.\n
Conclusion: Towards Smarter, Faster, and More Reliable FWI
\n\n\n The period from 2022 to 2025 has witnessed remarkable progress in applying deep learning to the complex challenges of Full Waveform Inversion. The field has matured beyond initial feasibility demonstrations to tackle fundamental issues like generalization to unseen geology, accurate recovery of fine details, robust handling of noisy data, and the principled integration of wave physics.\n
\n- \n
- \n More Sophisticated Architectures: We've seen the adoption of powerful architectures like Transformers (e.g., SVIT) capable of capturing global wavefield characteristics, complemented by specialized multi-stage networks (e.g., VIF-Net) designed to enhance specific aspects like interface sharpness.\n \n
- \n Deeper Physics Integration: Physics-informed approaches are gaining significant traction. Methods like IFWI (using neural networks as implicit representations of the velocity field) and FWI-GAN (leveraging adversarial learning constrained by physics simulation) reduce the dependence on vast labeled datasets and show promise in overcoming long-standing FWI challenges like cycle-skipping.\n \n
- \n Leveraging Data and Efficiency: The availability of large-scale, diverse benchmark datasets (OpenFWI) has been instrumental in driving progress and enabling rigorous model comparison. Concurrently, the emergence of FWI foundation models combined with parameter-efficient adaptation techniques like LoRA points towards a future where extremely powerful, general-purpose AI models can be rapidly and economically tailored for specific seismic surveys and geological targets, boosting both robustness and practical applicability.\n \n
There's no single \"best\" method; the optimal choice depends on the specific problem, available data, computational resources, and goals:
\n- \n
- For speed after training and if large labeled datasets are available: Supervised methods like U-Nets (e.g., InversionNet) or Transformers (e.g., SVIT) are strong candidates. Foundation models with PEFT offer efficiency if a good pre-trained model exists. \n
- If labeled velocity maps are scarce or unavailable, or robustness to cycle-skipping is paramount: Physics-informed methods like IFWI or FWI-GAN are attractive, as they learn directly from observed seismic data and physical principles. \n
- For resolving sharp interfaces: Specialized architectures like VIF-Net might outperform general-purpose models. \n
- For handling diverse data types or complex relationships: Transformers might offer advantages due to their global attention mechanism. \n
Often, hybrid approaches combining strengths (e.g., using a DL model for a good starting guess for traditional FWI, or incorporating physical constraints into a supervised network) show significant promise.
\n\n The trajectory suggests that the most fruitful path forward likely involves synergistic combinations: harnessing the pattern-recognition capabilities of advanced deep learning architectures, grounding them in the fundamental laws of wave physics, training them on diverse large-scale data, and utilizing efficient adaptation strategies for deployment. While significant challenges undoubtedly remain—particularly concerning the application to highly complex, noisy, and large-scale 3D field data—the pace of innovation is rapid. AI-driven FWI is steadily evolving into an indispensable tool in the geoscientist's toolkit for imaging and understanding the Earth's intricate subsurface.\n
\n\nSources & Further Reading
\n- \n {/* Merged and slightly deduplicated list */}\n
- Deng, Chen, et al. (2022). OpenFWI: Large-Scale Multi-Structural Benchmark Datasets for Seismic Full Waveform Inversion. Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track. (GitHub) \n
- Maiti, D., et al. (2024). Parameter Efficient Fine-Tuning Approach for Deep Learning Based Seismic Full Waveform Inversion. arXiv preprint arXiv:2402.19510. (arXiv) \n
- Li, Y., & Yang, B. (2023). Seismic velocity inversion transformer. GEOPHYSICS, 88(4), R513–R533. (SEG Library) \n
- Wang, Z., et al. (2024). Deep learning for interface completion constrained full-waveform inversion based on velocity interface fusion network. Computers & Geosciences, 174, 105834. (ScienceDirect) \n
- Sun, J., & Innanen, K. A. (2023). Implicit Full-Waveform Inversion With Deep Neural Representation. Journal of Geophysical Research: Solid Earth, 128(4), e2022JB025900. (Preprint: arXiv, Journal: JGR) \n
- Yang, F., & Ma, J. (2023). FWIGAN: Full-Waveform Inversion via a Physics-Informed Generative Adversarial Network. Journal of Geophysical Research: Solid Earth, 128(4), e2022JB025947. (JGR, Code: GitHub) \n
- Jin, B., et al. (2024). An empirical study of large-scale data-driven seismic full-waveform inversion. Scientific Reports, 14(1), 20034. (Nature Scientific Reports) \n
- Li, J., et al. (2023). TransInver: High-resolution 3-D seismic full-waveform inversion based on the self-attention mechanism. GEOPHYSICS, 88(6), R799-R817. (Advanced 3D Transformer FWI Example) (SEG Library) \n
- Chen, X., et al. (2023). DiffusionVel: Generative Diffusion Model based Seismic Velocity Inversion. arXiv preprint arXiv:2309.17309. (Diffusion models approach) (arXiv) \n
- Vaswani, A., et al. (2017). Attention is All You Need. Advances in Neural Information Processing Systems (NeurIPS). (Original Transformer Paper) (arXiv) \n
- Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. MICCAI. (Original U-Net Paper) (arXiv) \n
- Goodfellow, I. et al. (2014). Generative Adversarial Nets. Advances in Neural Information Processing Systems (NeurIPS). (Original GAN Paper) (arXiv) \n
- Arjovsky, M., Chintala, S., & Bottou, L. (2017). Wasserstein GAN. arXiv preprint arXiv:1701.07875. (arXiv) \n
- Gulrajani, I., et al. (2017). Improved Training of Wasserstein GANs. Advances in Neural Information Processing Systems (NeurIPS). (WGAN-GP Paper) (arXiv) \n
- Hu, E. J., et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. International Conference on Learning Representations (ICLR). (Original LoRA Paper) (arXiv) \n
- Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378, 686-707. (Key PINN Paper) (Journal) \n
- Virieux, J., & Operto, S. (2009). An overview of full-waveform inversion in exploration geophysics. Geophysics, 74(6), WCC1-WCC26. (Classic FWI Review) (SEG Library) \n
Previous Competitions Review (Insights for FWI)
\n\n Learning from past data science competitions that involved complex signals, inverse problems, or geophysical data can provide valuable strategies for the current FWI challenge. Here's a summary of insights, ordered by relevance:\n
\n\nFull Waveform Inversion Benchmarks (OpenFWI, 2021)
\n- \n
- Directly relevant: Provided benchmark datasets (synthetic seismic + velocity models) for ML-based FWI, similar to this competition's goal. \n
- Common Architecture: U-Net (CNN) architectures proved effective for the image-to-image task of mapping seismic data to velocity maps. \n
- Hybrid Models: Combining traditional physics-based inversion (e.g., for an initial model) with ML refinement often outperformed purely data-driven approaches. \n
- Multi-Scale Training: Mimicking classical FWI, training ML models progressively from low to high frequencies helped stability and accuracy. \n
- Data Diversity & Augmentation: Showcased the need for diverse geological scenarios in training data for generalization. Augmentation (noise, shifts, trace drops) improved robustness. \n
SEG AI Challenge: Seismic Facies Identification (2022)
\n- \n
- Relevant Domain: Focused on interpreting seismic images (post-stack data) to identify geological facies (segmentation). \n
- Dominant Architecture: U-Net CNN variants were heavily used and successful for pixel-wise classification/segmentation of seismic images. \n
- Ensembling: Top solutions frequently used ensembles of multiple U-Nets to improve robustness and generalization. \n
- Augmentation: Heavy data augmentation (noise, phase shifts, distortions) was crucial for handling real-world seismic complexities. \n
- Post-processing: Applying geological constraints (smoothing, filtering small patches) post-prediction improved the realism of results. \n
- Class Imbalance: Techniques to handle imbalanced facies data (e.g., weighted loss) might be relevant for FWI (e.g., weighting deeper features). \n
TGS Salt Identification Challenge (Kaggle 2018)
\n- \n
- Relevant Domain: Similar to SEG Facies, focused on segmenting salt bodies in seismic amplitude images. \n
- Dominant Architecture: U-Net CNNs were the core of all top solutions, confirming their suitability for seismic image interpretation. \n
- Key Techniques: Introduced useful tricks like Lovász hinge loss (optimizing IoU directly) and test-time augmentation (averaging predictions from augmented inputs) for better performance. \n
- Post-processing: Emphasized cleaning up predicted boundaries, analogous to enforcing smoothness or known constraints in FWI velocity models. \n
FastMRI Image Reconstruction Challenge (Facebook NYU 2020)
\n- \n
- Relevant Problem Type: An inverse problem (image reconstruction from under-sampled frequency-domain data), analogous to FWI recovering velocity from wave data. \n
- Physics-Guided AI: Winning solutions used physics-guided neural networks, embedding NNs within iterative physics-based reconstruction algorithms (unrolled optimization). \n
- Hybrid Methods: Showcased success by alternating between applying the physical forward model (MRI simulation) and a learned (CNN) correction step. This suggests potential for similar hybrid physics/ML cycles in FWI. \n
- Data Consistency: Enforcing that the output prediction, when run through the forward model, matches the original measurements improved results. \n
- Architecture: U-Nets were also used here for mapping sensor data to images. \n
Kaggle Ventilator Pressure Prediction (2021)
\n- \n
- Relevant Problem Type: Inverse problem (inferring internal pressure from time-series signals) using simulated, physics-based data. \n
- Sequence Modeling: Recurrent Neural Networks (RNNs, specifically LSTMs) were effective for capturing temporal dependencies in the waveform-like data. \n
- Physics Integration: Blending ML with physics knowledge (e.g., custom loss functions penalizing unphysical predictions, using physics equations) improved generalization. \n
- Ensembling: Ensembles of different models (e.g., LSTMs + Gradient Boosting) or across folds were crucial for top performance. \n
- Robustness: Techniques like robust loss functions (Huber loss) helped handle outliers or abrupt signal changes. \n
Radiant Earth Tropical Storm Wind Speed (DrivenData 2021)
\n- \n
- Relevant Data Type: Spatio-temporal signal processing (sequences of satellite images) to predict a physical quantity (wind speed). \n
- Combined Architectures: CNNs (for spatial features per image) combined with sequence models like LSTMs (for temporal evolution) were highly successful. \n
- Heavy Ensembling: The winning solution ensembled 51 diverse models, highlighting the power of model diversity for complex signal interpretation. \n
- Heavy Augmentation: Extensive augmentation (rotation, flipping, time-shifts, scaling) was critical for model invariance and generalization. \n
- Domain Knowledge: Although no explicit physics, successful models implicitly learned visual cues known to meteorologists (Dvorak technique). \n
LANL Earthquake Prediction (Kaggle 2019)
\n- \n
- Relevant Data Type: Directly used seismic time-series data (acoustic emissions). \n
- Signal Processing Features: Top solutions often relied on engineered features capturing signal statistics and frequency content changes, fed into boosting models. \n
- CNNs for Waveforms: 1D CNNs applied directly to raw waveforms were also effective. \n
- Frequency Analysis: Highlighted the importance of analyzing how frequency content changes over time, potentially relevant for analyzing different phases or features in FWI waveforms. \n
Weather4Cast Spatiotemporal Forecasting (NeurIPS 2022)
\n- \n
- Relevant Task: Forecasting the evolution of a physical field (precipitation radar) over time. \n
- Advanced Sequence Models: Top teams used Convolutional LSTMs and Transformers to model spatio-temporal dynamics. \n
- Auxiliary Input: Incorporating hints about motion (like optical flow) improved predictions, suggesting that providing initial velocity guesses or travel times might help FWI models. \n
- Transformers/Attention: Suggests potential for using attention mechanisms to focus FWI models on specific parts of the waveform data (e.g., first arrivals vs. reflections). \n