add result plots and model section

documentation/whitepaper.html

@@ -52,10 +52,10 @@ <h2 id="overview">Overview</h2>
 <div class="scholmd-float-caption"><figcaption><span class="scholmd-caption-head"><span class="scholmd-caption-head-prefix">Figure</span><span class="scholmd-caption-head-label">1</span></span><span class="scholmd-caption-text">Parallel batch job for modeling seismic data. Jobs are submitted to a pool using Batch Shipyard and individual tasks either run on separate nodes or as distributed workloads on small clusters. Each task is responsible for modeling the data of a given source location and stores the generated data in Blob storage upon completion.</span></figcaption></div>
 </figure>
 <h2 id="seismic-modeling-with-devito">Seismic modeling with Devito</h2>
-<p>Seismic modeling is used in geophysical exploration and monitoring to numerically predict data that is recorded in seismic surveys. Such surveys involve a seismic source being repeateldy fired within the target area, which causes waves to propagate through the subsurface. Where physical properties of the subsurface such as wave speed or density change, waves are reflected and travel back to the surface, where they are recorded by an array of seismic sensors (Figure 2a). The sensors record relative pressure changes in the water as a function of time, sensor number and the source location (Figure 2b). Realistic surveys involve thousands of seismic source locations and data has to be modeled for each location individually by numerically solving the corresponding wave equation.</p>
-<p>In this workflow, wave equations are discretized and solved with Devito, a Python package for finite-difference modeling and inversion. Devito’s application programming interface allows implementing wave equations as symbolic Python expressions that closely resemble the mathematical notation:</p>
+<p>Seismic modeling is used in geophysical exploration and monitoring to numerically predict data that is recorded in seismic surveys. Such surveys involve a seismic source being repeateldy fired within the target area, which causes waves to propagate through the subsurface. Where physical properties of the subsurface such as wave speed or density change, waves are reflected and travel back to the surface, where they are recorded by an array of seismic receivers (Figure 2a). The receivers record relative pressure changes in the water as a function of time, sensor number and the source location (Figure 2b). Industry-scale surveys involve thousands of seismic source locations and data has to be modeled for each location individually by numerically solving the corresponding wave equation.</p>
+<p>In this workflow, wave equations are discretized and solved with Devito, a Python package for finite-difference modeling and inversion. Devito’s application programming interface allows implementing wave equations as symbolic Python expressions that closely resemble the mathematical notation. For example, the acoustic isotropic wave equation with constant density is implemented as:</p>
 <pre><code>pde = model.m * u.dt2 - u.laplace + model.damp*u.dt</code></pre>
-<p>where <code>model.m</code> is a symbolic expression for the acoustic wave speed, <code>u</code> is the discretized acoustic wavefield and <code>u.dt2</code> and <code>u.lapace</code> are short-hand expressions for finite difference stencils of second temporal and spatial derivatives. The last expression implements absorbing boundaries to mimic wave propagation in an unbounded domain. During runtime, Devito automatically generates optimized C code for solving this equation from the symbolic expression, using its internal compiler.</p>
+<p>where <code>model.m</code> is a symbolic expression for the acoustic wave speed, <code>u</code> is the discretized acoustic wavefield and <code>u.dt2</code> and <code>u.lapace</code> are short-hand expressions for finite difference stencils of second temporal and spatial derivatives. The last expression implements absorbing boundaries to mimic wave propagation in an unbounded domain. Similarily, sources and receivers can be symbolically defined as well and added to this expression, making it possible to leverage Devito for real-world applications in exploration seismology. During runtime, Devito automatically generates optimized C code for solving this equation from the symbolic expression, using its internal compiler. Devito’s automated performance optimizations include equation clustering, FLOP-reduction optimization, SIMD-vectorization, loop-blocking, as well as the introduction of shared and/or distributed memory parallelism (i.e. multi-threading and domain decomposition).</p>
 <p>This tutorial demonstrates how use Azure Batch to model seismic data for a large number of source locations as an embarassingly parallel workload, making it possible to easily scale to relevant problem sizes as encountered in real-world scenarios. The software is deployed to a pool of parallel workers as a Docker container, which contains Devito implementations of the tilted transverse-isotropic (TTI) wave equation. This tutorial covers how to build the required Docker containers from scratch and how to manage the parallel pool and job submissions with Batch Shipyard.</p>
 <figure class="scholmd-float scholmd-figure" id="f2">
 <div class="scholmd-float-content"><figure class="scholmd-subfig" style="display: inline-block; width: 50%">
@@ -65,15 +65,27 @@ <h2 id="seismic-modeling-with-devito">Seismic modeling with Devito</h2>
 <img src="figures/shots.png" />
 <div class="scholmd-float-subcaption"><figcaption><span class="scholmd-caption-text">(b)</span></figcaption></div>
 </figure></div>
-<div class="scholmd-float-caption"><figcaption><span class="scholmd-caption-head"><span class="scholmd-caption-head-prefix">Figure</span><span class="scholmd-caption-head-label">2</span></span><span class="scholmd-caption-text">A marine seismic survey, in which a vessel tows a source and generates seismic waves that travel though the subsurface. At geological interfaces, wave are reflected back to the surface, where they are recorded by a set of seismic sensors (a). The sensors record pressure changes in the water as a function of time (vertical axis) and sensor number (horizontal axis) and the experiment is repeated for many source locations (b).</span></figcaption></div>
+<div class="scholmd-float-caption"><figcaption><span class="scholmd-caption-head"><span class="scholmd-caption-head-prefix">Figure</span><span class="scholmd-caption-head-label">2</span></span><span class="scholmd-caption-text">A marine seismic survey, in which a vessel tows a source and generates seismic waves that travel though the subsurface. At geological interfaces, wave are reflected back to the surface, where they are recorded by a set of seismic receivers (a). The receivers record pressure changes in the water as a function of time (vertical axis) and receiver number (horizontal axis) and the experiment is repeated for many source locations (b).</span></figcaption></div>
+</figure>
+<h2 id="experimental-set-up">Experimental set up</h2>
+<p>The model used in this tutorial is a 3D synthetic TTI model derived from the 3D SEG Salt and Overthrust models, with dimensions of 3.325 x 10 x 10 km. The model consists of six physical parameter, namely P-wave velocity, density, Thomsen parameters epsilon and delta, as well as tilt and azimuth of the anisotropy symmetry axes. The model is discretized with a 12.5 m cartesian grid, which results in 347 x 881 x 881 grid points, including 80 grid points for absorbing boundaries in each dimension. The seismic data is modeled for 2 seconds using a Ricker wavelet with 15 Hertz peak frequency, with data being recorded by 1,500 receivers that are randomly distributed along the ocean floor (Figure 3a). The source vessel fires the seismic source on a dense regular grid, consisting of 799 x 799 source locations (638,401 in total, Figure 3b). For modeling, source-receiver reciprocity is applied, which means that sources and receivers are interchangeable and data can be sorted into 1,500 shot records with 638,401 receivers each, which reduces the number of indivudal PDE solves to 1,500. For the numerical wave propagation, this tutorial provides Devito implementations of the pseudo-acoustic TTI wave equation with an 8th order finite-difference discretization in space and a 2nd order stencil in time.</p>
+<figure class="scholmd-float scholmd-figure" id="f3">
+<div class="scholmd-float-content"><figure class="scholmd-subfig" style="display: inline-block; width: 50%">
+<img src="figures/receiver_grid.png" />
+<div class="scholmd-float-subcaption"><figcaption><span class="scholmd-caption-text">(a)</span></figcaption></div>
+</figure><figure class="scholmd-subfig" style="display: inline-block; width: 50%">
+<img src="figures/source_grid.png" />
+<div class="scholmd-float-subcaption"><figcaption><span class="scholmd-caption-text">(b)</span></figcaption></div>
+</figure></div>
+<div class="scholmd-float-caption"><figcaption><span class="scholmd-caption-head"><span class="scholmd-caption-head-prefix">Figure</span><span class="scholmd-caption-head-label">3</span></span><span class="scholmd-caption-text">Randomized receiver grid with 1,500 ocean-bottom nodes (a) and dense source grid with 799 x 799 shot locations (b). For modeling, source-receiver reciprocity is applied, resuling in 1,500 shot location with a dense receiver grid.</span></figcaption></div>
 </figure>
 <h2 id="prerequisites">Prerequisites</h2>
 <ul>
 <li><p>Install Azure CLI, Batch Shipyard</p></li>
 <li><p>Optional: Batch Explorer</p></li>
 <li><p>(Or run everything from George’s docker container. Will it be made public?)</p></li>
 </ul>
-<h2 id="set-up">Set up</h2>
+<h2 id="azure-set-up">Azure Set up</h2>
 <h3 id="docker-container">Docker container</h3>
 <ul>
 <li><p>Optional: Build Docker containers and upload to Dockerhub</p></li>
@@ -86,7 +98,11 @@ <h3 id="upload-user-files">Upload user files</h3>
 </ul>
 <h3 id="batch-shipyard-configuration">Batch shipyard configuration</h3>
 <ul>
-<li>Fill out config files (<code>credentials.yaml</code>, <code>config.yaml</code>, <code>pool.yaml</code>, <code>jobs.yaml</code>)</li>
+<li><p>Computational set up: <code>HBv2</code> nodes, 120 CPU cores, 480 GB memory.</p></li>
+<li><p>Use one full node per source location. Each modeling task has 30 MPI ranks with 4 OMP threads each (i.e. we use all 120 cores per node).</p></li>
+<li><p>Pool size: 100 nodes.</p></li>
+<li><p>Job: Model xx of the 1,500 shots.</p></li>
+<li><p>How to fill out config files (<code>credentials.yaml</code>, <code>config.yaml</code>, <code>pool.yaml</code>, <code>jobs.yaml</code>)</p></li>
 </ul>
 <h2 id="submit-and-monitor-a-job">Submit and monitor a job</h2>
 <ul>
@@ -102,6 +118,16 @@ <h2 id="performance-analysis-and-results">Performance analysis and results</h2>
 <li><p>Plot performance, FLOPs</p></li>
 <li><p>Download + plot seismic data</p></li>
 </ul>
+<figure class="scholmd-float scholmd-figure" id="f4">
+<div class="scholmd-float-content"><figure class="scholmd-subfig" style="display: inline-block; width: 100%">
+<img src="figures/shots3D_notic.png" />
+<div class="scholmd-float-subcaption"><figcaption><span class="scholmd-caption-text">(a)</span></figcaption></div>
+</figure><br /><figure class="scholmd-subfig" style="display: inline-block; width: 100%">
+<img src="figures/slices3D_notic.png" />
+<div class="scholmd-float-subcaption"><figcaption><span class="scholmd-caption-text">(b)</span></figcaption></div>
+</figure></div>
+<div class="scholmd-float-caption"><figcaption><span class="scholmd-caption-head"><span class="scholmd-caption-head-prefix">Figure</span><span class="scholmd-caption-head-label">4</span></span><span class="scholmd-caption-text">Crossline samples of a 3D modeled seismic shot record (a) and time slices at increasing recording times (b).</span></figcaption></div>
+</figure>
 <h2 id="clean-up">Clean up</h2>
 <ul>
 <li><p>Kill job: <code>./shipyard jobs del -v</code></p></li>

documentation/whitepaper.md

@@ -48,23 +48,34 @@ The following sections provide a brief overview of seismic modeling with Devito
 
 ## Seismic modeling with Devito
 
-Seismic modeling is used in geophysical exploration and monitoring to numerically predict data that is recorded in seismic surveys. Such surveys involve a seismic source being repeateldy fired within the target area, which causes waves to propagate through the subsurface. Where physical properties of the subsurface such as wave speed or density change, waves are reflected and travel back to the surface, where they are recorded by an array of seismic sensors (Figure 2a). The sensors record relative pressure changes in the water as a function of time, sensor number and the source location (Figure 2b). Realistic surveys involve thousands of seismic source locations and data has to be modeled for each location individually by numerically solving the corresponding wave equation.
+Seismic modeling is used in geophysical exploration and monitoring to numerically predict data that is recorded in seismic surveys. Such surveys involve a seismic source being repeateldy fired within the target area, which causes waves to propagate through the subsurface. Where physical properties of the subsurface such as wave speed or density change, waves are reflected and travel back to the surface, where they are recorded by an array of seismic receivers (Figure 2a). The receivers record relative pressure changes in the water as a function of time, sensor number and the source location (Figure 2b). Industry-scale surveys involve thousands of seismic source locations and data has to be modeled for each location individually by numerically solving the corresponding wave equation.
 
-In this workflow, wave equations are discretized and solved with Devito, a Python package for finite-difference modeling and inversion. Devito's application programming interface allows implementing wave equations as symbolic Python expressions that closely resemble the mathematical notation:
+In this workflow, wave equations are discretized and solved with Devito, a Python package for finite-difference modeling and inversion. Devito's application programming interface allows implementing wave equations as symbolic Python expressions that closely resemble the mathematical notation. For example, the acoustic isotropic wave equation with constant density is implemented as:
 
 ```
 pde = model.m * u.dt2 - u.laplace + model.damp*u.dt
 ```
 
-where `model.m` is a symbolic expression for the acoustic wave speed, `u` is the discretized acoustic wavefield and `u.dt2` and `u.lapace` are short-hand expressions for finite difference stencils of second temporal and spatial derivatives. The last expression implements absorbing boundaries to mimic wave propagation in an unbounded domain. During runtime, Devito automatically generates optimized C code for solving this equation from the symbolic expression, using its internal compiler.
+where `model.m` is a symbolic expression for the acoustic wave speed, `u` is the discretized acoustic wavefield and `u.dt2` and `u.lapace` are short-hand expressions for finite difference stencils of second temporal and spatial derivatives. The last expression implements absorbing boundaries to mimic wave propagation in an unbounded domain. Similarily, sources and receivers can be symbolically defined as well and added to this expression, making it possible to leverage Devito for real-world applications in exploration seismology. During runtime, Devito automatically generates optimized C code for solving this equation from the symbolic expression, using its internal compiler. Devito's automated performance optimizations include equation clustering, FLOP-reduction optimization, SIMD-vectorization, loop-blocking, as well as the introduction of shared and/or distributed memory parallelism (i.e. multi-threading and domain decomposition).
 
 This tutorial demonstrates how use Azure Batch to model seismic data for a large number of source locations as an embarassingly parallel workload, making it possible to easily scale to relevant problem sizes as encountered in real-world scenarios. The software is deployed to a pool of parallel workers as a Docker container, which contains Devito implementations of the tilted transverse-isotropic (TTI) wave equation. This tutorial covers how to build the required Docker containers from scratch and how to manage the parallel pool and job submissions with Batch Shipyard.
 
 
 #### Figure: {#f2}
 ![(a)](figures/acquisition.png){width=50%}
 ![(b)](figures/shots.png){width=50%}
-: A marine seismic survey, in which a vessel tows a source and generates seismic waves that travel though the subsurface. At geological interfaces, wave are reflected back to the surface, where they are recorded by a set of seismic sensors (a). The sensors record pressure changes in the water as a function of time (vertical axis) and sensor number (horizontal axis) and the experiment is repeated for many source locations (b).
+: A marine seismic survey, in which a vessel tows a source and generates seismic waves that travel though the subsurface. At geological interfaces, wave are reflected back to the surface, where they are recorded by a set of seismic receivers (a). The receivers record pressure changes in the water as a function of time (vertical axis) and receiver number (horizontal axis) and the experiment is repeated for many source locations (b).
+
+
+## Experimental set up
+
+The model used in this tutorial is a 3D synthetic TTI model derived from the 3D SEG Salt and Overthrust models, with dimensions of 3.325 x 10 x 10 km. The model consists of six physical parameter, namely P-wave velocity, density, Thomsen parameters epsilon and delta, as well as tilt and azimuth of the anisotropy symmetry axes. The model is discretized with a 12.5 m cartesian grid, which results in 347 x 881 x 881 grid points, including 80 grid points for absorbing boundaries in each dimension. The seismic data is modeled for 2 seconds using a Ricker wavelet with 15 Hertz peak frequency, with data being recorded by 1,500 receivers that are randomly distributed along the ocean floor (Figure 3a). The source vessel fires the seismic source on a dense regular grid, consisting of 799 x 799 source locations (638,401 in total, Figure 3b). For modeling, source-receiver reciprocity is applied, which means that sources and receivers are interchangeable and data can be sorted into 1,500 shot records with 638,401 receivers each, which reduces the number of indivudal PDE solves to 1,500. For the numerical wave propagation, this tutorial provides Devito implementations of the pseudo-acoustic TTI wave equation with an 8th order finite-difference discretization in space and a 2nd order stencil in time.
+
+
+#### Figure: {#f3}
+![(a)](figures/receiver_grid.png){width=50%}
+![(b)](figures/source_grid.png){width=50%}
+: Randomized receiver grid with 1,500 ocean-bottom nodes (a) and dense source grid with 799 x 799 shot locations (b). For modeling, source-receiver reciprocity is applied, resuling in 1,500 shot location with a dense receiver grid.
 
 
 ## Prerequisites
@@ -76,8 +87,7 @@ This tutorial demonstrates how use Azure Batch to model seismic data for a large
 - (Or run everything from George's docker container. Will it be made public?)
 
 
-## Set up
-
+## Azure Set up
 
 ### Docker container
 
@@ -95,7 +105,15 @@ This tutorial demonstrates how use Azure Batch to model seismic data for a large
 
 ### Batch shipyard configuration
 
-- Fill out config files (`credentials.yaml`, `config.yaml`, `pool.yaml`, `jobs.yaml`)
+- Computational set up: `HBv2` nodes, 120 CPU cores, 480 GB memory. 
+
+- Use one full node per source location. Each modeling task has 30 MPI ranks with 4 OMP threads each (i.e. we use all 120 cores per node).
+
+- Pool size: 100 nodes.
+
+- Job: Model xx of the 1,500 shots.
+
+- How to fill out config files (`credentials.yaml`, `config.yaml`, `pool.yaml`, `jobs.yaml`)
 
 
 ## Submit and monitor a job
@@ -119,6 +137,11 @@ This tutorial demonstrates how use Azure Batch to model seismic data for a large
 
 - Download + plot seismic data
 
+#### Figure: {#f4}
+![(a)](figures/shots3D_notic.png){width=100%} \
+![(b)](figures/slices3D_notic.png){width=100%}
+: Crossline samples of a 3D modeled seismic shot record (a) and time slices at increasing recording times (b).
+
 
 ## Clean up