Skip to content

PAALS Tutorial 2015

Jump to bottom
Glen Hansen edited this page Feb 8, 2019 · 3 revisions

Exercise 1 - Parallel Mesh Generation

Suspension upright from the RPI Formula Hybrid race car (left) and Simmetrix SimModeler mesh generation and problem definition GUI (right).
  • Demonstrates
  • Source code
    • /projects/FASTMath/ATPESC-2015/install/pumi/src/generate.cc
  • Executing the example
    • Execution time: ~6 minutes
    • Number of cores: 128
    • Number of nodes: 16
    • Setup
mkdir $HOME/paals
cd $HOME/paals
cp -r /projects/FASTMath/ATPESC-2015/examples/paals/ex1/ .
cd ex1
  • Run the mesh generation tools and write scorec mesh files and paraview files.
qsub -A ATPESC2015 -q ATPESC2015 -O generate -n 16 --mode c8 --proccount 128 -t 10 ./generate upright.smd 13M
  • Examining results
    • Follow the output as the example runs tail -f generate.output, hold Ctrl then press c to exit.
    • Compare generate.output to out/out.txt.
    • The partition used by parallel mesh generation is balanced for its procedures. The image below shows 16 of the 128 parts.
      Parts 0-15 of initial 128 part partitioning of 13 million element mesh of upright model.
      In this figure parts appear to be fragmented (small disconnected components) and have a high part surface area - undesirable qualities for most analysis codes.
    • In Exercise 3 the mesh will be re-partitioned and the subsequent change to partition qualities computed.
    • Optional - Follow Exercise 2 to visualize the partition.

Exercise 2 - Optional - Mesh Visualization

  • Download ParaView
    • Download ParaView
    • Transfer the *vtu files to your machine
    • Load the mesh
      • Select the folder icon in the top left then browse to your home directory and select the pvtu file.
      • Click 'Apply' to render.
      • Select 'Surface With Edges' as shown.
        Enabling edge rendering in Paraview.
      • Select 'apf_parts' to color the mesh elements by their part id.
        Enabling part id coloring in Paraview.
      • Click 'Apply' to render.

Exercise 3 - Partitioning

Partition of 13 million element mesh of upright model from 128 parts (left; parts 0-15 shown) to 512 parts (middle and right; parts 0-63 shown) using Zoltan's interface to ParMETIS.
* Demonstrates * SCOREC API interfacing with Zoltan to control mesh partitioning * SCOREC ParMA diffusive partition improvement * Software * [SCOREC] * [Zoltan] * [ParMA] * Algorithms * [Recursive Inertial Bisection (RIB)] * [Unstructured mesh based diffusive improvement (ParMA)] * Source code * /projects/FASTMath/ATPESC-2015/install/pumi/src/ptnParma_nothread.cc * Executing the example * Execution time: ~5 mins * Number of cores: 512 * Number of nodes: 16 * Setup 
cd $HOME/paals
cp -r /projects/FASTMath/ATPESC-2015/examples/paals/ex3 .
cd ex3
tar xzf 13M.tar.gz
  • Run the partitioning tools and write mesh (*smb) files.
    • The arguments are model inputMesh outputMesh partitionFactor partitionMethod partitionApproach doLocalPartitioning 
qsub -A ATPESC2015 -q ATPESC2015 -O ptnParma -n 16 --mode c32 --proccount 512 -t 20 ./ptnParma_nothread upright.smd 13M/ 13M512p/ 4 rib reptn 0
  • Examining results
    • Compare ptnParma.output to out/out.txt.
    • Partition quality statistics are output for the initial, RIB, final ParMA partition; respectively marked with 'Initial', 'afterSplit', and 'final'. After RIB the number of disconnected components (disconnected <max avg>) increases, indicating degraded part connectivity, but the vertex and element imbalance (the first and forth value on the line entity imbalance <v e f r>) has decreased from 130%(2.30) to 34%(1.34) and 175%(2.75) to perfectly balanced(1.00), respectively. After ParMA improvement the average number of disconnected components per part is reduced from 4.8 to 0.316, entity imbalances are at most 6%(1.06), and the total number of vertices in the mesh (the first value on the line weighted vtx <tot max min avg>) is reduced by 0.6%.
    • Bonus question ( a cookie will be mailed to your home address if you can answer this... maybe ). How can the total number of vertices change after partitioning? Hint: think about the common/shared boundary of the parts.
    • Optional - Run ptnParma_nothread with the partitionMethod set to 'pmetis' and examine the results. How much longer did partitioning take (look for output lines 'planned Zoltan split factor' and 'elapsed migration time')? How does the partition quality compare to RIB?
    • Optional - Generate *vtu files via the render tool using the command mkdir vtu; qsub -A ATPESC2015 -q ATPESC2015 -O render -n 16 --mode c32 --proccount 512 -t 20 ./render upright.smd 13M512p/ vtu/13M512p and visualize them following the steps described in Exercise 2. Note the differences in shape and connectedness between the initial partition and the RIB+ParMA partition.

Exercise 4 - Thermomechanical response of a heated can

Heating of the can drives a decomposition process that pressurizes the can.

  • Background: The above figure shows the can near thermal equilibrium at 700 K. The can starts at room temperature and is heated in a fire. As the heating proceeds, material within the can decomposes and pressurizes the can. The thin wall deforms under pressure and stresses the weld at the top, where the can wall is welded to the lid.

  • Question: Do stresses in the weld reach the point of causing the weld to fail and vent the contents of the can into the atmosphere?

  • Problem Description

    • Can geometry
    • Initial conditions:
      • Uniform temperature of 300 K
      • Zero stress state, all displacements set to zero, zero traction and pressure
    • Boundary conditions:
      • Top of lid: Zero x,y, and z displacement
      • Center of can: Symmetry condition of zero axial displacement
      • Can bottom: Temperature starts at 300 K and ramps up to 800 K
      • Can pressure: Begins at 0 and stops at 160 MPa - developed by heat driven chemical decomposition of can contents
    • Linear elastic mechanics (thermo-elasticity)
      • Elastic Modulus: 170.3 GPa
        • Linear temperature coefficient: -0.01
      • Poisson's Ratio: 0.32
      • Thermal Conductivity: 45.0
      • Density: 7920
      • Thermal transient coefficient: 1.0
      • Thermal expansion coefficient: 1.0e-05
      • Heat capacity: 500.0

  • Demonstrates

    • PAALS adaptive loop driven by complex thermoelastic physics
    • Superconvergent patch recovery based error indicator driven adaptation, based on Cauchy stresses (Zienkiewicz-Zhu)
    • Software
    • Features
      • General mesh adaptation
      • SPR based error estimation

  • Example data files

    • Executable: /projects/FASTMath/ATPESC-2015/examples/paals/ex4/AlbanyT
    • Input files: /projects/FASTMath/ATPESC-2015/examples/paals/ex4/
    • Output files: /projects/FASTMath/ATPESC-2015/examples/paals/ex4/out/

  • Executing the example

    • Execution time: 30 mins
    • Number of cores: 64
    • Number of nodes: 4
    • Setup:
mkdir $HOME/paals
mkdir $HOME/paals/ex4
cd $HOME/paals
cp /projects/FASTMath/ATPESC-2015/examples/paals/ex4/mesh.tar.gz ex4
ln -s /projects/FASTMath/ATPESC-2015/install/Albany/bin/AlbanyT ex4
cd ex4
tar xf mesh.tar.gz
  • Run PAALS
qsub -A ATPESC2015 -q ATPESC2015 -O albany -n 4 -t 59 --mode c16 ./AlbanyT MechanicsWithTemperatureT.xml
  • Examining results
Heating and pressurization drives an evolution of stress in the root of the weld.
  • The above shows the final time value of the can deformation sequence, with the "warp based on displacements" function in Paraview. We are also plotting the projected nodal value of Cauchy stress at the indicated node over time in the diagram shown to the right. Note that in a mechanics calculation, the Cauchy stress is an integration point quantity, which is intrinsic to the finite element being examined. Most visualization packages will show such data as an element variable, which produces uneven coloration in the image. Here, we use L2 projection to cast the integration point values of Cauchy stress as node field values, which are rendered in Paraview.
Cauchy stress evolution within the can as the decompositio process increases the internal pressure.
  • The movie above shows the time evolution of the wall displacement (again "warp" is active in Paraview), with the surface colored base on L2 projected Cauchy stress.
Cauchy stress concentration near the weld root.
  • The above is a static closeup of the Cauchy stress near the root of the weld. The weld will fail if the stress near an imperfection is high enough to initiate a fracture in the weld region.
Temperature profile in the can near during the heating process.

  • The temperature profile shows the heating of the can from the initial condition of 300 K. The temperature increase drives a chemical decomposition process that pressurizes the can and deforms the can wall. The elevated temperature decreases the elastic modulus of the material in the weld which makes it less resistant to the deformation process that is underway. However, it also makes the material less brittle; the overall effect of the elevated temperature is hard to understand intuitively. Predicting what will actually happen requires experimentation or multiphysics analysis like we have just undertaken here. Mesh adaptation allows us to manage the spatial discretization error of the simulation as it evolves.

  • The Albany multiphysics code is open source, and is available at http://SNLComputation.github.io/Albany

Additional References

  • Thermomechanical response of a heated can derived from

    • V. Romero, A. Black, N. Breivik, G. Orient, J. Suo-Anttila, B. Antoun, A. Dodd, Advanced UQ and V&V Procedures applied to Thermal-Mechanical Response and Weld Failure in Heated Pressurizing Canisters, 2015 SAE World Congress April 21-23, Detroit, MI, SAND2015-3005C

  • SCOREC tools

    • M. Zhou, O. Sahni, T. Xie, M.S. Shephard and K.E. Jansen, Unstructured Mesh Partition Improvement for Implicit Finite Element at Extreme Scale, Journal of Supercomputing, 59(3): 1218-1228, 2012. DOI 10.1007s11227-010-0521-0
    • M. Zhou, T. Xie, S. Seol, M.S. Shephard, O. Sahni and K.E. Jansen, Tools to Support Mesh Adaptation on Massively Parallel Computers, Engineering with Computers, 28(3):287-301, 2012. DOI: 10.1007s00366-011-0218-x
    • M. Zhou, O. Sahni, M.S. Shephard, K.D. Devine and K.E. Jansen, Controlling unstructured mesh partitions for massively parallel simulations, SIAM J. Sci. Comp., 32(6):3201-3227, 2010. DOI: 10.1137090777323
    • M. Zhou, O. Sahni, H.J. Kim, C.A. Figueroa, C.A. Taylor, M.S. Shephard, and K.E. Jansen, Cardiovascular Flow Simulation at Extreme Scale, Computational Mechanics, 46:71-82, 2010. DOI: 10.1007s00466-009-0450-z

  • Mesh data and geometry interactions

    • Seol, E.S. and Shephard, M.S., Efficient distributed mesh data structure for parallel automated adaptive analysis, Engineering with Computers, 22(3-4):197-213, 2006, DOI: 10.1007s00366-006-0048-4
    • Beall, M.W., Walsh, J. and Shephard, M.S, A comparison of techniques for geometry access related to mesh generation, Engineering with Computers, 20(3):210-221, 2004, DOI: 10.1007s00366-004-0289-z.
    • Beall, M.W. and Shephard, M.S., A general topology-based mesh data structure, Int. J. Numer. Meth. Engng., 40(9):1573-1596, 1997, DOI: 10.1002(SICI)1097-0207(19970515)40:9<1573::AID-NME128>3.0.CO;2-9.

  • Adaptivity

    • "Aleksandr Ovcharenko, Parallel Anisotropic Mesh Adaptation with Boundary Layers, Ph.D. Dissertation, RPI, 2012":http://www.scorec.rpi.edu/REPORTS/2012-20.pdf
    • Q. Lu, M.S. Shephard, S. Tendulkar and M.W. Beall, Parallel Curved Mesh Adaptation for Large Scale High-Order Finite Element Simulations, Proc. 21 Roundtable, Springer, NY, pp. 419-436, 2012, DOI 10.1007978-3-642-33573-0.
    • A. Ovcharenko, K. Chitale, O. Sahni, K.E. Jansen and M.S. Shephard, S. Tendulkar and M.W. Beall, Parallel Adaptive Boundary Layer Meshing for CFD Analysis, Proc. 21st International Meshing Roundtable, Springer, NY, pp. 437-455, 2012, DOI 10.1007978-3-642-33573-0
    • X.-J. Luo, M.S. Shephard, L.-Q. Lee and C. Ng, Moving Curved Mesh Adaption for Higher Order Finite Element Simulations, Engineering with Computers, 27(1):41-50, 2011. DOI: 10.1007/s00366-010-0179-5
    • O. Sahni, X.J. Luo, K.E. Jansen, M.S. Shephard, Curved Boundary Layer Meshing for Adaptive Viscous Flow Simulations, Finite Elements in Analysis and Design, 46:132-139, 2010. DOI: 10.1007/s00366-008-0095-0
    • Alauzet, F., Li, X., Seol, E.S. and Shephard, M.S., Parallel Anisotropic 3D Mesh Adaptation by Mesh Modification, Engineering with Computers, 21(3):247-258, 2006, DOI: 10.1007s00366-005-0009-3
    • Li, X., Shephard, M.S. and Beall, M.W., 3-D Anisotropic Mesh Adaptation by Mesh Modifications, Comp. Meth. Appl. Mech. Engng., 194(48-49):4915-4950, 2005, doi:10.1016/j.cma.2004.11.019
    • Li, X., Shephard, M.S. and Beall, M.W., Accounting for curved domains in mesh adaptation, International Journal for Numerical Methods in Engineering, 58:246-276, 2003, DOI: 10.1002/nme.772

  • Albany

    • M. Gee, C. Siefert, J. Hu, R. Tuminaro, and M. Sala. ML 5.0 Smoothed Aggregation Users Guide. Technical Report SAND2006-2649, Sandia National Laboratories, 2006. http://trilinos.sandia.gov/packages/ml/mlguide5.pdf
    • Qiushi Chen, Jakob T. Ostien, Glen Hansen. Development of a Used Fuel Cladding Damage Model Incorporating Circumferential and Radial Hydride Responses. Journal of Nuclear Materials, 447(1-3):292-303, 2014. http://dx.doi.org/10.1016/j.jnucmat.2014.01.001
    • Michael A. Heroux, Roscoe A. Bartlett, Vicki E. Howle, Robert J. Hoekstra, Jonathan J. Hu, Tamara G. Kolda, Richard B. Lehoucq, Kevin R. Long, Roger P. Pawlowski, Eric T. Phipps, Andrew G. Salinger, Heidi K. Thornquist, Ray S. Tuminaro, James M. Willenbring, Alan Williams, and Kendall S. Stanley. An Overview of the Trilinos Package. ACM Trans. Math. Softw., 31(3):397–423, 2005. http://trilinos.sandia.gov
    • Roger P. Pawlowski, Eric T. Phipps, and Andrew G. Salinger. Automating Embedded Analysis Capabilities and Managing Software Complexity in Multiphysics Simulation, Part I: Template-based Generic Programming. Scientific Programming, 20(2):197–219, 2012. http://dx.doi.org/10.3233/SPR-2012-0350
    • Roger P. Pawlowski, Eric T. Phipps, Andrew G. Salinger, Steven J. Owen, Christopher M. Siefert, and Matthew L. Staten. Automating Embedded Analysis Capabilities and Managing Software Complexity in Multiphysics Simulation, Part II: Application to Partial Differential Equations. Scientific Programming, 20(3):327–345, 2012. http://dx.doi.org/10.3233/SPR-2012-0351
    • Eric Phipps. A Path Forward to Embedded Sensitivity Analysis, Uncertainty Quantification and Optimization. https://cfwebprod.sandia.gov/cfdocs/CompResearch/docs/Phipps_Embedded_SA_UQ_Opt_web.pdf
    • Eric Phipps and Roger Pawlowski. Efficient Expression Templates for Operator Overloading-based Automatic Differentiation. Preprint, 2012. http://arxiv.org/abs/1205.3506v1
    • Eric Phipps, H. Carter Edwards, Jonathan Hu, and Jakob T. Ostien. Exploring Emerging Manycore Architectures for Uncertainty Quantification through Embedded Stochastic Galerkin Methods. International Journal of Computer Mathematics, 91(4):707-729, 2014. http://www.tandfonline.com/doi/abs/10.1080/00207160.2013.840722
    • A. Salinger et al. Albany website. http://SNLComputation.github.io/Albany

Exercise 1

Exercise 2

Exercise 3

Exercise 4

Additional References

Clone this wiki locally