PECOS SUZERAIN: A SPECTRAL, COMPRESSIBLE DNS CODE

blahQ
Predictive Engineering and Computational Sciences Rhys Ulerich

Why Direct Numerical Simulation? Suzerain Code Requirements

PECOS calibration and validation requirements: • Target platforms are Hera-like clusters and uBGL-like systems
• UQ-driven approach requires significant amounts of calibration and validation data • Simulations will be both communication- and computation-intensive
• Anticipate and mitigate relatively small amounts of per-core memory
• To succeed at QoI prediction, turbulence modeling effort needs high quality data for effective calibration
• Scalability to tens of thousands of cores to allow both:
• Supersonic hero runs at previously unattained Reynolds numbers
DNS is uniquely suited to meet these data needs: • Quick turnaround on small and medium simulations during parametric studies
• DNS aids calibration and validation efforts by augmenting experimental data
• High throughput IO to speed both simulation and analysis
• DNS provides data with measurement uncertainties only limited by available compute resources
• Use existing, best-of-breed third party numerics libraries wherever possible (e.g. MKL/ESSL).
• DNS samples portions of the turbulence parameter space not explored in physical laboratories
• Tracing and performance quantification baked in using PECOS’ HPC Toolkit.
• DNS supports turbulence model improvement efforts by being the highest fidelity “model”
• Design for code extensibility and portability
• Complete documentation including:
• Function-level API documentation using Doxygen
Direct Numerical Simulation Goals • Simulation repeatability steps for all milestone simulations
• Comprehensive modeling and numerics document

• Support PECOS calibration and validation process

• Advance understanding of supersonic boundary layer physics Suzerain Functional Components
• Especially in presence of pressure gradients and ablation products
• Characterize what ablative conditions may sustain turbulence
• Create database of compressible turbulence statistics for modeler use • Fourier representation in the streamwise
• 100% simulation result traceability and repeatability and spanwise directions:
• Parallel, dealiased FFT performed using
scalable pencil decompositions
• Uses P3DFFT library from San Diego
Simulation Model Equations Supercomputing Center [Pek09]
• B-spline representation in the
wall-normal direction:
• Compressible Navier-Stokes using density (ρ), momentum (m
~ ), and total energy (e) for state • Provide favorable spectral properties [KMJ01]
• Builds atop GSL’s de Boor-based
• Perfect gas with a constant γ , constant Pr, power law viscosity, and Stokes hypothesis
routines [Gou09]
• SMR91 low-storage, hybrid
∂ implicit/explicit Runge-Kutta time
ρ = −∇ · m
~ stepping [SMR91]
∂t  
∂ m~ 1 ~ ↔ • Explicit operator used for expensive,
~
~ = −∇ ·
m ⊗m ~ − ∇p + ∇ · τ + ρf~b
~ communication-heavy nonlinear terms
∂t ρ Re • Implicit operator provides reasonable time
↔
∂ em~ 1 pm
~ 1 ~ τm~ steps despite acoustic modes
~
e = −∇ · + ~ ~ ~
∇ · µ∇T − ∇ · + ∇· + f~b · m
~ + ρqb
∂t ρ Re Pr (γ − 1) ρ Re ρ • BLAS-heavy implementation uses MKL/ESSL
during implicit computations
where
  • Physical space computations use Eigen
~ ·m
m ~ C++ serial tensor library
p = (γ − 1) e −
2ρ • Modern, self-documenting restart files
p contain repeatable scenario parameters
T =γ
ρ • Restart file infrastructure called ESIO shared
µ = Tβ with PSDNS PetaApps project
• Metadata-laden, HDF5 file format tuned for
2
λ=− µ high IO throughput
3
 T !   • C99/C++03 implementation replacing
↔ m
~
~ + ∇ ~ m
~ ~ · m
~ ↔
τ =µ ∇ +λ ∇ I legacy vector code
ρ ρ ρ

Test-Driven Development
Simulation Plans and Progression
• Automated verification test suite developed in tandem with code
• Functional test scope
• Simulations in support of PECOS’ compressible RANS UQ efforts
• Independent tests for each functional component
• Supersonic isothermal-wall channel flow [CKM95] • Unit test goal of 80% block coverage
• Supersonic boundary layer over a flat plate: • Both convergence and rate-of-convergence tests included
• Incorporating slow-growth terms to simulate successive streamwise stations [GMSW00] • System test scope
• Incorporating transpired wall boundary conditions to simulate idealized ablative conditions [VMN08] • Check successful integration of all functional components
• Under a variety of favorable pressure gradient conditions • E.g. propagation and decay of eigenfunctions
• Potentially adding an active scalar field • E.g. bitwise result agreement using in-memory versus restart file
• Mapping the reentry heat shield regions which can sustain turbulence • Performance test scope
• Baseline using scenario parameters from reentry stagnation point • Prevent compute performance regression
• Scenario-space study using PECOS reentry problem parameters
• Prevent communication performance regression
• Sensitivity to wall transpiration parameters
• Sensitivity to active scalar parameters • Build and test system employs GNU Autotools and Boost Test

Periodic Formulation for a Spatially Growing Boundary Layer References

[CKM95] G. N. Coleman, J. Kim, and R. D. Moser.

A numerical study of turbulent supersonic isothermal-wall channel flow.
Journal of Fluid Mechanics Digital Archive, 305(-1):159–183, 1995.
[GMSW00] Stephen E. Guarini, Robert D. Moser, Karim Shariff, and Alan Wray.
Direct numerical simulation of a supersonic turbulent boundary layer at mach 2.5.
Journal of Fluid Mechanics, 414:1–33, 2000.
[Gou09] Brian Gough.
Gnu Scientific Library Reference Manual.
• Boundary layer growth breaks periodicity in the streamwise direction x Network Theory Ltd, 3rd edition, 2009.
• Using a non-Fourier streamwise basis comes with penalties: [KMJ01] Wai Y. Kwok, Robert D. Moser, and Javier Jiménez.
• Spoils spectral derivative and other computation properties desirable in DNS A critical evaluation of the resolution properties of b-spline and compact finite difference
• Forces specifying time-dependent incoming and outgoing boundary conditions
methods.
• Spalart recast the boundary layer problem in a streamwise periodic form [Spa88]:
Journal of Computational Physics, 174(2):510–551, December 2001.
• Observed that layer thickness and energy have a slowly varying signal in the streamwise direction
• Defined wall-normal coordinate η so boundary & viscous sublayer thickness independent of x, i.e. [Pek09] Dmitry Pekurovsky.
u(x, η, z, t) = U (x, η) + A(x, η)up(x, η, z, t) P3dfft library.
where http://www.sdsc.edu/us/resources/p3dfft/, 2006–2009.
E1/2
[SMR91] Philippe R. Spalart, Robert D. Moser, and Michael M. Rogers.
D
2
U (x, η) := hu(x, η, z, t)iz,t , U 0(x, η, z, t) := u(x, η, z, t) − U (x, η), A(x, η) ∝ U 0(x, η, z, t)
z,t
Spectral methods for the navier-stokes equations with one infinite and two periodic directions.
• Assumed length and timescales of up(x, η, z, t) vary insignificantly compared to those of U (x, η), A(x, η) J. Comput. Phys., 96(2):297–324, 1991.
D E1/2
• Using these assumptions up is periodic, has zero mean in z, t, and u0p2 , and is independent of x
z,t [Spa88] Philippe R. Spalart.
• Derivatives of u now have multiple contributions because
Direct simulation of a turbulent boundary layer up to re = 1410.
∂u ∂U ∂A ∂up Journal of Fluid Mechanics, 187:61–98, 1988.
= + up + A
∂x |∂x {z∂x } ∂x}
[VMN08] Prem Venugopal, Robert D. Moser, and Fady M. Najjar.
| {z
slow terms fast term
• Applied scaling and multiscale analysis to add “slow growth terms” to Navier-Stokes Direct numerical simulation of turbulence in injection-driven plane channel flows.
• Spalart needed slow growth information to close the new terms [Spa88]: Physics of Fluids, 20(10), 2008.
• Chose to simulate a sequence of ”stations” along the boundary layer
• Each stations’ slow growth details fixed by the preceding upstream station
• In incompressible case, only mean-velocity and Reynolds stress profile information required from upstream
• Guarini et al. extended Spalart’s technique to the compressible case [GMSW00]

Institute for Computational Engineering and Sciences (ICES) rhys@ices.utexas.edu http://pecos.ices.utexas.edu