Executable Mode (Running real cases)

The directory structure is as follows:

.myCase
├── runPeregrine.py        # symlink to runPeregrine.py
├── Archive                # Folder to write archive results (*.h5, *.xmf)
├── Restart                # Folder to read/write restarts (q.h5, q.xmf)
├── Grid                   # Folder to read/write grid (g.h5, g.xmf)
├── Input                  # Folder for auxiliary inputs (profiles/, coproc.py, ...)
└── peregrine.yaml         # PEREGRINE config file (see /src/peregrinepy/files/configFile.py)

The grid file g.h5 is self-describing: it carries the BASE grid as it came from the mesher (coordinates as (nk, nj, ni) datasets, block connectivity in GridPro notation under /connectivity) and any number of decompositions of it (under /decompositions/{nProcs}x{ranksPerNode}, one per machine layout). A decomposition is a cut table (base block + node index ranges per piece), the pieces’ connectivity, and a rank per piece; pieces read their coordinates as hyperslabs of the base blocks, and the base grid never changes. g.xmf is only a ParaView view of the grid.

Restart and archive files are numbered by BASE block and written as (nk-1, nj-1, ni-1) cell datasets by hyperslab, so a run under one decomposition restarts under any other (different rank count, different cuts) without interpolation. Trace points (Input/tracePoints.npy) are (baseNblki, i, j, k) in base block cell indices.

Boundary Conditions

Boundary condition families live in peregrine.yaml under a top-level bcFams key, each family being {bcType, bcVals}. See templates.

Decomposition (cutting + load balancing)

Decompose the base grid for a machine layout (ranks and ranks per node). A cut plane must run through every block of its sheet (the family of parallel planes a conforming topology links together; a GridPro grid has a few dozen), cuts on different sheets commute, and a block’s piece count is the product over its three sheets of (cuts + 1). So the cheapest set of sheet cuts that brings every piece under the cap is solved exactly as a small integer program (-objective pieces for fewest new blocks, faces for least new halo area). The pieces are then placed over nodes first (METIS, minimizing network traffic) and ranks within each node (shared memory), and polished until every rank is within -tol of the mean cell count. Communication between blocks on the same rank is free. -granularity is the cap: pieces per rank the cutter aims for (omit it to place the base blocks uncut; 1 cuts only what cannot fit a rank; several values print the frontier). The result is verified for conformance and stored in the grid file:

utilities/decompose.py ./Grid -numProcs 384 -ranksPerNode 4 -granularity 1 1.5 2
utilities/decompose.py ./Grid -numProcs 384 -ranksPerNode 4 -granularity 1.5
utilities/verifyGrid.py ./Grid     # base grid + every stored decomposition

At startup, PEREGRINE detects the ranks per node from the MPI shared-memory split (override with io.ranksPerNode), picks the stored decomposition for the launch size (any layout for that rank count if the exact one is missing), and assumes one whole block per rank if none exists.

The step graph

A time step is a graph of kernel launches and halo exchanges (src/peregrinepy/_graph.py): every node declares the arrays it reads and writes and how far its stencil reaches, exchanges leave their variable valid only in the interior until they finish, and the executor runs each consumer on the core (interior at the needed depth) while the exchange is in flight and on the skin after, bit-identically to the sequential order. The integrators are whole-step graphs, so the state exchange of one stage overlaps the advective fluxes of the next and the gradient exchange overlaps the diffusive fluxes.

Kernel launches

Every cell kernel runs as one launch over all the rank’s blocks (a blockTable, see src/compute/blockTable_.hpp), so the launch count does not depend on the decomposition; only boundary conditions run per face. Tables cover a region of every block (all, core, or skin), and core plus skin tile every kernel range, which is how the interior cell state is computed while halos are in flight. Each table carries the execution-space instance it launches on (pgkokkos.partitionSpace(n)), so on asynchronous backends separate tables overlap. Bind OpenMP threads (OMP_PROC_BIND=spread OMP_PLACES=threads, OMP_WAIT_POLICY=active for many small blocks); runPeregrine.py warns when they are unset.

Chemistry

The Cantera mechanism yaml is the chemistry input: thermochem.mechanism names it (input dir, working dir, the PEREGRINE database, or a path) and thermochem.chemistry: true turns its reactions on. PEREGRINE parses the file itself (species composition, NASA7 thermo, Lennard-Jones transport; elementary, three-body, Lindemann and Troe falloff reactions with efficiencies, orders and unit conversion) into flat kinetics tables on thtrdat, and on the generic build one table-driven kernel evaluates any mechanism from them. Under the jit the mechanism is instead unrolled into straight-line source (caseKinetics.hpp in the cache: literal rate constants, explicit products, one block per reaction) and compiled with the case, which is about twice as fast as the table lookups. Nothing is checked in or run by hand per mechanism; a new mechanism is a new yaml file. thermochem.mechanism may still be a list of species names drawn from the PEREGRINE species library for non-reacting cases.

Finite-rate chemistry stays fully explicit. At every Runge-Kutta stage the source term is subcycled from the stage state with forward-Euler substeps, each the largest that keeps every mass fraction and the temperature non-negative, over a horizon fixed by the tableau (the stage’s largest Euler stretch: the full step for SSP RK2/RK3, half of it for the four-stage SSP scheme). The stage receives the reconstructed slope, the substep-weighted average of the rates, which conserves mass and elements exactly and is bounded regardless of stiffness; when one substep is feasible it is the base scheme’s slope exactly. There is no substep count to set. The implicit (Strang) path keeps its Backward Euler solver with its own step halving.

JIT specialization

runPeregrine.py recompiles the compute kernels for your case before solving: the species count, ghost-layer count, every thermo/transport coefficient and the mechanism’s kinetics tables are baked in as compile-time constants (per-thread scratch becomes stack arrays, loops get constant bounds). Builds are cached by content hash in ~/.cache/peregrine (override with PEREGRINE_CACHE), so only the first run of a new case/code combination pays the compile (~10 s + mechanism size). simulation.precision: single selects fp32 storage and kernel arithmetic (jit-only; the generic build is double). Set PEREGRINE_NO_JIT=1 to run on the generic build. If the toolchain or Kokkos install recorded at install time is unavailable, the run falls back to the generic build with a warning. Tests and utilities always use the generic build.

Config options

Every config key, its default, and the valid (registry-discovered) values:

utilities/configOptions.py

Old cases

Convert a pre-consolidation case (per-block grid files, loose conn.yaml, blocksForProcs.inp, bcFams.yaml):

utilities/convertCase.py ./myCase