PEREGRINE Documentation

Executable Mode

PEREGRINE can run in both a scriptable mode for simple cases, or executable mode for larger production runs. For examples of scripting modes, see the examples directory in the repository. For executable mode, see the case directory structure here, as well as the format the the input config file below. In executable mode, one creates an initialization file (a zero-th restart file) and begins from there. A scripted approach is best suited for this, read in a grid, use all the python goodies you want to create the initial flow field, then fire away:

import peregrinepy as pg
nblks = 10 #number of grid blocks
ns = 2 #number of species

# create multiblock restart
mb = pg.multiBlock.restart(nblks, ns)

# read in grid
pg.readers.readGrid(mb, "/path/to/g.*")

# iterate over each block and
# set values of initialization in q
# (can access the x,y,z coords of cell centers with
#  blk.array["xc"][:,:,:] and so on.)
for blk in mb:
    blk.array["q"][:,:,:, 0] = 101325.0 #pressure
    blk.array["q"][:,:,:, 1] = 1.0 # u velocity
    blk.array["q"][:,:,:, 2] = 0.0 # v velocity
    blk.array["q"][:,:,:, 3] = 0.0 # w velocity
    blk.array["q"][:,:,:, 4] = 0.0 # Temperature
    blk.array["q"][:,:,:, 5] = 0.0 # First species
    # ... and so on for ns-1 species.


# write out the zeroth restart file
pg.writers.writeRestart(mb,
                        "/path/to/Restart",
                        "/path/to/Grid")

CoProcessing

Coprocessing with ParaView is amazing, but takes effort to make it work well in my experience. It seems they have cleaned it up a lot from back in the day. To start download and install Paraview from source. I have tested up to 5.11 and it works well for me. To make coprocessing work, you need to compile paraview in catalyst mode. With cmake, pass the argument:

ccmake -DPARAVIEW_BUILD_EDITION:STRING=CATALYST /path/to/ParaView_src/

Assume you are installing it in /path/to/paraview.

Some Tips and Tricks

You must set the cmake flags:

PARAVIEW_USE_MPI=ON
PARAVIEW_USE_PYTHON=ON

Make sure ParaView finds the correct python you are planning to use with PEREGRINE.

Once you configure a bunch of times, look for a group of options that look like VTK_MODULE_USE_EXTERNAL_VTK_*. You want to turn on as many of those as you can, otherwise ParaView will download, and fail, to install many of these. Especially mpi4py, png, libxml and so on. Hopefully your cluster has these already and you can module load them, or your package manager probably has them too.

Running

To get it to work, you have to set these environment variables so python can find the catalyst install.

export PYTHONPATH=$PYTHONPATH:/path/to/paraview/lib/python3.XX/site-packages
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/paraview/lib
export PYTHONPATH=$PYTHONPATH:/path/to/paraview/lib

You will know it worked if from paraview.catalyst import bridge works in an interpreter.

Good Luck!

Code Structure

• Python + Kokkos
• Multiblock Structure
• Array, Mirror, View
• Boundary Conditions

Utilities

There are a bunch of utilities to make running with PEREGRINE easier, see the utilities folder. They are pretty obvious from the name. Here is a brief list and summary.

• decompose: Decompose the base grid for a machine layout (ranks x ranks per node): cut blocks where the rank count needs it, across the smallest cross-sections, and place the pieces over nodes then ranks, minimizing network traffic under a load-balance tolerance. Stored in g.h5 per layout. YAY MULTIBLOCK!

• generateTracePoints: Generate trace point input file for collecting data at points during a simulation.

• gridPro2pg: Grid Pro mesh to PEREGRINE mesh translation.

• icem2pg: ICEM mesh to PEREGRINE mesh translation.

• interpolate: Interpolate results from one mesh to another mesh.

• verifyGrid: Go block by block, face by face, and make sure everyone agrees who is connected to who, and in what orientation; then check every stored decomposition tiles the base grid exactly and passes the same geometric checks.

Config File

PEREGRINE run in executable mode requires an input configuration file (in yaml format) to tell it what to do. Here is a sample of such a file with comments.

# input output directories
io:
 gridDir: ./Grid
 inputDir: ./Input
 restartDir: ./Restart
 archiveDir: ./Archive

# simulation control
simulation:
  niter: 10
  dt: 0.9
  restartFrom: 0
  animateRestart: true
  animateArchive: true
  niterRestart: 1 #save out restart files
  niterArchive: 10 #save out light weight single precision files
  niterPrint: 1  #print to stdout case progress
  variableTimeStep: true
  maxCFL: 1.0
  checkNan: 1

solver: # solver selection
  timeIntegration: rk1

RHS: # RHS control
  shockHandling: null #method of shock handling
  primaryAdvFlux: secondOrderKEEP  # how to compute low dissipative adctive flux
  secondaryAdvFlux: null # how to compute low order stable advective flux
  switchAdvFlux: null # how to switch between the two
  diffusion: true #solve diffusion terms?
  subgrid: null #use a subgrid model?

thermochem: #thermo, transport, chem options
  mechanism: ["O2", "N2"] # list of library species names, or a Cantera mechanism yaml
  eos: cpg
  trans: constantProps #transport property calculations
  chemistry: false # finite-rate chemistry of the mechanism file

coprocess: # coprocessing
  catalyst: false # or path to paraview catalyst input python file
  trace: false # or path to trace points file
  niterTrace: 1