PEREGRINE: Accessible, Performant, Portable Multiphysics CFD

About

PEREGRINE is a second order, multiblock, structured-grid multiphysics, finite volume, 3D CFD solver. The main novelty of PEREGRINE is its implementation in Python for ease of development and use of Kokkos for performance portability. If you are unfamiliar with Kokkos, do a little digging, it is a great project with a healthy community and helpful developers. The TLDR; Kokkos is a C++ library (not a C++ language extension) that exposes useful abstractions for data management (i.e. multidimensional arrays) and kernel execution from CPU-Serial to GPU-Parallel. This allows a single source, multiple architecture, approach in PEREGRINE. In other words, you can run a case with PEREGRINE on your laptop, then without changing a single line of source code, run the same case on a AMD GPU based super computer. PEREGRINE is massively parallel inter-node via MPI communication.

Installation

One shot, from a fresh clone or tarball:

pip install .

This builds everything in Release mode. If an installed Kokkos (>= 5.2) is found via CMAKE_PREFIX_PATH or Kokkos_DIR it is used (and its enabled backends carry over); otherwise Kokkos 5.2.1 is fetched and built automatically with the SERIAL+OPENMP backends. pybind11 comes from pip (the vendored submodule is only a fallback), so even a non-recursive download installs cleanly.

Selecting a device backend

The Kokkos install controls the Host/Device execution parameters. On a machine with a site or self-built CUDA/HIP Kokkos (recommended for clusters):

CMAKE_PREFIX_PATH=/path/to/kokkos-cuda pip install .

Or have the automatic Kokkos build target a device by passing CMake defines through pip:

CMAKE_ARGS="-DKokkos_ENABLE_CUDA=ON -DKokkos_ARCH_VOLTA70=ON" pip install .

(equivalently pip install . -Ccmake.define.Kokkos_ENABLE_CUDA=ON ...).

Development Install

For C++ iteration, set PYTHONPATH to /path/to/PEREGRINE/src/ and build the compute module in place (installs the .so into src/peregrinepy/):

cd /path/to/PEREGRINE; mkdir build; cd build; ccmake ../; make -j install

To generate compile_commands.json,

cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON ../

Documentation

See the documentation here.

Profiling GPU via NVTX

Download and install the libraries found at here. At runtime, ensure the environment variable

$ export KOKKOS_PROFILE_LIBRARY=/path/to/kokkos-tools/kp_nvprof_connector.so

is set. Finally, run the simulation with nsys enabling cuda,nvtx trace options.

jsrun -p 1 -g 1 nsys profile -o outPutName --trace cuda,nvtx  -f true --stats=false python -m mpi4py pgScript.py

Performance

PEREGRINE is pretty fast by default. However, when running a simulation with multiple chemical species, it is recommended to turn on PEREGRINE_JIT in cmake, and then specify the value of PEREGRINE_NS. This will hard code ns at compile time, and gives a considerable performance improvement for EOS/transport calculations.

Parallel I/O

Parallel I/O can be achieved with a parallel capable h5py installation.

$ export CC=mpicc
$ export HDF5_MPI="ON"
$ export HDF5_DIR="/path/to/parallel/hdf5"  # If this isn't found by default
$ pip install h5py --no-binary=h5py

$HDF5_DIR must point to a parallel enabled HDF5 installation. Parallel I/O is only applicable when running simulations with config["io"]["lumpIO"]=true.

Attribution

Please use the following BibTex to cite PEREGRINE in scientific writing:

@misc{PEREGRINE,
   author = {Kyle A. Schau},
   year = {2021},
   note = {https://github.com/kaschau/PEREGRINE},
   title = {PEREGRINE: Accessible, Performant, Portable Multiphysics CFD}
}

License

PEREGRINE is released under the New BSD License (see the LICENSE file for details). Documentation is made available under a Creative Commons Attribution 4.0 license (see http://creativecommons.org/licenses/by/4.0/).