Integration of efficient compute kernels for phase field models into the waLBerla simulation framework using code generation and performance engineering techniques


Prof. Dr. Ulrich Rüde
Prof. Dr. Harald Köstler
Chair for Computer Science 10 – System simulation
Friedrich-Alexander-Universität Erlangen-Nürnberg

Project Summary

waLBerla is a modern Open-Source software framework whose focus is on the lattice Boltzmann method (LBM) as an alternative to classical Navier-Stokes solvers for computational fluid dynamics, but it can be used to efficiently implement many structured-grid based algorithms.
During this project, we developed a high-performance implementation of a multiphase lattice Boltzmann method based on the conservative Allen-Cahn model supporting high-density ratios and high Reynolds numbers. Code generation techniques were used to obtain optimized code for CPUs and GPUs automatically. The coupled model was specified in a high-level symbolic description and optimized through automatic transformations. The memory footprint of the resulting algorithm was reduced through the fusion of compute kernels. A roofline analysis demonstrated the excellent efficiency of the generated code on a single GPU. The resulting single GPU code has been integrated into waLBerla to run massively parallel simulations on large domains. Communication hiding and GPUDirect-enabled MPI yielded near-perfect scaling behaviour. Scaling experiments were conducted with up to 2048 GPUs, simulating several hundred fully resolved bubbles. Further, validation of the implementation was shown in a physically relevant scenario – a three-dimensional rising air bubble in water (cf. figure).

Resulting Publications during KONWIHR project:

Holzer, M., Bauer, M., Köstler, H., & Rüde, U. (2021). Highly efficient lattice Boltzmann multiphase simulations of immiscible fluids at high-density ratios on CPUs and GPUs through code generation. The International Journal of High Performance Computing Applications, 35(4), 413-427.

Mitchell, T., Holzer, M., Schwarzmeier, C., Bauer, M., Rüde, U., & Leonardi, C. (2021). Stability assessment of the phase-field lattice Boltzmann model and its application to Taylor bubbles in annular piping geometries. Physics of Fluids, 33(8), 083325.