[petsc-users] Tuning the parallel performance of a 3D FEM CFD code

[Henning Sauerland]

> How small is the Reynolds number? There is a difference between 0.1 and 100, although both may be laminar.
It's more like in the range of 100.

> I assume the XFEM basis is just resolving the jump across the interface. Does this mean the size of the matrix is changing as the interface moves? In any case, it looks like you can't amortize setup costs between time steps, so we need a solution short of a direct solve. 
Yes, exactly the pressure approximation space is enriched with discontinuous functions. Due to the XFEM basis I then get an additional degree of freedom at the nodes of elements cut by the interface.

> Unfortunately, multigrid methods for XFEM are a recent topic. Perhaps the best results I have seen (at conferences) use some geometric information in an otherwise algebraic framework. For this problem (unlike many fracture problems), the influence of the extended basis functions may be local enough that you can build a coarse level using the conventional part of the problem. The first thing I would try is probably to see if a direct solve with the conventional part makes an effective coarse level. If that works, I would see if ML or Hypre can do a reasonable job with that part of the problem.
> 
> I have no great confidence that this will work, it's highly dependent on how local the influence of the extended basis functions is. Perhaps you have enough experience with the method to hypothesize.
Unfortunately, I don't have any practical experience with multigrid methods, but I'm afraid that the conventional part won't make a good coarse level. Depending on the test case (large density ratios between the phases) the standard approximation won't do well. Anyway, I'll take a closer look at the multigrid topic.

> Note: for the conventional part of the problem, it is still incompressible flow. It sounds like you are using equal-order elements (Q1-Q1 stabilized; PSPG or Dohrmann&Bochev?).
Q1-Q1 SUPG, PSPG stabilized.

> For those elements, you will want to interlace the velocity and pressure degrees of freedom, then use a smoother that respects the block size. PETSc's ML and Hypre interfaces forward this information if you set the block size on the matrix. When using ML, you'll probably have to make the smoothers stronger. There are also some "energy minimization" options that may help.


Best,
Henning