<meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr"><div dir="ltr">Hao:<br></div><div>I would suggest to use a parallel sparse direct solver, e.g., superlu_dist or mumps. These solvers can take advantage of your sparse data structure. Once it works, then you may play with other preconditioners, such as bjacobi + lu/ilu. See</div><div><a href="https://www.mcs.anl.gov/petsc/miscellaneous/external.html">https://www.mcs.anl.gov/petsc/miscellaneous/external.html</a></div><div>Hong</div><div><br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font face="Courier"><span style="color:rgb(0,0,0)">Dear all, </span></font></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font face="Courier"><span><br>
</span></font></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font face="Courier"><span><br>
</span></font></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font face="Courier"><span style="color:rgb(0,0,0)">I have a few questions about the implementation of diagonal ILU PC in PETSc. I want to solve a very simple system with KSP (in parallel), the nature of the system (finite difference time-harmonic Maxwell)
is probably not important to the question itself. </span></font><span style="color:rgb(0,0,0);font-family:Courier">Long story short,</span><span style="color:rgb(0,0,0);font-family:Courier"> I
just need to solve a set of equations of Ax = b with a block diagonal system matrix, like (not sure if the mono font works): </span></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font face="Courier"><span><br>
</span></font></div>
<font face="Courier" style="color:rgb(255,255,255);font-size:12px"><span style="color:rgb(0,0,0)"> |X | </span></font>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font face="Courier"><span style="color:rgb(0,0,0)">A =| Y | </span></font></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font face="Courier"><span style="color:rgb(0,0,0)"> | Z| </span></font></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font face="Courier"><span><br>
</span></font></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font face="Courier"><span style="color:rgb(0,0,0)">Note that A is not really block-diagonal, it’s just a multi-diagonal matrix determined by the FD mesh, where most elements are close to diagonal. So instead of a full ILU decomposition, a D-ILU is a good
approximation as a preconditioner, and the number of blocks should not matter too much: </span></font></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<span style="color:rgb(0,0,0);font-family:Courier"><br>
</span></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font face="Courier"><span style="color:rgb(0,0,0)"> </span><span style="color:rgb(0,0,0)">|Lx | </span></font><span style="color:rgb(0,0,0);font-family:Courier">|Ux |</span></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font face="Courier"><span style="color:rgb(0,0,0)">L = </span><span style="color:rgb(0,0,0)">| Ly | and U = </span></font><span style="color:rgb(0,0,0);font-family:Courier">| Uy |</span></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font face="Courier"><span style="color:rgb(0,0,0)"> | Lz| </span></font><span style="color:rgb(0,0,0);font-family:Courier">| Uz|</span></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<span style="color:rgb(0,0,0);font-family:Courier"><br>
</span></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font color="#000000" face="Courier"><span>Where [Lx, Ux] = ILU0(X), etc. Indeed, the D-ILU preconditioner (with 3N blocks) is quite efficient with Krylov subspace methods like BiCGstab or QMR in my sequential Fortran/Matlab
code. </span></font></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font color="#000000" face="Courier"><span><br>
</span></font></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font color="#000000" face="Courier">So like most users, I am looking for a parallel implement with this problem in PETSc. After looking through the manual, it seems that the most straightforward way to do it is through PCBJACOBI. Not sure I understand it right,
I just setup a 3-block PCJACOBI and give each of the block a KSP with PCILU. Is this supposed to be equivalent to my D-ILU preconditioner? My little block of fortran code would look like: </font></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font color="#000000" face="Courier"><span>...</span></font></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)"><b> call</b></span><span style="font-variant-ligatures:no-common-ligatures"><span> </span>PCBJacobiSetTotalBlocks(pc_local,Ntotal, </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(200,20,201)">&</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures;color:rgb(117,117,117)"> </span><span style="font-variant-ligatures:no-common-ligatures"> <span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(200,20,201)">&</span><span style="font-variant-ligatures:no-common-ligatures"><span> </span>
isubs,ierr)</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures;color:rgb(117,117,117)"> </span><span style="font-variant-ligatures:no-common-ligatures"><span> </span> <span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)"><b>call</b></span><span style="font-variant-ligatures:no-common-ligatures"><span> </span>PCBJacobiSetLocalBlocks(pc_local,
Nsub, </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(200,20,201)">&</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures;color:rgb(117,117,117)"> </span><span style="font-variant-ligatures:no-common-ligatures"> <span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(200,20,201)">&</span><span style="font-variant-ligatures:no-common-ligatures">
isubs(istart:iend),ierr)</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(46,174,187)">
<span style="font-variant-ligatures:no-common-ligatures;color:rgb(117,117,117)"> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"><span> </span> <span> </span></span><span style="font-variant-ligatures:no-common-ligatures">!
set up the block jacobi structure</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures;color:rgb(117,117,117)"> </span><span style="font-variant-ligatures:no-common-ligatures"><span> </span> <span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)"><b>call</b></span><span style="font-variant-ligatures:no-common-ligatures"><span> </span>KSPSetup(ksp_local,ierr)</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(46,174,187)">
<span style="font-variant-ligatures:no-common-ligatures;color:rgb(117,117,117)"> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"><span> </span> <span> </span></span><span style="font-variant-ligatures:no-common-ligatures">!
allocate sub ksps</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures;color:rgb(117,117,117)"> </span><span style="font-variant-ligatures:no-common-ligatures"><span> </span> <span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(159,160,28)">allocate</span><span style="font-variant-ligatures:no-common-ligatures">(ksp_sub(Nsub))</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures;color:rgb(117,117,117)"> </span><span style="font-variant-ligatures:no-common-ligatures"><span> </span> <span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)"><b>call</b></span><span style="font-variant-ligatures:no-common-ligatures"><span> </span>PCBJacobiGetSubKSP(pc_local,Nsub,istart,
</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(200,20,201)">&</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures;color:rgb(117,117,117)"> </span><span style="font-variant-ligatures:no-common-ligatures"> <span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(200,20,201)">&</span><span style="font-variant-ligatures:no-common-ligatures"><span> </span>
ksp_sub,ierr)</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures;color:rgb(117,117,117)"> </span><span style="font-variant-ligatures:no-common-ligatures"><span> </span> <span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(159,160,28)">do</span><span style="font-variant-ligatures:no-common-ligatures"><span> </span>i</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(159,160,28)">=</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(193,101,28)">1</span><span style="font-variant-ligatures:no-common-ligatures">,Nsub</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures;color:rgb(117,117,117)"> </span><span style="font-variant-ligatures:no-common-ligatures"><span> </span> <span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)"><b>call</b></span><span style="font-variant-ligatures:no-common-ligatures"><span> </span>KSPGetPC(ksp_sub(i),pc_sub,ierr)</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(46,174,187)">
<span style="font-variant-ligatures:no-common-ligatures;color:rgb(117,117,117)"> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"><span> </span> <span> </span></span><span style="font-variant-ligatures:no-common-ligatures">!ILU
preconditioner</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures;color:rgb(117,117,117)"> </span><span style="font-variant-ligatures:no-common-ligatures"><span> </span> <span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)"><b>call</b></span><span style="font-variant-ligatures:no-common-ligatures"><span> </span>PCSetType(pc_sub,ptype,ierr)</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures;color:rgb(117,117,117)"> </span><span style="font-variant-ligatures:no-common-ligatures"><span> </span> <span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)"><b>call</b></span><span style="font-variant-ligatures:no-common-ligatures"><span> </span>PCFactorSetLevels(pc_sub,</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(193,101,28)">1</span><span style="font-variant-ligatures:no-common-ligatures">,ierr)<span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)">!
use ILU(1) here</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures;color:rgb(117,117,117)"> </span><span style="font-variant-ligatures:no-common-ligatures"><span> </span> <span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)"><b>call</b></span><span style="font-variant-ligatures:no-common-ligatures"><span> </span>KSPSetType(ksp_sub(i),KSPPREONLY,ierr)]</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(159,160,28)">
<span style="font-variant-ligatures:no-common-ligatures;color:rgb(117,117,117)"> </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"><span> </span> <span> </span></span><span style="font-variant-ligatures:no-common-ligatures">end
do</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures;color:rgb(117,117,117)"> </span><span style="font-variant-ligatures:no-common-ligatures"><span> </span> <span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)"><b>call</b></span><span style="font-variant-ligatures:no-common-ligatures"><span> </span>KSPSetTolerances(ksp_local,KSSiter%tol,PETSC_DEFAULT_REAL,<span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(200,20,201)">&</span></div>
<div style="margin:0px;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures;color:rgb(117,117,117)"> </span><span style="font-variant-ligatures:no-common-ligatures"> <span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(200,20,201)">&</span><span style="font-variant-ligatures:no-common-ligatures"><span> </span>
PETSC_DEFAULT_REAL,KSSiter%maxit,ierr)</span></div>
</div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
… </div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font color="#000000" face="Courier"><span><br>
</span></font></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<span style="color:rgb(0,0,0);font-family:Courier">I understand that the parallel performance may not be comparable, so I first set up a one-process test (with MPIAij, but all the rows are local since </span><font color="#000000" face="Courier">there
is only one process). The system is solved without any problem (identical results within error). But the performance is actually a lot worse (code built without debugging flags in performance tests) than my own home-brew implementation in Fortran (I wrote
my own ILU0 in CSR sparse matrix format), which is hard to believe. I suspect the difference is from the PC as the PETSc version took much more BiCGstab iterations (60-ish vs 100-ish) to converge to the same relative tol. </font></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font color="#000000" face="Courier"><br>
</font></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font color="#000000" face="Courier">This is </font><span style="color:rgb(0,0,0);font-family:Courier">further</span> <font color="#000000" face="Courier">confirmed when I change the setup of D-ILU (using 6 or 9 blocks instead
of 3). While my Fortran/Matlab codes see minimal performance difference (<5%) when I play with the D-ILU setup, increasing the number of D-ILU blocks from 3 to 9 caused the ksp setup with PCBJACOBI to suffer a performance decrease of more than 50% in sequential
test. So my <span>implementation</span> IS somewhat different in PETSc. Do I miss something in the PCBJACOBI setup? Or do I have some fundamental misunderstanding of how PCBJACOBI works in PETSc? </font></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font color="#000000" face="Courier"><br>
</font></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font color="#000000" face="Courier"><span>If this is not the right way to implement a block diagonal ILU as (parallel) PC, please kindly point me to the right direction. I searched through the mail list to find some answers,
only to find a couple of similar questions... An example would be nice.</span></font></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font color="#000000" face="Courier"><span><br>
</span></font></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font color="#000000" face="Courier">On the other hand, does PETSc support a simple way to use explicit L/U matrix as a preconditioner? I can import the D-ILU matrix (I already converted my A matrix into Mat) constructed in my Fortran code to make a better
comparison. Or do I have to construct my own PC using PCshell? If so, is there a good tutorial/example to learn about how to use PCSHELL (in a more advanced sense, like how to setup pc side and transpose)? </font></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font color="#000000" face="Courier"><br>
</font></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font color="#000000" face="Courier"><span>Thanks in advance, </span></font></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font color="#000000" face="Courier"><span><br>
</span></font></div>
<div style="color:rgb(255,255,255);font-family:Helvetica;font-size:12px">
<font color="#000000" face="Courier"><span>Hao</span></font></div>
<br>
</div>
</div>
</blockquote></div></div>