<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font face="Courier"><span style="color: rgb(0, 0, 0);">Dear all, </span></font></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font face="Courier"><span><br>
</span></font></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font face="Courier"><span><br>
</span></font></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font face="Courier"><span style="color: rgb(0, 0, 0);">I have a few questions about the implementation of diagonal ILU PC in PETSc. I want to solve a very simple system with KSP (in parallel), the nature of the system (finite difference time-harmonic Maxwell)
is probably not important to the question itself. </span></font><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Courier">Long story short,</span><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Courier"> I
just need to solve a set of equations of Ax = b with a block diagonal system matrix, like (not sure if the mono font works): </span></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font face="Courier"><span><br>
</span></font></div>
<font face="Courier" style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-size: 12px"><span style="color: rgb(0, 0, 0);"> |X | </span></font>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font face="Courier"><span style="color: rgb(0, 0, 0);">A =| Y | </span></font></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font face="Courier"><span style="color: rgb(0, 0, 0);"> | Z| </span></font></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font face="Courier"><span><br>
</span></font></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font face="Courier"><span style="color: rgb(0, 0, 0);">Note that A is not really block-diagonal, it’s just a multi-diagonal matrix determined by the FD mesh, where most elements are close to diagonal. So instead of a full ILU decomposition, a D-ILU is a good
approximation as a preconditioner, and the number of blocks should not matter too much: </span></font></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Courier"><br>
</span></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font face="Courier"><span style="color: rgb(0, 0, 0);"> </span><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0)">|Lx | </span></font><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Courier">|Ux |</span></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font face="Courier"><span style="color: rgb(0, 0, 0);">L = </span><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0)">| Ly | and U = </span></font><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Courier">| Uy |</span></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font face="Courier"><span style="color: rgb(0, 0, 0);"> | Lz| </span></font><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Courier">| Uz|</span></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Courier"><br>
</span></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font color="#000000" face="Courier"><span style="caret-color: rgb(0, 0, 0)">Where [Lx, Ux] = ILU0(X), etc. Indeed, the D-ILU preconditioner (with 3N blocks) is quite efficient with Krylov subspace methods like BiCGstab or QMR in my sequential Fortran/Matlab
code. </span></font></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font color="#000000" face="Courier"><span style="caret-color: rgb(0, 0, 0)"><br>
</span></font></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font color="#000000" face="Courier">So like most users, I am looking for a parallel implement with this problem in PETSc. After looking through the manual, it seems that the most straightforward way to do it is through PCBJACOBI. Not sure I understand it right,
I just setup a 3-block PCJACOBI and give each of the block a KSP with PCILU. Is this supposed to be equivalent to my D-ILU preconditioner? My little block of fortran code would look like: </font></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font color="#000000" face="Courier"><span style="caret-color: rgb(0, 0, 0)">...</span></font></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0)">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(46, 174, 187)"><b> call</b></span><span style="font-variant-ligatures: no-common-ligatures"><span> </span>PCBJacobiSetTotalBlocks(pc_local,Ntotal, </span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(200, 20, 201)">&</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0)">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> <span> </span></span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(200, 20, 201)">&</span><span style="font-variant-ligatures: no-common-ligatures"><span> </span>
isubs,ierr)</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0)">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"><span> </span> <span> </span></span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(46, 174, 187)"><b>call</b></span><span style="font-variant-ligatures: no-common-ligatures"><span> </span>PCBJacobiSetLocalBlocks(pc_local,
Nsub, </span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(200, 20, 201)">&</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0)">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> <span> </span></span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(200, 20, 201)">&</span><span style="font-variant-ligatures: no-common-ligatures">
isubs(istart:iend),ierr)</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(46, 174, 187)">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(0, 0, 0)"><span> </span> <span> </span></span><span style="font-variant-ligatures: no-common-ligatures">!
set up the block jacobi structure</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0)">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"><span> </span> <span> </span></span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(46, 174, 187)"><b>call</b></span><span style="font-variant-ligatures: no-common-ligatures"><span> </span>KSPSetup(ksp_local,ierr)</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(46, 174, 187)">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(0, 0, 0)"><span> </span> <span> </span></span><span style="font-variant-ligatures: no-common-ligatures">!
allocate sub ksps</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0)">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"><span> </span> <span> </span></span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(159, 160, 28)">allocate</span><span style="font-variant-ligatures: no-common-ligatures">(ksp_sub(Nsub))</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0)">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"><span> </span> <span> </span></span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(46, 174, 187)"><b>call</b></span><span style="font-variant-ligatures: no-common-ligatures"><span> </span>PCBJacobiGetSubKSP(pc_local,Nsub,istart,
</span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(200, 20, 201)">&</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0)">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> <span> </span></span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(200, 20, 201)">&</span><span style="font-variant-ligatures: no-common-ligatures"><span> </span>
ksp_sub,ierr)</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0)">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"><span> </span> <span> </span></span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(159, 160, 28)">do</span><span style="font-variant-ligatures: no-common-ligatures"><span> </span>i</span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(159, 160, 28)">=</span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(193, 101, 28)">1</span><span style="font-variant-ligatures: no-common-ligatures">,Nsub</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0)">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"><span> </span> <span> </span></span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(46, 174, 187)"><b>call</b></span><span style="font-variant-ligatures: no-common-ligatures"><span> </span>KSPGetPC(ksp_sub(i),pc_sub,ierr)</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(46, 174, 187)">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(0, 0, 0)"><span> </span> <span> </span></span><span style="font-variant-ligatures: no-common-ligatures">!ILU
preconditioner</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0)">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"><span> </span> <span> </span></span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(46, 174, 187)"><b>call</b></span><span style="font-variant-ligatures: no-common-ligatures"><span> </span>PCSetType(pc_sub,ptype,ierr)</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0)">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"><span> </span> <span> </span></span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(46, 174, 187)"><b>call</b></span><span style="font-variant-ligatures: no-common-ligatures"><span> </span>PCFactorSetLevels(pc_sub,</span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(193, 101, 28)">1</span><span style="font-variant-ligatures: no-common-ligatures">,ierr)<span> </span></span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(46, 174, 187)">!
use ILU(1) here</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0)">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"><span> </span> <span> </span></span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(46, 174, 187)"><b>call</b></span><span style="font-variant-ligatures: no-common-ligatures"><span> </span>KSPSetType(ksp_sub(i),KSPPREONLY,ierr)]</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(159, 160, 28)">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(0, 0, 0)"><span> </span> <span> </span></span><span style="font-variant-ligatures: no-common-ligatures">end
do</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0)">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"><span> </span> <span> </span></span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(46, 174, 187)"><b>call</b></span><span style="font-variant-ligatures: no-common-ligatures"><span> </span>KSPSetTolerances(ksp_local,KSSiter%tol,PETSC_DEFAULT_REAL,<span> </span></span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(200, 20, 201)">&</span></div>
<div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0)">
<span style="font-variant-ligatures: no-common-ligatures; color: rgb(117, 117, 117)"> </span><span style="font-variant-ligatures: no-common-ligatures"> <span> </span></span><span style="font-variant-ligatures: no-common-ligatures; color: rgb(200, 20, 201)">&</span><span style="font-variant-ligatures: no-common-ligatures"><span> </span>
PETSC_DEFAULT_REAL,KSSiter%maxit,ierr)</span></div>
</div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
… </div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font color="#000000" face="Courier"><span style="caret-color: rgb(0, 0, 0)"><br>
</span></font></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Courier">I understand that the parallel performance may not be comparable, so I first set up a one-process test (with MPIAij, but all the rows are local since </span><font color="#000000" face="Courier">there
is only one process). The system is solved without any problem (identical results within error). But the performance is actually a lot worse (code built without debugging flags in performance tests) than my own home-brew implementation in Fortran (I wrote
my own ILU0 in CSR sparse matrix format), which is hard to believe. I suspect the difference is from the PC as the PETSc version took much more BiCGstab iterations (60-ish vs 100-ish) to converge to the same relative tol. </font></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font color="#000000" face="Courier"><br>
</font></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font color="#000000" face="Courier">This is </font><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Courier">further</span> <font color="#000000" face="Courier">confirmed when I change the setup of D-ILU (using 6 or 9 blocks instead
of 3). While my Fortran/Matlab codes see minimal performance difference (<5%) when I play with the D-ILU setup, increasing the number of D-ILU blocks from 3 to 9 caused the ksp setup with PCBJACOBI to suffer a performance decrease of more than 50% in sequential
test. So my <span style="caret-color: rgb(0, 0, 0)">implementation</span> IS somewhat different in PETSc. Do I miss something in the PCBJACOBI setup? Or do I have some fundamental misunderstanding of how PCBJACOBI works in PETSc? </font></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font color="#000000" face="Courier"><br>
</font></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font color="#000000" face="Courier"><span style="caret-color: rgb(0, 0, 0)">If this is not the right way to implement a block diagonal ILU as (parallel) PC, please kindly point me to the right direction. I searched through the mail list to find some answers,
only to find a couple of similar questions... An example would be nice.</span></font></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font color="#000000" face="Courier"><span style="caret-color: rgb(0, 0, 0)"><br>
</span></font></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font color="#000000" face="Courier">On the other hand, does PETSc support a simple way to use explicit L/U matrix as a preconditioner? I can import the D-ILU matrix (I already converted my A matrix into Mat) constructed in my Fortran code to make a better
comparison. Or do I have to construct my own PC using PCshell? If so, is there a good tutorial/example to learn about how to use PCSHELL (in a more advanced sense, like how to setup pc side and transpose)? </font></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font color="#000000" face="Courier"><br>
</font></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font color="#000000" face="Courier"><span style="caret-color: rgb(0, 0, 0)">Thanks in advance, </span></font></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font color="#000000" face="Courier"><span style="caret-color: rgb(0, 0, 0)"><br>
</span></font></div>
<div style="caret-color: rgb(255, 255, 255); color: rgb(255, 255, 255); font-family: Helvetica; font-size: 12px">
<font color="#000000" face="Courier"><span style="caret-color: rgb(0, 0, 0)">Hao</span></font></div>
<br>
</div>
</body>
</html>