<div dir="ltr">Just some more background on that algorithm for others reading (which is obviously explained better in the paper, which Richard linked). As others point out, I don't think it fits your use case.<div><br></div><div>The motivation for the Chow-Patel algorithm is the fact that traditional ILU preconditioners don't work well in "fine-grained parallel" environments like GPUs. "Triangularity" is something associated with lots of data dependencies - think about Gaussian elimination - the whole idea is solving one equation at a time, based on the solutions of other equations. The Chow-Patel approach is to approach things in a clever way (solving a set of nonlinear equations describing the individual entries of the factors) to simultaneously compute all the entries of the triangular factors, asynchronously on lots of threads. That doesn't solve the problem of how to solve the resulting triangular systems in parallel, though, so that's done with an iterative approach (that is, you can approximate L^(-1) with a polynomial in L). It's a new approach and thus should be considered experimental. Key to note is that all of this is only explored or implemented on a single node (shared-memory domain), so if you want to use this preconditioner on multiple ranks it needs to be a sub-preconditioner in a block Jacobi, ASM, or other method with "local subsolves".</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Am Mo., 20. Jan. 2020 um 23:41 Uhr schrieb Mills, Richard Tran via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hi Xiangdong,</p>
<p>Maybe I am misunderstanding you, but it sounds like you want an exact direct solution, so I don't understand why you are using an incomplete factorization solver for this. SuperLU_DIST (as Mark has suggested) or MUMPS are two such packages that provide MPI-parallel
sparse LU factorization. If you need GPU support, SuperLU_DIST has such support. I don't know the status of our support for using the GPU capabilities of this, though -- I assume another developer can chime in regarding this.</p>
<p>Note that the ILU provided by "chowiluiennacl" employs a very different algorithm than the standard PCILU in PETSc, and you shouldn't expect to get the same incomplete factorization. The algorithm is described in this paper by Chow and Patel:</p>
<p><a href="https://www.cc.gatech.edu/~echow/pubs/parilu-sisc.pdf" target="_blank">https://www.cc.gatech.edu/~echow/pubs/parilu-sisc.pdf</a></p>
<p>Best regards,<br>
Richard<br>
</p>
<div>On 1/15/20 11:39 AM, Xiangdong wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr" style="color:rgb(0,0,0)">
<div>I just submitted the issue: <a href="https://gitlab.com/petsc/petsc/issues/535" target="_blank">https://gitlab.com/petsc/petsc/issues/535</a></div>
<div><br>
</div>
What I really want is an exact Block Tri-diagonal solver on GPU. Since for block tridiagonal system, ILU0 would be the same as ILU. So I tried the chowiluviennacl. but I found that the default parameters does not produce the same ILU0 factorization as the CPU
ones (PCILU). My guess is that if I increase the number of sweeps chow_patel_ilu_config.sweeps(3), it may give a better result. So the option Keys would be helpful.<br>
</div>
<div dir="ltr" style="color:rgb(0,0,0)"><br>
</div>
<div style="color:rgb(0,0,0)">Since Mark mentioned the Superlu's GPU feature, can I use superlu or hypre's GPU functionality through PETSc?</div>
<div style="color:rgb(0,0,0)"><br>
</div>
<div style="color:rgb(0,0,0)">Thank you.</div>
<div style="color:rgb(0,0,0)"><br>
</div>
<div style="color:rgb(0,0,0)">Xiangdong </div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Jan 15, 2020 at 2:22 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On Wed, Jan 15, 2020 at 1:48 PM Xiangdong <<a href="mailto:epscodes@gmail.com" target="_blank">epscodes@gmail.com</a>> wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">In the ViennaCL manual <a href="http://viennacl.sourceforge.net/doc/manual-algorithms.html" target="_blank">
http://viennacl.sourceforge.net/doc/manual-algorithms.html</a><br>
<br>
It did expose two parameters:<br>
<br>
// configuration of preconditioner:<br>
viennacl::linalg::chow_patel_tag chow_patel_ilu_config;<br>
chow_patel_ilu_config.sweeps(3); // three nonlinear sweeps<br>
chow_patel_ilu_config.jacobi_iters(2); // two Jacobi iterations per triangular 'solve' Rx=r<br>
<br>
and mentioned that:<br>
The number of nonlinear sweeps and Jacobi iterations need to be set problem-specific for best performance.<br>
<br>
In the PETSc' implementation:<br>
<br>
viennacl::linalg::chow_patel_tag ilu_tag;<br>
ViennaCLAIJMatrix *mat = (ViennaCLAIJMatrix*)gpustruct->mat;<br>
ilu->CHOWILUVIENNACL = new viennacl::linalg::chow_patel_ilu_precond<viennacl::compressed_matrix<PetscScalar> >(*mat, ilu_tag);<br>
<br>
The default is used. Is it possible to expose these two parameters so that user can change it through option keys?<br>
</div>
</blockquote>
<div><br>
</div>
<div>Yes. Do you mind making an issue for it? That way we can better keep track.</div>
<div><br>
</div>
<div><a href="https://gitlab.com/petsc/petsc/issues" target="_blank">https://gitlab.com/petsc/petsc/issues</a><br>
</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Thank you.<br>
<br>
Xiangdong</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Jan 15, 2020 at 12:40 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On Wed, Jan 15, 2020 at 9:59 AM Xiangdong <<a href="mailto:epscodes@gmail.com" target="_blank">epscodes@gmail.com</a>> wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Maybe I am not clear. I want to solve the block tridiagonal system Tx=b a few times with same T but different b. On CPU, I can have it by applying the ILU0 and reuse the factorization. Since it is block tridiagonal, ILU0 would give same results
as LU.
<div><br>
</div>
<div>I am trying to do the same thing on GPU with chowiluviennacl, but found default factorization does not produce the exact factorization for tridiagonal system. Can we tight the drop off tolerance so that it can work as LU for tridiagonal system?</div>
</div>
</blockquote>
<div><br>
</div>
<div>There are no options in our implementation. You could look at the ViennaCL manual to see if we missed something.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>Thank you.</div>
<div><br>
</div>
<div>Xiangdong<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Jan 15, 2020 at 9:41 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On Wed, Jan 15, 2020 at 9:36 AM Xiangdong <<a href="mailto:epscodes@gmail.com" target="_blank">epscodes@gmail.com</a>> wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Can chowiluviennacl do ilu0?
<div><br>
</div>
<div>I need to solve a tri-diagonal system directly. If I apply the PCILU, I will obtain the exact solution with preonly + pcilu. However, the preonly + chowiluviennacl will not provide the exact solution. Any option keys to set the CHOWILUVIENNACL filling
level or dropping off tolerance like the standard ilu?</div>
</div>
</blockquote>
<div><br>
</div>
<div>No. However, such a scheme makes less sense here. This algorithm spawns a individual threads for individual elements. Drop tolerance</div>
<div>is not less work, it is sparser, but that should not matter for a tridiagonal system. Levels also is not applicable since you have only 1 level.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>Thank you.</div>
<div><br>
</div>
<div>Best,</div>
<div>Xiangdong</div>
<div><br>
</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Jan 14, 2020 at 10:05 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On Tue, Jan 14, 2020 at 9:56 PM Xiangdong <<a href="mailto:epscodes@gmail.com" target="_blank">epscodes@gmail.com</a>> wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>Dear Developers,</div>
<div><br>
</div>
<div>I have a quick question about the chowiluviennacl. When I tried to use it, I found that it only works for np=1, not np>1. However, in the description of chowiluviennacl.cxx, it says "the ViennaCL Chow-Patel parallel ILU preconditioner".</div>
</div>
</blockquote>
<div><br>
</div>
<div>By parallel, this means shared memory parallelism on the GPU.</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>I am wondering whether I am using it correctly. Does chowiluviennacl work for np>1? </div>
</div>
</blockquote>
<div><br>
</div>
<div>I do not believe so. I do not see why it could not be extended, but that would mean writing some more code.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>In addition, are there option keys for the chowiluviennacl one can try?</div>
<div>Thank you.</div>
<div><br>
</div>
<div>Best,</div>
<div>Xiangdong</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote></div>