<div dir="ltr">Barry, we seem to get an error when you explicitly set this. <div><br></div><div>Garth, Maybe to set the default explicitly you need to use pc_type asm -sub_pc_type lu. That is the true default.</div><div><br></div><div>More below but this is the error message:</div><div><br></div><div><div style="font-size:12.8px">17:46 knepley/feature-plasma-example *= ~/Codes/petsc/src/ksp/ksp/<wbr>examples/tutorials$ /Users/markadams/Codes/petsc/<wbr>arch-macosx-gnu-g/bin/mpiexec -np 2 ./ex56 -ne 16 -pc_type gamg -ksp_view -mg_coarse_ksp_type preonly -mg_coarse_pc_type lu -mg_coarse_pc_factor_mat_<wbr>solver_package petsc</div><span class="gmail-im" style="font-size:12.8px"><div>[0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--</div><div>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/<wbr>linearsolvertable.html</a> for possible LU and Cholesky solvers</div><div>[0]PETSC ERROR: MatSolverPackage petsc does not support matrix type mpiaij</div></span><div style="font-size:12.8px">[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html</a> for trouble shooting.</div><div style="font-size:12.8px">[0]PETSC ERROR: Petsc Development GIT revision: v3.7.6-3658-g99fa2798da GIT Date: 2017-04-25 12:56:20 -0500</div><div style="font-size:12.8px">[0]PETSC ERROR: ./ex56 on a arch-macosx-gnu-g named MarksMac-5.local by markadams Wed Apr 26 17:46:28 2017</div><div style="font-size:12.8px">[0]PETSC ERROR: Configure options --with-cc=clang --with-cc++=clang++ COPTFLAGS="-g -O0 -mavx2" CXXOPTFLAGS="-g -O0 -mavx2" F</div><div style="font-size:12.8px"><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Apr 27, 2017 at 1:59 AM, Garth N. Wells <span dir="ltr"><<a href="mailto:gnw20@cam.ac.uk" target="_blank">gnw20@cam.ac.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-">On 27 April 2017 at 00:30, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br>
><br>
> Yes, you asked for LU so it used LU!<br>
><br>
> Of course for smaller coarse grids and large numbers of processes this is very inefficient.<br>
><br>
> The default behavior for GAMG is probably what you want. In that case it is equivalent to<br>
> -mg_coarse_pc_type bjacobi --mg_coarse_sub_pc_type lu. But GAMG tries hard to put all the coarse grid degrees<br>
> of freedom on the first process and none on the rest, so you do end up with the exact equivalent of a direct solver.<br>
> Try -ksp_view in that case.<br>
><br>
<br>
</span>Thanks, Barry.<br>
<br>
I'm struggling a little to understand the matrix data structure for<br>
the coarse grid. Is it just a mpiaji matrix, with all entries<br>
(usually) on one process?<br></blockquote><div><br></div><div>Yes.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Is there an options key prefix for the matrix on different levels?<br>
E.g., to turn on a viewer?<br></blockquote><div><br></div><div>something like -mg_level_1_ksp_view should work (run with -help to get the correct syntax).</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
If I get GAMG to use more than one process for the coarse grid (a GAMG<br>
setting), can I get a parallel LU (exact) solver to solve it using<br>
only the processes that store parts of the coarse grid matrix?<br></blockquote><div><br></div><div>No, we should make a sub communicator for the active processes only, but I am not too motivated to do this because the only reason that this matters is if 1) a solver (ie, the parallel direct solver) is lazy and puts reductions everywhere for not good reason, or 2) you use a Krylov solver (very uncommon). All of the communication in a non-krylov solver in point to point and there is no win that I know of with a sub communicator. </div><div><br></div><div>Note, the redundant coarse grid solver does use a subcommuncator, obviously, but I think it is hardwired to PETSC_COMM_SELF, but maybe not?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Related to all this, do the parallel LU solvers internally<br>
re-distribute a matrix over the whole MPI communicator as part of<br>
their re-ordering phase?<br></blockquote><div><br></div><div>They better not!</div><div><br></div><div>I doubt any solver would be that eager by default.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class="gmail-HOEnZb"><font color="#888888"><br>
Garth<br>
</font></span><div class="gmail-HOEnZb"><div class="gmail-h5"><br>
> There is also -mg_coarse_pc_type redundant -mg_coarse_redundant_pc_type lu. In that case it makes a copy of the coarse matrix on EACH process and each process does its own factorization and solve. This saves one phase of the communication for each V cycle since every process has the entire solution it just grabs from itself the values it needs without communication.<br>
><br>
><br>
><br>
><br>
>> On Apr 26, 2017, at 5:25 PM, Garth N. Wells <<a href="mailto:gnw20@cam.ac.uk">gnw20@cam.ac.uk</a>> wrote:<br>
>><br>
>> I'm a bit confused by the selection of the coarse grid solver for<br>
>> multigrid. For the demo ksp/ex56, if I do:<br>
>><br>
>> mpirun -np 1 ./ex56 -ne 16 -ksp_view -pc_type gamg<br>
>> -mg_coarse_ksp_type preonly -mg_coarse_pc_type lu<br>
>><br>
>> I see<br>
>><br>
>> Coarse grid solver -- level ------------------------------<wbr>-<br>
>> KSP Object: (mg_coarse_) 1 MPI processes<br>
>> type: preonly<br>
>> maximum iterations=10000, initial guess is zero<br>
>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<br>
>> left preconditioning<br>
>> using NONE norm type for convergence test<br>
>> PC Object: (mg_coarse_) 1 MPI processes<br>
>> type: lu<br>
>> out-of-place factorization<br>
>> tolerance for zero pivot 2.22045e-14<br>
>> matrix ordering: nd<br>
>> factor fill ratio given 5., needed 1.<br>
>> Factored matrix follows:<br>
>> Mat Object: 1 MPI processes<br>
>> type: seqaij<br>
>> rows=6, cols=6, bs=6<br>
>> package used to perform factorization: petsc<br>
>> total: nonzeros=36, allocated nonzeros=36<br>
>> total number of mallocs used during MatSetValues calls =0<br>
>> using I-node routines: found 2 nodes, limit used is 5<br>
>> linear system matrix = precond matrix:<br>
>> Mat Object: 1 MPI processes<br>
>> type: seqaij<br>
>> rows=6, cols=6, bs=6<br>
>> total: nonzeros=36, allocated nonzeros=36<br>
>> total number of mallocs used during MatSetValues calls =0<br>
>> using I-node routines: found 2 nodes, limit used is 5<br>
>><br>
>> which is what I expect. Increasing from 1 to 2 processes:<br>
>><br>
>> mpirun -np 2 ./ex56 -ne 16 -ksp_view -pc_type gamg<br>
>> -mg_coarse_ksp_type preonly -mg_coarse_pc_type lu<br>
>><br>
>> I see<br>
>><br>
>> Coarse grid solver -- level ------------------------------<wbr>-<br>
>> KSP Object: (mg_coarse_) 2 MPI processes<br>
>> type: preonly<br>
>> maximum iterations=10000, initial guess is zero<br>
>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<br>
>> left preconditioning<br>
>> using NONE norm type for convergence test<br>
>> PC Object: (mg_coarse_) 2 MPI processes<br>
>> type: lu<br>
>> out-of-place factorization<br>
>> tolerance for zero pivot 2.22045e-14<br>
>> matrix ordering: natural<br>
>> factor fill ratio given 0., needed 0.<br>
>> Factored matrix follows:<br>
>> Mat Object: 2 MPI processes<br>
>> type: superlu_dist<br>
>> rows=6, cols=6<br>
>> package used to perform factorization: superlu_dist<br>
>> total: nonzeros=0, allocated nonzeros=0<br>
>> total number of mallocs used during MatSetValues calls =0<br>
>> SuperLU_DIST run parameters:<br>
>> Process grid nprow 2 x npcol 1<br>
>> Equilibrate matrix TRUE<br>
>> Matrix input mode 1<br>
>> Replace tiny pivots FALSE<br>
>> Use iterative refinement FALSE<br>
>> Processors in row 2 col partition 1<br>
>> Row permutation LargeDiag<br>
>> Column permutation METIS_AT_PLUS_A<br>
>> Parallel symbolic factorization FALSE<br>
>> Repeated factorization SamePattern<br>
>> linear system matrix = precond matrix:<br>
>> Mat Object: 2 MPI processes<br>
>> type: mpiaij<br>
>> rows=6, cols=6, bs=6<br>
>> total: nonzeros=36, allocated nonzeros=36<br>
>> total number of mallocs used during MatSetValues calls =0<br>
>> using I-node (on process 0) routines: found 2 nodes, limit used is 5<br>
>><br>
>> Note that the coarse grid is now using superlu_dist. Is the coarse<br>
>> grid being solved in parallel?<br>
>><br>
>> Garth<br>
><br>
</div></div></blockquote></div><br></div></div></div>