<div dir="ltr"><div dir="ltr">On Fri, Jun 30, 2023 at 12:08 PM Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se">carl-johan.thore@liu.se</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-7772678620092781316">
<div lang="EN-US" style="overflow-wrap: break-word;">
<div class="m_-7772678620092781316WordSection1">
<p class="MsoNormal">“Possibly, but if you are doing FD, then there is built-in topology in DMDA that is not present in Plex, so<u></u><u></u></p>
<p class="MsoNormal">finding the neighbors in the right order is harder (possible, but harder, we address this in some new work that is not yet merged). There is also structured adaptive support with DMForest, but this also does not preserve the stencil.<u></u><u></u></p>
<p class="MsoNormal">“<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I’m using an FEM which doesn’t utilize such neighbor info, so perhaps Plex or DMForest could be easier to use then</p></div></div></div></blockquote><div><br></div><div>Oh yes. Then I would recommend at least looking at Plex and Forest. FEM is much better supported than in DMDA. By the end of the year, I should have everything in place to seamlessly support libCEED for assembly as well.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-7772678620092781316"><div lang="EN-US" style="overflow-wrap: break-word;"><div class="m_-7772678620092781316WordSection1">
<p class="MsoNormal">“The efficiency of active set VI solvers in PETSc demonstrates to me that solving reduced systems can be done efficiently with geometric multigrid using a structured grid so I would not suggest giving up on what you started. <u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> You can do it in two steps<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">1) Use PCREDISTRIBUTE but hack the code in redistribute.c to not move dof between MPI ranks, just have it remove the locked rows/columns (to start just run on one MPI rank since then nothing is moved) Then in your code you just need to
pull out the appropriate rows and columns of the interpolation that correspond to the dof you have kept and pass this smaller interpolation to the inner KSP PCMG. This is straightforward and like what is in DMSetVI. The MG convergence should be just as good
as on the full system.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">2) the only problem with 1 is it is likely to be poorly load balanced (but you can make some runs to see how imbalanced it is, that will depend exactly on what parts are locked and what MPI processes they are on). So if it is poorly balanced
then you would need to get out of redistribute.c a mapping for each kept dof to what MPI rank it is moved to and use that to move the entries in the reduced interpolation you have created. <u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> If you do succeed it would actually be useful code that we could add to PCREDISTRIBUTE for more general use by others.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> Barry<u></u><u></u></p>
<p class="MsoNormal">“<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks, that seems doable, if not super easy. I might try that
<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Kind regards<u></u><u></u></p>
<p class="MsoNormal">/Carl-Johan<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b>From:</b> Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> <br>
<b>Sent:</b> Friday, June 30, 2023 5:21 PM<br>
<b>To:</b> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Cc:</b> Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" target="_blank">carl-johan.thore@liu.se</a>>; <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><br>
<b>Subject:</b> Re: [petsc-users] PCMG with PCREDISTRIBUTE<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal"><br>
<br>
<u></u><u></u></p>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div>
<p class="MsoNormal">On Jun 30, 2023, at 10:22 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">On Fri, Jun 30, 2023 at 10:16 AM Carl-Johan Thore via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<u></u><u></u></p>
</div>
<div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal">Thanks for the quick reply and the suggestions!<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">“ … you should first check that the PCMG works quite well “<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Yes, the PCMG works very well for the full system.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">“I am guessing that your code is slightly different than ex42.c because you take the interpolation matrix provided by the DM <u></u><u></u></p>
<p class="MsoNormal">and give it to the inner KSP PCMG?. So you solve problem 2 but not problem 1.”<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Yes, it’s slightly different so problem 2 should be solved.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">It looked somewhat complicated to get PCMG to work with redistribute, so I’ll try with PCGAMG first<u></u><u></u></p>
<p class="MsoNormal">(it ran immediately with redistribute, but was slower than PCMG on my, very small, test problem. I’ll try<u></u><u></u></p>
<p class="MsoNormal">to tune the settings).
<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">A related question: I’m here using a DMDA for a structured grid but I’m locking so many DOFs that for many of the elements<u></u><u></u></p>
<p class="MsoNormal">all DOFs are locked. In such a case could it make sense to switch/convert the DMDA to a DMPlex containing only those<u></u><u></u></p>
<p class="MsoNormal">elements that actually have DOFs?<u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Possibly, but if you are doing FD, then there is built-in topology in DMDA that is not present in Plex, so<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">finding the neighbors in the right order is harder (possible, but harder, we address this in some new work that is not yet merged). There is also structured adaptive support with DMForest, but this also does not preserve the stencil.<u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"> The efficiency of active set VI solvers in PETSc demonstrates to me that solving reduced systems can be done efficiently with geometric multigrid using a structured grid so I would not suggest giving up on what you started. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> You can do it in two steps<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">1) Use PCREDISTRIBUTE but hack the code in redistribute.c to not move dof between MPI ranks, just have it remove the locked rows/columns (to start just run on one MPI rank since then nothing is moved) Then in your code you just need to
pull out the appropriate rows and columns of the interpolation that correspond to the dof you have kept and pass this smaller interpolation to the inner KSP PCMG. This is straightforward and like what is in DMSetVI. The MG convergence should be just as good
as on the full system.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">2) the only problem with 1 is it is likely to be poorly load balanced (but you can make some runs to see how imbalanced it is, that will depend exactly on what parts are locked and what MPI processes they are on). So if it is poorly balanced
then you would need to get out of redistribute.c a mapping for each kept dof to what MPI rank it is moved to and use that to move the entries in the reduced interpolation you have created. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> If you do succeed it would actually be useful code that we could add to PCREDISTRIBUTE for more general use by others.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> Barry<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<u></u><u></u></p>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div>
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> Thanks,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> Matt<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b>From:</b> Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>>
<br>
<b>Sent:</b> Friday, June 30, 2023 3:57 PM<br>
<b>To:</b> Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" target="_blank">carl-johan.thore@liu.se</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><br>
<b>Subject:</b> Re: [petsc-users] PCMG with PCREDISTRIBUTE<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<p class="MsoNormal"> Oh, I forgot to mention you should first check that the PCMG works quite well for the full system (without the PCREDISTRIBUTE); the convergence<u></u><u></u></p>
<div>
<p class="MsoNormal">on the redistributed system (assuming you did all the work to get PCMG to work for you) should be very similar to (but not measurably better) than the convergence on the full system.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><u></u> <u></u></p>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div>
<p class="MsoNormal">On Jun 30, 2023, at 9:17 AM, Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:<u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<p class="MsoNormal"> ex42.c provides directly the interpolation/restriction needed to move between levels in the loop<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"> for (k = 1; k < nlevels; k++) {<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> PetscCall(DMCreateInterpolation(da_list[k - 1], da_list[k], &R, NULL));<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> PetscCall(PCMGSetInterpolation(pc, k, R));<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> PetscCall(MatDestroy(&R));<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> }<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">The more standard alternative to this is to call KSPSetDM() and have the PCMG setup use the DM<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">to construct the interpolations (I don't know why ex42.c does this construction itself instead of having the KSPSetDM() process handle it but that doesn't matter). The end result
is the same in both cases.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Since PCREDISTRIBUTE builds its own new matrix (by using only certain rows and columns of the original matrix) the original interpolation<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">cannot be used for two reasons<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">1) (since it is for the full system) It is for the wrong problem. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2) In addition, if you ran with ex42.c the inner KSP does not have access to the interpolation that was constructed so you could not get PCMG to to work as indicated below.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I am guessing that your code is slightly different than ex42.c because you take the interpolation matrix provided by the DM <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">and give it to the inner KSP PCMG?. So you solve problem 2 but not problem 1.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">So the short answer is that there is no "canned" way to use the PCMG process trivially with PCDISTRIBUTE. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">To do what you want requires two additional steps<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">1) after you construct the full interpolation matrix (by using the DM) you need to remove the rows associated with the dof that have been removed by the "locked" variables (and
the columns that are associated with coarse grid points that live on the removed points) so that the interpolation is the correct "size" for the smaller problem<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2) since PCREDISTRIBUTE actually moves dof of freedom between MPI processes for load balancing after it has removed the locked variables you would need to do the exact same movement
for the rows of the interpolation matrix that you have constructed (after you have removed the "locked" rows of the interpolation.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Lots of bookkeeping to acheive 1 and 2 but conceptually simple.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">As an experiment you can try using PCGAMG on the redistributed matrix -redistribute_pc_type gamg to use algebraic multigrid just to see the time and convergence rates. Since GAMG
creates its own interpolation based on the matrix and it will be built on the smaller redistributed matrix there will no issue with the wrong "sized" interpolation. Of course you have the overhead of algebraic multigrid and cannot take advantage of geometric
multigrid. The GAMG approach may be satisfactory to your needs.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">If you are game for looking more closely at using redistribute with geometric multigrid and PETSc (which will require digging into PETSc source code and using internal information
in the PETSc source code) you can start by looking at how we solve variational problems with SNES using reduced space active set methods. SNESVINEWTONRSLS /src/snes/impls/vi/rs/virs.c This code solves problem 1 see() it builds the entire interpolation and
then pulls out the required non-locked part. Reduced space active set methods essentially lock the constrained dof and solve a smaller system without those dof at each iteration.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">But it does not solve problem 2. Moving the rows of the "smaller" interpolation to the correct MPI process based on where PCREDISTRIBUTE moved rows. To do this would requring looking
at the PCREDISTRUBUTE code and extracting the information of where each row was moving and performing the process for the interpolation matrix.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">src/ksp/pc/impls/redistribute/redistribute.c<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> Barry<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><u></u> <u></u></p>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div>
<p class="MsoNormal">On Jun 30, 2023, at 8:21 AM, Carl-Johan Thore via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Hi,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I'm trying to run an iterative solver (FGMRES for example) with PCMG as preconditioner. The setup of PCMG<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">is done roughly as in ex42 of the PETSc-tutorials (<a href="https://petsc.org/main/src/ksp/ksp/tutorials/ex42.c.html" target="_blank">https://petsc.org/main/src/ksp/ksp/tutorials/ex42.c.html</a>).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Since I have many locked degrees-of-freedom I would like to use PCREDISTRIBUTE. However, this<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">results in (30039 is the number of DOFs after redistribute and 55539 the number before):<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[0]PETSC ERROR: Nonconforming object sizes<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[0]PETSC ERROR: Matrix dimensions of A and P are incompatible for MatProductType PtAP: A 30039x30039, P 55539x7803<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[0]PETSC ERROR: See<span class="m_-7772678620092781316m-3777929816871973446apple-converted-space"> </span><a href="https://petsc.org/release/faq/" target="_blank">https://petsc.org/release/faq/</a><span class="m_-7772678620092781316m-3777929816871973446apple-converted-space"> </span>for
trouble shooting.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[0]PETSC ERROR: Petsc Development GIT revision: v3.19.0-238-g512d1ae6db4 GIT Date: 2023-04-24 16:37:00 +0200<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[0]PETSC ERROR: topopt on a arch-linux-c-opt Fri Jun 30 13:28:41 2023<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[0]PETSC ERROR: Configure options COPTFLAGS="-O3 -march=native" CXXOPTFLAGS="-O3 -march=native" FOPTFLAGS="-O3 -march=native" CUDAOPTFLAGS=-O3 --with-cuda --with-cusp --with-debugging=0
--download-scalapack --download-hdf5 --download-zlib --download-mumps --download-parmetis --download-metis --download-ptscotch --download-hypre --download-spai<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[0]PETSC ERROR: #1 MatProductSetFromOptions_Private() at /mnt/c/mathware/petsc/src/mat/interface/matproduct.c:420<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[0]PETSC ERROR: #2 MatProductSetFromOptions() at /mnt/c/mathware/petsc/src/mat/interface/matproduct.c:541<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[0]PETSC ERROR: #3 MatPtAP() at /mnt/c/mathware/petsc/src/mat/interface/matrix.c:9868<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[0]PETSC ERROR: #4 MatGalerkin() at /mnt/c/mathware/petsc/src/mat/interface/matrix.c:10899<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[0]PETSC ERROR: #5 PCSetUp_MG() at /mnt/c/mathware/petsc/src/ksp/pc/impls/mg/mg.c:1029<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[0]PETSC ERROR: #6 PCSetUp() at /mnt/c/mathware/petsc/src/ksp/pc/interface/precon.c:994<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[0]PETSC ERROR: #7 KSPSetUp() at /mnt/c/mathware/petsc/src/ksp/ksp/interface/itfunc.c:406<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[0]PETSC ERROR: #8 PCSetUp_Redistribute() at /mnt/c/mathware/petsc/src/ksp/pc/impls/redistribute/redistribute.c:327<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[0]PETSC ERROR: #9 PCSetUp() at /mnt/c/mathware/petsc/src/ksp/pc/interface/precon.c:994<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[0]PETSC ERROR: #10 KSPSetUp() at /mnt/c/mathware/petsc/src/ksp/ksp/interface/itfunc.c:406<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[0]PETSC ERROR: #11 KSPSolve_Private() at /mnt/c/mathware/petsc/src/ksp/ksp/interface/itfunc.c:824<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[0]PETSC ERROR: #12 KSPSolve() at /mnt/c/mathware/petsc/src/ksp/ksp/interface/itfunc.c:1070<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">It’s clear what happens I think, and it kind of make since not all levels are redistributed as they should (?).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Is it possible to use PCMG with PCREDISTRIBUTE in an easy way?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Kind regards,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Carl-Johan<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><span class="m_-7772678620092781316gmailsignatureprefix">-- </span><u></u><u></u></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div></blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>