Today I solved the bug for more than three levels with MULTILEVEL BDDC. If I have time, tomorrow I will adapt the code to deal with local matrices having a NearNullSpace object attached. Then the next step will be remove explicit calls to METIS and use MatPartitioning routines. Jed: can you give me some quick hints on how they behave? In particular, I wish to know if I can assemble the adjacency on N procs and then partition it in M procs (with M<N).<br>
<br><div class="gmail_quote">2012/1/9 Stefano Zampini <span dir="ltr"><<a href="mailto:stefano.zampini@gmail.com">stefano.zampini@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br><br><div class="gmail_quote"><div class="im">2012/1/9 Jed Brown <span dir="ltr"><<a href="mailto:jedbrown@mcs.anl.gov" target="_blank">jedbrown@mcs.anl.gov</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="gmail_quote">Cc'ing petsc-dev to get other opinions.</div><div class="gmail_quote"><br></div></blockquote></div><div><br>Great.<br> <br></div><div class="im"><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="gmail_quote"></div><div class="gmail_quote"><div>On Mon, Jan 9, 2012 at 06:56, Stefano Zampini <span dir="ltr"><<a href="mailto:stefano.zampini@gmail.com" target="_blank">stefano.zampini@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>Because of the way I built the code, different coarse problem types set communicators and operators in a different way. Specifically,<br>
<br>- MULTILEVEL_BDDC: mat_type=MATIS, coarse_comm=comm of parent, coarse_pc=PCBDDC.<br>- PARALLEL_BDDC: mat_type=MATMPIAIJ, coarse_comm=comm of parent, coarse_pc=PCREDUNDANT.<br></div></blockquote><div><br></div></div><div>
The only distinction here is the matrix type.</div><div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>
- SEQUENTIAL_BDDC (only rank 0 creates it): mat_type=MATSEQAIJ, coarse_comm=PETSC_COMM_SELF, coarse_pc=PCLU.<br></div></blockquote><div><br></div></div><div>This should be handled through a mechanism like PCREDUNDANT so that the caller does not need to know. This "reduce to subcommunicator, broadcast the result" would be simple to add to PCREDUNDANT.</div>
<div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>
- REPLICATED_BDDC (all ranks creates the same sequential ksp): mat_type=MATSEQAIJ, coarse_comm=PETSC_COMM_SELF, coarse_pc=PCLU.<br></div></blockquote><div><br></div></div><div>What does this have over using PCREDUNDANT as in case 2?</div>
<div>
<div></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
</div><div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div> indeed a possible user code<br>
<br>PCBDDCGetCoarseKSP(pcbddc,&coarse_ksp)<br>
KSPGetPC(coarse_ksp,&coarse_pc)<br>PCSetType(coarse_pc,PCBDDC) (or something else)<br>PCSetup(pcbddc)<br><br>breaks at the second line since pcbddc->coarse_ksp doesn't exist before calling PCSetup. </div></blockquote>
<div><br></div></div><div>You can make PCBDDCGetCoarseKSP() return a KSP that has not had its type set yet.</div><div><div> </div></div></div></blockquote></div><div><br>By creating the KSP object internally to the function call? In this case the problem remains since the user should need to know a priori the type of communicator associated with the KSP. Can we change the communicator after we created the KSP object?</div>
</blockquote></div></div><br><div>No, but I think that the coarse communicator should probably always be the same as for the whole problem, and then use PCREDUNDANT or similar if we want to reduce to a subcomm.</div><div>
<br></div></blockquote></div><div><br><br>
Except for MULTILEVEL_BDDC, all other approaches solve exactly the
coarse problem thus there are no differences in theoretical results. In practice,
they are there for performances optimizations. REPLICATED_BDDC is
performed directly using allgatherv operations and it is an heritage
from my previous codes. I can drop it. If you can manage the
SEQUENTIAL_BDDC case directly in PCREDUNDANT, then we will be half-way. Enums on coarse communications type will be eliminated too as communications can be all made using VecScatters.<br>
<br> </div><div class="im"><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div></div>
<div>But this brings up an issue that also appears in GAMG, how to expose user access to coarse KSPs even though they are created on adaptively sized subcomms. If we configure strictly through the options database, then we are okay, but what if we have to pass any information directly to the KSP? We can set operators and call PCSetUp(), then pull out levels, but what should be returned on ranks that are not participating?</div>
</blockquote></div></div><br>Coarse KSP in my multilevel BDDC implementation is created on the same comm of parent; ranks not partecipating in coarse comm will create all objects needed my the levels even if they have a local dimension equal to zero.<span class="HOEnZb"><font color="#888888"><br>
<br><br><br clear="all"><br>-- <br>Stefano<br>
</font></span></blockquote></div><br><br clear="all"><br>-- <br>Stefano<br>