<br><br><div class="gmail_quote">2012/1/9 Jed Brown <span dir="ltr"><<a href="mailto:jedbrown@mcs.anl.gov">jedbrown@mcs.anl.gov</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="gmail_quote">Cc'ing petsc-dev to get other opinions.</div><div class="gmail_quote"><br></div></blockquote><div><br>Great.<br> <br></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="gmail_quote"></div><div class="gmail_quote"><div class="im">On Mon, Jan 9, 2012 at 06:56, Stefano Zampini <span dir="ltr"><<a href="mailto:stefano.zampini@gmail.com" target="_blank">stefano.zampini@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>Because of the way I built the code, different coarse problem types set communicators and operators in a different way. Specifically,<br>
<br>- MULTILEVEL_BDDC: mat_type=MATIS, coarse_comm=comm of parent, coarse_pc=PCBDDC.<br>- PARALLEL_BDDC: mat_type=MATMPIAIJ, coarse_comm=comm of parent, coarse_pc=PCREDUNDANT.<br></div></blockquote><div><br></div></div><div>
The only distinction here is the matrix type.</div><div class="im">
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>
- SEQUENTIAL_BDDC (only rank 0 creates it): mat_type=MATSEQAIJ, coarse_comm=PETSC_COMM_SELF, coarse_pc=PCLU.<br></div></blockquote><div><br></div></div><div>This should be handled through a mechanism like PCREDUNDANT so that the caller does not need to know. This "reduce to subcommunicator, broadcast the result" would be simple to add to PCREDUNDANT.</div>
<div class="im">
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>
- REPLICATED_BDDC (all ranks creates the same sequential ksp): mat_type=MATSEQAIJ, coarse_comm=PETSC_COMM_SELF, coarse_pc=PCLU.<br></div></blockquote><div><br></div></div><div>What does this have over using PCREDUNDANT as in case 2?</div>
<div class="im">
<div></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
</div><div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div> indeed a possible user code<br>
<br>PCBDDCGetCoarseKSP(pcbddc,&coarse_ksp)<br>
KSPGetPC(coarse_ksp,&coarse_pc)<br>PCSetType(coarse_pc,PCBDDC) (or something else)<br>PCSetup(pcbddc)<br><br>breaks at the second line since pcbddc->coarse_ksp doesn't exist before calling PCSetup. </div></blockquote>
<div><br></div></div><div>You can make PCBDDCGetCoarseKSP() return a KSP that has not had its type set yet.</div><div><div> </div></div></div></blockquote></div><div><br>By creating the KSP object internally to the function call? In this case the problem remains since the user should need to know a priori the type of communicator associated with the KSP. Can we change the communicator after we created the KSP object?</div>
</blockquote></div></div><br><div>No, but I think that the coarse communicator should probably always be the same as for the whole problem, and then use PCREDUNDANT or similar if we want to reduce to a subcomm.</div><div>
<br></div></blockquote><div><br><br>
Except for MULTILEVEL_BDDC, all other approaches solve exactly the
coarse problem thus there are no differences in theoretical results. In practice,
they are there for performances optimizations. REPLICATED_BDDC is
performed directly using allgatherv operations and it is an heritage
from my previous codes. I can drop it. If you can manage the
SEQUENTIAL_BDDC case directly in PCREDUNDANT, then we will be half-way. Enums on coarse communications type will be eliminated too as communications can be all made using VecScatters.<br>
<br> </div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div></div>
<div>But this brings up an issue that also appears in GAMG, how to expose user access to coarse KSPs even though they are created on adaptively sized subcomms. If we configure strictly through the options database, then we are okay, but what if we have to pass any information directly to the KSP? We can set operators and call PCSetUp(), then pull out levels, but what should be returned on ranks that are not participating?</div>
</blockquote></div><br>Coarse KSP in my multilevel BDDC implementation is created on the same comm of parent; ranks not partecipating in coarse comm will create all objects needed my the levels even if they have a local dimension equal to zero.<br>
<br><br><br clear="all"><br>-- <br>Stefano<br>