[petsc-users] Using BDDC preconditioner for assembled matrices

Thu Oct 25 09:19:53 CDT 2018

Right now, one to four. I am just running some tests with small matrices.
Later on, I am planning to do large scale tests hopefully up to 1024
processes. I was worried that iteration numbers may get worse.

I actually use hybridization and I was reading the preprint "Algebraic
Hybridization and Static Condensation with Application to Scalable H(div)
Preconditioning" by Dobrev et al. ( https://arxiv.org/abs/1801.08914 ) and
they show that multigrid is optimal for the grad-div problem discretized
with H(div) conforming FEMs when hybridized. That is actually why I think
that BDDC also would be optimal. I will look into ngsolve to see if I can
have such a domain decomposition. Maybe I can do it manually just as proof
of concept.

I am using GMRES. I was wondering if the application of BDDC is a linear
operator, if it is not maybe I should use FGMRES. But I could not find any
comments about that.

I will recompile PETSc with ParMETIS and try your suggestions. Thank you! I
will update you soon.

Best wishes,
Abdullah Ali Sivas

On Thu, 25 Oct 2018 at 09:53, Stefano Zampini <stefano.zampini at gmail.com>
wrote:

> How many processes (subdomains) are you using?
> I would not say the number of iterations is bad, and it also seems to
> plateau.
> The grad-div problem is quite hard to be solved (unless you use
> hybridization), you can really benefit from the "Neumann" assembly.
> I believe you are using GMRES, as the preconditioned operator (i.e
> M_BDDC^-1 A) is not guaranteed to be positive definite when you use the
> automatic disassembling.
> You may slightly improve the quality of the disassembling by using
> -mat_is_disassemble_l2g_type nd -mat_partitioning_type parmetis if you have
> PETSc compiled with ParMETIS support.
>
>
> Il giorno mer 24 ott 2018 alle ore 20:59 Abdullah Ali Sivas <
> abdullahasivas at gmail.com> ha scritto:
>
>> Hi Stefano,
>>
>> I am trying to solve the div-div problem (or grad-div problem in strong
>> form) with a H(div)-conforming FEM. I am getting the matrices from an
>> external source (to be clear, from an ngsolve script) and I am not sure if
>> it is possible to get a MATIS matrix out of that. So I am just treating it
>> as if I am not able to access the assembly code. The results are 2, 31, 26,
>> 27, 31 iterations, respectively, for matrix sizes 282, 1095, 4314, 17133,
>> 67242, 267549. However, norm of the residual also grows significantly;
>> 7.38369e-09 for 1095 and 5.63828e-07 for 267549. I can try larger sizes, or
>> maybe this is expected for this case.
>>
>> As a side question, if we are dividing the domain into number of MPI
>> processes subdomains, does it mean that convergence is affected negatively
>> by the increasing number of processes? I know that alternating Schwarz
>> method and some other domain decomposition methods sometimes suffer from
>> the decreasing radius of the subdomains. It sounds like BDDC is pretty
>> similar to those by your description.
>>
>> Best wishes,
>> Abdullah Ali Sivas
>>
>> On Wed, 24 Oct 2018 at 05:28, Stefano Zampini <stefano.zampini at gmail.com>
>> wrote:
>>
>>> Abdullah,
>>>
>>> The "Neumann" problems Jed is referring to result from assembling your
>>> problem on each subdomain ( = MPI process) separately.
>>> Assuming you are using FEM, these problems have been historically  named
>>> "Neumann" as they correspond to a problem with natural boundary conditions
>>> (Neumann bc for Poisson).
>>> Note that in PETSc the subdomain decomposition is associated with the
>>> mesh decomposition.
>>>
>>> When converting from an assembled AIJ matrix to a MATIS format, such
>>> "Neumann" information is lost.
>>> You can disassemble an AIJ matrix, in the sense that you can find local
>>> matrices A_j such that A = \sum_j R^T_j A_j R_j (as it is done in ex72.c),
>>> but you cannot guarantee (unless if you solve an optimization problem) that
>>> the disassembling will produce subdomain Neumann problems that are
>>> consistent with your FEM problem.
>>>
>>> I have added such disassembling code a few months ago, just to have
>>> another alternative for preconditioning AIJ matrices in PETSc; there are
>>> few tweaks one can do to improve the quality of the disassembling, but I
>>> discourage its usage unless you don't have access to the FEM assembly code.
>>>
>>> With that said, what problem are you trying to solve? Are you using DMDA
>>> or DMPlex? What are the results you obtained with using the automatic
>>> disassembling?
>>>
>>> Il giorno mer 24 ott 2018 alle ore 08:14 Abdullah Ali Sivas <
>>> abdullahasivas at gmail.com> ha scritto:
>>>
>>>> Hi Jed,
>>>>
>>>> Thanks for your reply. The assembled matrix I have corresponds to the
>>>> full problem on the full mesh. There are no "Neumann" problems (or any sort
>>>> of domain decomposition) defined in the code generates the matrix. However,
>>>> I think assembling the full problem is equivalent to implicitly assembling
>>>> the "Neumann" problems, since the system can be partitioned as;
>>>>
>>>> [A_{LL} | A_{LI}]  [u_L]     [F]
>>>> -----------|------------ -------- = -----
>>>> [A_{IL}  |A_{II} ]   [u_I]      [G]
>>>>
>>>> and G should correspond to the Neumann problem. I might be thinking
>>>> wrong (or maybe I completely misunderstood the idea), if so please correct
>>>> me. But I think that the problem is that I am not explicitly telling PCBDDC
>>>> which dofs are interface dofs.
>>>>
>>>> Regards,
>>>> Abdullah Ali Sivas
>>>>
>>>> On Tue, 23 Oct 2018 at 23:16, Jed Brown <jed at jedbrown.org> wrote:
>>>>
>>>>> Did you assemble "Neumann" problems that are compatible with your
>>>>> definition of interior/interface degrees of freedom?
>>>>>
>>>>> Abdullah Ali Sivas <abdullahasivas at gmail.com> writes:
>>>>>
>>>>> > Dear all,
>>>>> >
>>>>> > I have a series of linear systems coming from a PDE for which BDDC
>>>>> is an
>>>>> > optimal preconditioner. These linear systems are assembled and I
>>>>> read them
>>>>> > from a file, then convert into MATIS as required (as in
>>>>> >
>>>>> https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex72.c.html
>>>>> > ). I expect each of the systems converge to the solution in almost
>>>>> same
>>>>> > number of iterations but I don't observe it. I think it is because I
>>>>> do not
>>>>> > provide enough information to the preconditioner. I can get a list
>>>>> of inner
>>>>> > dofs and interface dofs. However, I do not know how to use them. Has
>>>>> anyone
>>>>> > have any insights about it or done something similar?
>>>>> >
>>>>> > Best wishes,
>>>>> > Abdullah Ali Sivas
>>>>>
>>>>
>>>
>>> --
>>> Stefano
>>>
>>
>
> --
> Stefano
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181025/56c4a119/attachment.html>