[petsc-users] DMLocalToLocal with DMPlex in Fortran

[Matthew Knepley]

On Tue, Oct 11, 2022 at 6:04 PM Mike Michell <mi.mike1021 at gmail.com> wrote:

> Thank you for the reply and checking.
> Indeed, it seems that local-to-local halo is still not implemented for
> DMPlex. But I believe this is a very required feature for large 3D
> simulations with DMPlex.
>

It is possible that it will make a difference. An explicit code, running
completely from local vectors, which assembles its own residuals without
mapping to global vectors,
could probably realize some gain. It would be interesting to see the
numbers.


> Would you mind if I ask for any estimated timeline to fix this issue and
> put it on the official version of PETSc? If I remember correctly, we had a
> similar discussion a few months ago.
>

I cannot do it until November since we have a big review coming up.

  Thanks,

     Matt


> Thanks,
> Mike
>
> On Mon, Oct 10, 2022 at 10:41 PM Mike Michell <mi.mike1021 at gmail.com>
>> wrote:
>>
>>> Hi, I was wondering if there is any comment on the example file that I
>>> can refer to.
>>>
>>
>> I see the problem. Local2Local is not implemented for Plex. I thought we
>> had this automated, but it was only
>> coded for DMDA. It is a fairly mechanical transformation of the
>> Global2Local, just remapping indices, but it
>> will take some time since there is a lot of backlog this semester.
>>
>> I have fixed the error message so now it is obvious what the problem is.
>>
>>   Thanks,
>>
>>       Matt
>>
>>
>>> Thanks,
>>> Mike
>>>
>>>
>>>> Thank you for the reply.
>>>> Sure, a short example code is attached here with a square box mesh and
>>>> a run script.
>>>> Inside the source, you may find two versions of halo exchange; one is
>>>> for local to global (Version-1) and another one is for local to local
>>>> (Version-2), which is not working in my case.
>>>> In the output.vtu, you will see the halo exchanged vector resolved to
>>>> each vertex with (myrank + 1), so if the code is running with 2procs, at
>>>> the parallel boundary, you will see 3. In this example, there is no ghost
>>>> layer.
>>>>
>>>> Thanks,
>>>> Mike
>>>>
>>>>
>>>>> On Sat, Oct 1, 2022 at 8:51 PM Mike Michell <mi.mike1021 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thank you for the reply. There is that file in
>>>>>> src/dm/interface/ftn-auto/ for me, instead of the path you mentioned.
>>>>>>
>>>>>> After "make allfortranstubs" was done and, PETSc reconfigured and
>>>>>> reinstalled.
>>>>>>
>>>>>> However, I still have the same problem at the line in which
>>>>>> DMLocalToLocalBegin() is used. What I am doing to setup halo exchange is as
>>>>>> follows;
>>>>>> - declare DMPlex
>>>>>> - PetscSectionCreate()
>>>>>> - PetscSectionSetNumFields()
>>>>>> - PetscSectionSetFieldComponents()
>>>>>> - PetscSectionSetChart()
>>>>>> - do loop over dofs: PetscSectionSetDof() and
>>>>>> PetscSectionSetFieldDof()
>>>>>> - PetscSectionSetUp()
>>>>>> - DMSetLocalSection()
>>>>>> - PetscSectionDestroy()
>>>>>> - DMGetSectionSF()
>>>>>> - PetscSFSetUp()
>>>>>>
>>>>>> Then, the halo exchange is performed as follows;
>>>>>> - DMGetLocalVector()
>>>>>> - Fill the local vector
>>>>>> - DMLocalToLocalBegin() --(*)
>>>>>> - DMLocalToLocalEnd()
>>>>>> - DMRestoreLocalVector()
>>>>>>
>>>>>> Then, the code crashes at (*).
>>>>>>
>>>>>
>>>>> Can you send something I can run? Then I will find the problem and fix
>>>>> it.
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>      Matt
>>>>>
>>>>>
>>>>>> Previously(at the time PETSc did not support LocalToLocal for DMPlex
>>>>>> in Fortran), the part above, "DMLocalToLocalBegin() and
>>>>>> DMLocalToLocalEnd()", consisted of;
>>>>>> - DMLocalToGlobalBegin()
>>>>>> - DMLocalToGlobalEnd()
>>>>>> - DMGlobalToLocalBegin()
>>>>>> - DMGlobalToLocalEnd()
>>>>>> and it worked okay.
>>>>>>
>>>>>> I am unclear which part is causing the problem. Shall I define the
>>>>>> PetscSection and PetscSF in different ways in case of Local to Local Halo
>>>>>> exchange?
>>>>>> Any comment will be appreciated.
>>>>>>
>>>>>> Thanks,
>>>>>> Mike
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Fri, Sep 30, 2022 at 4:14 PM Mike Michell <mi.mike1021 at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> As a follow-up to this email thread,
>>>>>>>> https://www.mail-archive.com/petsc-users@mcs.anl.gov/msg44070.html
>>>>>>>>
>>>>>>>> Are DMLocalToLocalBegin() and DMLocalToLocalEnd() really available
>>>>>>>> for DMPlex with Fortran on the latest version of PETSc (3.17.99 from
>>>>>>>> GitLab)? Matt commented that the Fortran bindings were updated so that
>>>>>>>> those functions must be available in the latest version of PETSc, however,
>>>>>>>> it seems still they are not working from my test with DMPlex in Fortran.
>>>>>>>> Can anyone provide some comments? Probably I am missing some mandatory
>>>>>>>> header file? Currently, I have headers;
>>>>>>>>
>>>>>>>> #include "petsc/finclude/petscvec.h"
>>>>>>>> #include "petsc/finclude/petscdmplex.h"
>>>>>>>> #include "petsc/finclude/petscdmlabel.h"
>>>>>>>> #include "petsc/finclude/petscdm.h"
>>>>>>>>
>>>>>>>
>>>>>>> The source for these functions is in
>>>>>>>
>>>>>>>   src/dm/ftn-auto/dmf.c
>>>>>>>
>>>>>>> Is it there for you? If not, you can run
>>>>>>>
>>>>>>>   make allfortranstubs
>>>>>>>
>>>>>>> Fortran functions are not declared, so the header should not matter
>>>>>>> for compilation, just the libraries for linking.
>>>>>>>
>>>>>>>   Thanks,
>>>>>>>
>>>>>>>      Matt
>>>>>>>
>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Mike
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20221012/c18b69ad/attachment-0001.html>