[petsc-users] Is it possible to keep track of original elements # after a call to DMPlexDistribute ?
Matthew Knepley
knepley at gmail.com
Wed Oct 6 20:23:05 CDT 2021
On Wed, Oct 6, 2021 at 5:43 PM Eric Chamberland <
Eric.Chamberland at giref.ulaval.ca> wrote:
> Hi Matthew,
>
> we tried to use that. Now, we discovered that:
>
> 1- even if we "ask" for sfNatural creation with DMSetUseNatural, it is not
> created because DMPlexCreateGlobalToNaturalSF looks for a "section": this
> is not documented in DMSetUseNaturalso we are asking ourselfs: "is this a
> permanent feature or a temporary situation?"
>
> I think explaining this will help clear up a lot.
What the Natural2Global map does is permute a solution vector into the
ordering that it would have had prior to mesh distribution.
Now, in order to do this permutation, I need to know the original (global)
data layout. If it is not specified _before_ distribution, we
cannot build the permutation. The section describes the data layout, so I
need it before distribution.
I cannot think of another way that you would implement this, but if you
want something else, let me know.
> 2- We then tried to create a "section" in different manners: we took the
> code into the example petsc/src/dm/impls/plex/tests/ex15.c. However, we
> ended up with a segfault:
>
> corrupted size vs. prev_size
> [rohan:07297] *** Process received signal ***
> [rohan:07297] Signal: Aborted (6)
> [rohan:07297] Signal code: (-6)
> [rohan:07297] [ 0] /lib64/libpthread.so.0(+0x13f80)[0x7f6f13be3f80]
> [rohan:07297] [ 1] /lib64/libc.so.6(gsignal+0x10b)[0x7f6f109b718b]
> [rohan:07297] [ 2] /lib64/libc.so.6(abort+0x175)[0x7f6f109b8585]
> [rohan:07297] [ 3] /lib64/libc.so.6(+0x7e2f7)[0x7f6f109fb2f7]
> [rohan:07297] [ 4] /lib64/libc.so.6(+0x857ea)[0x7f6f10a027ea]
> [rohan:07297] [ 5] /lib64/libc.so.6(+0x86036)[0x7f6f10a03036]
> [rohan:07297] [ 6] /lib64/libc.so.6(+0x861a3)[0x7f6f10a031a3]
> [rohan:07297] [ 7] /lib64/libc.so.6(+0x88740)[0x7f6f10a05740]
> [rohan:07297] [ 8] /lib64/libc.so.6(__libc_malloc+0x1b8)[0x7f6f10a070c8]
> [rohan:07297] [ 9]
> /lib64/libc.so.6(__backtrace_symbols+0x134)[0x7f6f10a8b064]
> [rohan:07297] [10]
> /home/mefpp_ericc/GIREF/bin/MEF++.dev(_Z12reqBacktraceRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x4e)[0x4538ce]
> [rohan:07297] [11]
> /home/mefpp_ericc/GIREF/bin/MEF++.dev(_Z15attacheDebuggerv+0x120)[0x4523c0]
> [rohan:07297] [12]
> /home/mefpp_ericc/GIREF/lib/libgiref_dev_Util.so(traitementSignal+0x612)[0x7f6f28f503a2]
> [rohan:07297] [13] /lib64/libc.so.6(+0x3a210)[0x7f6f109b7210]
>
> [rohan:07297] [14]
> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscTrMallocDefault+0x6fd)[0x7f6f22f1b8ed]
> [rohan:07297] [15]
> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscMallocA+0x5cd)[0x7f6f22f19c2d]
> [rohan:07297] [16]
> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscSFCreateSectionSF+0xb48)[0x7f6f23268e18]
> [rohan:07297] [17]
> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(DMPlexCreateGlobalToNaturalSF+0x13b2)[0x7f6f241a5602]
> [rohan:07297] [18]
> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(DMPlexDistribute+0x39b1)[0x7f6f23fdca21]
>
> I am not sure what happened here, but if you could send a sample code, I
will figure it out.
> If we do not create a section, the call to DMPlexDistribute is successful,
> but DMPlexGetGlobalToNaturalSF return a null SF pointer...
>
Yes, it just ignores it in this case because it does not have a global
layout.
> Here are the operations we are calling ( this is almost the code we are
> using, I just removed verifications and creation of the connectivity which
> use our parallel structure and code):
>
> ===========
>
> PetscInt* lCells = 0;
> PetscInt lNumCorners = 0;
> PetscInt lDimMail = 0;
> PetscInt lnumCells = 0;
>
> //At this point we create the cells for PETSc expected input for
> DMPlexBuildFromCellListParallel and set lNumCorners, lDimMail and lnumCells
> to correct values.
> ...
>
> DM lDMBete = 0
> DMPlexCreate(lMPIComm,&lDMBete);
>
> DMSetDimension(lDMBete, lDimMail);
>
> DMPlexBuildFromCellListParallel(lDMBete,
> lnumCells,
> PETSC_DECIDE,
>
> pLectureElementsLocaux.reqNbTotalSommets(),
> lNumCorners,
> lCells,
> PETSC_NULL);
>
> DM lDMBeteInterp = 0;
> DMPlexInterpolate(lDMBete, &lDMBeteInterp);
> DMDestroy(&lDMBete);
> lDMBete = lDMBeteInterp;
>
> DMSetUseNatural(lDMBete,PETSC_TRUE);
>
> PetscSF lSFMigrationSansOvl = 0;
> PetscSF lSFMigrationOvl = 0;
> DM lDMDistribueSansOvl = 0;
> DM lDMAvecOverlap = 0;
>
> PetscPartitioner lPart;
> DMPlexGetPartitioner(lDMBete, &lPart);
> PetscPartitionerSetFromOptions(lPart);
>
> PetscSection section;
> PetscInt numFields = 1;
> PetscInt numBC = 0;
> PetscInt numComp[1] = {1};
> PetscInt numDof[4] = {1, 0, 0, 0};
> PetscInt bcFields[1] = {0};
> IS bcPoints[1] = {NULL};
>
> DMSetNumFields(lDMBete, numFields);
>
> DMPlexCreateSection(lDMBete, NULL, numComp, numDof, numBC, bcFields,
> bcPoints, NULL, NULL, §ion);
> DMSetLocalSection(lDMBete, section);
>
> DMPlexDistribute(lDMBete, 0, &lSFMigrationSansOvl,
> &lDMDistribueSansOvl); // segfault!
>
> ===========
>
> So we have other question/remarks:
>
> 3- Maybe PETSc expect something specific that is missing/not verified: for
> example, we didn't gave any coordinates since we just want to partition and
> compute overlap for the mesh... and then recover our element numbers in a
> "simple way"
>
> 4- We are telling ourselves it is somewhat a "big price to pay" to have to
> build an unused section to have the global to natural ordering set ? Could
> this requirement be avoided?
>
I don't think so. There would have to be _some_ way of describing your data
layout in terms of mesh points, and I do not see how you could use less
memory doing that.
> 5- Are there any improvement towards our usages in 3.16 release?
>
Let me try and run the code above.
Thanks,
Matt
> Thanks,
>
> Eric
>
>
> On 2021-09-29 7:39 p.m., Matthew Knepley wrote:
>
> On Wed, Sep 29, 2021 at 5:18 PM Eric Chamberland <
> Eric.Chamberland at giref.ulaval.ca> wrote:
>
>> Hi,
>>
>> I come back with _almost_ the original question:
>>
>> I would like to add an integer information (*our* original element
>> number, not petsc one) on each element of the DMPlex I create with
>> DMPlexBuildFromCellListParallel.
>>
>> I would like this interger to be distribruted by or the same way
>> DMPlexDistribute distribute the mesh.
>>
>> Is it possible to do this?
>>
>
> I think we already have support for what you want. If you call
>
> https://petsc.org/main/docs/manualpages/DM/DMSetUseNatural.html
>
> before DMPlexDistribute(), it will compute a PetscSF encoding the global
> to natural map. You
> can get it with
>
>
> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetGlobalToNaturalSF.html
>
> and use it with
>
>
> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGlobalToNaturalBegin.html
>
> Is this sufficient?
>
> Thanks,
>
> Matt
>
>
>> Thanks,
>>
>> Eric
>>
>> On 2021-07-14 1:18 p.m., Eric Chamberland wrote:
>> > Hi,
>> >
>> > I want to use DMPlexDistribute from PETSc for computing overlapping
>> > and play with the different partitioners supported.
>> >
>> > However, after calling DMPlexDistribute, I noticed the elements are
>> > renumbered and then the original number is lost.
>> >
>> > What would be the best way to keep track of the element renumbering?
>> >
>> > a) Adding an optional parameter to let the user retrieve a vector or
>> > "IS" giving the old number?
>> >
>> > b) Adding a DMLabel (seems a wrong good solution)
>> >
>> > c) Other idea?
>> >
>> > Of course, I don't want to loose performances with the need of this
>> > "mapping"...
>> >
>> > Thanks,
>> >
>> > Eric
>> >
>> --
>> Eric Chamberland, ing., M. Ing
>> Professionnel de recherche
>> GIREF/Université Laval
>> (418) 656-2131 poste 41 22 42
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
> --
> Eric Chamberland, ing., M. Ing
> Professionnel de recherche
> GIREF/Université Laval
> (418) 656-2131 poste 41 22 42
>
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20211006/186fb72f/attachment.html>
More information about the petsc-users
mailing list