[petsc-users] Is it possible to keep track of original elements # after a call to DMPlexDistribute ?
Eric Chamberland
Eric.Chamberland at giref.ulaval.ca
Wed Oct 20 14:06:36 CDT 2021
Hi Matthew,
we tried to reproduce the error in a simple example.
The context is the following: We hard coded the mesh and initial
partition into the code (see sConnectivity and sInitialPartition) for 2
ranks and try to create a section in order to use the
DMPlexNaturalToGlobalBegin function to retreive our initial element numbers.
Now the call to DMPlexDistribute give different errors depending on what
type of component we ask the field to be created. For our objective, we
would like a global field to be created on elements only (like a P0
interpolation).
We now have the following error generated:
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Petsc has generated inconsistent data
[0]PETSC ERROR: Inconsistency in indices, 18 should be 17
[0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html
for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.15.0, Mar 30, 2021
[0]PETSC ERROR: ./bug on a named rohan by ericc Wed Oct 20 14:52:36 2021
[0]PETSC ERROR: Configure options
--prefix=/opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7
--with-mpi-compilers=1 --with-mpi-dir=/opt/openmpi-4.1.0_gcc7
--with-cxx-dialect=C++14 --with-make-np=12 --with-shared-libraries=1
--with-debugging=yes --with-memalign=64 --with-visibility=0
--with-64-bit-indices=0 --download-ml=yes --download-mumps=yes
--download-superlu=yes --download-hpddm=yes --download-slepc=yes
--download-superlu_dist=yes --download-parmetis=yes
--download-ptscotch=yes --download-metis=yes --download-strumpack=yes
--download-suitesparse=yes --download-hypre=yes
--with-blaslapack-dir=/opt/intel/oneapi/mkl/2021.1.1/env/../lib/intel64
--with-mkl_pardiso-dir=/opt/intel/oneapi/mkl/2021.1.1/env/..
--with-mkl_cpardiso-dir=/opt/intel/oneapi/mkl/2021.1.1/env/..
--with-scalapack=1
--with-scalapack-include=/opt/intel/oneapi/mkl/2021.1.1/env/../include
--with-scalapack-lib="-L/opt/intel/oneapi/mkl/2021.1.1/env/../lib/intel64
-lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64"
[0]PETSC ERROR: #1 PetscSFCreateSectionSF() at
/tmp/ompi-opt/petsc-3.15.0-debug/src/vec/is/sf/utils/sfutils.c:409
[0]PETSC ERROR: #2 DMPlexCreateGlobalToNaturalSF() at
/tmp/ompi-opt/petsc-3.15.0-debug/src/dm/impls/plex/plexnatural.c:184
[0]PETSC ERROR: #3 DMPlexDistribute() at
/tmp/ompi-opt/petsc-3.15.0-debug/src/dm/impls/plex/plexdistribute.c:1733
[0]PETSC ERROR: #4 main() at bug_section.cc:159
[0]PETSC ERROR: No PETSc Option Table entries
[0]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------
Hope the attached code is self-explaining, note that to make it short,
we have not included the final part of it, just the buggy part we are
encountering right now...
Thanks for your insights,
Eric
On 2021-10-06 9:23 p.m., Matthew Knepley wrote:
> On Wed, Oct 6, 2021 at 5:43 PM Eric Chamberland
> <Eric.Chamberland at giref.ulaval.ca
> <mailto:Eric.Chamberland at giref.ulaval.ca>> wrote:
>
> Hi Matthew,
>
> we tried to use that. Now, we discovered that:
>
> 1- even if we "ask" for sfNatural creation with DMSetUseNatural,
> it is not created because DMPlexCreateGlobalToNaturalSF looks for
> a "section": this is not documented in DMSetUseNaturalso we are
> asking ourselfs: "is this a permanent feature or a temporary
> situation?"
>
> I think explaining this will help clear up a lot.
>
> What the Natural2Global map does is permute a solution vector into the
> ordering that it would have had prior to mesh distribution.
> Now, in order to do this permutation, I need to know the original
> (global) data layout. If it is not specified _before_ distribution, we
> cannot build the permutation. The section describes the data layout,
> so I need it before distribution.
>
> I cannot think of another way that you would implement this, but if
> you want something else, let me know.
>
> 2- We then tried to create a "section" in different manners: we
> took the code into the example
> petsc/src/dm/impls/plex/tests/ex15.c. However, we ended up with a
> segfault:
>
> corrupted size vs. prev_size
> [rohan:07297] *** Process received signal ***
> [rohan:07297] Signal: Aborted (6)
> [rohan:07297] Signal code: (-6)
> [rohan:07297] [ 0] /lib64/libpthread.so.0(+0x13f80)[0x7f6f13be3f80]
> [rohan:07297] [ 1] /lib64/libc.so.6(gsignal+0x10b)[0x7f6f109b718b]
> [rohan:07297] [ 2] /lib64/libc.so.6(abort+0x175)[0x7f6f109b8585]
> [rohan:07297] [ 3] /lib64/libc.so.6(+0x7e2f7)[0x7f6f109fb2f7]
> [rohan:07297] [ 4] /lib64/libc.so.6(+0x857ea)[0x7f6f10a027ea]
> [rohan:07297] [ 5] /lib64/libc.so.6(+0x86036)[0x7f6f10a03036]
> [rohan:07297] [ 6] /lib64/libc.so.6(+0x861a3)[0x7f6f10a031a3]
> [rohan:07297] [ 7] /lib64/libc.so.6(+0x88740)[0x7f6f10a05740]
> [rohan:07297] [ 8]
> /lib64/libc.so.6(__libc_malloc+0x1b8)[0x7f6f10a070c8]
> [rohan:07297] [ 9]
> /lib64/libc.so.6(__backtrace_symbols+0x134)[0x7f6f10a8b064]
> [rohan:07297] [10]
> /home/mefpp_ericc/GIREF/bin/MEF++.dev(_Z12reqBacktraceRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x4e)[0x4538ce]
> [rohan:07297] [11]
> /home/mefpp_ericc/GIREF/bin/MEF++.dev(_Z15attacheDebuggerv+0x120)[0x4523c0]
> [rohan:07297] [12]
> /home/mefpp_ericc/GIREF/lib/libgiref_dev_Util.so(traitementSignal+0x612)[0x7f6f28f503a2]
> [rohan:07297] [13] /lib64/libc.so.6(+0x3a210)[0x7f6f109b7210]
>
> [rohan:07297] [14]
> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscTrMallocDefault+0x6fd)[0x7f6f22f1b8ed]
> [rohan:07297] [15]
> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscMallocA+0x5cd)[0x7f6f22f19c2d]
> [rohan:07297] [16]
> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscSFCreateSectionSF+0xb48)[0x7f6f23268e18]
> [rohan:07297] [17]
> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(DMPlexCreateGlobalToNaturalSF+0x13b2)[0x7f6f241a5602]
> [rohan:07297] [18]
> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(DMPlexDistribute+0x39b1)[0x7f6f23fdca21]
>
> I am not sure what happened here, but if you could send a sample code,
> I will figure it out.
>
> If we do not create a section, the call to DMPlexDistribute is
> successful, but DMPlexGetGlobalToNaturalSF return a null SF pointer...
>
> Yes, it just ignores it in this case because it does not have a global
> layout.
>
> Here are the operations we are calling ( this is almost the code
> we are using, I just removed verifications and creation of the
> connectivity which use our parallel structure and code):
>
> ===========
>
> PetscInt* lCells = 0;
> PetscInt lNumCorners = 0;
> PetscInt lDimMail = 0;
> PetscInt lnumCells = 0;
>
> //At this point we create the cells for PETSc expected input for
> DMPlexBuildFromCellListParallel and set lNumCorners, lDimMail and
> lnumCells to correct values.
> ...
>
> DM lDMBete = 0
> DMPlexCreate(lMPIComm,&lDMBete);
>
> DMSetDimension(lDMBete, lDimMail);
>
> DMPlexBuildFromCellListParallel(lDMBete,
> lnumCells,
> PETSC_DECIDE,
> pLectureElementsLocaux.reqNbTotalSommets(),
> lNumCorners,
> lCells,
> PETSC_NULL);
>
> DM lDMBeteInterp = 0;
> DMPlexInterpolate(lDMBete, &lDMBeteInterp);
> DMDestroy(&lDMBete);
> lDMBete = lDMBeteInterp;
>
> DMSetUseNatural(lDMBete,PETSC_TRUE);
>
> PetscSF lSFMigrationSansOvl = 0;
> PetscSF lSFMigrationOvl = 0;
> DM lDMDistribueSansOvl = 0;
> DM lDMAvecOverlap = 0;
>
> PetscPartitioner lPart;
> DMPlexGetPartitioner(lDMBete, &lPart);
> PetscPartitionerSetFromOptions(lPart);
>
> PetscSection section;
> PetscInt numFields = 1;
> PetscInt numBC = 0;
> PetscInt numComp[1] = {1};
> PetscInt numDof[4] = {1, 0, 0, 0};
> PetscInt bcFields[1] = {0};
> IS bcPoints[1] = {NULL};
>
> DMSetNumFields(lDMBete, numFields);
>
> DMPlexCreateSection(lDMBete, NULL, numComp, numDof, numBC,
> bcFields, bcPoints, NULL, NULL, §ion);
> DMSetLocalSection(lDMBete, section);
>
> DMPlexDistribute(lDMBete, 0, &lSFMigrationSansOvl,
> &lDMDistribueSansOvl); // segfault!
>
> ===========
>
> So we have other question/remarks:
>
> 3- Maybe PETSc expect something specific that is missing/not
> verified: for example, we didn't gave any coordinates since we
> just want to partition and compute overlap for the mesh... and
> then recover our element numbers in a "simple way"
>
> 4- We are telling ourselves it is somewhat a "big price to pay" to
> have to build an unused section to have the global to natural
> ordering set ? Could this requirement be avoided?
>
> I don't think so. There would have to be _some_ way of describing your
> data layout in terms of mesh points, and I do not see how you could
> use less memory doing that.
>
> 5- Are there any improvement towards our usages in 3.16 release?
>
> Let me try and run the code above.
>
> Thanks,
>
> Matt
>
> Thanks,
>
> Eric
>
>
> On 2021-09-29 7:39 p.m., Matthew Knepley wrote:
>> On Wed, Sep 29, 2021 at 5:18 PM Eric Chamberland
>> <Eric.Chamberland at giref.ulaval.ca
>> <mailto:Eric.Chamberland at giref.ulaval.ca>> wrote:
>>
>> Hi,
>>
>> I come back with _almost_ the original question:
>>
>> I would like to add an integer information (*our* original
>> element
>> number, not petsc one) on each element of the DMPlex I create
>> with
>> DMPlexBuildFromCellListParallel.
>>
>> I would like this interger to be distribruted by or the same way
>> DMPlexDistribute distribute the mesh.
>>
>> Is it possible to do this?
>>
>>
>> I think we already have support for what you want. If you call
>>
>> https://petsc.org/main/docs/manualpages/DM/DMSetUseNatural.html
>> <https://petsc.org/main/docs/manualpages/DM/DMSetUseNatural.html>
>>
>> before DMPlexDistribute(), it will compute a PetscSF encoding the
>> global to natural map. You
>> can get it with
>>
>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetGlobalToNaturalSF.html
>> <https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetGlobalToNaturalSF.html>
>>
>> and use it with
>>
>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGlobalToNaturalBegin.html
>> <https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGlobalToNaturalBegin.html>
>>
>> Is this sufficient?
>>
>> Thanks,
>>
>> Matt
>>
>> Thanks,
>>
>> Eric
>>
>> On 2021-07-14 1:18 p.m., Eric Chamberland wrote:
>> > Hi,
>> >
>> > I want to use DMPlexDistribute from PETSc for computing
>> overlapping
>> > and play with the different partitioners supported.
>> >
>> > However, after calling DMPlexDistribute, I noticed the
>> elements are
>> > renumbered and then the original number is lost.
>> >
>> > What would be the best way to keep track of the element
>> renumbering?
>> >
>> > a) Adding an optional parameter to let the user retrieve a
>> vector or
>> > "IS" giving the old number?
>> >
>> > b) Adding a DMLabel (seems a wrong good solution)
>> >
>> > c) Other idea?
>> >
>> > Of course, I don't want to loose performances with the need
>> of this
>> > "mapping"...
>> >
>> > Thanks,
>> >
>> > Eric
>> >
>> --
>> Eric Chamberland, ing., M. Ing
>> Professionnel de recherche
>> GIREF/Université Laval
>> (418) 656-2131 poste 41 22 42
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to
>> which their experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>
> --
> Eric Chamberland, ing., M. Ing
> Professionnel de recherche
> GIREF/Université Laval
> (418) 656-2131 poste 41 22 42
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
--
Eric Chamberland, ing., M. Ing
Professionnel de recherche
GIREF/Université Laval
(418) 656-2131 poste 41 22 42
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20211020/da1720e1/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bug_section.cc
Type: text/x-c++src
Size: 4728 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20211020/da1720e1/attachment-0001.bin>
More information about the petsc-users
mailing list