<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <p>Hi Matthew,</p>
    <p>ok, good news then! :)<br>
    </p>
    <p>Meanwhile, I tested it the "other way" with
      DMPlexGlobalToNaturalBegin/End and it does not give a "sensed"
      result neither.  <br>
    </p>
    <p>Probably it does something like "GlobalToSequential" too?</p>
    <p>Thanks!</p>
    <p>Eric</p>
    <p><br>
    </p>
    <div class="moz-cite-prefix">On 2021-11-01 3:25 p.m., Matthew
      Knepley wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAMYG4Gkr7efVgpZPuYFgM6oWBJw68pkD_T87dFrL+3Ht2KPy0w@mail.gmail.com">
      <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      <div dir="ltr">
        <div dir="ltr">On Sun, Oct 31, 2021 at 10:07 AM Eric Chamberland
          <<a href="mailto:Eric.Chamberland@giref.ulaval.ca"
            moz-do-not-send="true">Eric.Chamberland@giref.ulaval.ca</a>>
          wrote:<br>
        </div>
        <div class="gmail_quote">
          <blockquote class="gmail_quote" style="margin:0px 0px 0px
            0.8ex;border-left:1px solid
            rgb(204,204,204);padding-left:1ex">
            <div>
              <p>Hi Matthew,</p>
              <p>we do not know if DMPlexNaturalToGlobalBegin/End is
                buggy or if it is our comprehension of what it should do
                that is wrong...<br>
              </p>
              <p>Would you just check if what we try to do from line 313
                to 356 is good or wrong?</p>
              <p>The expected result is that the global vector
                "lGlobalVec" contains the element numbers from the
                initial partition that have been put into "lNatVec".<br>
              </p>
              <p>Thanks a lot for any insights!!</p>
            </div>
          </blockquote>
          <div>Okay, I have found the problem.I confess to not actually
            looking very closely at the implementation before.
            GlobalToNatural was assuming that</div>
          <div>the "natural" was also sequential, and looking at it all
            the tests are eq-to-parallel distribution. I need to fix it
            for a parallel natural state. Hopefully</div>
          <div>it will not take me long.</div>
          <div><br>
          </div>
          <div>  Thanks,</div>
          <div><br>
          </div>
          <div>     Matt </div>
          <blockquote class="gmail_quote" style="margin:0px 0px 0px
            0.8ex;border-left:1px solid
            rgb(204,204,204);padding-left:1ex">
            <div>
              <p>Eric<br>
              </p>
              <p><br>
              </p>
              <div>On 2021-10-27 2:32 p.m., Eric Chamberland wrote:<br>
              </div>
              <blockquote type="cite">
                <p>Hi Matthew,</p>
                <p>we continued the example.  Now it must be our misuse
                  of PETSc that produced the wrong result.</p>
                <p>As stated into the code:</p>
                <p>// The call to DMPlexNaturalToGlobalBegin/End does
                  not produce our expected result...<br>
                    // In lGlobalVec, we expect to have:<br>
                    /*<br>
                     * Process [0]<br>
                     * 2.<br>
                     * 4.<br>
                     * 8.<br>
                     * 3.<br>
                     * 9.<br>
                     * Process [1]<br>
                     * 1.<br>
                     * 5.<br>
                     * 7.<br>
                     * 0.<br>
                     * 6.<br>
                     *<br>
                     * but we obtained:<br>
                     *<br>
                     * Process [0]<br>
                     * 2.<br>
                     * 4.<br>
                     * 8.<br>
                     * 0.<br>
                     * 0.<br>
                     * Process [1]<br>
                     * 0.<br>
                     * 0.<br>
                     * 0.<br>
                     * 0.<br>
                     * 0.<br>
                     */</p>
                <p>(see attached ex44.c)</p>
                <p>Thanks,</p>
                <p>Eric<br>
                </p>
                <div>On 2021-10-27 1:25 p.m., Eric Chamberland wrote:<br>
                </div>
                <blockquote type="cite">
                  <p>Great!</p>
                  <p>Thanks Matthew, it is working for me up to that
                    point!</p>
                  <p>We are continuing the ex44.c and forward it to you
                    at the next blocking point...</p>
                  <p>Eric<br>
                  </p>
                  <div>On 2021-10-27 11:14 a.m., Matthew Knepley wrote:<br>
                  </div>
                  <blockquote type="cite">
                    <div dir="ltr">
                      <div dir="ltr">On Wed, Oct 27, 2021 at 8:29 AM
                        Eric Chamberland <<a
                          href="mailto:Eric.Chamberland@giref.ulaval.ca"
                          target="_blank" moz-do-not-send="true">Eric.Chamberland@giref.ulaval.ca</a>>
                        wrote:<br>
                      </div>
                      <div class="gmail_quote">
                        <blockquote class="gmail_quote"
                          style="margin:0px 0px 0px
                          0.8ex;border-left:1px solid
                          rgb(204,204,204);padding-left:1ex">
                          <div>
                            <p>Hi Matthew,</p>
                            <p>the smallest mesh which crashes the code
                              is a 2x5 mesh:</p>
                            <p><img
                                src="cid:part3.D30D23C7.AB83C5CF@giref.ulaval.ca"
                                alt="" class=""></p>
                            <p>See the modified ex44.c</p>
                            <p>With smaller meshes(2x2, 2x4, etc), it
                              passes...  But it bugs latter when I try
                              to use DMPlexNaturalToGlobalBegin but
                              let's keep that other problem for later...</p>
                            <p>Thanks a lot for helping digging into
                              this! :)</p>
                          </div>
                        </blockquote>
                        <div>I have made a small fix in this branch</div>
                        <div><br>
                        </div>
                        <div>  <a
                            href="https://gitlab.com/petsc/petsc/-/commits/knepley/fix-plex-g2n"
                            target="_blank" moz-do-not-send="true">https://gitlab.com/petsc/petsc/-/commits/knepley/fix-plex-g2n</a></div>
                        <div><br>
                        </div>
                        <div>It seems to run for me. Can you check it?</div>
                        <div><br>
                        </div>
                        <div>  Thanks,</div>
                        <div><br>
                        </div>
                        <div>     Matt</div>
                        <div> </div>
                        <blockquote class="gmail_quote"
                          style="margin:0px 0px 0px
                          0.8ex;border-left:1px solid
                          rgb(204,204,204);padding-left:1ex">
                          <div>
                            <p>Eric</p>
                            <p>(sorry if you received this for a  2nd
                              times, I have trouble with my mail)<br>
                            </p>
                            <div>On 2021-10-26 4:35 p.m., Matthew
                              Knepley wrote:<br>
                            </div>
                            <blockquote type="cite">
                              <div dir="ltr">
                                <div dir="ltr">On Tue, Oct 26, 2021 at
                                  1:35 PM Eric Chamberland <<a
                                    href="mailto:Eric.Chamberland@giref.ulaval.ca"
                                    target="_blank"
                                    moz-do-not-send="true">Eric.Chamberland@giref.ulaval.ca</a>>
                                  wrote:<br>
                                </div>
                                <div class="gmail_quote">
                                  <blockquote class="gmail_quote"
                                    style="margin:0px 0px 0px
                                    0.8ex;border-left:1px solid
                                    rgb(204,204,204);padding-left:1ex">
                                    <div>
                                      <p>Here is a screenshot of the
                                        partition I hard coded (top) and
                                        vertices/element numbers (down):<br>
                                      </p>
                                      <p><img
                                          src="cid:part6.FA10DB20.57B6D759@giref.ulaval.ca"
                                          alt="" class=""></p>
                                      <p>I have not yet modified the
                                        ex44.c example to properly
                                        assign the coordinates... <br>
                                      </p>
                                      <p>(but I would not have done it
                                        like it is in the last version
                                        because the sCoords array is the
                                        global array with global
                                        vertices number)</p>
                                      <p>I will have time to do this
                                        tomorrow...</p>
                                      <p>Maybe I can first try to
                                        reproduce all this with a
                                        smaller mesh?<br>
                                      </p>
                                    </div>
                                  </blockquote>
                                  <div><br>
                                  </div>
                                  <div>That might make it easier to find
                                    a problem.</div>
                                  <div><br>
                                  </div>
                                  <div>  Thanks!</div>
                                  <div><br>
                                  </div>
                                  <div>     Matt</div>
                                  <div> </div>
                                  <blockquote class="gmail_quote"
                                    style="margin:0px 0px 0px
                                    0.8ex;border-left:1px solid
                                    rgb(204,204,204);padding-left:1ex">
                                    <div>
                                      <p> </p>
                                      <p>Eric<br>
                                      </p>
                                      <div>On 2021-10-26 9:46 a.m.,
                                        Matthew Knepley wrote:<br>
                                      </div>
                                      <blockquote type="cite">
                                        <div dir="ltr">Okay, I ran it.
                                          Something seems off with the
                                          mesh. First, I cannot simply
                                          explain the partition. The
                                          number of shared vertices and
                                          edges
                                          <div>does not seem to come
                                            from a straight cut. Second,
                                            the mesh look scrambled on
                                            output.</div>
                                          <div><br>
                                          </div>
                                          <div>  Thanks,</div>
                                          <div><br>
                                          </div>
                                          <div>    Matt</div>
                                        </div>
                                        <br>
                                        <div class="gmail_quote">
                                          <div dir="ltr"
                                            class="gmail_attr">On Sun,
                                            Oct 24, 2021 at 11:49 PM
                                            Eric Chamberland <<a
                                              href="mailto:Eric.Chamberland@giref.ulaval.ca"
                                              target="_blank"
                                              moz-do-not-send="true">Eric.Chamberland@giref.ulaval.ca</a>>
                                            wrote:<br>
                                          </div>
                                          <blockquote
                                            class="gmail_quote"
                                            style="margin:0px 0px 0px
                                            0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
                                            <div>
                                              <p>Hi Matthew,</p>
                                              <p>ok, I started back from
                                                your ex44.c example and
                                                added the global array
                                                of coordinates.  I just
                                                have to code the
                                                creation of the local
                                                coordinates now.</p>
                                              <p>Eric<br>
                                              </p>
                                              <div>On 2021-10-20 6:55
                                                p.m., Matthew Knepley
                                                wrote:<br>
                                              </div>
                                              <blockquote type="cite">
                                                <div dir="ltr">
                                                  <div dir="ltr">On Wed,
                                                    Oct 20, 2021 at 3:06
                                                    PM Eric Chamberland
                                                    <<a
                                                      href="mailto:Eric.Chamberland@giref.ulaval.ca"
                                                      target="_blank"
                                                      moz-do-not-send="true">Eric.Chamberland@giref.ulaval.ca</a>>
                                                    wrote:<br>
                                                  </div>
                                                  <div
                                                    class="gmail_quote">
                                                    <blockquote
                                                      class="gmail_quote"
                                                      style="margin:0px
                                                      0px 0px
                                                      0.8ex;border-left:1px
                                                      solid
                                                      rgb(204,204,204);padding-left:1ex">
                                                      <div>
                                                        <p>Hi Matthew,</p>
                                                        <p>we tried to
                                                          reproduce the
                                                          error in a
                                                          simple
                                                          example.</p>
                                                        <p>The context
                                                          is the
                                                          following: We
                                                          hard coded the
                                                          mesh and
                                                          initial
                                                          partition into
                                                          the code (see
                                                          sConnectivity
                                                          and
                                                          sInitialPartition)
                                                          for 2 ranks
                                                          and try to
                                                          create a
                                                          section in
                                                          order to use
                                                          the
                                                          DMPlexNaturalToGlobalBegin
                                                          function to
                                                          retreive our
                                                          initial
                                                          element
                                                          numbers.</p>
                                                        <p>Now the call
                                                          to
                                                          DMPlexDistribute
                                                          give different
                                                          errors
                                                          depending on
                                                          what type of
                                                          component we
                                                          ask the field
                                                          to be
                                                          created.  For
                                                          our objective,
                                                          we would like
                                                          a global field
                                                          to be created
                                                          on elements
                                                          only (like a
                                                          P0
                                                          interpolation).<br>
                                                        </p>
                                                        <p>We now have
                                                          the following
                                                          error
                                                          generated:</p>
                                                        <p>[0]PETSC
                                                          ERROR:
                                                          ---------------------
                                                          Error Message
--------------------------------------------------------------<br>
                                                          [0]PETSC
                                                          ERROR: Petsc
                                                          has generated
                                                          inconsistent
                                                          data<br>
                                                          [0]PETSC
                                                          ERROR:
                                                          Inconsistency
                                                          in indices, 18
                                                          should be 17<br>
                                                          [0]PETSC
                                                          ERROR: See <a
href="https://www.mcs.anl.gov/petsc/documentation/faq.html"
                                                          target="_blank"
moz-do-not-send="true">https://www.mcs.anl.gov/petsc/documentation/faq.html</a>
                                                          for trouble
                                                          shooting.<br>
                                                          [0]PETSC
                                                          ERROR: Petsc
                                                          Release
                                                          Version
                                                          3.15.0, Mar
                                                          30, 2021 <br>
                                                          [0]PETSC
                                                          ERROR: ./bug
                                                          on a  named
                                                          rohan by ericc
                                                          Wed Oct 20
                                                          14:52:36 2021<br>
                                                          [0]PETSC
                                                          ERROR:
                                                          Configure
                                                          options
                                                          --prefix=/opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7
--with-mpi-compilers=1 --with-mpi-dir=/opt/openmpi-4.1.0_gcc7
                                                          --with-cxx-dialect=C++14
--with-make-np=12 --with-shared-libraries=1 --with-debugging=yes
                                                          --with-memalign=64
--with-visibility=0 --with-64-bit-indices=0 --download-ml=yes
                                                          --download-mumps=yes
--download-superlu=yes --download-hpddm=yes --download-slepc=yes
                                                          --download-superlu_dist=yes
--download-parmetis=yes --download-ptscotch=yes --download-metis=yes
                                                          --download-strumpack=yes
--download-suitesparse=yes --download-hypre=yes
--with-blaslapack-dir=/opt/intel/oneapi/mkl/2021.1.1/env/../lib/intel64
--with-mkl_pardiso-dir=/opt/intel/oneapi/mkl/2021.1.1/env/..
                                                          --with-mkl_cpardiso-dir=/opt/intel/oneapi/mkl/2021.1.1/env/..
--with-scalapack=1
                                                          --with-scalapack-include=/opt/intel/oneapi/mkl/2021.1.1/env/../include
--with-scalapack-lib="-L/opt/intel/oneapi/mkl/2021.1.1/env/../lib/intel64
-lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64"<br>
                                                          [0]PETSC
                                                          ERROR: #1
                                                          PetscSFCreateSectionSF()
                                                          at
/tmp/ompi-opt/petsc-3.15.0-debug/src/vec/is/sf/utils/sfutils.c:409<br>
                                                          [0]PETSC
                                                          ERROR: #2
                                                          DMPlexCreateGlobalToNaturalSF()
                                                          at
                                                          /tmp/ompi-opt/petsc-3.15.0-debug/src/dm/impls/plex/plexnatural.c:184<br>
                                                          [0]PETSC
                                                          ERROR: #3
                                                          DMPlexDistribute()
                                                          at
                                                          /tmp/ompi-opt/petsc-3.15.0-debug/src/dm/impls/plex/plexdistribute.c:1733<br>
                                                          [0]PETSC
                                                          ERROR: #4
                                                          main() at
                                                          bug_section.cc:159<br>
                                                          [0]PETSC
                                                          ERROR: No
                                                          PETSc Option
                                                          Table entries<br>
                                                          [0]PETSC
                                                          ERROR:
                                                          ----------------End
                                                          of Error
                                                          Message
                                                          -------send
                                                          entire error
                                                          message to <a
href="mailto:petsc-maint@mcs.anl.gov" target="_blank"
                                                          moz-do-not-send="true">petsc-maint@mcs.anl.gov</a>----------<br>
                                                          <br>
                                                        </p>
                                                        <p>Hope the
                                                          attached code
                                                          is
                                                          self-explaining,
                                                          note that to
                                                          make it short,
                                                          we have not
                                                          included the
                                                          final part of
                                                          it, just the
                                                          buggy part we
                                                          are
                                                          encountering
                                                          right now...</p>
                                                        <p>Thanks for
                                                          your insights,</p>
                                                      </div>
                                                    </blockquote>
                                                    <div>Thanks for
                                                      making the
                                                      example. I tweaked
                                                      it slightly. I put
                                                      in a test case
                                                      that just makes a
                                                      parallel 7 x 10
                                                      quad mesh. This
                                                      works</div>
                                                    <div>fine. Thus I
                                                      think it must be
                                                      something
                                                      connected with the
                                                      original mesh. It
                                                      is hard to get a
                                                      handle on it
                                                      without the
                                                      coordinates.</div>
                                                    <div>Do you think
                                                      you could put the
                                                      coordinate array
                                                      in? I have added
                                                      the code to load
                                                      them (see attached
                                                      file).</div>
                                                    <div><br>
                                                    </div>
                                                    <div>  Thanks,</div>
                                                    <div><br>
                                                    </div>
                                                    <div>     Matt </div>
                                                    <blockquote
                                                      class="gmail_quote"
                                                      style="margin:0px
                                                      0px 0px
                                                      0.8ex;border-left:1px
                                                      solid
                                                      rgb(204,204,204);padding-left:1ex">
                                                      <div>
                                                        <p>Eric<br>
                                                        </p>
                                                        <div>On
                                                          2021-10-06
                                                          9:23 p.m.,
                                                          Matthew
                                                          Knepley wrote:<br>
                                                        </div>
                                                        <blockquote
                                                          type="cite">
                                                          <div dir="ltr">
                                                          <div dir="ltr">On
                                                          Wed, Oct 6,
                                                          2021 at 5:43
                                                          PM Eric
                                                          Chamberland
                                                          <<a
                                                          href="mailto:Eric.Chamberland@giref.ulaval.ca"
target="_blank" moz-do-not-send="true">Eric.Chamberland@giref.ulaval.ca</a>>
                                                          wrote:<br>
                                                          </div>
                                                          <div
                                                          class="gmail_quote">
                                                          <blockquote
                                                          class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
                                                          rgb(204,204,204);padding-left:1ex">
                                                          <div>
                                                          <p>Hi Matthew,</p>
                                                          <p>we tried to
                                                          use that. 
                                                          Now, we
                                                          discovered
                                                          that:</p>
                                                          <p>1- even if
                                                          we "ask" for
                                                          sfNatural
                                                          creation with
DMSetUseNatural, it is not created because
DMPlexCreateGlobalToNaturalSF looks for a "section": this is not
                                                          documented in
DMSetUseNaturalso we are asking ourselfs: "is this a permanent feature
                                                          or a temporary
                                                          situation?"<br>
                                                          </p>
                                                          </div>
                                                          </blockquote>
                                                          <div>I think
                                                          explaining
                                                          this will help
                                                          clear up a
                                                          lot.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>What the
Natural2Global map does is permute a solution vector into the ordering
                                                          that it would
                                                          have had prior
                                                          to mesh
                                                          distribution.</div>
                                                          <div>Now, in
                                                          order to do
                                                          this
                                                          permutation, I
                                                          need to know
                                                          the original
                                                          (global) data
                                                          layout. If it
                                                          is not
                                                          specified
                                                          _before_
                                                          distribution,
                                                          we</div>
                                                          <div>cannot
                                                          build the
                                                          permutation. 
                                                          The section
                                                          describes the
                                                          data layout,
                                                          so I need it
                                                          before
                                                          distribution.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>I cannot
                                                          think of
                                                          another way
                                                          that you would
                                                          implement
                                                          this, but if
                                                          you want
                                                          something
                                                          else, let me
                                                          know.</div>
                                                          <blockquote
                                                          class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
                                                          rgb(204,204,204);padding-left:1ex">
                                                          <div>
                                                          <p>2- We then
                                                          tried to
                                                          create a
                                                          "section" in
                                                          different
                                                          manners: we
                                                          took the code
                                                          into the
                                                          example
                                                          petsc/src/dm/impls/plex/tests/ex15.c. 
                                                          However, we
                                                          ended up with
                                                          a segfault:</p>
                                                          <p>corrupted
                                                          size vs.
                                                          prev_size<br>
                                                          [rohan:07297]
                                                          *** Process
                                                          received
                                                          signal ***<br>
                                                          [rohan:07297]
                                                          Signal:
                                                          Aborted (6)<br>
                                                          [rohan:07297]
                                                          Signal code: 
                                                          (-6)<br>
                                                          [rohan:07297]
                                                          [ 0]
/lib64/libpthread.so.0(+0x13f80)[0x7f6f13be3f80]<br>
                                                          [rohan:07297]
                                                          [ 1]
/lib64/libc.so.6(gsignal+0x10b)[0x7f6f109b718b]<br>
                                                          [rohan:07297]
                                                          [ 2]
/lib64/libc.so.6(abort+0x175)[0x7f6f109b8585]<br>
                                                          [rohan:07297]
                                                          [ 3]
/lib64/libc.so.6(+0x7e2f7)[0x7f6f109fb2f7]<br>
                                                          [rohan:07297]
                                                          [ 4]
/lib64/libc.so.6(+0x857ea)[0x7f6f10a027ea]<br>
                                                          [rohan:07297]
                                                          [ 5]
/lib64/libc.so.6(+0x86036)[0x7f6f10a03036]<br>
                                                          [rohan:07297]
                                                          [ 6]
/lib64/libc.so.6(+0x861a3)[0x7f6f10a031a3]<br>
                                                          [rohan:07297]
                                                          [ 7]
/lib64/libc.so.6(+0x88740)[0x7f6f10a05740]<br>
                                                          [rohan:07297]
                                                          [ 8]
/lib64/libc.so.6(__libc_malloc+0x1b8)[0x7f6f10a070c8]<br>
                                                          [rohan:07297]
                                                          [ 9]
/lib64/libc.so.6(__backtrace_symbols+0x134)[0x7f6f10a8b064]<br>
                                                          [rohan:07297]
                                                          [10]
/home/mefpp_ericc/GIREF/bin/MEF++.dev(_Z12reqBacktraceRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x4e)[0x4538ce]<br>
                                                          [rohan:07297]
                                                          [11]
/home/mefpp_ericc/GIREF/bin/MEF++.dev(_Z15attacheDebuggerv+0x120)[0x4523c0]<br>
                                                          [rohan:07297]
                                                          [12]
/home/mefpp_ericc/GIREF/lib/libgiref_dev_Util.so(traitementSignal+0x612)[0x7f6f28f503a2]<br>
                                                          [rohan:07297]
                                                          [13]
/lib64/libc.so.6(+0x3a210)[0x7f6f109b7210]</p>
                                                          <p>[rohan:07297]
                                                          [14]
/opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscTrMallocDefault+0x6fd)[0x7f6f22f1b8ed]<br>
                                                          [rohan:07297]
                                                          [15]
/opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscMallocA+0x5cd)[0x7f6f22f19c2d]<br>
                                                          [rohan:07297]
                                                          [16]
/opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscSFCreateSectionSF+0xb48)[0x7f6f23268e18]<br>
                                                          [rohan:07297]
                                                          [17]
/opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(DMPlexCreateGlobalToNaturalSF+0x13b2)[0x7f6f241a5602]<br>
                                                          [rohan:07297]
                                                          [18]
/opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(DMPlexDistribute+0x39b1)[0x7f6f23fdca21]<br>
                                                          </p>
                                                          </div>
                                                          </blockquote>
                                                          <div>I am not
                                                          sure what
                                                          happened here,
                                                          but if you
                                                          could send a
                                                          sample code, I
                                                          will figure it
                                                          out. </div>
                                                          <blockquote
                                                          class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
                                                          rgb(204,204,204);padding-left:1ex">
                                                          <div>
                                                          <p>If we do
                                                          not create a
                                                          section, the
                                                          call to
                                                          DMPlexDistribute
                                                          is successful,
                                                          but
DMPlexGetGlobalToNaturalSF return a null SF pointer...<br>
                                                          </p>
                                                          </div>
                                                          </blockquote>
                                                          <div>Yes, it
                                                          just ignores
                                                          it in this
                                                          case because
                                                          it does not
                                                          have a global
                                                          layout. </div>
                                                          <blockquote
                                                          class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
                                                          rgb(204,204,204);padding-left:1ex">
                                                          <div>
                                                          <p> </p>
                                                          <p>Here are
                                                          the operations
                                                          we are calling
                                                          ( this is
                                                          almost the
                                                          code we are
                                                          using, I just
                                                          removed
                                                          verifications
                                                          and creation
                                                          of the
                                                          connectivity
                                                          which use our
                                                          parallel
                                                          structure and
                                                          code):<br>
                                                          </p>
                                                          <p>===========</p>
                                                          <p>  PetscInt*
                                                          lCells      =
                                                          0;<br>
                                                            PetscInt 
                                                          lNumCorners =
                                                          0;<br>
                                                            PetscInt 
                                                          lDimMail    =
                                                          0;<br>
                                                            PetscInt 
                                                          lnumCells   =
                                                          0;<br>
                                                          <br>
                                                            //At this
                                                          point we
                                                          create the
                                                          cells for
                                                          PETSc expected
                                                          input for
DMPlexBuildFromCellListParallel and set lNumCorners, lDimMail and
                                                          lnumCells to
                                                          correct
                                                          values.<br>
                                                            ...<br>
                                                            <br>
                                                            DM      
                                                          lDMBete = 0<br>
                                                           
                                                          DMPlexCreate(lMPIComm,&lDMBete);<br>
                                                          <br>
                                                           
                                                          DMSetDimension(lDMBete,
                                                          lDimMail);<br>
                                                          <br>
                                                           
                                                          DMPlexBuildFromCellListParallel(lDMBete,<br>
                                  lnumCells,<br>
                                  PETSC_DECIDE,<br>
                                 
                                                          pLectureElementsLocaux.reqNbTotalSommets(),<br>
                                  lNumCorners,<br>
                                  lCells,<br>
                                  PETSC_NULL);<br>
                                                          <br>
                                                            DM
                                                          lDMBeteInterp
                                                          = 0;<br>
                                                           
                                                          DMPlexInterpolate(lDMBete,
&lDMBeteInterp);<br>
                                                           
                                                          DMDestroy(&lDMBete);<br>
                                                            lDMBete =
                                                          lDMBeteInterp;<br>
                                                          <br>
                                                           
                                                          DMSetUseNatural(lDMBete,PETSC_TRUE);<br>
                                                          <br>
                                                            PetscSF
                                                          lSFMigrationSansOvl
                                                          = 0;<br>
                                                            PetscSF
                                                          lSFMigrationOvl
                                                          = 0;<br>
                                                            DM
                                                          lDMDistribueSansOvl
                                                          = 0;<br>
                                                            DM
                                                          lDMAvecOverlap
                                                          = 0;<br>
                                                          <br>
                                                           
                                                          PetscPartitioner
                                                          lPart;<br>
                                                           
                                                          DMPlexGetPartitioner(lDMBete,
                                                          &lPart);<br>
                                                           
                                                          PetscPartitionerSetFromOptions(lPart);<br>
                                                          <br>
                                                           
                                                          PetscSection  
                                                          section;<br>
                                                           
                                                          PetscInt      
                                                          numFields   =
                                                          1;<br>
                                                           
                                                          PetscInt      
                                                          numBC       =
                                                          0;<br>
                                                           
                                                          PetscInt      
                                                          numComp[1]  =
                                                          {1};<br>
                                                           
                                                          PetscInt      
                                                          numDof[4]   =
                                                          {1, 0, 0, 0};<br>
                                                           
                                                          PetscInt      
                                                          bcFields[1] =
                                                          {0};<br>
                                                           
                                                          IS            
                                                          bcPoints[1] =
                                                          {NULL};<br>
                                                          <br>
                                                           
                                                          DMSetNumFields(lDMBete,
                                                          numFields);<br>
                                                          <br>
                                                           
                                                          DMPlexCreateSection(lDMBete,
                                                          NULL, numComp,
                                                          numDof, numBC,
                                                          bcFields,
                                                          bcPoints,
                                                          NULL, NULL,
                                                          &section);<br>
                                                           
                                                          DMSetLocalSection(lDMBete,
                                                          section);<br>
                                                          <br>
                                                           
                                                          DMPlexDistribute(lDMBete,
                                                          0,
                                                          &lSFMigrationSansOvl,
&lDMDistribueSansOvl); // segfault!<br>
                                                          </p>
                                                          <p>===========<br>
                                                          </p>
                                                          <p>So we have
                                                          other
                                                          question/remarks:</p>
                                                          <p>3- Maybe
                                                          PETSc expect
                                                          something
                                                          specific that
                                                          is missing/not
                                                          verified: for
                                                          example, we
                                                          didn't gave
                                                          any
                                                          coordinates
                                                          since we just
                                                          want to
                                                          partition and
                                                          compute
                                                          overlap for
                                                          the mesh...
                                                          and then
                                                          recover our
                                                          element
                                                          numbers in a
                                                          "simple way"</p>
                                                          <p>4- We are
                                                          telling
                                                          ourselves it
                                                          is somewhat a
                                                          "big price to
                                                          pay" to have
                                                          to build an
                                                          unused section
                                                          to have the
                                                          global to
                                                          natural
                                                          ordering set
                                                          ?  Could this
                                                          requirement be
                                                          avoided?</p>
                                                          </div>
                                                          </blockquote>
                                                          <div>I don't
                                                          think so.
                                                          There would
                                                          have to be
                                                          _some_ way of
                                                          describing
                                                          your data
                                                          layout in
                                                          terms of mesh
                                                          points, and I
                                                          do not see how
                                                          you could use
                                                          less memory
                                                          doing that. </div>
                                                          <blockquote
                                                          class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
                                                          rgb(204,204,204);padding-left:1ex">
                                                          <div>
                                                          <p>5- Are
                                                          there any
                                                          improvement
                                                          towards our
                                                          usages in 3.16
                                                          release?</p>
                                                          </div>
                                                          </blockquote>
                                                          <div>Let me
                                                          try and run
                                                          the code
                                                          above.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>  Thanks,</div>
                                                          <div><br>
                                                          </div>
                                                          <div>   
                                                           Matt </div>
                                                          <blockquote
                                                          class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
                                                          rgb(204,204,204);padding-left:1ex">
                                                          <div>
                                                          <p>Thanks,</p>
                                                          <p>Eric</p>
                                                          <p><br>
                                                          </p>
                                                          <div>On
                                                          2021-09-29
                                                          7:39 p.m.,
                                                          Matthew
                                                          Knepley wrote:<br>
                                                          </div>
                                                          <blockquote
                                                          type="cite">
                                                          <div dir="ltr">
                                                          <div dir="ltr">On
                                                          Wed, Sep 29,
                                                          2021 at 5:18
                                                          PM Eric
                                                          Chamberland
                                                          <<a
                                                          href="mailto:Eric.Chamberland@giref.ulaval.ca"
target="_blank" moz-do-not-send="true">Eric.Chamberland@giref.ulaval.ca</a>>
                                                          wrote:<br>
                                                          </div>
                                                          <div
                                                          class="gmail_quote">
                                                          <blockquote
                                                          class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
                                                          rgb(204,204,204);padding-left:1ex">Hi,<br>
                                                          <br>
                                                          I come back
                                                          with _almost_
                                                          the original
                                                          question:<br>
                                                          <br>
                                                          I would like
                                                          to add an
                                                          integer
                                                          information
                                                          (*our*
                                                          original
                                                          element <br>
                                                          number, not
                                                          petsc one) on
                                                          each element
                                                          of the DMPlex
                                                          I create with
                                                          <br>
DMPlexBuildFromCellListParallel.<br>
                                                          <br>
                                                          I would like
                                                          this interger
                                                          to be
                                                          distribruted
                                                          by or the same
                                                          way <br>
DMPlexDistribute distribute the mesh.<br>
                                                          <br>
                                                          Is it possible
                                                          to do this?<br>
                                                          </blockquote>
                                                          <div><br>
                                                          </div>
                                                          <div>I think
                                                          we already
                                                          have support
                                                          for what you
                                                          want. If you
                                                          call</div>
                                                          <div><br>
                                                          </div>
                                                          <div>  <a
                                                          href="https://petsc.org/main/docs/manualpages/DM/DMSetUseNatural.html"
target="_blank" moz-do-not-send="true">https://petsc.org/main/docs/manualpages/DM/DMSetUseNatural.html</a></div>
                                                          <div><br>
                                                          </div>
                                                          <div>before
                                                          DMPlexDistribute(),
                                                          it will
                                                          compute a
                                                          PetscSF
                                                          encoding the
                                                          global to
                                                          natural map.
                                                          You</div>
                                                          <div>can get
                                                          it with</div>
                                                          <div><br>
                                                          </div>
                                                          <div>  <a
href="https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetGlobalToNaturalSF.html"
target="_blank" moz-do-not-send="true">https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetGlobalToNaturalSF.html</a></div>
                                                          <div><br>
                                                          </div>
                                                          <div>and use
                                                          it with</div>
                                                          <div><br>
                                                          </div>
                                                          <div>  <a
href="https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGlobalToNaturalBegin.html"
target="_blank" moz-do-not-send="true">https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGlobalToNaturalBegin.html</a></div>
                                                          <div><br>
                                                          </div>
                                                          <div>Is this
                                                          sufficient?</div>
                                                          <div><br>
                                                          </div>
                                                          <div>  Thanks,</div>
                                                          <div><br>
                                                          </div>
                                                          <div>     Matt</div>
                                                          <div> </div>
                                                          <blockquote
                                                          class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
                                                          rgb(204,204,204);padding-left:1ex">
                                                          Thanks,<br>
                                                          <br>
                                                          Eric<br>
                                                          <br>
                                                          On 2021-07-14
                                                          1:18 p.m.,
                                                          Eric
                                                          Chamberland
                                                          wrote:<br>
                                                          > Hi,<br>
                                                          ><br>
                                                          > I want to
                                                          use
                                                          DMPlexDistribute
                                                          from PETSc for
                                                          computing
                                                          overlapping <br>
                                                          > and play
                                                          with the
                                                          different
                                                          partitioners
                                                          supported.<br>
                                                          ><br>
                                                          > However,
                                                          after calling
DMPlexDistribute, I noticed the elements are <br>
                                                          >
                                                          renumbered and
                                                          then the
                                                          original
                                                          number is
                                                          lost.<br>
                                                          ><br>
                                                          > What
                                                          would be the
                                                          best way to
                                                          keep track of
                                                          the element
                                                          renumbering?<br>
                                                          ><br>
                                                          > a) Adding
                                                          an optional
                                                          parameter to
                                                          let the user
                                                          retrieve a
                                                          vector or <br>
                                                          > "IS"
                                                          giving the old
                                                          number?<br>
                                                          ><br>
                                                          > b) Adding
                                                          a DMLabel
                                                          (seems a wrong
                                                          good solution)<br>
                                                          ><br>
                                                          > c) Other
                                                          idea?<br>
                                                          ><br>
                                                          > Of
                                                          course, I
                                                          don't want to
                                                          loose
                                                          performances
                                                          with the need
                                                          of this <br>
                                                          >
                                                          "mapping"...<br>
                                                          ><br>
                                                          > Thanks,<br>
                                                          ><br>
                                                          > Eric<br>
                                                          ><br>
                                                          -- <br>
                                                          Eric
                                                          Chamberland,
                                                          ing., M. Ing<br>
                                                          Professionnel
                                                          de recherche<br>
GIREF/Université Laval<br>
                                                          (418) 656-2131
                                                          poste 41 22 42<br>
                                                          <br>
                                                          </blockquote>
                                                          </div>
                                                          <br
                                                          clear="all">
                                                          <div><br>
                                                          </div>
                                                          -- <br>
                                                          <div dir="ltr">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>What most
                                                          experimenters
                                                          take for
                                                          granted before
                                                          they begin
                                                          their
                                                          experiments is
                                                          infinitely
                                                          more
                                                          interesting
                                                          than any
                                                          results to
                                                          which their
                                                          experiments
                                                          lead.<br>
                                                          -- Norbert
                                                          Wiener</div>
                                                          <div><br>
                                                          </div>
                                                          <div><a
                                                          href="http://www.cse.buffalo.edu/~knepley/"
target="_blank" moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          <pre cols="72">-- 
Eric Chamberland, ing., M. Ing
Professionnel de recherche
GIREF/Université Laval
(418) 656-2131 poste 41 22 42</pre>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br
                                                          clear="all">
                                                          <div><br>
                                                          </div>
                                                          -- <br>
                                                          <div dir="ltr">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>What most
                                                          experimenters
                                                          take for
                                                          granted before
                                                          they begin
                                                          their
                                                          experiments is
                                                          infinitely
                                                          more
                                                          interesting
                                                          than any
                                                          results to
                                                          which their
                                                          experiments
                                                          lead.<br>
                                                          -- Norbert
                                                          Wiener</div>
                                                          <div><br>
                                                          </div>
                                                          <div><a
                                                          href="http://www.cse.buffalo.edu/~knepley/"
target="_blank" moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                        </blockquote>
                                                        <pre cols="72">-- 
Eric Chamberland, ing., M. Ing
Professionnel de recherche
GIREF/Université Laval
(418) 656-2131 poste 41 22 42</pre>
                                                      </div>
                                                    </blockquote>
                                                  </div>
                                                  <br clear="all">
                                                  <div><br>
                                                  </div>
                                                  -- <br>
                                                  <div dir="ltr">
                                                    <div dir="ltr">
                                                      <div>
                                                        <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>What most
                                                          experimenters
                                                          take for
                                                          granted before
                                                          they begin
                                                          their
                                                          experiments is
                                                          infinitely
                                                          more
                                                          interesting
                                                          than any
                                                          results to
                                                          which their
                                                          experiments
                                                          lead.<br>
                                                          -- Norbert
                                                          Wiener</div>
                                                          <div><br>
                                                          </div>
                                                          <div><a
                                                          href="http://www.cse.buffalo.edu/~knepley/"
target="_blank" moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                        </div>
                                                      </div>
                                                    </div>
                                                  </div>
                                                </div>
                                              </blockquote>
                                              <pre cols="72">-- 
Eric Chamberland, ing., M. Ing
Professionnel de recherche
GIREF/Université Laval
(418) 656-2131 poste 41 22 42</pre>
                                            </div>
                                          </blockquote>
                                        </div>
                                        <br clear="all">
                                        <div><br>
                                        </div>
                                        -- <br>
                                        <div dir="ltr">
                                          <div dir="ltr">
                                            <div>
                                              <div dir="ltr">
                                                <div>
                                                  <div dir="ltr">
                                                    <div>What most
                                                      experimenters take
                                                      for granted before
                                                      they begin their
                                                      experiments is
                                                      infinitely more
                                                      interesting than
                                                      any results to
                                                      which their
                                                      experiments lead.<br>
                                                      -- Norbert Wiener</div>
                                                    <div><br>
                                                    </div>
                                                    <div><a
                                                        href="http://www.cse.buffalo.edu/~knepley/"
                                                        target="_blank"
moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
                                                    </div>
                                                  </div>
                                                </div>
                                              </div>
                                            </div>
                                          </div>
                                        </div>
                                      </blockquote>
                                      <pre cols="72">-- 
Eric Chamberland, ing., M. Ing
Professionnel de recherche
GIREF/Université Laval
(418) 656-2131 poste 41 22 42</pre>
                                    </div>
                                  </blockquote>
                                </div>
                                <br clear="all">
                                <div><br>
                                </div>
                                -- <br>
                                <div dir="ltr">
                                  <div dir="ltr">
                                    <div>
                                      <div dir="ltr">
                                        <div>
                                          <div dir="ltr">
                                            <div>What most experimenters
                                              take for granted before
                                              they begin their
                                              experiments is infinitely
                                              more interesting than any
                                              results to which their
                                              experiments lead.<br>
                                              -- Norbert Wiener</div>
                                            <div><br>
                                            </div>
                                            <div><a
                                                href="http://www.cse.buffalo.edu/~knepley/"
                                                target="_blank"
                                                moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
                                            </div>
                                          </div>
                                        </div>
                                      </div>
                                    </div>
                                  </div>
                                </div>
                              </div>
                            </blockquote>
                            <pre cols="72">-- 
Eric Chamberland, ing., M. Ing
Professionnel de recherche
GIREF/Université Laval
(418) 656-2131 poste 41 22 42</pre>
                          </div>
                        </blockquote>
                      </div>
                      <br clear="all">
                      <div><br>
                      </div>
                      -- <br>
                      <div dir="ltr">
                        <div dir="ltr">
                          <div>
                            <div dir="ltr">
                              <div>
                                <div dir="ltr">
                                  <div>What most experimenters take for
                                    granted before they begin their
                                    experiments is infinitely more
                                    interesting than any results to
                                    which their experiments lead.<br>
                                    -- Norbert Wiener</div>
                                  <div><br>
                                  </div>
                                  <div><a
                                      href="http://www.cse.buffalo.edu/~knepley/"
                                      target="_blank"
                                      moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
                                  </div>
                                </div>
                              </div>
                            </div>
                          </div>
                        </div>
                      </div>
                    </div>
                  </blockquote>
                  <pre cols="72">-- 
Eric Chamberland, ing., M. Ing
Professionnel de recherche
GIREF/Université Laval
(418) 656-2131 poste 41 22 42</pre>
                </blockquote>
                <pre cols="72">-- 
Eric Chamberland, ing., M. Ing
Professionnel de recherche
GIREF/Université Laval
(418) 656-2131 poste 41 22 42</pre>
              </blockquote>
              <pre cols="72">-- 
Eric Chamberland, ing., M. Ing
Professionnel de recherche
GIREF/Université Laval
(418) 656-2131 poste 41 22 42</pre>
            </div>
          </blockquote>
        </div>
        <br clear="all">
        <div><br>
        </div>
        -- <br>
        <div dir="ltr" class="gmail_signature">
          <div dir="ltr">
            <div>
              <div dir="ltr">
                <div>
                  <div dir="ltr">
                    <div>What most experimenters take for granted before
                      they begin their experiments is infinitely more
                      interesting than any results to which their
                      experiments lead.<br>
                      -- Norbert Wiener</div>
                    <div><br>
                    </div>
                    <div><a href="http://www.cse.buffalo.edu/~knepley/"
                        target="_blank" moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
                    </div>
                  </div>
                </div>
              </div>
            </div>
          </div>
        </div>
      </div>
    </blockquote>
    <pre class="moz-signature" cols="72">-- 
Eric Chamberland, ing., M. Ing
Professionnel de recherche
GIREF/Université Laval
(418) 656-2131 poste 41 22 42</pre>
  </body>
</html>