[petsc-users] DMPlex: Mapping cells before and after partitioning
Matthew Knepley
knepley at gmail.com
Wed Jan 15 21:05:18 CST 2020
On Wed, Jan 15, 2020 at 4:08 PM Matthew Knepley <knepley at gmail.com> wrote:
> On Wed, Jan 15, 2020 at 3:47 PM 'Bisht, Gautam' via tdycores-dev <
> tdycores-dev at googlegroups.com> wrote:
>
>> Hi Matt,
>>
>> I’m running into error while using DMPlexNaturalToGlobalBegin/End and am
>> hoping you have some insights in what I’m doing incorrectly. I create a
>> 2x2x2 grid and distribute it across processors (N=1,2). I create a natural
>> and a global vector; and then call DMPlexNaturalToGlobalBegin/End. Here are
>> the two issues:
>>
>> - When N = 1, PETSc complains about DMSetUseNatural() not being called
>> before DMPlexDistribute(), which is certainly not the case.
>> - For N=1 and 2, global vector doesn’t have valid entries.
>>
>> I’m not sure how to create the natural vector and have used
>> DMCreateGlobalVector() to create the natural vector, which could be the
>> issue.
>>
>> Attached is the sample code to reproduce the error and below is the
>> screen output.
>>
>
> Cool. I will run it and figure out the problem.
>
1) There was bad error reporting there. I am putting the fix in a new
branch. It did not check for being on one process. If you run with
knepley/fix-dm-g2n-serial
It will work correctly in serial.
2) The G2N needs a serial data layout to work, so you have to make a
Section _before_ distributing. I need to put that in the docs. I have
fixed your example to do this and attached it. I run it with
master *:~/Downloads/tmp/Gautam$ /PETSc3/petsc/bin/mpiexec -n 1
./ex_test -dm_plex_box_faces 2,2,2 -dm_view
DM Object: 1 MPI processes
type: plex
DM_0x84000000_0 in 3 dimensions:
0-cells: 27
1-cells: 54
2-cells: 36
3-cells: 8
Labels:
marker: 1 strata with value/size (1 (72))
Face Sets: 6 strata with value/size (6 (4), 5 (4), 3 (4), 4 (4), 1 (4), 2
(4))
depth: 4 strata with value/size (0 (27), 1 (54), 2 (36), 3 (8))
Field p:
adjacency FVM++
Natural vector:
Vec Object: 1 MPI processes
type: seq
0.
1.
2.
3.
4.
5.
6.
7.
Global vector:
Vec Object: 1 MPI processes
type: seq
0.
1.
2.
3.
4.
5.
6.
7.
Information about the mesh:
[0] cell = 00; (0.250000, 0.250000, 0.250000); is_local = 1
[0] cell = 01; (0.750000, 0.250000, 0.250000); is_local = 1
[0] cell = 02; (0.250000, 0.750000, 0.250000); is_local = 1
[0] cell = 03; (0.750000, 0.750000, 0.250000); is_local = 1
[0] cell = 04; (0.250000, 0.250000, 0.750000); is_local = 1
[0] cell = 05; (0.750000, 0.250000, 0.750000); is_local = 1
[0] cell = 06; (0.250000, 0.750000, 0.750000); is_local = 1
[0] cell = 07; (0.750000, 0.750000, 0.750000); is_local = 1
master *:~/Downloads/tmp/Gautam$ /PETSc3/petsc/bin/mpiexec -n 2 ./ex_test
-dm_plex_box_faces 2,2,2 -dm_view
DM Object: Parallel Mesh 2 MPI processes
type: plex
Parallel Mesh in 3 dimensions:
0-cells: 27 27
1-cells: 54 54
2-cells: 36 36
3-cells: 8 8
Labels:
depth: 4 strata with value/size (0 (27), 1 (54), 2 (36), 3 (8))
marker: 1 strata with value/size (1 (72))
Face Sets: 6 strata with value/size (1 (4), 2 (4), 3 (4), 4 (4), 5 (4), 6
(4))
Field p:
adjacency FVM++
Natural vector:
Vec Object: 2 MPI processes
type: mpi
Process [0]
0.
1.
2.
3.
Process [1]
4.
5.
6.
7.
Global vector:
Vec Object: 2 MPI processes
type: mpi
Process [0]
2.
3.
6.
7.
Process [1]
0.
1.
4.
5.
Information about the mesh:
[0] cell = 00; (0.250000, 0.750000, 0.250000); is_local = 1
[0] cell = 01; (0.750000, 0.750000, 0.250000); is_local = 1
[0] cell = 02; (0.250000, 0.750000, 0.750000); is_local = 1
[0] cell = 03; (0.750000, 0.750000, 0.750000); is_local = 1
[0] cell = 04; (0.250000, 0.250000, 0.250000); is_local = 0
[0] cell = 05; (0.750000, 0.250000, 0.250000); is_local = 0
[0] cell = 06; (0.250000, 0.250000, 0.750000); is_local = 0
[0] cell = 07; (0.750000, 0.250000, 0.750000); is_local = 0
[1] cell = 00; (0.250000, 0.250000, 0.250000); is_local = 1
[1] cell = 01; (0.750000, 0.250000, 0.250000); is_local = 1
[1] cell = 02; (0.250000, 0.250000, 0.750000); is_local = 1
[1] cell = 03; (0.750000, 0.250000, 0.750000); is_local = 1
[1] cell = 04; (0.250000, 0.750000, 0.250000); is_local = 0
[1] cell = 05; (0.750000, 0.750000, 0.250000); is_local = 0
[1] cell = 06; (0.250000, 0.750000, 0.750000); is_local = 0
[1] cell = 07; (0.750000, 0.750000, 0.750000); is_local = 0
Thanks,
Matt
> Thanks,
>
> Matt
>
>
>> >make ex_test
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >$PETSC_DIR/$PETSC_ARCH/bin/mpiexec -np 1 ./ex_test
>> Natural vector:
>>
>> Vec Object: 1 MPI processes
>> type: seq
>> 0.
>> 1.
>> 2.
>> 3.
>> 4.
>> 5.
>> 6.
>> 7.
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Object is in wrong state
>> [0]PETSC ERROR: DM global to natural SF was not created.
>> You must call DMSetUseNatural() before DMPlexDistribute().
>>
>> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Development GIT revision: v3.12.2-537-g5f77d1e0e5
>> GIT Date: 2019-12-21 14:33:27 -0600
>> [0]PETSC ERROR: ./ex_test on a darwin-gcc8 named WE37411 by bish218 Wed
>> Jan 15 12:34:03 2020
>> [0]PETSC ERROR: Configure options
>> --with-blaslapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate
>> --download-parmetis=yes --download-metis=yes --with-hdf5-dir=/opt/local
>> --download-zlib --download-exodusii=yes --download-hdf5=yes
>> --download-netcdf=yes --download-pnetcdf=yes --download-hypre=yes
>> --download-mpich=yes --download-mumps=yes --download-scalapack=yes
>> --with-cc=/opt/local/bin/gcc-mp-8 --with-cxx=/opt/local/bin/g++-mp-8
>> --with-fc=/opt/local/bin/gfortran-mp-8 --download-sowing=1
>> PETSC_ARCH=darwin-gcc8
>> [0]PETSC ERROR: #1 DMPlexNaturalToGlobalBegin() line 289 in
>> /Users/bish218/projects/petsc/petsc_v3.12.2/src/dm/impls/plex/plexnatural.c
>>
>> Global vector:
>>
>> Vec Object: 1 MPI processes
>> type: seq
>> 0.
>> 0.
>> 0.
>> 0.
>> 0.
>> 0.
>> 0.
>> 0.
>>
>> Information about the mesh:
>>
>> Rank = 0
>> local_id = 00; (0.250000, 0.250000, 0.250000); is_local = 1
>> local_id = 01; (0.750000, 0.250000, 0.250000); is_local = 1
>> local_id = 02; (0.250000, 0.750000, 0.250000); is_local = 1
>> local_id = 03; (0.750000, 0.750000, 0.250000); is_local = 1
>> local_id = 04; (0.250000, 0.250000, 0.750000); is_local = 1
>> local_id = 05; (0.750000, 0.250000, 0.750000); is_local = 1
>> local_id = 06; (0.250000, 0.750000, 0.750000); is_local = 1
>> local_id = 07; (0.750000, 0.750000, 0.750000); is_local = 1
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>> >$PETSC_DIR/$PETSC_ARCH/bin/mpiexec -np 2 ./ex_test
>> Natural vector:
>>
>> Vec Object: 2 MPI processes
>> type: mpi
>> Process [0]
>> 0.
>> 1.
>> 2.
>> 3.
>> Process [1]
>> 4.
>> 5.
>> 6.
>> 7.
>>
>> Global vector:
>>
>> Vec Object: 2 MPI processes
>> type: mpi
>> Process [0]
>> 0.
>> 0.
>> 0.
>> 0.
>> Process [1]
>> 0.
>> 0.
>> 0.
>> 0.
>>
>> Information about the mesh:
>>
>> Rank = 0
>> local_id = 00; (0.250000, 0.750000, 0.250000); is_local = 1
>> local_id = 01; (0.750000, 0.750000, 0.250000); is_local = 1
>> local_id = 02; (0.250000, 0.750000, 0.750000); is_local = 1
>> local_id = 03; (0.750000, 0.750000, 0.750000); is_local = 1
>> local_id = 04; (0.250000, 0.250000, 0.250000); is_local = 0
>> local_id = 05; (0.750000, 0.250000, 0.250000); is_local = 0
>> local_id = 06; (0.250000, 0.250000, 0.750000); is_local = 0
>> local_id = 07; (0.750000, 0.250000, 0.750000); is_local = 0
>>
>> Rank = 1
>> local_id = 00; (0.250000, 0.250000, 0.250000); is_local = 1
>> local_id = 01; (0.750000, 0.250000, 0.250000); is_local = 1
>> local_id = 02; (0.250000, 0.250000, 0.750000); is_local = 1
>> local_id = 03; (0.750000, 0.250000, 0.750000); is_local = 1
>> local_id = 04; (0.250000, 0.750000, 0.250000); is_local = 0
>> local_id = 05; (0.750000, 0.750000, 0.250000); is_local = 0
>> local_id = 06; (0.250000, 0.750000, 0.750000); is_local = 0
>> local_id = 07; (0.750000, 0.750000, 0.750000); is_local = 0
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>> -Gautam
>>
>>
>>
>>
>> On Jan 9, 2020, at 4:57 PM, 'Bisht, Gautam' via tdycores-dev <
>> tdycores-dev at googlegroups.com> wrote:
>>
>>
>>
>> On Jan 9, 2020, at 4:25 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>
>> On Thu, Jan 9, 2020 at 1:35 PM 'Bisht, Gautam' via tdycores-dev <
>> tdycores-dev at googlegroups.com> wrote:
>>
>>
>> > On Jan 9, 2020, at 2:58 PM, Jed Brown <jed at jedbrown.org> wrote:
>> >
>> > "'Bisht, Gautam' via tdycores-dev" <tdycores-dev at googlegroups.com>
>> writes:
>> >
>> >>> Do you need to rely on the element number, or would coordinates (of a
>> >>> centroid?) be sufficient for your purposes?
>> >>
>> >> I do need to rely on the element number. In my case, I have a mapping
>> file that remaps data from one grid onto another grid. Though I’m currently
>> creating a hexahedron mesh, in the future I would be reading in an
>> unstructured grid from a file for which I cannot rely on coordinates.
>> >
>> > How does the mapping file work and how is it generated?
>>
>> In CESM/E3SM, the mapping file is used to map fluxes or states between
>> grids of two components (e.g. land & atmosphere). The mapping method can be
>> conservative, nearest neighbor, bilinear, etc. While CESM/E3SM uses
>> ESMF_RegridWeightGen to generate the mapping file, I’m using by own MATLAB
>> script to create the mapping file.
>>
>> I’m surprised that this is not an issue for other codes that are using
>> DMPlex. E.g In PFLOTRAN, when a user creates a custom unstructured grid,
>> they can specify material property for each grid cell. So, there should be
>> a way to create a vectorscatter that will scatter material property read in
>> the “application”-order (i.e. order before calling DMPlexDistribute() ) to
>> ghosted-order (i.e. order after calling DMPlexDistribute()).
>>
>>
>> We did build something specific for this because some people wanted it. I
>> wish I could purge this from all simulations. Its
>> definitely destructive, but this is the way the world currently is.
>>
>> You want this:
>>
>>
>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexNaturalToGlobalBegin.html
>> <https://protect2.fireeye.com/v1/url?k=5fd6bd0b-036383c4-5fd6971e-0cc47adc5e60-2156f2075f6da02c&q=1&e=986b333d-b320-4f42-9c3c-fd4c4b3e0e40&u=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fpetsc-current%2Fdocs%2Fmanualpages%2FDMPLEX%2FDMPlexNaturalToGlobalBegin.html>
>>
>>
>> Perfect.
>>
>> Thanks.
>> -Gautam
>>
>>
>>
>> Thanks,
>>
>> Matt
>>
>>
>> > We can locate points and create interpolation with unstructured grids.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "tdycores-dev" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an email to tdycores-dev+unsubscribe at googlegroups.com.
>> > To view this discussion on the web visit
>> https://protect2.fireeye.com/v1/url?k=b265c01b-eed0fed4-b265ea0e-0cc47adc5e60-1707adbf1790c7e4&q=1&e=0962f8e1-9155-4d9c-abdf-2b6481141cd0&u=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2F8736come4e.fsf%2540jedbrown.org
>> .
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "tdycores-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to tdycores-dev+unsubscribe at googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/tdycores-dev/9AB001AF-8857-446A-AE69-E8D6A25CB8FA%40pnnl.gov
>> <https://protect2.fireeye.com/v1/url?k=711264f2-2da75a3d-71124ee7-0cc47adc5e60-a1731ca51d9b5203&q=1&e=4ec2bd09-0f1d-461c-bde7-4a0d8446ce3d&u=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2F9AB001AF-8857-446A-AE69-E8D6A25CB8FA%2540pnnl.gov>
>> .
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "tdycores-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to tdycores-dev+unsubscribe at googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/tdycores-dev/CAMYG4Gm%3DSY%3DyDiYOdBm1j_KZO5NYhu80ZhbFTV23O%2Bv-zVvFnA%40mail.gmail.com
>> <https://protect2.fireeye.com/v1/url?k=ce43b754-92f6899b-ce439d41-0cc47adc5e60-0a864c0f80b4c75d&q=1&e=4ec2bd09-0f1d-461c-bde7-4a0d8446ce3d&u=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2FCAMYG4Gm%253DSY%253DyDiYOdBm1j_KZO5NYhu80ZhbFTV23O%252Bv-zVvFnA%2540mail.gmail.com%3Futm_medium%3Demail%26utm_source%3Dfooter>
>> .
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "tdycores-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to tdycores-dev+unsubscribe at googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/tdycores-dev/7C23ABBA-2F76-4EAB-9834-9391AD77E18B%40pnnl.gov
>> <https://protect2.fireeye.com/v1/url?k=365c6512-6ae95bdd-365c4f07-0cc47adc5e60-f9b3b6f5e8918ac5&q=1&e=986b333d-b320-4f42-9c3c-fd4c4b3e0e40&u=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2F7C23ABBA-2F76-4EAB-9834-9391AD77E18B%2540pnnl.gov%3Futm_medium%3Demail%26utm_source%3Dfooter>
>> .
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "tdycores-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to tdycores-dev+unsubscribe at googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/tdycores-dev/8A7925AE-08F5-4F81-AAA5-B2FDC3D833B0%40pnnl.gov
>> <https://groups.google.com/d/msgid/tdycores-dev/8A7925AE-08F5-4F81-AAA5-B2FDC3D833B0%40pnnl.gov?utm_medium=email&utm_source=footer>
>> .
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200115/05926b09/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex_test.c
Type: application/octet-stream
Size: 3579 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200115/05926b09/attachment-0001.obj>
More information about the petsc-users
mailing list