[petsc-users] DMPlex: Mapping cells before and after partitioning

Wed Jan 15 14:47:25 CST 2020

Hi Matt,

I’m running into error while using DMPlexNaturalToGlobalBegin/End and am hoping you have some insights in what I’m doing incorrectly. I create a 2x2x2 grid and distribute it across processors (N=1,2). I create a natural and a global vector; and then call DMPlexNaturalToGlobalBegin/End. Here are the two issues:

- When N = 1, PETSc complains about DMSetUseNatural() not being called before DMPlexDistribute(), which is certainly not the case.
- For N=1 and 2, global vector doesn’t have valid entries.

I’m not sure how to create the natural vector and have used DMCreateGlobalVector() to create the natural vector, which could be the issue.

Attached is the sample code to reproduce the error and below is the screen output.

>make ex_test

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>$PETSC_DIR/$PETSC_ARCH/bin/mpiexec -np 1 ./ex_test
Natural vector:

Vec Object: 1 MPI processes
  type: seq
0.
1.
2.
3.
4.
5.
6.
7.
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: DM global to natural SF was not created.
You must call DMSetUseNatural() before DMPlexDistribute().

[0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.12.2-537-g5f77d1e0e5  GIT Date: 2019-12-21 14:33:27 -0600
[0]PETSC ERROR: ./ex_test on a darwin-gcc8 named WE37411 by bish218 Wed Jan 15 12:34:03 2020
[0]PETSC ERROR: Configure options --with-blaslapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --download-parmetis=yes --download-metis=yes --with-hdf5-dir=/opt/local --download-zlib --download-exodusii=yes --download-hdf5=yes --download-netcdf=yes --download-pnetcdf=yes --download-hypre=yes --download-mpich=yes --download-mumps=yes --download-scalapack=yes --with-cc=/opt/local/bin/gcc-mp-8 --with-cxx=/opt/local/bin/g++-mp-8 --with-fc=/opt/local/bin/gfortran-mp-8 --download-sowing=1 PETSC_ARCH=darwin-gcc8
[0]PETSC ERROR: #1 DMPlexNaturalToGlobalBegin() line 289 in /Users/bish218/projects/petsc/petsc_v3.12.2/src/dm/impls/plex/plexnatural.c

Global vector:

Vec Object: 1 MPI processes
  type: seq
0.
0.
0.
0.
0.
0.
0.
0.

Information about the mesh:

Rank = 0
local_id = 00; (0.250000, 0.250000, 0.250000); is_local = 1
local_id = 01; (0.750000, 0.250000, 0.250000); is_local = 1
local_id = 02; (0.250000, 0.750000, 0.250000); is_local = 1
local_id = 03; (0.750000, 0.750000, 0.250000); is_local = 1
local_id = 04; (0.250000, 0.250000, 0.750000); is_local = 1
local_id = 05; (0.750000, 0.250000, 0.750000); is_local = 1
local_id = 06; (0.250000, 0.750000, 0.750000); is_local = 1
local_id = 07; (0.750000, 0.750000, 0.750000); is_local = 1

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

>$PETSC_DIR/$PETSC_ARCH/bin/mpiexec -np 2 ./ex_test
Natural vector:

Vec Object: 2 MPI processes
  type: mpi
Process [0]
0.
1.
2.
3.
Process [1]
4.
5.
6.
7.

Global vector:

Vec Object: 2 MPI processes
  type: mpi
Process [0]
0.
0.
0.
0.
Process [1]
0.
0.
0.
0.

Information about the mesh:

Rank = 0
local_id = 00; (0.250000, 0.750000, 0.250000); is_local = 1
local_id = 01; (0.750000, 0.750000, 0.250000); is_local = 1
local_id = 02; (0.250000, 0.750000, 0.750000); is_local = 1
local_id = 03; (0.750000, 0.750000, 0.750000); is_local = 1
local_id = 04; (0.250000, 0.250000, 0.250000); is_local = 0
local_id = 05; (0.750000, 0.250000, 0.250000); is_local = 0
local_id = 06; (0.250000, 0.250000, 0.750000); is_local = 0
local_id = 07; (0.750000, 0.250000, 0.750000); is_local = 0

Rank = 1
local_id = 00; (0.250000, 0.250000, 0.250000); is_local = 1
local_id = 01; (0.750000, 0.250000, 0.250000); is_local = 1
local_id = 02; (0.250000, 0.250000, 0.750000); is_local = 1
local_id = 03; (0.750000, 0.250000, 0.750000); is_local = 1
local_id = 04; (0.250000, 0.750000, 0.250000); is_local = 0
local_id = 05; (0.750000, 0.750000, 0.250000); is_local = 0
local_id = 06; (0.250000, 0.750000, 0.750000); is_local = 0
local_id = 07; (0.750000, 0.750000, 0.750000); is_local = 0

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

-Gautam

On Jan 9, 2020, at 4:57 PM, 'Bisht, Gautam' via tdycores-dev <tdycores-dev at googlegroups.com<mailto:tdycores-dev at googlegroups.com>> wrote:

On Jan 9, 2020, at 4:25 PM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Thu, Jan 9, 2020 at 1:35 PM 'Bisht, Gautam' via tdycores-dev <tdycores-dev at googlegroups.com<mailto:tdycores-dev at googlegroups.com>> wrote:

> On Jan 9, 2020, at 2:58 PM, Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>> wrote:
>
> "'Bisht, Gautam' via tdycores-dev" <tdycores-dev at googlegroups.com<mailto:tdycores-dev at googlegroups.com>> writes:
>
>>> Do you need to rely on the element number, or would coordinates (of a
>>> centroid?) be sufficient for your purposes?
>>
>> I do need to rely on the element number.  In my case, I have a mapping file that remaps data from one grid onto another grid. Though I’m currently creating a hexahedron mesh, in the future I would be reading in an unstructured grid from a file for which I cannot rely on coordinates.
>
> How does the mapping file work and how is it generated?

In CESM/E3SM, the mapping file is used to map fluxes or states between grids of two components (e.g. land & atmosphere). The mapping method can be conservative, nearest neighbor, bilinear, etc. While CESM/E3SM uses ESMF_RegridWeightGen to generate the mapping file, I’m using by own MATLAB script to create the mapping file.

I’m surprised that this is not an issue for other codes that are using DMPlex. E.g In PFLOTRAN, when a user creates a custom unstructured grid, they can specify material property for each grid cell. So, there should be a way to create a vectorscatter that will scatter material property read in the “application”-order (i.e. order before calling DMPlexDistribute() ) to ghosted-order (i.e. order after calling DMPlexDistribute()).

We did build something specific for this because some people wanted it. I wish I could purge this from all simulations. Its
definitely destructive, but this is the way the world currently is.

You want this:

  https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexNaturalToGlobalBegin.html<https://protect2.fireeye.com/v1/url?k=5fd6bd0b-036383c4-5fd6971e-0cc47adc5e60-2156f2075f6da02c&q=1&e=986b333d-b320-4f42-9c3c-fd4c4b3e0e40&u=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fpetsc-current%2Fdocs%2Fmanualpages%2FDMPLEX%2FDMPlexNaturalToGlobalBegin.html>

Perfect.

Thanks.
-Gautam

    Thanks,

     Matt

> We can locate points and create interpolation with unstructured grids.
>
> --
> You received this message because you are subscribed to the Google Groups "tdycores-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to tdycores-dev+unsubscribe at googlegroups.com<mailto:tdycores-dev%2Bunsubscribe at googlegroups.com>.
> To view this discussion on the web visit https://protect2.fireeye.com/v1/url?k=b265c01b-eed0fed4-b265ea0e-0cc47adc5e60-1707adbf1790c7e4&q=1&e=0962f8e1-9155-4d9c-abdf-2b6481141cd0&u=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2F8736come4e.fsf%2540jedbrown.org.

--
You received this message because you are subscribed to the Google Groups "tdycores-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tdycores-dev+unsubscribe at googlegroups.com<mailto:tdycores-dev%2Bunsubscribe at googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/tdycores-dev/9AB001AF-8857-446A-AE69-E8D6A25CB8FA%40pnnl.gov<https://protect2.fireeye.com/v1/url?k=711264f2-2da75a3d-71124ee7-0cc47adc5e60-a1731ca51d9b5203&q=1&e=4ec2bd09-0f1d-461c-bde7-4a0d8446ce3d&u=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2F9AB001AF-8857-446A-AE69-E8D6A25CB8FA%2540pnnl.gov>.

--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>

--
You received this message because you are subscribed to the Google Groups "tdycores-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tdycores-dev+unsubscribe at googlegroups.com<mailto:tdycores-dev+unsubscribe at googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/tdycores-dev/CAMYG4Gm%3DSY%3DyDiYOdBm1j_KZO5NYhu80ZhbFTV23O%2Bv-zVvFnA%40mail.gmail.com<https://protect2.fireeye.com/v1/url?k=ce43b754-92f6899b-ce439d41-0cc47adc5e60-0a864c0f80b4c75d&q=1&e=4ec2bd09-0f1d-461c-bde7-4a0d8446ce3d&u=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2FCAMYG4Gm%253DSY%253DyDiYOdBm1j_KZO5NYhu80ZhbFTV23O%252Bv-zVvFnA%2540mail.gmail.com%3Futm_medium%3Demail%26utm_source%3Dfooter>.

--
You received this message because you are subscribed to the Google Groups "tdycores-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tdycores-dev+unsubscribe at googlegroups.com<mailto:tdycores-dev+unsubscribe at googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/tdycores-dev/7C23ABBA-2F76-4EAB-9834-9391AD77E18B%40pnnl.gov<https://protect2.fireeye.com/v1/url?k=365c6512-6ae95bdd-365c4f07-0cc47adc5e60-f9b3b6f5e8918ac5&q=1&e=986b333d-b320-4f42-9c3c-fd4c4b3e0e40&u=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2F7C23ABBA-2F76-4EAB-9834-9391AD77E18B%2540pnnl.gov%3Futm_medium%3Demail%26utm_source%3Dfooter>.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200115/09816f05/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex_test.c
Type: application/octet-stream
Size: 3806 bytes
Desc: ex_test.c
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200115/09816f05/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: makefile
Type: application/octet-stream
Size: 299 bytes
Desc: makefile
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200115/09816f05/attachment-0003.obj>