[petsc-users] reading and writing periodic DMPlex to file

Matthew Knepley knepley at gmail.com
Thu Dec 15 11:56:47 CST 2022 

On Wed, Dec 14, 2022 at 3:58 AM Berend van Wachem <berend.vanwachem at ovgu.de>
wrote:

>
> Dear PETSc team and users,
>
> I have asked a few times about this before, but we haven't really gotten
> this to work yet.
>
> In our code, we use the DMPlex framework and are also interested in
> periodic geometries.
>
> As our simulations typically require many time-steps, we would like to
> be able to save the DM to file and to read it again to resume the
> simulation (a restart).
>
> Although this works for a non-periodic DM, we haven't been able to get
> this to work for a periodic one. To illustrate this, I have made a
> working example, consisting of 2 files, createandwrite.c and
> readandcreate.c. I have attached these 2 working examples. We are using
> Petsc-3.18.2.
>
> In the first file (createandwrite.c) a DMPlex is created and written to
> a file. Periodicity is activated on lines 52-55 of the code.
>
> In the second file (readandcreate.c) a DMPlex is read from the file.
> When a periodic DM is read, this does not work. Also, trying to
> 'enforce' periodicity, lines 55 - 66, does not work if the number of
> processes is larger than 1 - the code "hangs" without producing an error.
>
> Could you indicate what I am missing? I have really tried many different
> options, without finding a solution.
>

Hi Berend,

There are several problems. I will eventually fix all of them, but I think
we can get this working quickly.

1) Periodicity information is not saved. I will fix this, but forcing it
should work.

2) You were getting a hang because the blocksize on the local coordinates
was not set correctly after loading
     since the vector had zero length. This does not happen in any test
because HDF5 loads a global vector, but
     most other things create local coordinates. I have a fix for this,
which I will get in an MR, Also, I moved DMLocalizeCoordinates()
     after distribution, since this is where it belongs.

knepley/fix-plex-periodic-faces *$:/PETSc3/petsc/petsc-dev$ git diff
diff --git a/src/dm/interface/dmcoordinates.c
b/src/dm/interface/dmcoordinates.c
index a922348f95b..6437e9f7259 100644
--- a/src/dm/interface/dmcoordinates.c
+++ b/src/dm/interface/dmcoordinates.c
@@ -551,10 +551,14 @@ PetscErrorCode DMGetCoordinatesLocalSetUp(DM dm)
   PetscFunctionBegin;
   PetscValidHeaderSpecific(dm, DM_CLASSID, 1);
   if (!dm->coordinates[0].xl && dm->coordinates[0].x) {
-    DM cdm = NULL;
+    DM       cdm = NULL;
+    PetscInt bs;

     PetscCall(DMGetCoordinateDM(dm, &cdm));
     PetscCall(DMCreateLocalVector(cdm, &dm->coordinates[0].xl));
+    // If the size of the vector is 0, it will not get the right block size
+    PetscCall(VecGetBlockSize(dm->coordinates[0].x, &bs));
+    PetscCall(VecSetBlockSize(dm->coordinates[0].xl, bs));
     PetscCall(PetscObjectSetName((PetscObject)dm->coordinates[0].xl,
"coordinates"));
     PetscCall(DMGlobalToLocalBegin(cdm, dm->coordinates[0].x,
INSERT_VALUES, dm->coordinates[0].xl));
     PetscCall(DMGlobalToLocalEnd(cdm, dm->coordinates[0].x, INSERT_VALUES,
dm->coordinates[0].xl));

 3) If I comment out forcing the periodicity, your example does not run for
me. I will try to figure it out

[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Nonconforming object sizes
[0]PETSC ERROR: SF roots 4400 < pEnd 6000
[1]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: WARNING! There are option(s) set that were not used! Could
be the program crashed before they were used or a spelling mistake, etc!
[1]PETSC ERROR: Nonconforming object sizes
[0]PETSC ERROR: Option left: name:-start_in_debugger_no (no value) source:
command line
[1]PETSC ERROR: SF roots 4400 < pEnd 6000
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.18.1-494-g16200351da0
 GIT Date: 2022-12-12 23:42:20 +0000
[1]PETSC ERROR: WARNING! There are option(s) set that were not used! Could
be the program crashed before they were used or a spelling mistake, etc!
[1]PETSC ERROR: Option left: name:-start_in_debugger_no (no value) source:
command line
[0]PETSC ERROR: ./readandcreate on a arch-master-debug named
MacBook-Pro.cable.rcn.com by knepley Thu Dec 15 12:50:26 2022
[1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Configure options --PETSC_ARCH=arch-master-debug
--download-bamg --download-bison --download-chaco --download-ctetgen
--download-egads --download-eigen --download-exodusii --download-fftw
--download-hpddm --download-ks --download-libceed --download-libpng
--download-metis --download-ml --download-mumps --download-muparser
--download-netcdf --download-opencascade --download-p4est
--download-parmetis --download-pnetcdf --download-pragmatic
--download-ptscotch --download-scalapack --download-slepc
--download-suitesparse --download-superlu_dist --download-tetgen
--download-triangle --with-cmake-exec=/PETSc3/petsc/apple/bin/cmake
--with-ctest-exec=/PETSc3/petsc/apple/bin/ctest
--with-hdf5-dir=/PETSc3/petsc/apple --with-mpi-dir=/PETSc3/petsc/apple
--with-petsc4py=1 --with-shared-libraries --with-slepc --with-zlib
[1]PETSC ERROR: Petsc Development GIT revision: v3.18.1-494-g16200351da0
 GIT Date: 2022-12-12 23:42:20 +0000
[0]PETSC ERROR: #1 PetscSectionCreateGlobalSection() at
/PETSc3/petsc/petsc-dev/src/vec/is/section/interface/section.c:1322
[1]PETSC ERROR: ./readandcreate on a arch-master-debug named
MacBook-Pro.cable.rcn.com by knepley Thu Dec 15 12:50:26 2022
[0]PETSC ERROR: #2 DMGetGlobalSection() at
/PETSc3/petsc/petsc-dev/src/dm/interface/dm.c:4527
[1]PETSC ERROR: Configure options --PETSC_ARCH=arch-master-debug
--download-bamg --download-bison --download-chaco --download-ctetgen
--download-egads --download-eigen --download-exodusii --download-fftw
--download-hpddm --download-ks --download-libceed --download-libpng
--download-metis --download-ml --download-mumps --download-muparser
--download-netcdf --download-opencascade --download-p4est
--download-parmetis --download-pnetcdf --download-pragmatic
--download-ptscotch --download-scalapack --download-slepc
--download-suitesparse --download-superlu_dist --download-tetgen
--download-triangle --with-cmake-exec=/PETSc3/petsc/apple/bin/cmake
--with-ctest-exec=/PETSc3/petsc/apple/bin/ctest
--with-hdf5-dir=/PETSc3/petsc/apple --with-mpi-dir=/PETSc3/petsc/apple
--with-petsc4py=1 --with-shared-libraries --with-slepc --with-zlib
[0]PETSC ERROR: #3 DMPlexSectionLoad_HDF5_Internal() at
/PETSc3/petsc/petsc-dev/src/dm/impls/plex/plexhdf5.c:2750
[1]PETSC ERROR: #1 PetscSectionCreateGlobalSection() at
/PETSc3/petsc/petsc-dev/src/vec/is/section/interface/section.c:1322
[0]PETSC ERROR: #4 DMPlexSectionLoad() at
/PETSc3/petsc/petsc-dev/src/dm/impls/plex/plex.c:2364
[1]PETSC ERROR: #2 DMGetGlobalSection() at
/PETSc3/petsc/petsc-dev/src/dm/interface/dm.c:4527
[0]PETSC ERROR: #5 main() at
/Users/knepley/Downloads/tmp/Berend/readandcreate.c:85
[1]PETSC ERROR: #3 DMPlexSectionLoad_HDF5_Internal() at
/PETSc3/petsc/petsc-dev/src/dm/impls/plex/plexhdf5.c:2750
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -malloc_debug (source: environment)
[1]PETSC ERROR: #4 DMPlexSectionLoad() at
/PETSc3/petsc/petsc-dev/src/dm/impls/plex/plex.c:2364
[1]PETSC ERROR: #5 main() at
/Users/knepley/Downloads/tmp/Berend/readandcreate.c:85
[0]PETSC ERROR: -start_in_debugger_no (source: command line)
[1]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------
[1]PETSC ERROR: -malloc_debug (source: environment)
application called MPI_Abort(MPI_COMM_SELF, 60) - process 0
[1]PETSC ERROR: -start_in_debugger_no (source: command line)
[1]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_SELF, 60) - process 0

4) We now have parallel HDF5 loading, so you should not have to manually
distribute. I will change your example to use it
     and send it back when I am done.

  Thanks!

     Matt

Many thanks and kind regards,
> Berend.



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20221215/0d927313/attachment.html>

More information about the petsc-users mailing list