[petsc-dev] Checkpoint-restart with DMPlex objects

[Matthew Knepley]

On Fri, Jul 20, 2018 at 5:34 AM Lawrence Mitchell <wence at gmx.li> wrote:

> Dear petsc-dev,
>
> I'm once again revisiting doing "proper" checkpoint-restart cycles.  I
> would like to leverage the existing PETSc stuff for this as much as
> possible, but I am a bit lost as to what is implemented, and what is
> missing.
>
> I have:
>
> - A (distributed) DMPlex defining the topology
>
> - Some number of fields defined on this topology, each described by:
>
>   - a Section (with a point permutation set)
>   - a Vec of coefficients
>   - Some extra information that describes what the coefficients mean, but
> let's assume I know how to handle that.
>
> (Aside, for Vecs with a block size > 1, I actually have a section that
> indexes the blocks, which probably means I need to unroll into an unblocked
> version first).
>

Sections usually are {point --> {dofs}}. Not sure how it uses blocks
instead of points.


> I would like:
>
> - To be able to dump the DMPlex, and fields, on N processes
>

I think the current HDF5 does what you want.


> - To be able to load the DMPlex, and fields, on P processes.  In the first
> instance, to get things going, I am happy if P=1.
>

I think this also works with arbitrary P, although the testing can be
described as extremely thin.


> For dumping, I think I can do DMView(dm) in PETSc "native" format, and
> that will write out the topology in a global numbering.
>

I would use HDF5.


> For the field coefficients, I can just VecView(vec).  But there does not
> appear to be any way of saving the Section so that I can actually attach
> those coefficients to points in the mesh.
>

Hmm, I will check this right now. If it does not exist, I will write it.


> I can do PetscSectionCreateGlobalSection(section), so that I have a the
> global numbering for offsets, but presumably for the point numbering, I
> need to convert the local chart into global point numbers using
> DMPlexCreatePointNumbering?
>

No, all Sections use local points. We do not use global point numbers
anywhere in Plex.


> For load, I can do DMLoad(dm), which only loads on rank-0 for now.


I do not think that is true.


> Then VecLoad for the coefficients, and (presumably) a putative
> PetscSectionLoad so that I can associate coefficients to points in the
> topology.
>

Okay, I am remembering more now. I just use the PetscDS to automatically
create the Section, and pray that
it matches the Vec that we saved. This is a real design limitation, since
the Section is associated with the DM
not the Vec, you have to assume all stored Vecs were created with the
default Section in the DM. Now, if this
is true, then you could just load the Section, set it as the default for
the DM, and load the Vec.


> So it feels like the crucial part of this is a native (HDF5) based viewing
> for PetscSection.
>

Yep. I thought this worked, but maybe I only decided it should work.


> For each section (I'm ignoring fields, because those are just sections
> anyway), there are two explicit, and one implicit, pieces of information:
>
> 1. The number of dofs per point
>

Yep.


> 2. The offset of the dof (this is the global numbering)
>

No, you should store the local Section. You use the SF for the DM to
automatically get the global section.


> 3 (implicit). The association of the local chart numbering to the global
> chart.
>

Above.


> Saving the first two is easy, how best to save the last so that I can load
> easily.  My thought is to, for each point in the section chart, save a
> three-tuple: (global-point-number, dofs-per-point, global-dof-number)
>

Hmm, I have to think a little bit more. Where are you at  with this now?

  Matt


> Then, I can easily read this data in and correctly map from the on-disk
> coefficients into any section I might have built on the newly loaded DM.
>
> Does this all sound reasonable?  Or have I missed either an existing
> implementation, or other issues?
>
> Cheers,
>
> Lawrence



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20181214/ea50c5cb/attachment.html>