[petsc-dev] Checkpoint-restart with DMPlex objects

[Lawrence Mitchell]

Dear petsc-dev,

I'm once again revisiting doing "proper" checkpoint-restart cycles.  I would like to leverage the existing PETSc stuff for this as much as possible, but I am a bit lost as to what is implemented, and what is missing.

I have:

- A (distributed) DMPlex defining the topology

- Some number of fields defined on this topology, each described by:
 
  - a Section (with a point permutation set)
  - a Vec of coefficients
  - Some extra information that describes what the coefficients mean, but let's assume I know how to handle that.

(Aside, for Vecs with a block size > 1, I actually have a section that indexes the blocks, which probably means I need to unroll into an unblocked version first).

I would like:

- To be able to dump the DMPlex, and fields, on N processes

- To be able to load the DMPlex, and fields, on P processes.  In the first instance, to get things going, I am happy if P=1.

For dumping, I think I can do DMView(dm) in PETSc "native" format, and that will write out the topology in a global numbering.

For the field coefficients, I can just VecView(vec).  But there does not appear to be any way of saving the Section so that I can actually attach those coefficients to points in the mesh.

I can do PetscSectionCreateGlobalSection(section), so that I have a the global numbering for offsets, but presumably for the point numbering, I need to convert the local chart into global point numbers using DMPlexCreatePointNumbering?


For load, I can do DMLoad(dm), which only loads on rank-0 for now.  Then VecLoad for the coefficients, and (presumably) a putative PetscSectionLoad so that I can associate coefficients to points in the topology.

So it feels like the crucial part of this is a native (HDF5) based viewing for PetscSection.

For each section (I'm ignoring fields, because those are just sections anyway), there are two explicit, and one implicit, pieces of information:

1. The number of dofs per point
2. The offset of the dof (this is the global numbering)

3 (implicit). The association of the local chart numbering to the global chart.

Saving the first two is easy, how best to save the last so that I can load easily.  My thought is to, for each point in the section chart, save a three-tuple: (global-point-number, dofs-per-point, global-dof-number)

Then, I can easily read this data in and correctly map from the on-disk coefficients into any section I might have built on the newly loaded DM.

Does this all sound reasonable?  Or have I missed either an existing implementation, or other issues?

Cheers,

Lawrence