<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Matt, great that your reminded this email. I actually completely missed it that time.<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 14 Dec 2018, at 19:54, Matthew Knepley via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov" class="">petsc-dev@mcs.anl.gov</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<div class="gmail_quote">
<div dir="ltr" class="">On Fri, Jul 20, 2018 at 5:34 AM Lawrence Mitchell <<a href="mailto:wence@gmx.li" class="">wence@gmx.li</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
Dear petsc-dev,<br class="">
<br class="">
I'm once again revisiting doing "proper" checkpoint-restart cycles. I would like to leverage the existing PETSc stuff for this as much as possible, but I am a bit lost as to what is implemented, and what is missing.<br class="">
<br class="">
I have:<br class="">
<br class="">
- A (distributed) DMPlex defining the topology<br class="">
<br class="">
- Some number of fields defined on this topology, each described by:<br class="">
<br class="">
- a Section (with a point permutation set)<br class="">
- a Vec of coefficients<br class="">
- Some extra information that describes what the coefficients mean, but let's assume I know how to handle that.<br class="">
<br class="">
(Aside, for Vecs with a block size > 1, I actually have a section that indexes the blocks, which probably means I need to unroll into an unblocked version first).<br class="">
</blockquote>
<div class=""><br class="">
</div>
<div class="">Sections usually are {point --> {dofs}}. Not sure how it uses blocks instead of points.</div>
<div class=""> </div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
I would like:<br class="">
<br class="">
- To be able to dump the DMPlex, and fields, on N processes<br class="">
</blockquote>
<div class=""><br class="">
</div>
<div class="">I think the current HDF5 does what you want.</div>
<div class=""> </div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
- To be able to load the DMPlex, and fields, on P processes. In the first instance, to get things going, I am happy if P=1.<br class="">
</blockquote>
<div class=""><br class="">
</div>
<div class="">I think this also works with arbitrary P, although the testing can be described as extremely thin.</div>
</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>I think we need to be much more precise here. First off, there are now two HDF5 formats:<br class="">
1) PETSC_VIEWER_HDF5_PETSC - store Plex graph serialization
<div class="">2) PETSC_VIEWER_HDF5_XDMF - store XDMF-compatible representation of vertices and cells
<div class="">3) PETSC_VIEWER_HDF5_VIZ slightly extends 2) with some stuff for visualization, you perhaps understand it better</div>
<div class=""><br class="">
</div>
<div class="">PETSC_VIEWER_DEFAULT/PETSC_VIEWER_NATIVE mean store all three above. I think what Lawrence calls Native should be 1).</div>
<div class=""><br class="">
</div>
<div class="">The format 1) is currently written in parallel but loaded sequentially</div>
<div class=""><a href="https://bitbucket.org/petsc/petsc/src/fbb1886742ac2bbe3b4d1df09bff9724d3fee060/src/dm/impls/plex/plexhdf5.c#lines-834" class="">https://bitbucket.org/petsc/petsc/src/fbb1886742ac2bbe3b4d1df09bff9724d3fee060/src/dm/impls/plex/plexhdf5.c#lines-834</a></div>
<div class=""><br class="">
</div>
<div class="">I don't understand, how it can work correctly for a distributed mesh while the Point SF (connecting partitions) is not stored FWICS. I think there's even no PetscSFView_HDF5(). I will check it more deeply soon.</div>
<div class=""><br class="">
</div>
<div class="">The format 2) is for which I implemented parallel DMLoad().</div>
<div class="">Unfortunately, I can't declare it bulletproof until we declare parallel DMPlexInterpolate() as 100% working. I did quite some work towards it in</div>
<div class=""><a href="https://bitbucket.org/petsc/petsc/pull-requests/1227/dmplexintepolate-fix-orientation-of-faces/" class="">https://bitbucket.org/petsc/petsc/pull-requests/1227/dmplexintepolate-fix-orientation-of-faces/</a></div>
<div class="">but as stated in the PR summary, there are still some examples failing because of the wrong Point SF which is partly fixed in knepley/fix-plex-interpolate-sf but it seems it's not yet finished. Matt, is there any chance you could look at it at
some point in near future?</div>
<div class=""><br class="">
</div>
<div class="">I think for Lawrence's purposes, 2) can be used to read the initial mesh file but for checkpointing 1) seems to be better ATM because it dumps everything including interpolated edges & faces, labels and perhaps some more additional information.</div>
<div class=""><br class="">
</div>
<div class="">I will nevertheless keep on working to improve 2) so that it can store edges & faces & labels in the XDMF-compatible way.</div>
</div>
</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<div class="gmail_quote">
<div class=""> </div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
For dumping, I think I can do DMView(dm) in PETSc "native" format, and that will write out the topology in a global numbering.<br class="">
</blockquote>
<div class=""><br class="">
</div>
<div class="">I would use HDF5.</div>
<div class=""> </div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
For the field coefficients, I can just VecView(vec). But there does not appear to be any way of saving the Section so that I can actually attach those coefficients to points in the mesh.<br class="">
</blockquote>
<div class=""><br class="">
</div>
<div class="">Hmm, I will check this right now. If it does not exist, I will write it.</div>
</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>No, it certainly doesn't exist. There is only ASCII view implemented.</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<div class="gmail_quote">
<div class=""> </div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
I can do PetscSectionCreateGlobalSection(section), so that I have a the global numbering for offsets, but presumably for the point numbering, I need to convert the local chart into global point numbers using DMPlexCreatePointNumbering?<br class="">
</blockquote>
<div class=""><br class="">
</div>
<div class="">No, all Sections use local points. We do not use global point numbers anywhere in Plex.</div>
</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>True. DMPlex is partition-wise sequential. The only thing which connects the submeshes is the Point SF.</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<div class="gmail_quote">
<div class=""> </div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
For load, I can do DMLoad(dm), which only loads on rank-0 for now. </blockquote>
<div class=""><br class="">
</div>
<div class="">I do not think that is true.</div>
</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>Actually it is true if we talk about PETSC_VIEWER_HDF5_PETSC.</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<div class="gmail_quote">
<div class=""> </div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
Then VecLoad for the coefficients, and (presumably) a putative PetscSectionLoad so that I can associate coefficients to points in the topology.<br class="">
</blockquote>
<div class=""><br class="">
</div>
<div class="">Okay, I am remembering more now. I just use the PetscDS to automatically create the Section, and pray that</div>
<div class="">it matches the Vec that we saved. This is a real design limitation, since the Section is associated with the DM</div>
<div class="">not the Vec, you have to assume all stored Vecs were created with the default Section in the DM. Now, if this</div>
<div class="">is true, then you could just load the Section, set it as the default for the DM, and load the Vec.</div>
<div class=""> </div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
So it feels like the crucial part of this is a native (HDF5) based viewing for PetscSection.<br class="">
</blockquote>
<div class=""><br class="">
</div>
<div class="">Yep. I thought this worked, but maybe I only decided it should work.</div>
<div class=""> </div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
For each section (I'm ignoring fields, because those are just sections anyway), there are two explicit, and one implicit, pieces of information:<br class="">
<br class="">
1. The number of dofs per point<br class="">
</blockquote>
<div class=""><br class="">
</div>
<div class="">Yep.</div>
<div class=""> </div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
2. The offset of the dof (this is the global numbering)<br class="">
</blockquote>
<div class=""><br class="">
</div>
<div class="">No, you should store the local Section. You use the SF for the DM to automatically get the global section.</div>
</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>True, see above.</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<div class="gmail_quote">
<div class=""> </div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
3 (implicit). The association of the local chart numbering to the global chart.<br class="">
</blockquote>
<div class=""><br class="">
</div>
<div class="">Above.</div>
<div class=""> </div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
Saving the first two is easy, how best to save the last so that I can load easily. My thought is to, for each point in the section chart, save a three-tuple: (global-point-number, dofs-per-point, global-dof-number)<br class="">
</blockquote>
<div class=""><br class="">
</div>
<div class="">Hmm, I have to think a little bit more. Where are you at with this now?</div>
<div class=""><br class="">
</div>
<div class=""> Matt</div>
<div class=""> </div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
Then, I can easily read this data in and correctly map from the on-disk coefficients into any section I might have built on the newly loaded DM.<br class="">
<br class="">
Does this all sound reasonable? Or have I missed either an existing implementation, or other issues?<br class="">
<br class="">
Cheers,<br class="">
<br class="">
Lawrence</blockquote>
</div>
<br clear="all" class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>I think there should be no big problem implementing PetscSection{View,Load}_HDF5(). I would suggest to do make use of ISView() to store the integer arrays (as I did with matrices). The same for PetscSF.</div>
<div><br class="">
</div>
<div>
<div class="">Note it seems to me XDMF could possibly store also those coefficients (mapped by PetscSection on PETSc side)</div>
<div class=""><a href="http://www.xdmf.org/index.php/XDMF_Model_and_Format#Attribute" class="">http://www.xdmf.org/index.php/XDMF_Model_and_Format#Attribute</a></div>
<div class="">I would be thankful for feedback on this.</div>
<div class=""><br class="">
</div>
<div class="">Vaclav</div>
<div class=""><br class="">
</div>
</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<div class=""><br class="">
</div>
--<span class="Apple-converted-space"> </span><br class="">
<div dir="ltr" class="gmail_signature">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">
-- Norbert Wiener</div>
<div class=""><br class="">
</div>
<div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</body>
</html>