[petsc-dev] DMplex reader / viewers

[Tim Tautges (ANL)]

On 01/21/2014 05:58 PM, Gorman, Gerard J wrote:
>
>>>
>>> I am not sure if Exodus has a good solution here. As far as I understand, exodus is inherently sequential, even
>>> when implemented with HDF5 instead of netcdf. I would also worry about third party support for exodus files using
>>> HDF5 as their storage format. Exodus has an parallel extension called nemesis, but I can?t figure out how how
>>> their concept of ghost point and cells works. The documentation on this point is really unclear.
>>>
>
>
> I have to admit I was kind of hoping that ExodusII folks would have come on a bit more on the parallel IO front (I’m
> assuming those guys also run large simulations…). That said, I see this as a two stage process: first integrate with
> DMPlex as that should give the high level abstraction for read/write to file; secondly extend the family of
> readers/writers. At least this way there will be some agility and interoperability between different formats, and it
> will not be too disruptive to the application codes when a different formats adopted.
>

My impression is that the ExodusII people are working within the context of code frameworks more than disk file formats 
to do this, e.g. in Trilinos and Sierra.  I don't think the ExoII file format by itself is very conducive to 
representing parallel, which is why Nemesis writes an annotation (though, I haven't followed ExoII developments closely 
since they went open source several years back).

>
>
>>> b. Do ?poor man? parallel I/O where each CPU does its own I/O, and possibly create interface matching files ? la
>>> nemesis or SILO. Maybe, we can save enough information on the parallel layout in order to easily write an
>>> un-partitionner as a post-processor.
>
> I am pretty sure that if we are writing everything in slabs to a HDF5 container we do not have to worry too much
> about the parallel layout although some clear optimisations are possible. In the  worst case it is a three stage
> process of where we perform a parallel read of the connectivity, scatter/gather for continuous numbering, parallel
> repartitioning and subsequent parallel read of remaining data. Importantly, it is at least scalable.
>

We've seen fragmentation with unstructured meshes being a problem too, and you won't escape that even with renumbering 
(though reading then migrating would address that, at the cost of some additional communication and possibly reading to 
figure out where things need to go).

>>>
>>
>> Depending on the degree of direct interaction/automation in those interactions between the mesh and Petsc, there
>> are other options as well.  One that we're developing, based on the MOAB library, can read/write (in serial)
>> ExodusII, and also supports parallel read/write using its own HDF5-based format.  Parallel I/O robustness has been
>> iffy above ~16k procs and 32M-64M hex/tet elements, but for smaller problems it should work.  We're in the process
>> of developing direct support for going between a mesh defined with fields (called tags in MOAB) and petsc vectors.
>> MOAB has pretty solid support for things like computing sharing and ghosting between procs and exchanging/reducing
>> field values on those entities.  Viz is supported either by compiling a VTK/Paraview plugin that pulls the
>> mesh/fields through MOAB or by translating to VTK (also supported directly from MOAB); Visit also has a plugin you
>> can enable.  See http://trac.mcs.anl.gov/projects/ITAPS/wiki/MOAB for details of MOAB; the petsc integration stuff
>> is on a bitbucket branch of petsc owned by Vijay Mahadevan.
>
> Another reason this could be of great interest is that MOAB supports (according to the docs) geometric topology which
> could be used when adapting the mesh on a curved surface for example - another item on my which list.

Yes, MOAB's file format can handle definitions of groupings (and relations between the groups) necessary to represent 
geometric topology.  What you use for the shape evaluation of those geometric surfaces is another question though.  If 
you're wanting to do that using a reconstructed continuous patch, MOAB has some code to do that too, though it's not as 
good as the latest stuff in that area (from Jiao at Stony Brook).


Is it
> integrated to PETSc via the plex or does this essentially replace the functionality of the plex?

It's not via plex, but I'm pretty sure all the mesh-related functionality available through plex is available through 
different API functions in MOAB.

Why does it break
> down for more than 16k procs?

It's a combination of things:
- maximizing generality means we're using more than just 2 or 3 tables, because in addition to nodes and elements, we 
need groupings, whose membership determines whether a group is resident on a given processor, etc, and that strongly 
affects scaling
- that same generality causes us to hit an MPI/IO bug on IBM (though we haven't checked on Q yet to see if that's been 
addressed, it might have been); we've worked with ANL I/O guys off and on on this, and hope to get back to that on Q soon
- we do single file parallel I/O, without any 2-phase (communicate down to I/O nodes then do I/O), and that hits HDF5 
pretty hard; we're working with hdfgroup to explore that

We haven't done any benchmarking on a Lustre system yet, but I expect that to do worse than IBM, because of the many 
tables thing (my impression is that Lustre doesn't handle frequent metadata reads well)

is it just a case that Lustre gets hammered? What magic sauce is used by high order FEM
> codes such as nek500 that can run on ~1m cores?
>

Those codes go for a much more restricted I/O data case, which allows them to specialize and do their own implementation 
of parallel I/O.  So, Nek5000 has its own implementation of poor man's parallel, they repeat vertices in the file that 
are logically the same (shared between hexes), and they don't really do subsets.  I think that's great to do if you have 
to, but I'm still hoping for more support in that direction from general libraries.

>>
>> libmesh also maintains its own DMlibmesh, but I'm not sure how solid their support for large mesh / parallel I/O is
>> (but they've been working on it recently I know).
>>
>
>
> Are there any other formats that we should be considering? It’s a few years since I tried playing about with CGNS -
> at the time its parallel IO was non-existent and I have not seen it being pushed since. XDMF looks interesting as it
> is essentially some xml metadata and a HDF5 bucket. Is anyone championing this?
>

Don't know about XDMF.  I know there's been a bunch of work on SILO and its parallel performance fairly recently (3 or 4 
yrs ago) and it's used heavily inside LLNL.

- tim

> Cheers Gerard
>

-- 
================================================================
"You will keep in perfect peace him whose mind is
   steadfast, because he trusts in you."               Isaiah 26:3

              Tim Tautges            Argonne National Laboratory
          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
  phone (gvoice): (608) 354-1459      1500 Engineering Dr.
             fax: (608) 263-4499      Madison, WI 53706