[petsc-dev] DMplex reader / viewers

Tue Jan 21 15:07:38 CST 2014

On 01/21/2014 12:00 PM, petsc-dev-request at mcs.anl.gov wrote:

> Date: Tue, 21 Jan 2014 17:56:35 +0000
> From: Blaise A Bourdin <bourdin at lsu.edu>
> To: "Gorman, Gerard J" <g.gorman at imperial.ac.uk>
> Cc: petsc-dev <petsc-dev at mcs.anl.gov>
> Subject: Re: [petsc-dev] DMplex reader / viewers
> Message-ID: <ECA3EBD3-7618-490A-B167-B562428ECC7B at lsu.edu>
> Content-Type: text/plain; charset="Windows-1252"
>
>
> On Jan 21, 2014, at 10:23 AM, Gorman, Gerard J <g.gorman at imperial.ac.uk> wrote:
>
>> Hi Blaise
>>
>> This is pretty much music to my ears. Interoperability is a major headache, most file formats are rubbish (e.g. cannot support for boundaries, limited meta data etc etc) most lack reasonable parallel IO solutions (i.e. something other than 1 file per process)?and so on. At the moment a few groups here at IC (this mostly means Michael Lange ;) are integrating DMPlex with a range of FEM codes to try to rationalise this. There are certainly some features missing (we?ve bumped into most of that you?ve listed below) but all indications are that DMPlex has the right design to be expanded to cover these use cases.
>>
>> On 21 Jan 2014, at 15:30, Blaise A Bourdin <bourdin at lsu.edu> wrote:
>>
>>> Hi,
>>>
>>> It looks like DMplex is steadily gaining maturity but I/O is lagging behind. As far as I understand, right now, PETSc can _read_ a mesh in exodus format, and  write binary VTS format, but many issues remain, IMHO:
>>>   - The exodus reader relies on a hard-coded nodeset named ?marker?. Generating such a nodeset is not trivial
>>>     (at least not for complex meshes generated with Cubit / Trelis).
>>>   - Reading from or writing to exodus files is not supported.
>>
>> So we *really* want this as well for the purpose of dumping results and checkpointing. Trying to shoehorn high order and DG data into VTK is complete crap. I love VTK because its got loads of functionality and it is easy to throw stuff together, but we are getting ourselves into the position that we are just layering hack after hack to deal with the fact that we cannot write all required data into a vtk file. These days VTK/paraview has its own ExodusII reader so we have a route to nice seamless integration.
>>
>>
>>>   - The VTS viewer only allows to read and write _all_ fields in a DM. This may be overkill if one only
>>>     wants to read boundary values, for instance.
>>
>> or only writing out prognostic fields for example.
>>
>>>   - The VTS viewer loses all informations on exodus nodesets and cell sets. These may have some significance
>>>     and may be required to exploit the output of a computations.
>>
>> Right - this includes boundary labels, and it is just a fluff to have to write this out into VTK. You would have to write a separate vtu or something resulting in more messiness (and I already have enough problems on LUSTER from having too many files).
>>
>>
>>>   - VTS seems to have a concept of ?blocks?. My understanding is that the parallel VTS viewer uses blocks to
>>>     save subdomains, and that continuity of piecewise linear fields across subdomain boundaries is lost.
>>>     It is not entirely clear to me if with this layout, it would be possible to reopen a file with a
>>>     different processor count.
>>
>> I think you just do not want to go there? For a start your vtk file would not be a valid checkpoint to consider restarting on a different number of processes. And secondly, it would just be a mess to program.
>>
>>>
>>> I can dedicate some resources to improving DMplex I/O. Perhaps we can start a discussion by listing the desired features such readers / writers should have. I will pitch in by listing what matters to me:
>>
>> Keep talking?we have also an FTE working on this currently but this is a long wish list and a lot of effort is required if this is to be done within a reasonable time frame. It would be great if more people were working on this.
> I have a postdoc here who will devote some of his time to this task.
>
>>
>>>   - A well documented and adopted file format that most post-processors / visualization tools can use
>>
>> ExodusII appears to be the current favoured option.
> Sadly yes? But SILO may be a close second and has a more modern interface.
>
>
>>
>>>   - Ability to read / write individual fields
>>>   - Preserve _all_ information from the exodus file (node / side / cell sets), do not lose continuity of fields
>>>     across subdomain boundaries.
>>>   - Ability to reopen file on a different cpu count
>>
>> So this is where we need to have parallel IO support. Quoting from petcs?s exodus.py
>> ??"
>> # ExodusII does not call HDF5 directly, but it does call nc_def_var_deflate(), which is only
>> # part of libnetcdf when built using --enable-netcdf-4.  Currently --download-netcdf (netcdf.py)
>> # sets --enable-netcdf-4 only when HDF5 is enabled.
>> ??"
>> So, there may be some rebuilding required to ensure that all the dependencies are built properly but it?s clearly there.
>
> I am not sure if Exodus has a good solution here. As far as I understand, exodus is inherently sequential, even when implemented with HDF5 instead of netcdf. I would also worry about third party support for exodus files using HDF5 as their storage format.
> Exodus has an parallel extension called nemesis, but I can?t figure out how how their concept of ghost point and cells works. The documentation on this point is really unclear.
>
> Considering the dismal state of parallel FE formats and libraries, it seems to me that one needs to chose between two options:
>
>   a. scatter back to a single I/O node and use sequential I/O using the ordering of the original (exodus) mesh. This allows reading and writing on an arbitrary number of processors, but has potential memory footprint and performance issues. How large a mesh can we reasonably expect to be able to handle this way?
>   b. Do ?poor man? parallel I/O where each CPU does its own I/O, and possibly create interface matching files ? la nemesis or SILO. Maybe, we can save enough information on the parallel layout in order to easily write an un-partitionner as a post-processor.
>
> Unless one can come up with a better solution than a or b, I?d like to start by writing a very simple ASCII viewer demonstrating the communication involved, then modify it to use exodus, SILO or HDF5 format.
>
> Blaise
>
>

Depending on the degree of direct interaction/automation in those interactions between the mesh and Petsc, there are 
other options as well.  One that we're developing, based on the MOAB library, can read/write (in serial) ExodusII, and 
also supports parallel read/write using its own HDF5-based format.  Parallel I/O robustness has been iffy above ~16k 
procs and 32M-64M hex/tet elements, but for smaller problems it should work.  We're in the process of developing direct 
support for going between a mesh defined with fields (called tags in MOAB) and petsc vectors.  MOAB has pretty solid 
support for things like computing sharing and ghosting between procs and exchanging/reducing field values on those 
entities.  Viz is supported either by compiling a VTK/Paraview plugin that pulls the mesh/fields through MOAB or by 
translating to VTK (also supported directly from MOAB); Visit also has a plugin you can enable.  See 
http://trac.mcs.anl.gov/projects/ITAPS/wiki/MOAB for details of MOAB; the petsc integration stuff is on a bitbucket 
branch of petsc owned by Vijay Mahadevan.

libmesh also maintains its own DMlibmesh, but I'm not sure how solid their support for large mesh / parallel I/O is (but 
they've been working on it recently I know).

- tim

-- 
================================================================
"You will keep in perfect peace him whose mind is
   steadfast, because he trusts in you."               Isaiah 26:3

              Tim Tautges            Argonne National Laboratory
          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
  phone (gvoice): (608) 354-1459      1500 Engineering Dr.
             fax: (608) 263-4499      Madison, WI 53706