[MOAB-dev] file formats

[Vijay S. Mahadevan]

Hi NIco,

Our current HDF5-based I/O workflow supports adding arbitrary tags if
you want to store time-stepping data. This is what Lukasz is doing for
his needs and should extend arbitrarily to other applications using
MOAB. As Lukasz also mentioned, the visualization is the tricky bit
here since none of the main tools (VisIt, Paraview) understand the
details of the layout explicitly and so only expose part of the data
available in h5m files. I've also been think about enhancements to
support multiple HDF5 files, where the mesh is written out only
initially and when it changes. But this is a little more involved for
the book-keeping to remain general.

On a related note, we are working with LLNL to add a native MOAB
plugin to VisIt, that directly uses MOAB underneath, in parallel, to
query and visualize solution. Once that is in place, adding capability
to show time-series data can be added out of the box. Note that MOAB
already supports only loading partial sets and tags, if you are
concerned about the data movement with I/O for large simulation runs.

However, we have also been considering alternatives that support
parallel I/O. Yes, from our discussions, I believe we converged on
XDMF and Exodus/Nemesis. Currently, this is a bottleneck on the
resources available to us and so has been in the back burner. There is
native support for time-series data with the pure NetCDF route
already, written more focused on climate applications [1, 2]. We can
also talk about extending this to more general use cases, if we
understand your needs better.

Thoughts ?

Vijay

[1] https://www.ncl.ucar.edu/ParVis.shtml
[2] https://cfwebprod.sandia.gov/cfdocs/CompResearch/docs/template.pdf

On Wed, Apr 6, 2016 at 6:34 AM, Lukasz Kaczmarczyk
<Lukasz.Kaczmarczyk at glasgow.ac.uk> wrote:
> Hello,
>
> I am not convinced that this is the best approach, however I can tell how I
> do it.
>
> I create meshset with number of contained meshsets equal to number of time
> steps. Each of contained meshsets has tag, which data on all entities on
> meshset which I store for time step. That can be saved in native HD5 file
> and read  back if needed without data redundancies and allow to add time
> steps. However is a issue of post-processing, that will need some proper
> reader for post-procssor which I not currently have.
>
> There are additional complexities, when you store multiple fields and you
> store data for variable number of entities for each step. In may case I
> implemented it like tape recorder, that I have different tapes for which I
> record and play tapes depending on need. If this will be any use for you
> pleas look at example,
> http://mofem.eng.gla.ac.uk/mofem/html/record__series__atom_8cpp_source.html
> In example I have two tapes there and two moab instances, one use for
> recording another for reading for testing proposes. This works ok, but is
> missing proper reader for paraview and at the end I produce series of VTKs
> for paraview. This is very specific to my development by you can use it as
> idea if you like it.
>
> I would be great to have some common interface to record and read time
> series of tags for moab. That if one will write reader for paraview it could
> be used by several developers woking on different codes.
>
>
> Regards,
> Lukasz
>
>
>
>
>
> On 6 Apr 2016, at 10:02, Nico Schlömer <nico.schloemer at gmail.com> wrote:
>
> Hi everyone,
>
> I'll start tackling a time-series problem rather soon and I'm wondering
> about file formats here. Ideally, I'd have one time series per file
> (particularly since the mesh isn't changing over time, perhaps that could be
> made use of), but I'm not sure if MOAB's custom HDF5 supports that yet.
>
> Thinking of which, I faintly remember there had been talk about switching to
> a better-supported format in the future (xdmf and exodus came to mind). Is
> this still a thing?
>
> Cheers,
> Nico
>
>