[MOAB-dev] file formats
Grindeanu, Iulian R.
iulian at mcs.anl.gov
Fri Apr 8 08:35:45 CDT 2016
Hi Jacob,
This is very interesting, do you have a small example with xmdf and h5m that you can share?
I don't know about paraview, but visit uses an hdf5 reader that is not compiled with parallel option.
So that means that the xmdf -> hdf5 datasets reads are serial, in my understanding (at least when used from visit)
I see some notes in the page you are pointing that xml has "light" data, while hdf5 files would have "heavy" data.
For h5m / moab, connectivity would qualify as heavy or light ?
Thanks,
Iulian
________________________________________
From: moab-dev-bounces at mcs.anl.gov [moab-dev-bounces at mcs.anl.gov] on behalf of Jacob Hochhalter [Jacob.D.Hochhalter at nasa.gov]
Sent: Friday, April 08, 2016 7:43 AM
To: moab-dev at mcs.anl.gov
Subject: Re: [MOAB-dev] file formats
All,
We have had this issue come up too. Our solution, which we have been
very happy with so far, is to use the .h5m file as part of the XDMF
concept (see http://www.xdmf.org/index.php/Main_Page). With that
approach, we simply 'wrap' the .h5m file with a xml file, which
describes which data we want to visualize (and how). There is no
duplication of the mesh or results data, since the xml file simply
'points' to datasets in the h5m file. The xml file very generation is
very quick with a simple python script (we have written to general
python classes for this, if you're interested). An added bonus is that
both VisIt and Paraview support XDMF inherently.
- Jake
On 04/08/2016 03:31 AM, Nico Schlömer wrote:
> Thanks for the comment!
>
> To be able to do I/O from a visualizable format directly (without
> mbconvert or any other tool) is in fact more important to me than not
> wasting space on storing a mesh more than once (although also
> interesting). Perhaps these two things can be aligned.
>
> Cheers,
> Nico
>
> On Thu, Apr 7, 2016 at 11:06 PM Vijay S. Mahadevan <vijay.m at gmail.com
> <mailto:vijay.m at gmail.com>> wrote:
>
> Nico, if the time series data are stored as separate tags, like
> [SOLUTION_STEP0, SOLUTION_STEP1, .., SOLUTION_STEPN], any of the
> conversion tools should provide the right data output that can be
> visualized easily.
>
> You should be able to use mbconvert that gets installed with MOAB for
> the conversion. Of course, if you want exodus output, you will need
> the NetCDF dependency enabled. VTK is available out of the box though.
>
> We can also discuss more about the time series data and not writing
> the mesh more than once, especially for static non-AMR simulations. We
> should be able to find some time in the summer or a little later to
> implement this functionality within MOAB.
>
> Vijay
>
> On Wed, Apr 6, 2016 at 2:46 PM, Nico Schlömer
> <nico.schloemer at gmail.com <mailto:nico.schloemer at gmail.com>> wrote:
> > Thanks for the comments!
> >
> >> the visualization is the tricky bit tere since none of the main
> tools
> >> (VisIt, Paraview) understand the
> >> details of the layout explicitly
> >> [...]
> >> if we understand your needs better.
> >
> > The question I have to answer the most is how to visualize H5M
> files. I
> > usually tell people to grab meshio [1] and convert to a format
> they use. I
> > guess I could extend meshio such that it can translate h5m's tag
> time series
> > data into another format with native series. Anyhow, more work
> for me and
> > for the users (to convert), so I'm not so happy with that yet.
> >
> > Cheers,
> > Nico
> >
> > [1] https://pypi.python.org/pypi/meshio
> >
> > On Wed, Apr 6, 2016 at 4:49 PM Vijay S. Mahadevan
> <vijay.m at gmail.com <mailto:vijay.m at gmail.com>> wrote:
> >>
> >> Hi NIco,
> >>
> >> Our current HDF5-based I/O workflow supports adding arbitrary
> tags if
> >> you want to store time-stepping data. This is what Lukasz is
> doing for
> >> his needs and should extend arbitrarily to other applications using
> >> MOAB. As Lukasz also mentioned, the visualization is the tricky bit
> >> here since none of the main tools (VisIt, Paraview) understand the
> >> details of the layout explicitly and so only expose part of the
> data
> >> available in h5m files. I've also been think about enhancements to
> >> support multiple HDF5 files, where the mesh is written out only
> >> initially and when it changes. But this is a little more
> involved for
> >> the book-keeping to remain general.
> >>
> >> On a related note, we are working with LLNL to add a native MOAB
> >> plugin to VisIt, that directly uses MOAB underneath, in
> parallel, to
> >> query and visualize solution. Once that is in place, adding
> capability
> >> to show time-series data can be added out of the box. Note that
> MOAB
> >> already supports only loading partial sets and tags, if you are
> >> concerned about the data movement with I/O for large simulation
> runs.
> >>
> >> However, we have also been considering alternatives that support
> >> parallel I/O. Yes, from our discussions, I believe we converged on
> >> XDMF and Exodus/Nemesis. Currently, this is a bottleneck on the
> >> resources available to us and so has been in the back burner.
> There is
> >> native support for time-series data with the pure NetCDF route
> >> already, written more focused on climate applications [1, 2].
> We can
> >> also talk about extending this to more general use cases, if we
> >> understand your needs better.
> >>
> >> Thoughts ?
> >>
> >> Vijay
> >>
> >> [1] https://www.ncl.ucar.edu/ParVis.shtml
> >> [2]
> https://cfwebprod.sandia.gov/cfdocs/CompResearch/docs/template.pdf
> >>
> >> On Wed, Apr 6, 2016 at 6:34 AM, Lukasz Kaczmarczyk
> >> <Lukasz.Kaczmarczyk at glasgow.ac.uk
> <mailto:Lukasz.Kaczmarczyk at glasgow.ac.uk>> wrote:
> >> > Hello,
> >> >
> >> > I am not convinced that this is the best approach, however I
> can tell
> >> > how I
> >> > do it.
> >> >
> >> > I create meshset with number of contained meshsets equal to
> number of
> >> > time
> >> > steps. Each of contained meshsets has tag, which data on all
> entities on
> >> > meshset which I store for time step. That can be saved in
> native HD5
> >> > file
> >> > and read back if needed without data redundancies and allow
> to add time
> >> > steps. However is a issue of post-processing, that will need
> some proper
> >> > reader for post-procssor which I not currently have.
> >> >
> >> > There are additional complexities, when you store multiple
> fields and
> >> > you
> >> > store data for variable number of entities for each step. In
> may case I
> >> > implemented it like tape recorder, that I have different
> tapes for which
> >> > I
> >> > record and play tapes depending on need. If this will be any
> use for you
> >> > pleas look at example,
> >> >
> >> >
> http://mofem.eng.gla.ac.uk/mofem/html/record__series__atom_8cpp_source.html
> >> > In example I have two tapes there and two moab instances, one
> use for
> >> > recording another for reading for testing proposes. This
> works ok, but
> >> > is
> >> > missing proper reader for paraview and at the end I produce
> series of
> >> > VTKs
> >> > for paraview. This is very specific to my development by you
> can use it
> >> > as
> >> > idea if you like it.
> >> >
> >> > I would be great to have some common interface to record and
> read time
> >> > series of tags for moab. That if one will write reader for
> paraview it
> >> > could
> >> > be used by several developers woking on different codes.
> >> >
> >> >
> >> > Regards,
> >> > Lukasz
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On 6 Apr 2016, at 10:02, Nico Schlömer
> <nico.schloemer at gmail.com <mailto:nico.schloemer at gmail.com>> wrote:
> >> >
> >> > Hi everyone,
> >> >
> >> > I'll start tackling a time-series problem rather soon and I'm
> wondering
> >> > about file formats here. Ideally, I'd have one time series
> per file
> >> > (particularly since the mesh isn't changing over time,
> perhaps that
> >> > could be
> >> > made use of), but I'm not sure if MOAB's custom HDF5 supports
> that yet.
> >> >
> >> > Thinking of which, I faintly remember there had been talk about
> >> > switching to
> >> > a better-supported format in the future (xdmf and exodus came
> to mind).
> >> > Is
> >> > this still a thing?
> >> >
> >> > Cheers,
> >> > Nico
> >> >
> >> >
>
More information about the moab-dev
mailing list