[Nek5000-users] Writing 2d output files for 3d data

Tue Jun 25 10:37:24 CDT 2013

I think there are two ways to approach this problem:

- Discover why you run out of memory with VisIt? Is this to be expected? How
many elements and what polynomial order are you using? How much memory
memory does your machine have? Which VisIt version are you using? May be
VisIt is running into a memory leak?
- Let's assume you can't do it with VisIt (for whatever reason). I guess one
option you may want to consider is: Get the data for a defined set of points
(e.g slice grid) using the parallel interpolation routines and dump the data
into a single or shared (MPI-IO) file. Make sure that your file format is
supported by VisIt. This works really well even with huge data sets.

Hth,
Stefan

Von:  <nek5000-users at lists.mcs.anl.gov>
Antworten an:  <nek5000-users at lists.mcs.anl.gov>
Datum:  Tue, 25 Jun 2013 08:16:37 -0700
An:  <nek5000-users at lists.mcs.anl.gov>
Betreff:  Re: [Nek5000-users] Writing 2d output files for 3d data

Paul,

Thanks for the starting points. I have a few more questions, especially
about the parallelization that's necessary for these huge systems. I don't
really care about speed here, I just want to output a 2d slice and I don't
plan on doing it very often.

1.) for z_slice_g(), what is ezi? Say I want to output all the data at z=0
(i.e. the bottom plate). What would I use for ezi? Or do I have to create
some routine to find ezi for this case?

2.) I am a little confused about the parallel version of outfld2d(). It
looks like outfld2d_p gets called by buff_2d_out(). Would you suggest doing
this?

Thanks,

Janet

>> 
>> Hi Janet,
>> 
>> I added z_slice_g, a generalized z_slice() extraction routine
>> to the repo in navier5.f.
>> 
>> I've not tested it.
>> 
>> Please let me know if this works for you.   Note that all it
>> does is to extract a scalar.  You would call this once for each
>> variable of interest and then dump them as a field using some
>> type of output routine --- outfld2d() and outfld2d_p() look
>> like decent starting points but are not quite what you need.
>> 
>> If I were you, I would start with the binary .fld (not .f0000)
>> format from a single processor and see how long that takes.
>> 
>> I think the _p() variant does the following ---
>> 
>> For several processors, it holds the data from different output
>> times (e.g., proc 0 holds time 1, proc 64 holds time 2, etc.).
>> Then, at an agreed-upon timestep, each of these opens a file,
>> in parallel, and writes the entire z-slice that it holds as a copy.
>> 
>> Since each processor gets an entire z-slice from the above routine,
>> this is not a big deal, and you still get parallel I/O --- it's
>> time parallel instead of space parallel.   Here's a cartoon:
>> 
>> 
>> time/space:        p0         p4         p12        p16
>> 
>> step 1000      get slice  get slice  get slice  get slice
>> step 1000     save slice
>> 
>> step 2000      get slice  get slice  get slice  get slice
>> step 2000                save slice
>> 
>> step 3000      get slice  get slice  get slice  get slice
>> step 3000                           save slice
>> 
>> step 4000      get slice  get slice  get slice  get slice
>> step 4000                                      save slice
>> 
>> step 4000      write f1   write f2   write f3   write f4   <-- Parallel IO
>> 
>> The attraction of this approach is that the write() looks
>> like serial I/O because the requisite data is fully resident
>> on the processor that is doing the write, so it is simpler
>> to code.
>> 
>> 
>> Paul
>> 
>> On Tue, 25 Jun 2013, nek5000-users at lists.mcs.anl.gov wrote:
>> 
>>> 
>>> Hi Janet,
>>> 
>>> I think that z_slice() and outfld2d would be reasonable
>>> starting points.
>>> 
>>> As I look at the code, z_slice takes element 1, slice 1
>>> in the z direction.   You would want to generalize this.
>>> 
>>> Also, it looks outfld2d() is set up only for ascii output
>>> at the moment.   I would suggest conversion to binary.
>>> 
>>> I can take a pass at making these changes but will not have
>>> time to verify functionality for a few days.  I could however
>>> work with you on that.
>>> 
>>> Paul
>>> 
>>> 
>>> On Tue, 25 Jun 2013, nek5000-users at lists.mcs.anl.gov wrote:
>>> 
>>>> We are generating huge 3d data files (>8GB with multiple I/O) which are so
>>>> large, our version of visit does not have enough memory to load them.
>>>> Oftentimes we only want to visualize a 2d slice anyway. Is there a routine
>>>> in nek5000 that does this? Or one that can be easily modified? To be more
>>>> specific, we want to output, for example, the coordinates and the velocity
>>>> field for some fixed z-value, into a blah.f00001 file that can then be read
>>>> by visit.
>>>> Thanks,
>>>> Janet
>>>> -- 
>>>> Janet Scheel
>>>> Assistant Professor of Physics
>>>> Occidental College
>>>> 1600 Campus Road, M21
>>>> Los Angeles, CA 90041
>>>> (323) 259-2777 <tel:%28323%29%20259-2777>
>>>> jscheel at oxy.edu
>>> _______________________________________________
>>> Nek5000-users mailing list
>>> Nek5000-users at lists.mcs.anl.gov
>>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
>>> 
>> _______________________________________________
>> Nek5000-users mailing list
>> Nek5000-users at lists.mcs.anl.gov
>> https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users
> 

-- 
Janet Scheel
Assistant Professor of Physics
Occidental College
1600 Campus Road, M21
Los Angeles, CA 90041
(323) 259-2777
jscheel at oxy.edu _______________________________________________
Nek5000-users mailing list Nek5000-users at lists.mcs.anl.gov
https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/nek5000-users/attachments/20130625/2cae17f1/attachment-0001.html>