<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; "><div>I think there are two ways to approach this problem:</div><div><br></div><div>- Discover why you run out of memory with VisIt? Is this to be expected? How many elements and what polynomial order are you using? How much memory memory does your machine have? Which VisIt version are you using? May be VisIt is running into a memory leak?</div><div>- Let's assume you can't do it with VisIt (for whatever reason). I guess one option you may want to consider is: Get the data for a defined set of points (e.g slice grid) using the parallel interpolation routines and dump the data into a single or shared (MPI-IO) file. Make sure that your file format is supported by VisIt. This works really well even with huge data sets.</div><div><br></div><div>Hth,</div><div>Stefan</div><div><br></div><span id="OLK_SRC_BODY_SECTION"><div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt"><span style="font-weight:bold">Von: </span> <<a href="mailto:nek5000-users@lists.mcs.anl.gov">nek5000-users@lists.mcs.anl.gov</a>><br><span style="font-weight:bold">Antworten an: </span> <<a href="mailto:nek5000-users@lists.mcs.anl.gov">nek5000-users@lists.mcs.anl.gov</a>><br><span style="font-weight:bold">Datum: </span> Tue, 25 Jun 2013 08:16:37 -0700<br><span style="font-weight:bold">An: </span> <<a href="mailto:nek5000-users@lists.mcs.anl.gov">nek5000-users@lists.mcs.anl.gov</a>><br><span style="font-weight:bold">Betreff: </span> Re: [Nek5000-users] Writing 2d output files for 3d data<br></div><div><br></div>Paul,<br><div class="gmail_quote"><br>Thanks for the starting points. I have a few more questions, especially about the parallelization that's necessary for these huge systems. I don't really care about speed here, I just want to output a 2d slice and I don't plan on doing it very often.<br><br>1.) for z_slice_g(), what is ezi? Say I want to output all the data at z=0 (i.e. the bottom plate). What would I use for ezi? Or do I have to create some routine to find ezi for this case?<br><br>2.) I am a little confused about the parallel version of outfld2d(). It looks like outfld2d_p gets called by buff_2d_out(). Would you suggest doing this?<br><br>Thanks,<br><br>Janet<br><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><blockquote type="cite"><div><br>Hi Janet,<br><br>
I added z_slice_g, a generalized z_slice() extraction routine<br>to the repo in navier5.f.<br><br>I've not tested it.<br><br>Please let me know if this works for you. Note that all it<br>does is to extract a scalar. You would call this once for each<br>
variable of interest and then dump them as a field using some<br>type of output routine --- outfld2d() and outfld2d_p() look<br>like decent starting points but are not quite what you need.<br><br>If I were you, I would start with the binary .fld (not .f0000)<br>
format from a single processor and see how long that takes.<br><br>I think the _p() variant does the following ---<br><br>For several processors, it holds the data from different output<br>times (e.g., proc 0 holds time 1, proc 64 holds time 2, etc.).<br>
Then, at an agreed-upon timestep, each of these opens a file,<br>in parallel, and writes the entire z-slice that it holds as a copy.<br><br>Since each processor gets an entire z-slice from the above routine,<br>this is not a big deal, and you still get parallel I/O --- it's<br>
time parallel instead of space parallel. Here's a cartoon:<br><br><br>time/space: p0 p4 p12 p16<br><br>step 1000 get slice get slice get slice get slice<br>step 1000 save slice<br><br>step 2000 get slice get slice get slice get slice<br>step 2000 save slice<br><br>step 3000 get slice get slice get slice get slice<br>step 3000 save slice<br><br>
step 4000 get slice get slice get slice get slice<br>step 4000 save slice<br><br>step 4000 write f1 write f2 write f3 write f4 <-- Parallel IO<br><br>The attraction of this approach is that the write() looks<br>
like serial I/O because the requisite data is fully resident<br>on the processor that is doing the write, so it is simpler<br>to code.<div><div class="h5"><br><br>Paul<br><br>On Tue, 25 Jun 2013, <a href="mailto:nek5000-users@lists.mcs.anl.gov" target="_blank">nek5000-users@lists.mcs.anl.gov</a> wrote:<br><br><blockquote type="cite"><br></blockquote><blockquote type="cite">Hi Janet,<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I think that z_slice() and outfld2d would be reasonable<br></blockquote><blockquote type="cite">starting points.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">As I look at the code, z_slice takes element 1, slice 1<br></blockquote><blockquote type="cite">in the z direction. You would want to generalize this.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Also, it looks outfld2d() is set up only for ascii output<br></blockquote><blockquote type="cite">at the moment. I would suggest conversion to binary.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I can take a pass at making these changes but will not have<br></blockquote><blockquote type="cite">time to verify functionality for a few days. I could however<br></blockquote><blockquote type="cite">work with you on that.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Paul<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">On Tue, 25 Jun 2013, <a href="mailto:nek5000-users@lists.mcs.anl.gov" target="_blank">nek5000-users@lists.mcs.anl.gov</a> wrote:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite">We are generating huge 3d data files (>8GB with multiple I/O) which are so<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">large, our version of visit does not have enough memory to load them.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Oftentimes we only want to visualize a 2d slice anyway. Is there a routine<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">
in nek5000 that does this? Or one that can be easily modified? To be more<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">specific, we want to output, for example, the coordinates and the velocity<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">field for some fixed z-value, into a blah.f00001 file that can then be read<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">
by visit.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Thanks,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Janet<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">-- <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Janet Scheel<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Assistant Professor of Physics<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Occidental College<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">1600 Campus Road, M21<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Los Angeles, CA 90041<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><a href="tel:%28323%29%20259-2777" value="+13232592777" target="_blank">(323) 259-2777</a><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><a href="mailto:jscheel@oxy.edu" target="_blank">jscheel@oxy.edu</a><br></blockquote></blockquote><blockquote type="cite">_______________________________________________<br></blockquote><blockquote type="cite">Nek5000-users mailing list<br></blockquote><blockquote type="cite"><a href="mailto:Nek5000-users@lists.mcs.anl.gov" target="_blank">Nek5000-users@lists.mcs.anl.gov</a><br></blockquote><blockquote type="cite"><a href="https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users</a><br></blockquote><blockquote type="cite"><br></blockquote>
_______________________________________________<br>Nek5000-users mailing list<br><a href="mailto:Nek5000-users@lists.mcs.anl.gov" target="_blank">Nek5000-users@lists.mcs.anl.gov</a><br><a href="https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users</a><br></div></div></div></blockquote></div><br></div></blockquote></div><br><br clear="all"><br>-- <br>Janet Scheel<br>Assistant Professor of Physics<br>Occidental College<br>1600 Campus Road, M21<br>Los Angeles, CA 90041<br>(323) 259-2777<br><a href="mailto:jscheel@oxy.edu">jscheel@oxy.edu</a>
_______________________________________________
Nek5000-users mailing list
<a href="mailto:Nek5000-users@lists.mcs.anl.gov">Nek5000-users@lists.mcs.anl.gov</a>
<a href="https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users">https://lists.mcs.anl.gov/mailman/listinfo/nek5000-users</a>
</span></body></html>