<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style id="owaParaStyle" type="text/css">P {margin-top:0;margin-bottom:0;}</style>
</head>
<body ocsi="0" fpstyle="1" style="word-wrap:break-word">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">Hi Jim,<br>
DMMoab loading just delegates to moab's hdf5 reader<br>
<br>
We were seeing performance degradations for larger number of tasks, on the order of 8k, 16k tasks<br>
<br>
<br>
can you say something about your:<br>
hdf5 version, config flags (I assume optimized, production )<br>
machine characteristics (I/O performance)?<br>
mpi version / compiler/ mpi IO version?<br>
<br>
for more fine grain profiling, can you use this branch, and use CPUTIME option at reading?<br>
<a href="https://bitbucket.org/fathomteam/moab/branch/iulian07/timing_h5mreader" target="_blank">https://bitbucket.org/fathomteam/moab/branch/iulian07/timing_h5mreader</a><br>
<br>
<br>
hanging out for a mesh of 30 million looks like a problem/issue. <br>
Do you have empty partitions? I assume not (you would have seen a warning at partitioning).<br>
<br>
Iulian<br>
<br>
<br>
<br>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<hr tabindex="-1">
<div style="direction: ltr;" id="divRpF24892"><font size="2" color="#000000" face="Tahoma"><b>From:</b> moab-dev-bounces@mcs.anl.gov [moab-dev-bounces@mcs.anl.gov] on behalf of WARNER, JAMES E. (LARC-D309) [james.e.warner@nasa.gov]<br>
<b>Sent:</b> Monday, December 14, 2015 5:25 PM<br>
<b>To:</b> moab-dev@mcs.anl.gov<br>
<b>Subject:</b> [MOAB-dev] DMMoabLoadFromFile() - parallel performance issue<br>
</font><br>
</div>
<div></div>
<div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px">Hi Vijay & <span style="white-space:pre-wrap; widows:1">Iulian,</span></div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px"><span style="white-space:pre-wrap; widows:1"><br>
</span></div>
<div style="widows:auto"><span style="widows:1"><font face="Calibri,sans-serif"><span style="white-space:pre-wrap">Hope you are doing well!
</span></font></span><font face="Calibri,sans-serif"><span style="white-space:pre-wrap">I have a question regarding some strange behavior we’re seeing with the
</span></font>DMMoabLoadFromFile() function…</div>
<div style="widows:auto"><br>
</div>
<div style="widows:auto">After doing some recent profiling of our MOAB-based finite element code, we noticed that we are spending a disproportionate amount of CPU time within the DMMoabLoadFromFile() function, which gets slower / remains constant as we increase
the number of processors. We also recently attempted a scalability test with ~30M FEM nodes on 500 processors which hung in DMMoabLoadFromFile() for about 45 minutes before we killed the job. We then re-ran the test on one processor and it made it through
successfully in several seconds. </div>
<div style="widows:auto"><br>
</div>
<div style="widows:auto">To reproduce the problem we’re seeing, we wrote a test case (attached here) that simply loads a smaller mesh with approximately 16K nodes and prints the run time. When I run the code on an increasing number of processors, I get something
like:</div>
<div><span style="widows:1"><font face="Calibri,sans-serif"><span style="white-space:pre-wrap"><br>
</span></font></span></div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px">NP=1: Time to read file: 0.0416839 [sec.]</div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px">NP=2: Time to read file: 1.42497 [sec.]</div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px">NP=4: Time to read file: 1.13678 [sec.]</div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px">NP=8: Time to read file: 1.0475 [sec.]</div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px">…</div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px"><br>
</div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px">If it is relevant/helpful – we are using the mbpart tool to partition the mesh. Do you have any ideas why we are not seeing scalability here? Any thoughts/tips would be appreciated!
Let me know if you would like any more information. </div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px"><br>
</div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px">Thanks,</div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px">Jim</div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px"><br>
</div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px">
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</body>
</html>