[MOAB-dev] DMMoabLoadFromFile() - parallel performance issue

[Grindeanu, Iulian R.]

Hi Jim,
DMMoab loading just delegates to moab's hdf5 reader

We were seeing performance degradations for larger number of tasks, on the order of 8k, 16k tasks


can you say something about your:
 hdf5 version, config flags (I assume optimized, production )
 machine characteristics (I/O performance)?
mpi version / compiler/ mpi IO version?

for more fine grain profiling, can you use this branch, and use CPUTIME option at reading?
https://bitbucket.org/fathomteam/moab/branch/iulian07/timing_h5mreader


hanging out for a mesh of 30 million looks like a problem/issue.
Do you have empty partitions? I assume not (you would have seen a warning at partitioning).

Iulian



________________________________
From: moab-dev-bounces at mcs.anl.gov [moab-dev-bounces at mcs.anl.gov] on behalf of WARNER, JAMES E. (LARC-D309) [james.e.warner at nasa.gov]
Sent: Monday, December 14, 2015 5:25 PM
To: moab-dev at mcs.anl.gov
Subject: [MOAB-dev] DMMoabLoadFromFile() - parallel performance issue

Hi Vijay & Iulian,

Hope you are doing well! I have a question regarding some strange behavior we’re seeing with the DMMoabLoadFromFile() function…

After doing some recent profiling of our MOAB-based finite element code, we noticed that we are spending a disproportionate amount of CPU time within the DMMoabLoadFromFile() function, which gets slower / remains constant as we increase the number of processors. We also recently attempted a scalability test with ~30M FEM nodes  on 500 processors which hung in DMMoabLoadFromFile() for about 45 minutes before we killed the job. We then re-ran the test on one processor and it made it through successfully in several seconds.

To reproduce the problem we’re seeing, we wrote a test case (attached here) that simply loads a smaller mesh with approximately 16K nodes and prints the run time. When I run the code on an increasing number of processors, I get something like:

NP=1: Time to read file: 0.0416839 [sec.]
NP=2: Time to read file: 1.42497 [sec.]
NP=4: Time to read file: 1.13678 [sec.]
NP=8: Time to read file: 1.0475 [sec.]
…

If it is relevant/helpful – we are using the mbpart tool to partition the mesh.  Do you have any ideas why we are not seeing scalability here? Any thoughts/tips would be appreciated! Let me know if you would like any more information.

Thanks,
Jim



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/moab-dev/attachments/20151215/44fa5070/attachment.html>