<html><body><div style="color:#000; background-color:#fff; font-family:verdana, helvetica, sans-serif;font-size:10pt"><div id="yiv3958718385"><div><div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); font-family: verdana, helvetica, sans-serif; font-size: 10pt;"><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_724" style=""><span class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_879" style="">On unit test for my project, I am trying to use ParallelComm::assign_global_id() to get the global ids for vertices of a simple grid</span></div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_724" style="background-color:transparent;"> // Mesh Ids
</div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_724" style="background-color:transparent;"> // 2.0 6 ------- 7 -------- 8 -------- 9 --------- 10
</div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_724" style="background-color:transparent;"> // | | | | | </div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_724" style="background-color:transparent;"> // | 1 | 2 | 3 | 4 | </div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_724" style="background-color:transparent;"> // |
| |
| | </div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_724" style="background-color:transparent;"> // 1.0 1 ------- 2 -------- 3 -------- 4 --------- 5</div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_724" style="background-color:transparent;"><span class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_881" style=""></span></div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_724" style="background-color:transparent;"> // 0.0 1.0 2.0 3.0 4.0 </div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_724"
style="background-color:transparent;"><br clear="none" style=""></div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_724" style="background-color: transparent; color: rgb(0, 0, 0); font-size: 13px; font-family: verdana, helvetica, sans-serif; font-style: normal;">and got global ids as following</div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_724" style="background-color:transparent;">// Global Ids </div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_724" style="background-color:transparent;"> // 2.0 4 ------- 3 -------- 6 -------- 10 --------- 14
</div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_724" style="background-color:transparent;"> // | |
| | | </div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_724" style="background-color:transparent;"> // | | | | | </div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_724" style="background-color:transparent;"> // | | | | |
</div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_724" style="background-color:transparent;"> // 1.0 1 ------- 2 -------- 5 -------- 9 --------- 13</div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_724" style="background-color:transparent;"> // 0.0 1.0 2.0 3.0 4.0 </div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_724" style="background-color:transparent;"><br clear="none" id="yiv3958718385yui_3_16_0_1_1400642133071_22203"></div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_724" style="background-color: transparent; color: rgb(0, 0, 0); font-size: 13px; font-family: verdana, helvetica, sans-serif; font-style: normal;">Don't know if this is what
Anton got through. </div><div class="yiv3958718385 yui_3_16_0_1_1400595764048_145087" id="yiv3958718385yui_3_16_0_8_1400642133071_724" style="background-color:transparent;color:rgb(0, 0, 0);font-size:13px;font-family:verdana, helvetica, sans-serif;font-style:normal;"><br></div><div class="yiv3958718385 yui_3_16_0_1_1400595764048_145087" id="yiv3958718385yui_3_16_0_8_1400642133071_724" style="background-color:transparent;color:rgb(0, 0, 0);font-size:13px;font-family:verdana, helvetica, sans-serif;font-style:normal;">Iulian, if you are interested to see the test, please let me know, and I'll send to you.</div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_726" style=""></div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_728" style=""> </div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_730" style="">Jane </div><div class="yiv3958718385"
id="yiv3958718385yui_3_16_0_1_1400642133071_17750" style=""><br clear="none" class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_732" style=""><br clear="none" class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_734" style=""></div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_736" style="">Asst. Researcher<br clear="none" class="yiv3958718385" style="">Dept. of Engineering Physics<br clear="none" class="yiv3958718385" style="">UW @ Madison</div><br clear="none" class="yiv3958718385" style=""><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_1_1400642133071_17748" style=""><br clear="none" class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_739" style=""></div><div class="yiv3958718385" id="yiv3958718385yui_3_16_0_8_1400642133071_741" style="">"And we know that for those who love God, that is, for those who are called according to his purpose, all things are working together for
good." (Romans 8:28)</div><div class="yiv3958718385yqt9405446528" id="yiv3958718385yqt52686"><div class="yiv3958718385yahoo_quoted" id="yiv3958718385yui_3_16_0_8_1400642133071_744" style="display: block;"> <div class="yiv3958718385" style="font-family: verdana, helvetica, sans-serif; font-size: 10pt;"> <div class="yiv3958718385" style="font-family: HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; font-size: 12pt;"> <div class="yiv3958718385" dir="ltr" style=""> <font class="yiv3958718385" size="2" face="Arial" style=""> On Tuesday, May 20, 2014 5:36 PM, "Grindeanu, Iulian R." <iulian@mcs.anl.gov> wrote:<br clear="none" class="yiv3958718385" style=""> </font> </div> <br clear="none" class="yiv3958718385" style=""><br clear="none" class="yiv3958718385" style=""> <div class="yiv3958718385" style=""><div class="yiv3958718385" id="yiv3958718385" style=""><style type="text/css">#yiv3958718385 P
{margin-top:0;margin-bottom:0;}</style><div class="yiv3958718385" style="">
<div class="yiv3958718385" style="direction: ltr; font-family: Tahoma; color: rgb(0, 0, 0); font-size: 10pt;">hmmm,<br clear="none" class="yiv3958718385" style="">
this is not good :(<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
Are you running this on mira? Do you have a small file for a laptop/workstation?<br clear="none" class="yiv3958718385" style="">
Maybe I can create one similar.<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
Do you see this only on 1024 processes or can it be lower count?<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
How does your model look like?<br clear="none" class="yiv3958718385" style="">
Any processor should not communicate with more than 64 other processes, maybe after ghosting this number is reached.<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
Can you run a debug version of this ? maybe some asserts are not triggered in optimized mode.<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
Is your file somewhere on mira I can get to it? <br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
Iulian<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
<div class="yiv3958718385" id="yiv3958718385yqt75616" style=""><div class="yiv3958718385" style="font-family: 'Times New Roman'; color: rgb(0, 0, 0); font-size: 16px;">
<hr class="yiv3958718385" tabindex="-1" style="">
<div class="yiv3958718385" id="yiv3958718385divRpF370222" style="direction:ltr;"><font class="yiv3958718385" color="#000000" face="Tahoma" size="2" style=""><b class="yiv3958718385" style="">From:</b> moab-dev-bounces@mcs.anl.gov [moab-dev-bounces@mcs.anl.gov] on behalf of kanaev@ibrae.ac.ru [kanaev@ibrae.ac.ru]<br clear="none" class="yiv3958718385" style="">
<b class="yiv3958718385" style="">Sent:</b> Tuesday, May 20, 2014 5:05 PM<br clear="none" class="yiv3958718385" style="">
<b class="yiv3958718385" style="">To:</b> MOAB dev<br clear="none" class="yiv3958718385" style="">
<b class="yiv3958718385" style="">Subject:</b> Re: [MOAB-dev] Fwd: Re: Job exiting early [ThermHydraX]<br clear="none" class="yiv3958718385" style="">
</font><br clear="none" class="yiv3958718385" style="">
</div>
<div class="yiv3958718385" style=""></div>
<div class="yiv3958718385" style="">
<div class="yiv3958718385" style="">The problem is still here </div>
<div class="yiv3958718385" style="">I've made a simple program performing certain numbers of exchange_tags calls within a loop
</div>
<div class="yiv3958718385" style="">If you run it on several processors with any mesh file it will eventually crash with the following message from every core:
</div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style="">Fatal error in PMPI_Isend: Internal MPI error!, error stack: </div>
<div class="yiv3958718385" style="">PMPI_Isend(148): MPI_Isend(buf=0xd0f300, count=4, MPI_UNSIGNED_CHAR, dest=1, tag=6, MPI_COMM_WORLD, request=0xcde354) failed
</div>
<div class="yiv3958718385" style="">(unknown)(): Internal MPI error! </div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style="">Thanks </div>
<div class="yiv3958718385" style="">Anton </div>
<div class="yiv3958718385" style=""><br clear="none" class="yiv3958718385" style="">
</div>
<div class="yiv3958718385" style="">On Tue, 20 May 2014 04:40:03 -0400, wrote: </div>
<blockquote class="yiv3958718385" style="padding-left:5px;border-left-color:#1010ff;border-left-width:2px;border-left-style:solid;margin-left:5px;width:100%;">
<div class="yiv3958718385" style="">Please disregard that, the global_id space for Quads was incontinuous in my mesh file
</div>
<div class="yiv3958718385" style="">Will check back with correct mesh </div>
<div class="yiv3958718385" style="">Anton<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
On Mon, 19 May 2014 15:17:14 -0400, wrote: </div>
<blockquote class="yiv3958718385" style="padding-left:5px;border-left-color:#1010ff;border-left-width:2px;border-left-style:solid;margin-left:5px;width:100%;">
<div class="yiv3958718385" style="">Hello MOAB dev, </div>
<div class="yiv3958718385" style="">I've attached a simplified version of my program that crashes presumably after particular numbers calls of exchange_tags
</div>
<div class="yiv3958718385" style="">I ran it couple of times on Mira on 1024 cores (64 nodes in --mode c16) </div>
<div class="yiv3958718385" style="">It performs around 524378 iterations and then crushes (error file attached) </div>
<div class="yiv3958718385" style="">Can you please take a look at what Scott Parker from ALCF suggests about it: </div>
<div class="yiv3958718385" style="">-------- Original Message -------- </div>
<table class="yiv3958718385" border="0" cellpadding="0" cellspacing="0" style=""><tbody class="yiv3958718385" style=""><tr class="yiv3958718385" style=""><th align="right" colspan="1" rowspan="1" valign="baseline" class="yiv3958718385" style="">Subject: </th><td colspan="1" rowspan="1" class="yiv3958718385" style="">Re: Job exiting early [ThermHydraX]</td></tr><tr class="yiv3958718385" style=""><th align="right" colspan="1" rowspan="1" valign="baseline" class="yiv3958718385" style="">Date: </th><td colspan="1" rowspan="1" class="yiv3958718385" style="">Fri, 9 May 2014 18:48:25 -0500</td></tr><tr class="yiv3958718385" style=""><th align="right" colspan="1" rowspan="1" valign="baseline" class="yiv3958718385" style="">From: </th><td colspan="1" rowspan="1" class="yiv3958718385" style="">Scott Parker </td></tr><tr class="yiv3958718385" style=""><th align="right" colspan="1" rowspan="1" valign="baseline" class="yiv3958718385" style="">To: </th><td colspan="1"
rowspan="1" class="yiv3958718385" style=""> </td></tr></tbody></table>
<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
Anton-<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
I took a look at the core files and from the stack trace it appears that the code is failing in an MPI_Isend call<br clear="none" class="yiv3958718385" style="">
that is called from moab.ParallelComm::recv_buffer which is called from moab::ParallelComm::exchange_tags<br clear="none" class="yiv3958718385" style="">
called from main(). The stack looks like:<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
main<br clear="none" class="yiv3958718385" style="">
moab::ParallelComm::exchange_tags<br clear="none" class="yiv3958718385" style="">
moab.ParallelComm::recv_buffer<br clear="none" class="yiv3958718385" style="">
MPI_Isend<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
I've been able to get the same failure and error message you are seeing by having an MPI process call MPI_Isend<br clear="none" class="yiv3958718385" style="">
when there are no matching receives. After 2 million Isends the program exits with the error you are seeing. So<br clear="none" class="yiv3958718385" style="">
I'm pretty sure your ending up with a large number of outstanding requests and the program is failing because<br clear="none" class="yiv3958718385" style="">
it can't allocate space for new MPI_Request objects.<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
I'd suggest looking at how Moab is calling MPI and how many requests might be outstanding at any one time.<br clear="none" class="yiv3958718385" style="">
Since the code is running for 5 hours and looks to be executing hundreds of thousands of iterations I wonder<br clear="none" class="yiv3958718385" style="">
if there is some sort of send-receive mismatch that is letting requests accumulate. I think your best bet is to<br clear="none" class="yiv3958718385" style="">
talk to the Moab folks and see if they have any ideas about why this might be happening.
<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
One possibility is a load imbalance between processes - if you don't have any MPI_Barriers or other collectives in<br clear="none" class="yiv3958718385" style="">
your code you could try adding a barrier to synchronize the processes.<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
If the Moab guys can't help you and adding a barrier doesn't help I can work with you to instrument the code to<br clear="none" class="yiv3958718385" style="">
collect more information on how MPI is being called and we could possibly pin down the source of the problem<br clear="none" class="yiv3958718385" style="">
that way.<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
Scott<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
<div class="yiv3958718385" style="">On 5/2/14, 11:14 PM, <a rel="nofollow" shape="rect" class="yiv3958718385" ymailto="mailto:kanaev@ibrae.ac.ru" target="_blank" href="mailto:kanaev@ibrae.ac.ru" style="">
kanaev@ibrae.ac.ru</a> wrote:<br clear="none" class="yiv3958718385" style="">
</div>
<blockquote class="yiv3958718385" style="">
<div class="yiv3958718385" style="">Hello Scott </div>
<div class="yiv3958718385" style="">The dir is cd /gpfs/mira-fs0/projects/ThermHydraX/kanaev/pinit </div>
<div class="yiv3958718385" style="">The run produced core files is 253035 </div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style="">I took another run with the line MPI_Allreduce(&r,&rmax,1,MPI_REAL8,MPI_MAX,MPI_COMM_WORLD); commented and it stopped the same very time iteration #524378, just passed some more lines
</div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style="">I use MOAB library and its function of exchanging data between processors so i think i cannot really count MPI requests
</div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style="">Anton </div>
<div class="yiv3958718385" style=""><br clear="none" class="yiv3958718385" style="">
On Mon, 28 Apr 2014 16:45:41 -0500, Scott Parker wrote: </div>
<blockquote class="yiv3958718385" style="padding-left:5px;border-left-color:#1010ff;border-left-width:2px;border-left-style:solid;margin-left:5px;width:100%;">
<br clear="none" class="yiv3958718385" style="">
Can you point me to the directory where your binary and core files are?<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
The stack trace you sent shows a call to MPI_Waitany, do you know how many MPI requests<br clear="none" class="yiv3958718385" style="">
the code generally has outstanding at any time?<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
-Scott<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
<div class="yiv3958718385" style="">On 4/28/14, 4:30 PM, <a rel="nofollow" shape="rect" class="yiv3958718385" ymailto="mailto:kanaev@ibrae.ac.ru" target="_blank" href="mailto:kanaev@ibrae.ac.ru" style="">
kanaev@ibrae.ac.ru</a> wrote:<br clear="none" class="yiv3958718385" style="">
</div>
<blockquote class="yiv3958718385" style="">
<div class="yiv3958718385" style="">Hello Scott, </div>
<div class="yiv3958718385" style="">I took rerun with the mentioned keys. The code was freshly compiled with makefile attached just in case.
</div>
<div class="yiv3958718385" style="">I've got 1024 core files. Two of them are attached. </div>
<div class="yiv3958718385" style="">I run <span class="yiv3958718385" style="font-family: 'Lucida Grande', Verdana, Arial, Helvetica, sans-serif;">bgq_stack for core.0 and here's what i got:</span>
</div>
<div class="yiv3958718385" style=""><span class="yiv3958718385" style="font-family: 'Lucida Grande', Verdana, Arial, Helvetica, sans-serif;"></span> [akanaev@miralac1 pinit]$bgq_stack pinit core.0
</div>
<div class="yiv3958718385" style="">------------------------------------------------------------------------ </div>
<div class="yiv3958718385" style="">Program : /gpfs/mira-fs0/projects/ThermHydraX/kanaev/pinit/pinit </div>
<div class="yiv3958718385" style="">------------------------------------------------------------------------ </div>
<div class="yiv3958718385" style="">+++ID Rank: 0, TGID: 1, Core: 0, HWTID:0 TID: 1 State: RUN </div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style="">00000000018334c0 </div>
<div class="yiv3958718385" style="">_ZN4PAMI6Device2MU7Factory12advance_implEPNS1_7ContextEmm </div>
<div class="yiv3958718385" style="">/bgsys/source/srcV1R2M1.17463/comm/sys/buildtools/pami/components/devices/bgq/mu2/Factory.h:269
</div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style="">000000000170da28 </div>
<div class="yiv3958718385" style="">PAMI_Context_trylock_advancev </div>
<div class="yiv3958718385" style="">/bgsys/source/srcV1R2M1.17463/comm/sys/buildtools/pami/api/c/pami.cc:554 </div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style="">000000000155d0dc </div>
<div class="yiv3958718385" style="">PMPI_Waitany </div>
<div class="yiv3958718385" style="">/bgsys/source/srcV1R2M1.17463/comm/lib/dev/mpich2/src/mpid/pamid/include/../src/mpid_progress.h:239
</div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style="">00000000010e84e4 </div>
<div class="yiv3958718385" style="">00000072.long_branch_r2off.H5Dget_space+0 </div>
<div class="yiv3958718385" style="">:0 </div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style="">0000000001042be0 </div>
<div class="yiv3958718385" style="">00000012.long_branch_r2off._ZNSt14basic_ifstreamIcSt11char_traitsIcEEC1EPKcSt13_Ios_Openmode+0
</div>
<div class="yiv3958718385" style="">:0 </div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style="">00000000019de058 </div>
<div class="yiv3958718385" style="">generic_start_main </div>
<div class="yiv3958718385" style="">/bgsys/drivers/V1R2M1/ppc64/toolchain/gnu/glibc-2.12.2/csu/../csu/libc-start.c:226
</div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style="">00000000019de354 </div>
<div class="yiv3958718385" style="">__libc_start_main </div>
<div class="yiv3958718385" style="">/bgsys/drivers/V1R2M1/ppc64/toolchain/gnu/glibc-2.12.2/csu/../sysdeps/unix/sysv/linux/powerpc/libc-start.c:194
</div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style="">0000000000000000 </div>
<div class="yiv3958718385" style="">?? </div>
<div class="yiv3958718385" style="">??:0 </div>
<div class="yiv3958718385" style=""><br clear="none" class="yiv3958718385" style="">
</div>
<div class="yiv3958718385" style="">> Have these sort of runs succeeded in the past using the same code base with no changes and similar input data?
</div>
<div class="yiv3958718385" style="">That is the first time i'm trying to run this code for that long time </div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style="">Thanks </div>
<div class="yiv3958718385" style="">Anton </div>
<div class="yiv3958718385" style=""><br clear="none" class="yiv3958718385" style="">
On Thu, 24 Apr 2014 15:45:49 -0500, Scott Parker wrote: </div>
<blockquote class="yiv3958718385" style="padding-left:5px;border-left-color:#1010ff;border-left-width:2px;border-left-style:solid;margin-left:5px;width:1109px;">
<br clear="none" class="yiv3958718385" style="">
Anton-<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
Thanks, it's aborting because of a runtime error that appears to be in the mpich layer. <br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
Can you rerun with "--env BG_COREDUMPONEXIT=1:BG_COREDUMPONERROR=1" added to your qsub line - that should<br clear="none" class="yiv3958718385" style="">
generate some core files on which you can run bgq_stack.<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
The system software (driver) on Mira was updated this week and I'd like to get a clearer picture of <br clear="none" class="yiv3958718385" style="">
whether that could be related to you problem, so<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
Has your code been recompiled since Monday? If not can you recompile and try running again<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
Have these sort of runs succeeded in the past using the same code base with no changes and similar input data?<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
-Scott<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
<div class="yiv3958718385" style="">On 4/24/14, 2:59 PM, <a rel="nofollow" shape="rect" class="yiv3958718385" ymailto="mailto:kanaev@ibrae.ac.ru" target="_blank" href="mailto:kanaev@ibrae.ac.ru" style="">kanaev@ibrae.ac.ru</a> wrote:<br clear="none" class="yiv3958718385" style="">
</div>
<blockquote class="yiv3958718385" style="">
<div class="yiv3958718385" style="">Sorry about the attached files, here the are </div>
<div class="yiv3958718385" style="">There's no core files after exiting, looks like stopping because of time requested expires but you can see from cobaltlog it's about 5 hours passed (10 hours was requested) before exit
</div>
<div class="yiv3958718385" style="">Anton </div>
<div class="yiv3958718385" style="">On Thu, 24 Apr 2014 14:07:07 -0500, Scott Parker wrote: </div>
<blockquote class="yiv3958718385" style="padding-left:5px;border-left-color:#1010ff;border-left-width:2px;border-left-style:solid;margin-left:5px;width:1029px;">
<br clear="none" class="yiv3958718385" style="">
Anton-<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
Please send these emails to <a rel="nofollow" shape="rect" class="yiv3958718385" ymailto="mailto:support@alcf.anl.gov" target="_blank" href="mailto:support@alcf.anl.gov" style="">support@alcf.anl.gov</a> as I may not always be available to investigate.<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
I don't see the cobalt or error files attached, so I can't really say anything about why your job may be<br clear="none" class="yiv3958718385" style="">
failing. Do you get core files when the job crashed? If so I'd recommend using 'bgq_stack '<br clear="none" class="yiv3958718385" style="">
to try and get the file and line number where the failure occurred. Knowing the line may be enough<br clear="none" class="yiv3958718385" style="">
to let you figure it out, if not you'll need to dump the values of the variables at the time of the crash<br clear="none" class="yiv3958718385" style="">
to get a clearer picture of what is going on.<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
Scott<br clear="none" class="yiv3958718385" style="">
<br clear="none" class="yiv3958718385" style="">
<div class="yiv3958718385" style="">On 4/24/14, 1:36 PM, <a rel="nofollow" shape="rect" class="yiv3958718385" ymailto="mailto:kanaev@ibrae.ac.ru" target="_blank" href="mailto:kanaev@ibrae.ac.ru" style="">kanaev@ibrae.ac.ru</a> wrote:<br clear="none" class="yiv3958718385" style="">
</div>
<blockquote class="yiv3958718385" style="">
<div class="yiv3958718385" style="">Hello Scott, </div>
<div class="yiv3958718385" style="">I've tried twice to run 10 hours 1024 cores job on Mira in mode c16 with </div>
<div class="yiv3958718385" style="">qsub -n 64 -t 10:00:00 --mode c16 -A ThermHydraX pinit Lid_128x128x1p1024.h5m </div>
<div class="yiv3958718385" style="">Both times job exited earlier than expected on the same iteration after the same error during executing the following section (it's between two couts):
</div>
<div class="yiv3958718385" style=""> ... </div>
<div class="yiv3958718385" style="">//LOOP OVER OWNED CELLS </div>
<div class="yiv3958718385" style=""> double r = 0; </div>
<div class="yiv3958718385" style=""> for (moab::Range::iterator it = owned_ents.begin(); it != owned_ents.end(); it++) {
</div>
<div class="yiv3958718385" style=""> EntityHandle ent = *it; </div>
<div class="yiv3958718385" style=""> int cellid = mb->id_from_handle(ent); </div>
<div class="yiv3958718385" style=""> double Vol_; </div>
<div class="yiv3958718385" style=""> double u_; </div>
<div class="yiv3958718385" style=""> double v_; </div>
<div class="yiv3958718385" style=""> double w_; </div>
<div class="yiv3958718385" style=""> double r1,r2,r3; </div>
<div class="yiv3958718385" style=""> double tmp; </div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style=""> result = mb->tag_get_data(u, &ent, 1, &u_); </div>
<div class="yiv3958718385" style=""> PRINT_LAST_ERROR; </div>
<div class="yiv3958718385" style=""> result = mb->tag_get_data(v, &ent, 1, &v_); </div>
<div class="yiv3958718385" style=""> PRINT_LAST_ERROR; </div>
<div class="yiv3958718385" style=""> result = mb->tag_get_data(w, &ent, 1, &w_); </div>
<div class="yiv3958718385" style=""> PRINT_LAST_ERROR; </div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style=""> double result; </div>
<div class="yiv3958718385" style=""> SCALAR_PRODUCT(result,u_,v_,w_,CG[cellid][2].lx,CG[cellid][2].ly,CG[cellid][2].lz);
</div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style=""> r1 = (sound + fabs(result))/CG[cellid][2].length; </div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style=""> SCALAR_PRODUCT(result,u_,v_,w_,CG[cellid][3].lx,CG[cellid][3].ly,CG[cellid][3].lz);
</div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style=""> r2 = (sound + fabs(result))/CG[cellid][3].length; </div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style=""> SCALAR_PRODUCT(result,u_,v_,w_,CG[cellid][5].lx,CG[cellid][5].ly,CG[cellid][5].lz);
</div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style=""> r3 = (sound + fabs(result))/CG[cellid][5].length; </div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style=""> tmp = MAX3(r1,r2,r3); </div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style=""> r = MAX(tmp,r); </div>
<div class="yiv3958718385" style=""> } </div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style=""> double rmax; </div>
<div class="yiv3958718385" style=""> MPI_Allreduce(&r,&rmax,1,MPI_REAL8,MPI_MAX,MPI_COMM_WORLD); </div>
<div class="yiv3958718385" style=""> tau = CFL/rmax; </div>
<div class="yiv3958718385" style=""> ttime+=tau; </div>
<div class="yiv3958718385" style="">... </div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style="">So it may be Allreduce </div>
<div class="yiv3958718385" style="">I've attached cobaltlog and error files of both runs </div>
<div class="yiv3958718385" style="">Can you please take a look and suggest a further debugging </div>
<div class="yiv3958718385" style=""> </div>
<div class="yiv3958718385" style="">Thanks </div>
<div class="yiv3958718385" style="">Anton </div>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<br clear="none" class="yiv3958718385" style="">
</blockquote>
<div class="yiv3958718385" style=""> </div>
</blockquote>
<br clear="none" class="yiv3958718385" style="">
</blockquote>
<div class="yiv3958718385" style=""> </div>
</blockquote>
<div class="yiv3958718385" style=""> </div>
</div>
</div></div>
</div>
</div></div><br clear="none" class="yiv3958718385" style=""><br clear="none" class="yiv3958718385" style=""></div> </div> </div> </div></div> </div></div></div></div></body></html>