<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Jayesh,<div><br></div><div>First of all, thank you for the quick response. To answer your questions, this is a new cluster and it is still going through its paces before being put online. </div><div>I did run the cpi.exe and another MPI program from one of my colleagues across all the nodes without any incident.</div><div><br></div><div>When I came into work today, I restarted all of the nodes, ensured that their firewalls were down for MPICH2 (smpd.exe and mpiexec.exe)</div><div>and tried executing the MCNPX program again. As if by magic, that did the trick and the program executed without error.</div><div><br></div><div>However a new issue has cropped up. Running on a comparable single core, the MCNPX simulation in question takes 632 cpu minutes to complete.</div><div>In the first run through MPI, the simulation took 2875 cpu minutes over 12 cores and 16 processes (hyperthreading on four of the cores). </div><div>After disabling the hyperthreading on all of the cores, I was able to bring the total cpu time down to 1420 minutes.</div><div>I'm trying to understand why this is occurring -- why would running the MPI version take over double the computational time?</div><div><br></div><div>Thanks,</div><div><br></div><div>-- Matt</div><div><br></div><div><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">___<br>Matthew J. Riblett<br>Nuclear Engineering Class '12<br>Rensselaer Polytechnic Institute</div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Rensselaer Radiation Measurement and Dosimetry Group</div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">American Nuclear Society, Section President<br>MANE Department Student Advisory Council<br><br>Email: <a href="mailto:riblem@rpi.edu">riblem@rpi.edu</a><br>Main: +1.646.843.9596<br>Mobile: +1.804.245.0578<br>Web: <a href="http://riblem.rpians.org">http://riblem.rpians.org</a></div></span><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div class=""><br></div></div></span></div></span><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br class="Apple-interchange-newline"></div></span></div></span></div></span></div></span><br class="Apple-interchange-newline"></span><br class="Apple-interchange-newline">
</div>
<br><div><div>On Jun 21, 2012, at 4:29 PM, Jayesh Krishna wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Hi,<br><br># Did you run cpi across all the nodes (the nodes used for running MCNPX)?<br># Have you tried running other MPI programs on the cluster (Is it a new cluster?)?<br><br>Regards,<br>Jayesh<br><br>----- Original Message -----<br>From: "Matthew Riblett" <<a href="mailto:riblem@rpi.edu">riblem@rpi.edu</a>><br>To: <a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a><br>Sent: Wednesday, June 20, 2012 2:18:52 PM<br>Subject: [mpich-discuss] Issue running MCNPX on small cluster: Error sending<span class="Apple-tab-span" style="white-space:pre"> </span>static commons.<br><br><br>Hello, <br><br><br>I am attempting to run MCNPX in an MPI environment on a small cluster of computers (Dell PowerEdge servers running 64-bit Windows Server 2008 Standard). <br>I am using the precompiled 64-bit MPI executables from RSICC. <br>I've had success running the process on each of four test servers when configured to run on only one host and can escalate to run multiple processes on single hosts. <br>When I attempt to run the program across multiple hosts (ex: -hosts 4 Mercury-1 Mercury-2 Mercury-3 Mercury-4) it returns a fatal error: <br><br><br>master starting 3 by 1 subtasks 06/20/12 15:06:29 <br>master sending static commons... <br>Fatal error in MPI_Send: Other MPI error, error stack <br>MPI_Send(173)................: MPI_Send(buf=0000000020E00000, count=236236, MPI_PACKED, dest=1, tag=4 MPI_COMM_WORLD) failed <br>MPIDI_CH3I_Progress(402)........: <br>MPID_nem_mpich2_blocking_recv(905)...: <br>MPID_nem_newtcp_module_poll(37)......: <br>MPID_nem_newtcp_module_connpoll(2656): <br>gen_cnting_fail_handler(1739)........: connect failed - the semaphore timeout period has expired (errno 121) <br><br><br>job aborted: <br>rank: node: exit code[: error message] <br>0: Mercury-1: 1: process 0 exited without calling finalize <br>1: Mercury-2: 123 <br>2: Mercury-3: 123 <br>3: Mercury-4: 123 <br><br><br>I've looked at several of the archived posts that seemed to have similar problems, such as <a href="http://lists.mcs.anl.gov/pipermail/mpich-discuss/2011-August/010696.html">http://lists.mcs.anl.gov/pipermail/mpich-discuss/2011-August/010696.html</a> . <br>In each case they passed the static commons sending point and got to the point where the program was sending dynamic commons. <br><br><br>This is a rather large simulation ~600Mb and I was curious as to whether or not its size may be playing a role in this error. <br>Running the cpi.exe example, the hosts communicate with one another and there is no problem in execution. <br><br><br>I don't think this is a firewall issue as both smpd.exe and mpiexec.exe are granted exceptions in the Windows Firewall. <br><br><br>Thanks in advance, <br><br><br>-- Matt <br><br><br><br><br>___ <br>Matthew J. Riblett <br>Nuclear Engineering Class '12 <br>Rensselaer Polytechnic Institute <br>Rensselaer Radiation Measurement and Dosimetry Group <br>American Nuclear Society, Section President <br>MANE Department Student Advisory Council <br><br>Email: <a href="mailto:riblem@rpi.edu">riblem@rpi.edu</a> <br>Main: +1.646.843.9596 <br>Mobile: +1.804.245.0578 <br>Web: <a href="http://riblem.rpians.org">http://riblem.rpians.org</a> <br><br><br><br><br><br><br><br><br><br><br>_______________________________________________<br>mpich-discuss mailing list <a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a><br>To manage subscription options or unsubscribe:<br><a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a><br><br></div></blockquote></div><br></div></body></html>