<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Can you send me your darshan_config.conf file? Also what is the bb_dragonfly_sim executable? I’m assuming a model you wrote that uses the dragonfly network model. If so, how can I access that code?<div class=""><br class=""></div><div class="">Do you know if your model is deterministic at all (or close to it)? For some smaller run (like maybe decreasing the amount of time you’re simulating and/or use a smaller network), run it with sync=1 and with sync=3. Does the number of Net Events given at the end of the simulation match? We know that in CODES there are some issues with the models being non-deterministic in the net events. If a parallel run of your model is pretty far off from the sequential run on the number of net events, there’s probably a reverse computation issue that is some how causing these errors you get in ROSS.</div><div class=""><br class=""></div><div class="">Caitlin<br class=""><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Aug 9, 2018, at 5:39 PM, Jian Peng <<a href="mailto:jpeng10@hawk.iit.edu" class="">jpeng10@hawk.iit.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class=""><div class="">Tried the newer version, the error still exists. The MPI I'm using is [MPICH2 3.2] The command I'm using is:<br class=""><br class="">mpirun -f ./hosts -n 33 ./bb_dragonfly_sim --extramem=100000 --nkp=128 --sync=3 --batch=1 --gvt-interval=32 -- /home/cc/share/sim_configs/darshan_config.conf<br class=""><br class=""></div>Also, another long-existing error sometimes pops up, which I think might be related to the gvt:<br class=""><br class="">node: 27: error: /home/cc/Project/NERSC/ROSS-master/core/gvt/mpi_allreduce.c:180: PE 27 GVT decreased 1.42151 -> 1.36322<br class="">node: 5: error: /home/cc/Project/NERSC/ROSS-master/core/gvt/mpi_allreduce.c:180: PE 5 GVT decreased 1.42151 -> 1.36322<br class="">node: 29: error: /home/cc/Project/NERSC/ROSS-master/core/gvt/mpi_allreduce.c:180: PE 29 GVT decreased 1.42151 -> 1.36322<br class="">node: 28: error: /home/cc/Project/NERSC/ROSS-master/core/gvt/mpi_allreduce.c:180: PE 28 GVT decreased 1.42151 -> 1.36322<br class="">node: 13: error: /home/cc/Project/NERSC/ROSS-master/core/gvt/mpi_allreduce.c:180: PE 13 GVT decreased 1.42151 -> 1.36322<br class="">.......<br class=""><br class=""></div>My current walk-around solution to the later issue is changing the "nkp" parameter.<br class=""></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Thu, Aug 9, 2018 at 12:49 PM, Caitlin Ross <span dir="ltr" class=""><<a href="mailto:rossc3@rpi.edu" target="_blank" class="">rossc3@rpi.edu</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class="">It’s saying that a PE received an event that has a time stamp less than the current GVT, which shouldn’t be possible. <div class=""><br class=""></div><div class="">But your line number for the error in network-mpi.c is off from what it is currently in the master branch of ROSS. There’s been some changes in the MPI layer of ROSS relatively recently (in May/June), so my first recommendation is to update your version of ROSS and see if you still get the error. If you do still get the error, could you also send some more details on the simulation run that causes this error?</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Caitlin</div><div class=""><br class=""><div class=""><blockquote type="cite" class=""><div class=""><div class="h5"><div class="">On Aug 9, 2018, at 2:20 PM, Jian Peng <<a href="mailto:jpeng10@hawk.iit.edu" target="_blank" class="">jpeng10@hawk.iit.edu</a>> wrote:</div><br class="m_-7741574670078658731Apple-interchange-newline"></div></div><div class=""><div class=""><div class="h5"><div dir="ltr" class=""><div class="">Hi All, <br class=""></div><div class=""><br class=""></div><div class="">Just ran into an issue with error:<br class=""><br class=""> "error: network-mpi.c:388: 1:Received straggler from 7: 2938789.459012 3193751.109728 (0)" . Which is cause by failure of <br class=""><div style="color:rgb(212,212,212);background-color:rgb(30,30,30);font-family:Menlo,Monaco,"Courier New",monospace;font-weight:normal;font-size:12px;line-height:18px;white-space:pre-wrap" class=""><div class=""><span style="color:rgb(197,134,192)" class="">if</span><span style="color:rgb(212,212,212)" class="">(e-></span><span style="color:rgb(156,220,254)" class="">recv_ts</span><span style="color:rgb(212,212,212)" class=""> </span><span style="color:rgb(212,212,212)" class=""><</span><span style="color:rgb(212,212,212)" class=""> me-></span><span style="color:rgb(156,220,254)" class="">GVT</span><span style="color:rgb(212,212,212)" class="">)</span></div></div>in<br class=""><div style="color:rgb(212,212,212);background-color:rgb(30,30,30);font-family:Menlo,Monaco,"Courier New",monospace;font-weight:normal;font-size:12px;line-height:18px;white-space:pre-wrap" class=""><div class=""><span style="color:rgb(220,220,170)" class="">recv_finish</span><span style="color:rgb(212,212,212)" class="">(tw_pe </span><span style="color:rgb(212,212,212)" class="">*</span><span style="color:rgb(212,212,212)" class="">me, tw_event </span><span style="color:rgb(212,212,212)" class="">*</span><span style="color:rgb(212,212,212)" class="">e, </span><span style="color:rgb(86,156,214)" class="">char</span><span style="color:rgb(212,212,212)" class=""> </span><span style="color:rgb(212,212,212)" class="">*</span><span style="color:rgb(212,212,212)" class=""> buffer)</span></div></div><br class=""></div><div class="">Any suggestion of fixing this issue? Thanks!<br class=""><br class=""></div><div class=""><br class=""></div></div></div></div><span class="">
______________________________<wbr class="">_________________<br class="">codes-ross-users mailing list<br class=""><a href="mailto:codes-ross-users@lists.mcs.anl.gov" target="_blank" class="">codes-ross-users@lists.mcs.<wbr class="">anl.gov</a><br class=""><a href="https://lists.mcs.anl.gov/mailman/listinfo/codes-ross-users" target="_blank" class="">https://lists.mcs.anl.gov/<wbr class="">mailman/listinfo/codes-ross-<wbr class="">users</a><br class=""></span></div></blockquote></div><br class=""></div></div></blockquote></div><br class=""></div>
</div></blockquote></div><br class=""></div></div></body></html>