<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>Hi Maxime,</div>
<div><br>
</div>
<div>I ran the HPL traces with no MPI data type on the simulation and here are some observations. I disabled any synchronizations (wait, wait-alls) in the simulation so that it only matches the MPI sends with the receives and does nothing else. </div>
<div><br>
</div>
<div>- Rank 0 expects 192 messages from Rank 1 but it instead receives 192 messages from Rank 2.</div>
<div>- Rank 1 receives 192 messages from rank 0 but there are no corresponding receives posted so the messages remain unmatched. -</div>
<div>- Rank 2 is expecting 192 messages from Rank 0 but they don’t arrive (probably because they arrived at Rank 1).</div>
<div><br>
</div>
<div>Is it possible that having no MPI data type resulted in missing messages that introduced these discrepancies? Or maybe the application is terminating earlier than usual? </div>
<div><br>
</div>
<div>I will try the version with MPI data types and let you know if the results are different. </div>
<div><br>
</div>
<div>Thanks,</div>
<div>Misbah</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span><<a href="mailto:codes-ross-users-bounces@lists.mcs.anl.gov">codes-ross-users-bounces@lists.mcs.anl.gov</a>> on behalf of Maxime Chevalier <<a href="mailto:maxime.chevalier@inria.fr">maxime.chevalier@inria.fr</a>><br>
<span style="font-weight:bold">Date: </span>Sunday, June 4, 2017 at 12:20 PM<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:codes-ross-users@lists.mcs.anl.gov">codes-ross-users@lists.mcs.anl.gov</a>" <<a href="mailto:codes-ross-users@lists.mcs.anl.gov">codes-ross-users@lists.mcs.anl.gov</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [codes-ross-users] Replay HPL's dumpi trace on CODES<br>
</div>
<div><br>
</div>
<div>
<div>
<div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000">
<div><span style="background-color: rgb(253, 253, 253); font-size: 12pt;"><br>
</span></div>
<div><span style="background-color: rgb(253, 253, 253); font-size: 12pt;">Hi Misbah,</span></div>
<div>
<div style="background-color: #fdfdfd;" data-mce-style="background-color: #fdfdfd;">
Thanks for your help, you can find dumpi traces with "UNDEFINED DATA TYPE" and without via the link below. Codes-workload-dump utility is very usefull, thanks for that (I was using dumpistat).</div>
<div style="background-color: #fdfdfd;" data-mce-style="background-color: #fdfdfd;">
<br>
</div>
<div style="background-color: #fdfdfd;" data-mce-style="background-color: #fdfdfd;">
<a href="https://1drv.ms/f/s!Ati25f8zqy9lnNFi7EX8u1tmdJ4rfw">https://1drv.ms/f/s!Ati25f8zqy9lnNFi7EX8u1tmdJ4rfw</a></div>
<div style="background-color: #fdfdfd;" data-mce-style="background-color: #fdfdfd;">
<br>
</div>
<div style="background-color: #fdfdfd;" data-mce-style="background-color: #fdfdfd;">
Regards,</div>
<div style="background-color: #fdfdfd;" data-mce-style="background-color: #fdfdfd;">
Maxime</div>
</div>
<hr id="zwchr">
<blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;">
<b>De: </b>"Misbah Mubarak" <<a href="mailto:mmubarak@anl.gov">mmubarak@anl.gov</a>><br>
<b>À: </b>"Maxime Chevalier" <<a href="mailto:maxime.chevalier@inria.fr">maxime.chevalier@inria.fr</a>>,
<a href="mailto:codes-ross-users@lists.mcs.anl.gov">codes-ross-users@lists.mcs.anl.gov</a><br>
<b>Envoyé: </b>Vendredi 2 Juin 2017 18:54:13<br>
<b>Objet: </b>Re: [codes-ross-users] Replay HPL's dumpi trace on CODES<br>
<div><br>
</div>
<div>Hi Maxime,</div>
<div><br>
</div>
<div>There is a codes-workload-dump utility that helps you inspect the traces and provides detailed information on the individual MPI operations such as number of bytes transmitted (which is derived by the data type and count). If you could run the utility
with one of the traces and send me the output, I can have a look at whats going on. Alternatively, if you could share the traces, I can have a look at those.</div>
<div><br>
</div>
<div>Using the utility is simple, here is some documentation on how to run it:</div>
<div><br>
</div>
<div><a href="https://xgitlab.cels.anl.gov/codes/codes/wikis/codes-dumpi-workload" target="_blank">https://xgitlab.cels.anl.gov/codes/codes/wikis/codes-dumpi-workload</a></div>
<div><br>
</div>
<div>Thanks,</div>
<div>Misbah</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span><<a href="mailto:codes-ross-users-bounces@lists.mcs.anl.gov" target="_blank">codes-ross-users-bounces@lists.mcs.anl.gov</a>> on behalf of Maxime Chevalier <<a href="mailto:maxime.chevalier@inria.fr" target="_blank">maxime.chevalier@inria.fr</a>><br>
<span style="font-weight:bold">Date: </span>Friday, June 2, 2017 at 8:52 AM<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:codes-ross-users@lists.mcs.anl.gov" target="_blank">codes-ross-users@lists.mcs.anl.gov</a>" <<a href="mailto:codes-ross-users@lists.mcs.anl.gov" target="_blank">codes-ross-users@lists.mcs.anl.gov</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [codes-ross-users] Replay HPL's dumpi trace on CODES<br>
</div>
<div><br>
</div>
<div>
<div>
<div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000">
<div>
<div>Hi Misbah,</div>
<div>Thanks for your fast response. I was looking for the data type, but I don't really understand. I have figured out how to avoid "UNDEFINED DATA TYPE" errors by compiling HPL whit "HPL_NO_MPI_DATATYPE", but the output is quite the same (see trace below).
I don't know if it's a step forward or backward...</div>
</div>
<div></div>
<div><span id="spans0e0" class="sac" style="margin: 0px; padding: 0px; border: 0px; outline: 0px; vertical-align: baseline; cursor: pointer; color: green; font-family: Arial, serif; font-size: 14px;"><br>
</span></div>
<div>Regards,</div>
<div>Maxime</div>
<div><br>
</div>
<div>Trace :</div>
<div>
<p style="margin: 0px;">Fri Jun 2 09:15:49 2017</p>
<p style="margin: 0px;">ROSS Revision: 4c6a7d8eb9c784797d900edfc76725d62ec25941</p>
<p style="margin: 0px;">tw_net_start: Found world size to be 1</p>
<p style="margin: 0px;">ROSS Core Configuration: <br>
Total Nodes 1<br>
Total Processors [Nodes (1) x PE_per_Node (1)] 1<br>
Total KPs [Nodes (1) x KPs (16)] 16<br>
Total LPs 54<br>
Simulation End Time 300000000000.00<br>
LP-to-PE Mapping model defined</p>
<p style="margin: 0px;"><br>
ROSS Event Memory Allocation:<br>
Model events 13825<br>
Network events 50000<br>
Total events 63824</p>
<p style="margin: 0px;">*** START SEQUENTIAL SIMULATION ***</p>
<p style="margin: 0px;">*** END SIMULATION ***</p>
<p style="margin: 0px;"><br>
LP 1 unmatched irecvs 1 unmatched sends 0 Total sends 0 receives 2 collectives 0 delays 8 wait alls 0 waits 0 send time 0.000000 wait 0.000000<br>
LP 3 unmatched irecvs 1 unmatched sends 0 Total sends 1 receives 1 collectives 0 delays 10 wait alls 0 waits 0 send time 3.202149 wait 0.000000<br>
LP 5 unmatched irecvs 0 unmatched sends 0 Total sends 0 receives 1 collectives 0 delays 7 wait alls 0 waits 0 send time 0.000000 wait 0.000000<br>
LP 7 unmatched irecvs 1 unmatched sends 0 Total sends 1 receives 1 collectives 0 delays 10 wait alls 0 waits 0 send time 3.189207 wait 0.000000<br>
: Running Time = 0.0001 seconds</p>
<p style="margin: 0px;">TW Library Statistics:<br>
Total Events Processed 56<br>
Events Aborted (part of RBs) 0<br>
Events Rolled Back 0<br>
Event Ties Detected in PE Queues 0<br>
Efficiency 100.00 %<br>
Total Remote (shared mem) Events Processed 0<br>
Percent Remote Events 0.00 %<br>
Total Remote (network) Events Processed 0<br>
Percent Remote Events 0.00 %</p>
<p style="margin: 0px;">Total Roll Backs 0<br>
Primary Roll Backs 0<br>
Secondary Roll Backs 0<br>
Fossil Collect Attempts 0<br>
Total GVT Computations 0</p>
<p style="margin: 0px;">Net Events Processed 56<br>
Event Rate (events/sec) 823529.4<br>
Total Events Scheduled Past End Time 0</p>
<p style="margin: 0px;">TW Memory Statistics:<br>
Events Allocated 63825<br>
Memory Allocated 62573<br>
Memory Wasted 683</p>
<p style="margin: 0px;">TW Data Structure sizes in bytes (sizeof):<br>
PE struct 608<br>
KP struct 144<br>
LP struct 128<br>
LP Model struct 760<br>
LP RNGs 80<br>
Total LP 968<br>
Event struct 144<br>
Event struct with Model 928</p>
<p style="margin: 0px;">TW Clock Cycle Statistics (MAX values in secs at 1.0000 GHz):<br>
Priority Queue (enq/deq) 0.0000<br>
AVL Tree (insert/delete) 0.0000<br>
LZ4 (de)compression 0.0000<br>
Buddy system 0.0000<br>
Event Processing 0.0000<br>
Event Cancel 0.0000<br>
Event Abort 0.0000</p>
<p style="margin: 0px;">GVT 0.0000<br>
Fossil Collect 0.0000<br>
Primary Rollbacks 0.0000<br>
Network Read 0.0000<br>
Statistics Computation 0.0000<br>
Statistics Write 0.0000<br>
Total Time (Note: Using Running Time above for Speedup) 0.0002</p>
<p style="margin: 0px;">TW GVT Statistics: MPI AllReduce<br>
GVT Interval 16<br>
GVT Real Time Interval (cycles) 0<br>
GVT Real Time Interval (sec) 0.00000000<br>
Batch Size 16</p>
<p style="margin: 0px;">Forced GVT 0<br>
Total GVT Computations 0<br>
Total All Reduce Calls 0<br>
Average Reduction / GVT -nan</p>
<p style="margin: 0px;">Total bytes sent 8 recvd 20 <br>
max runtime 0.000000 ns avg runtime 0.000000 <br>
max comm time 0.000000 avg comm time -69573.000000 <br>
max send time 3.202149 avg send time 1.597839 <br>
max recv time 45682.609151 avg recv time 11420.652288 <br>
max wait time 0.000000 avg wait time 0.000000</p>
</div>
<hr id="zwchr">
<blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;">
<b>De: </b>"Misbah Mubarak" <<a href="mailto:mmubarak@anl.gov" target="_blank">mmubarak@anl.gov</a>><br>
<b>À: </b>"Maxime Chevalier" <<a href="mailto:maxime.chevalier@inria.fr" target="_blank">maxime.chevalier@inria.fr</a>>,
<a href="mailto:codes-ross-users@lists.mcs.anl.gov" target="_blank">codes-ross-users@lists.mcs.anl.gov</a><br>
<b>Envoyé: </b>Mardi 30 Mai 2017 18:12:46<br>
<b>Objet: </b>Re: [codes-ross-users] Replay HPL's dumpi trace on CODES<br>
<div><br>
</div>
<div>Hi Maxime,</div>
<div><br>
</div>
<div>Thanks for your message. There seems to be a data type that is either not supported by DUMPI or CODES. Are you familiar with what data types are being used by the HPL trace? I will find out if the support for them can be added in the code. </div>
<div><br>
</div>
<div>Regards,</div>
<div>Misbah</div>
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span><<a href="mailto:codes-ross-users-bounces@lists.mcs.anl.gov" target="_blank">codes-ross-users-bounces@lists.mcs.anl.gov</a>> on behalf of Maxime Chevalier <<a href="mailto:maxime.chevalier@inria.fr" target="_blank">maxime.chevalier@inria.fr</a>><br>
<span style="font-weight:bold">Date: </span>Monday, May 29, 2017 at 3:51 AM<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:codes-ross-users@lists.mcs.anl.gov" target="_blank">codes-ross-users@lists.mcs.anl.gov</a>" <<a href="mailto:codes-ross-users@lists.mcs.anl.gov" target="_blank">codes-ross-users@lists.mcs.anl.gov</a>><br>
<span style="font-weight:bold">Subject: </span>[codes-ross-users] Replay HPL's dumpi trace on CODES<br>
</div>
<div><br>
</div>
<div>
<div>
<div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000">
<div>Hi,</div>
<div>I'm trying to replay HPL's DUMPI trace generated on my computer with CODES. Unfortunately, I get a lot of "Undefined data type" errors (see the trace below).</div>
<div>I have already replayed AMG traces (downloaded <a href="http://portal.nersc.gov/project/CAL/designforward.htm" title="AMG traces" target="_blank">
here</a>) and replayed my own generated AMG traces. It has worked fine.</div>
<div>So I'm wondering if I did something wrong, or if it's HPL fault.</div>
<div><br>
</div>
<div>Best regards,</div>
<div>Maxime</div>
<div><br>
</div>
<div><br>
</div>
<div>Trace :</div>
<blockquote>
<div><br>
</div>
</blockquote>
<div>
<blockquote>
<p style="margin: 0px;">ROSS Revision: 4c6a7d8eb9c784797d900edfc76725d62ec25941</p>
<p style="margin: 0px;">tw_net_start: Found world size to be 1</p>
<p style="margin: 0px;">ROSS Core Configuration: <br>
Total Nodes 1<br>
Total Processors [Nodes (1) x PE_per_Node (1)] 1<br>
Total KPs [Nodes (1) x KPs (16)] 16<br>
Total LPs 5<br>
Simulation End Time 300000000000.00<br>
LP-to-PE Mapping model defined</p>
<p style="margin: 0px;"><br>
ROSS Event Memory Allocation:<br>
Model events 1281<br>
Network events 50000<br>
Total events 51280</p>
<p style="margin: 0px;"><br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type <br>
Undefined data type *** START SEQUENTIAL SIMULATION ***</p>
<p style="margin: 0px;">*** END SIMULATION ***</p>
<p style="margin: 0px;"><br>
LP 1 unmatched irecvs 1 unmatched sends 0 Total sends 0 receives 1 collectives 0 delays 7 wait alls 0 waits 0 send time 0.000000 wait 0.000000<br>
: Running Time = 0.0000 seconds</p>
<p style="margin: 0px;">TW Library Statistics:<br>
Total Events Processed 8<br>
Events Aborted (part of RBs) 0<br>
Events Rolled Back 0<br>
Event Ties Detected in PE Queues 0<br>
Efficiency 100.00 %<br>
Total Remote (shared mem) Events Processed 0<br>
Percent Remote Events 0.00 %<br>
Total Remote (network) Events Processed 0<br>
Percent Remote Events 0.00 %</p>
<p style="margin: 0px;">Total Roll Backs 0<br>
Primary Roll Backs 0<br>
Secondary Roll Backs 0<br>
Fossil Collect Attempts 0<br>
Total GVT Computations 0</p>
<p style="margin: 0px;">Net Events Processed 8<br>
Event Rate (events/sec) 307692.3<br>
Total Events Scheduled Past End Time 0</p>
<p style="margin: 0px;">TW Memory Statistics:<br>
Events Allocated 51281<br>
Memory Allocated 51168<br>
Memory Wasted 720</p>
<p style="margin: 0px;">TW Data Structure sizes in bytes (sizeof):<br>
PE struct 608<br>
KP struct 144<br>
LP struct 128<br>
LP Model struct 760<br>
LP RNGs 80<br>
Total LP 968<br>
Event struct 144<br>
Event struct with Model 928</p>
<p style="margin: 0px;">TW Clock Cycle Statistics (MAX values in secs at 1.0000 GHz):<br>
Priority Queue (enq/deq) 0.0000<br>
AVL Tree (insert/delete) 0.0000<br>
LZ4 (de)compression 0.0000<br>
Buddy system 0.0000<br>
Event Processing 0.0000<br>
Event Cancel 0.0000<br>
Event Abort 0.0000</p>
<p style="margin: 0px;">GVT 0.0000<br>
Fossil Collect 0.0000<br>
Primary Rollbacks 0.0000<br>
Network Read 0.0000<br>
Statistics Computation 0.0000<br>
Statistics Write 0.0000<br>
Total Time (Note: Using Running Time above for Speedup) 0.0001</p>
<p style="margin: 0px;">TW GVT Statistics: MPI AllReduce<br>
GVT Interval 16<br>
GVT Real Time Interval (cycles) 0<br>
GVT Real Time Interval (sec) 0.00000000<br>
Batch Size 16</p>
<p style="margin: 0px;">Forced GVT 0<br>
Total GVT Computations 0<br>
Total All Reduce Calls 0<br>
Average Reduction / GVT -nan</p>
<p style="margin: 0px;">Total bytes sent 0 recvd 4 <br>
max runtime 0.000000 ns avg runtime 0.000000 <br>
max comm time 0.000000 avg comm time -66232.000000 <br>
max send time 0.000000 avg send time 0.000000 <br>
max recv time 0.000000 avg recv time 0.000000 <br>
max wait time 0.000000 avg wait time 0.000000 <br>
LP-IO: writing output to hpl-trace-25282-1495543803/<br>
LP-IO: data files:<br>
hpl-trace-25282-1495543803/mpi-replay-stats<br>
hpl-trace-25282-1495543803/model-net-category-all</p>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
</div>
</div>
</div>
</span></blockquote>
<div><br>
</div>
</div>
</div>
</div>
</span>
</body>
</html>