<div dir="ltr"><div>Peder</div><div><br></div><div>I have slightly modified your code and I confirm the bug.</div><div>The bug is not with the MatMatTranspose operation; it is within the HDF5 reader. I will soon open an MR with the code and discussing the issues.</div><div><br></div><div>Thanks for reporting the issue</div><div>Stefano<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno mer 21 apr 2021 alle ore 12:22 Peder Jørgensgaard Olesen via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div id="gmail-m_-5517018130510300103divtagdefaultwrapper" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif" dir="ltr">
<p>Dear Hong</p>
<p><br>
</p>
<p>Thank your for your reply.</p>
<p><br>
</p>
<p>I have a hunch that the issue goes beyond the minor differences that might arise from floating-point computation order, however.</p>
<p><br>
</p>
<p>Writing the product matrix to a binary file using MatView() and inspecting the output shows very different entries depending on the number of processes. Here are the first three rows and columns of the product matrix obtained in a sequential run:</p>
<p>2.58348 1.68202 1.66302</p>
<p><span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px">1.68202 4.27506 1.91897</span><br>
</p>
<p><span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px"><span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px">1.66302
1.91897 2.70028</span><br>
</span></p>
<p><br>
</p>
<p>- and the corresponding part of the product matrix obtained on one node (40 processes):</p>
<p>4.43536 2.17261 0.16430</p>
<p>2.17261 4.53224 2.53210</p>
<p><span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px">0.16430 2.53210 4.73234</span><br>
</p>
<p><br>
</p>
<p>The parallel result is not even close to the sequential one. Trying different numbers of processes produces yet different results.</p>
<p><br>
</p>
<p><span style="font-size:12pt">Also, the eigenvectors that I subsequently determine using a </span><span style="font-size:12pt">SLEPC solver do not form </span><span style="font-size:12pt">a proper basis for the column space of the data matrix as they
must, <span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px">
which is hardly a surprise given the variability of results indicated above -</span><span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px"> </span>except
</span><span style="font-size:12pt">when the code is run on just </span><span style="font-size:12pt">a single process. Forming such a basis central to the intended application, and g</span><span style="font-size:12pt">iven that it would need to work on </span><span style="font-size:12pt">rather
large data sets, running on a </span><span style="font-size:12pt">single process is hardly a viable solution.</span></p>
<p><span style="font-size:12pt"><br>
</span></p>
<p><span style="font-size:12pt">Best regards</span></p>
<p><span style="font-size:12pt">Peder</span></p>
</div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-5517018130510300103divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>Fra:</b> Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>><br>
<b>Sendt:</b> 19. april 2021 18:34:31<br>
<b>Til:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>; Peder Jørgensgaard Olesen<br>
<b>Emne:</b> Re: Rather different matrix product results on multiple processes</font>
<div> </div>
</div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;background-color:rgb(255,255,255);display:inline">Peder,</span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;background-color:rgb(255,255,255);display:inline">I tested your code on
a linux machine. I got</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;background-color:rgb(255,255,255);display:inline">$ ./acorr_mwe
<div>Data matrix norm: 5.0538e+01</div>
Autocorrelation matrix norm: 1.0473e+03<br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;background-color:rgb(255,255,255);display:inline"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;background-color:rgb(255,255,255);display:inline">mpiexec -n 40 ./acorr_mwe -matmattransmult_mpidense_mpidense_via
allgatherv (default)
<div>Data matrix norm: 5.0538e+01</div>
Autocorrelation matrix norm: 1.0363e+03<br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;background-color:rgb(255,255,255);display:inline"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;background-color:rgb(255,255,255);display:inline">mpiexec -n 20 ./acorr_mwe
<div>Data matrix norm: 5.0538e+01</div>
Autocorrelation matrix norm: 1.0897e+03<br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;background-color:rgb(255,255,255);display:inline"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;background-color:rgb(255,255,255);display:inline">mpiexec -n 40 ./acorr_mwe
-matmattransmult_mpidense_mpidense_via cyclic
<div>Data matrix norm: 5.0538e+01</div>
Autocorrelation matrix norm: 1.0363e+03</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;background-color:rgb(255,255,255);display:inline"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;background-color:rgb(255,255,255);display:inline">I use petsc 'main' branch
(same as the latest release). You can remove MatAssemblyBegin/End calls after <span style="background-color:rgb(255,255,255);display:inline">MatMatTransposeMult():</span></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;background-color:rgb(255,255,255);display:inline">MatMatTransposeMult(data_mat,
data_mat, MAT_INITIAL_MATRIX, PETSC_DEFAULT, &corr_mat); <br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;background-color:rgb(255,255,255);display:inline">//ierr = MatAssemblyBegin(corr_mat,
MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); </span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;background-color:rgb(255,255,255);display:inline">//ierr = MatAssemblyEnd(corr_mat,
MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);<br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;background-color:rgb(255,255,255);display:inline"><br>
</span></div>
<div style="font-size:12pt;color:rgb(0,0,0)"><font face="Calibri, Helvetica, sans-serif, EmojiFont, Apple Color Emoji, Segoe UI Emoji, NotoColorEmoji, Segoe UI Symbol, Android Emoji, EmojiSymbols">The communication patterns of parallel implementation
led to different order of floating-point computation, thus slightly different matrix norm of R. </font></div>
<div style="font-size:12pt;color:rgb(0,0,0)"><font face="Calibri, Helvetica, sans-serif, EmojiFont, Apple Color Emoji, Segoe UI Emoji, NotoColorEmoji, Segoe UI Symbol, Android Emoji, EmojiSymbols">Hong</font></div>
<div id="gmail-m_-5517018130510300103appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-5517018130510300103divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> petsc-users <<a href="mailto:petsc-users-bounces@mcs.anl.gov" target="_blank">petsc-users-bounces@mcs.anl.gov</a>> on behalf of Peder Jørgensgaard Olesen via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Sent:</b> Monday, April 19, 2021 7:57 AM<br>
<b>To:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> [petsc-users] Rather different matrix product results on multiple processes</font>
<div> </div>
</div>
<div dir="ltr">
<div id="gmail-m_-5517018130510300103x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif">
<p style="margin-top:0px;margin-bottom:0px">Hello,</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">When computing a matrix product of the type R = D.DT using MatMatTransposeMult() I find I get rather different results depending on the number of processes. In one example using a data set that is small compared
to the application I get Frobenius norms |R| = 1.047e3 on a single process, 1.0363e3 on a single HPC node (40 cores), and 9.7307e2 on two nodes.</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">I have ascertained that the single process result is indeed the correct one (i.e., eigenvectors of R form a proper basis for the columns of D), so naturally I'd love to be able to reproduce this result across
different parallel setups. How might I achieve this?</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">I'm attaching MWE code and the data set used for the example.</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">Thanks in advance!</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<div id="gmail-m_-5517018130510300103x_Signature">
<div id="gmail-m_-5517018130510300103x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top:0px;margin-bottom:0px">Best Regards</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">Peder Jørgensgaard Olesen</p>
<p style="margin-top:0px;margin-bottom:0px"><span style="font-size:10pt;color:rgb(117,123,128)">PhD Student, Turbulence Research Lab</span></p>
<p style="margin-top:0px;margin-bottom:0px"><span style="font-size:10pt;color:rgb(117,123,128)">Dept. of Mechanical Engineering</span></p>
<p style="margin-top:0px;margin-bottom:0px"><span style="font-size:10pt;color:rgb(117,123,128)">Technical University of Denmark</span></p>
<p style="margin-top:0px;margin-bottom:0px"><span style="font-size:10pt;color:rgb(117,123,128)">Niels Koppels Allé</span></p>
<p style="margin-top:0px;margin-bottom:0px"><span style="font-size:10pt;color:rgb(117,123,128)">Bygning 403, Rum 105</span></p>
<p style="margin-top:0px;margin-bottom:0px"><span style="font-size:10pt;color:rgb(117,123,128)">DK-2800 Kgs. Lyngby</span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature">Stefano</div>