<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family:Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;background-color:rgb(255, 255, 255);display:inline !important">Peder,</span><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family:Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;background-color:rgb(255, 255, 255);display:inline !important">I tested your code on
a linux machine. I got</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family:Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;background-color:rgb(255, 255, 255);display:inline !important">$ ./acorr_mwe
<div>Data matrix norm: 5.0538e+01</div>
Autocorrelation matrix norm: 1.0473e+03<br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family:Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;background-color:rgb(255, 255, 255);display:inline !important"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family:Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;background-color:rgb(255, 255, 255);display:inline !important">mpiexec -n 40 ./acorr_mwe -matmattransmult_mpidense_mpidense_via
allgatherv (default)
<div>Data matrix norm: 5.0538e+01</div>
Autocorrelation matrix norm: 1.0363e+03<br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family:Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;background-color:rgb(255, 255, 255);display:inline !important"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family:Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;background-color:rgb(255, 255, 255);display:inline !important">mpiexec -n 20 ./acorr_mwe
<div>Data matrix norm: 5.0538e+01</div>
Autocorrelation matrix norm: 1.0897e+03<br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family:Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;background-color:rgb(255, 255, 255);display:inline !important"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family:Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;background-color:rgb(255, 255, 255);display:inline !important">mpiexec -n 40 ./acorr_mwe
-matmattransmult_mpidense_mpidense_via cyclic
<div>Data matrix norm: 5.0538e+01</div>
Autocorrelation matrix norm: 1.0363e+03</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family:Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;background-color:rgb(255, 255, 255);display:inline !important"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family:Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;background-color:rgb(255, 255, 255);display:inline !important">I use petsc 'main' branch
(same as the latest release). You can remove MatAssemblyBegin/End calls after <span style="background-color:rgb(255, 255, 255);display:inline !important">MatMatTransposeMult():</span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family:Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;background-color:rgb(255, 255, 255);display:inline !important">MatMatTransposeMult(data_mat,
data_mat, MAT_INITIAL_MATRIX, PETSC_DEFAULT, &corr_mat); <br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family:Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;background-color:rgb(255, 255, 255);display:inline !important">//ierr = MatAssemblyBegin(corr_mat,
MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); </span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family:Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;background-color:rgb(255, 255, 255);display:inline !important">//ierr = MatAssemblyEnd(corr_mat,
MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);<br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family:Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;background-color:rgb(255, 255, 255);display:inline !important"><br>
</span></div>
<div style="font-size: 12pt; color: rgb(0, 0, 0);"><font face="Calibri, Helvetica, sans-serif, EmojiFont, Apple Color Emoji, Segoe UI Emoji, NotoColorEmoji, Segoe UI Symbol, Android Emoji, EmojiSymbols">The communication patterns of parallel implementation
led to different order of floating-point computation, thus slightly different matrix norm of R. </font></div>
<div style="font-size: 12pt; color: rgb(0, 0, 0);"><font face="Calibri, Helvetica, sans-serif, EmojiFont, Apple Color Emoji, Segoe UI Emoji, NotoColorEmoji, Segoe UI Symbol, Android Emoji, EmojiSymbols">Hong</font></div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> petsc-users <petsc-users-bounces@mcs.anl.gov> on behalf of Peder Jørgensgaard Olesen via petsc-users <petsc-users@mcs.anl.gov><br>
<b>Sent:</b> Monday, April 19, 2021 7:57 AM<br>
<b>To:</b> petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
<b>Subject:</b> [petsc-users] Rather different matrix product results on multiple processes</font>
<div> </div>
</div>
<div dir="ltr">
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<p style="margin-top: 0px; margin-bottom: 0px;">Hello,</p>
<p style="margin-top: 0px; margin-bottom: 0px;"><br>
</p>
<p style="margin-top: 0px; margin-bottom: 0px;">When computing a matrix product of the type R = D.DT using MatMatTransposeMult() I find I get rather different results depending on the number of processes. In one example using a data set that is small compared
to the application I get Frobenius norms |R| = 1.047e3 on a single process, 1.0363e3 on a single HPC node (40 cores), and 9.7307e2 on two nodes.</p>
<p style="margin-top: 0px; margin-bottom: 0px;"><br>
</p>
<p style="margin-top: 0px; margin-bottom: 0px;">I have ascertained that the single process result is indeed the correct one (i.e., eigenvectors of R form a proper basis for the columns of D), so naturally I'd love to be able to reproduce this result across
different parallel setups. How might I achieve this?</p>
<p style="margin-top: 0px; margin-bottom: 0px;"><br>
</p>
<p style="margin-top: 0px; margin-bottom: 0px;">I'm attaching MWE code and the data set used for the example.</p>
<p style="margin-top: 0px; margin-bottom: 0px;"><br>
</p>
<p style="margin-top: 0px; margin-bottom: 0px;">Thanks in advance!</p>
<p style="margin-top: 0px; margin-bottom: 0px;"><br>
</p>
<div id="x_Signature">
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top: 0px; margin-bottom: 0px;">Best Regards</p>
<p style="margin-top: 0px; margin-bottom: 0px;"><br>
</p>
<p style="margin-top: 0px; margin-bottom: 0px;">Peder Jørgensgaard Olesen</p>
<p style="margin-top: 0px; margin-bottom: 0px;"><span style="font-size:10pt; color:rgb(117,123,128)">PhD Student, Turbulence Research Lab</span></p>
<p style="margin-top: 0px; margin-bottom: 0px;"><span style="font-size:10pt; color:rgb(117,123,128)">Dept. of Mechanical Engineering</span></p>
<p style="margin-top: 0px; margin-bottom: 0px;"><span style="font-size:10pt; color:rgb(117,123,128)">Technical University of Denmark</span></p>
<p style="margin-top: 0px; margin-bottom: 0px;"><span style="font-size:10pt; color:rgb(117,123,128)">Niels Koppels Allé</span></p>
<p style="margin-top: 0px; margin-bottom: 0px;"><span style="font-size:10pt; color:rgb(117,123,128)">Bygning 403, Rum 105</span></p>
<p style="margin-top: 0px; margin-bottom: 0px;"><span style="font-size:10pt; color:rgb(117,123,128)">DK-2800 Kgs. Lyngby</span></p>
</div>
</div>
</div>
</div>
</body>
</html>