<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Hi Petsc-Users,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I am trying to understand an issue where PetscCommDuplicate() calls are taking an increasing percentage of time as I run a fixed-sized problem on more processes.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I am using the FEM to solve the steady-state heat transfer equation (K.x = q) using a PC direct solver, like MUMPS.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I am running on the NERSC Cray X30, which has two Xeon's per node with 12 cores each, and profiling the code using CrayPat sampling.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>On a typical problem (1E+6 finite elements), running on a single node:<o:p></o:p></p><p class=MsoNormal>-for 2 cores (1 on each Xeon), about 1% of time is PetscCommDuplicate (on process 1, but on the root it is less), and (for reference) 9% of total time is for MUMPS.<o:p></o:p></p><p class=MsoNormal>-for 8 cores (4 on each Xeon), over 6% of time is PetscCommDuplicate (on every process except the root, where it is <1%), and 9-10% of total time is for MUMPS.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>What is the large PetscCommDuplicate time connected to, an increasing number of messages (tags)? Would using fewer MatSetValues() and VecSetValues() calls (with longer message lengths) alleviate this?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>For reference, the PETSc calling sequence in the code is as follows.<o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> // Create the solution and RHS vectors<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = VecCreate(petscData->mpicomm,&mesh->hpx);<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = PetscObjectSetName((PetscObject) mesh->hpx, "Solution");<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = VecSetSizes(mesh->hpx,mesh->lxN,mesh->neqns); // size = # of equations; distribution to match mesh<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = VecSetFromOptions(mesh->hpx); // allow run time options<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = VecDuplicate(mesh->hpx,&q); // create the RHS vector<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> // Create the stiffnexx matrix<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = MatCreate(petscData->mpicomm,&K);<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = MatSetSizes(K,mesh->lxN,mesh->lxN,mesh->neqns,mesh->neqns);<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = MatSetType(K,MATAIJ); // default sparse type<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> // Do preallocation<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = MatMPIAIJSetPreallocation(K,d_nz,NULL,o_nz,NULL);<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = MatSeqAIJSetPreallocation(K,d_nz,NULL);<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = MatSetOption(K,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE);<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = MatSetUp(K);<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> // Create and set up the KSP context as a PreConditioner Only (Direct) Solution<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = KSPCreate(petscData->mpicomm,&ksp);<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = KSPSetOperators(ksp,K,K);<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = KSPSetType(ksp,KSPPREONLY);<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> // Set the temperature vector<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = VecSet(mesh->hpx,mesh->Tmin);<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> // Set the default PC method as MUMPS<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = KSPGetPC(ksp,&pc); // extract the preconditioner<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = PCSetType(pc,PCLU); // set pc options<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = PCFactorSetMatSolverPackage(pc,MATSOLVERMUMPS);<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = KSPSetFromOptions(ksp);<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> // Set the values for the K matrix and q vector<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> // which involves a lot of these calls<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = MatSetValues(K,mrows,idxm,ncols,idxn,pKe,ADD_VALUES); // 1 call per matrix row (equation)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = VecSetValues(q,nqe,ixn,pqe,ADD_VALUES); // 1 call per element<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = VecAssemblyBegin(q);<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = MatAssemblyBegin(K,MAT_FINAL_ASSEMBLY);<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = VecAssemblyEnd(q);<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = MatAssemblyEnd(K,MAT_FINAL_ASSEMBLY);<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> // Solve //////////////////////////////////////<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> ierr = KSPSolve(ksp,q,mesh->hpx);<o:p></o:p></span></p><p class=MsoNormal>...<o:p></o:p></p><p class=MsoNormal>*Note that the code evenly divides the finite elements over the total number of processors, and I am using ghosting of the FE vertices vector to handle the vertices that are needed on more than 1 process.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks in advance for your help,<o:p></o:p></p><p class=MsoNormal>Matt Overholt<o:p></o:p></p><p class=MsoNormal>CapeSym, Inc.<o:p></o:p></p></div><br />
<table style="border-top: 1px solid #D3D4DE;">
<tr>
<td style="width: 55px; padding-top: 13px;"><a href="https://www.avast.com/en-us/lp-safe-emailing-2270-b?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=oa-2270-b" target="_blank"><img src="https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-tick-v1.gif" width="46" height="29" style="width: 46px; height: 29px;" /></a></td>
<td style="width: 470px; padding-top: 12px; color: #41424e; font-size: 13px; font-family: Arial, Helvetica, sans-serif; line-height: 18px;">Virus-free. <a href="https://www.avast.com/en-us/lp-safe-emailing-2270-b?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=oa-2270-b" target="_blank" style="color: #4453ea;">www.avast.com</a>
</td>
</tr>
</table></body></html>