<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Wed, Oct 5, 2016 at 2:30 PM, Matthew Overholt <span dir="ltr"><<a href="mailto:overholt@capesim.com" target="_blank">overholt@capesim.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div class="m_-5981765143895589735WordSection1"><p class="MsoNormal">Hi Petsc-Users,<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">I am trying to understand an issue where PetscCommDuplicate() calls are taking an increasing percentage of time as I run a fixed-sized problem on more processes.<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">I am using the FEM to solve the steady-state heat transfer equation (K.x = q) using a PC direct solver, like MUMPS.<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">I am running on the NERSC Cray X30, which has two Xeon's per node with 12 cores each, and profiling the code using CrayPat sampling.<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">On a typical problem (1E+6 finite elements), running on a single node:<u></u><u></u></p><p class="MsoNormal">-for 2 cores (1 on each Xeon), about 1% of time is PetscCommDuplicate (on process 1, but on the root it is less), and (for reference) 9% of total time is for MUMPS.<u></u><u></u></p><p class="MsoNormal">-for 8 cores (4 on each Xeon), over 6% of time is PetscCommDuplicate (on every process except the root, where it is <1%), and 9-10% of total time is for MUMPS.<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">What is the large PetscCommDuplicate time connected to, an increasing number of messages (tags)? Would using fewer MatSetValues() and VecSetValues() calls (with longer message lengths) alleviate this?</p></div></div></blockquote><div><br></div><div>1) I am skeptical of the result. Can you do a run with direct measurement rather than sampling?</div><div><br></div><div>2) Can we see the output of -log_view ?</div><div><br></div><div>3) Are you sure you configured with --with-debugging=0 ?</div><div><br></div><div>4) If this result is true, it could be coming from bad behavior with PetscSpinlock on this machine. We need to see configure.log</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div class="m_-5981765143895589735WordSection1"><p class="MsoNormal">For reference, the PETSc calling sequence in the code is as follows.<u></u><u></u></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> // Create the solution and RHS vectors<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = VecCreate(petscData->mpicomm,&<wbr>mesh->hpx);<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = PetscObjectSetName((<wbr>PetscObject) mesh->hpx, "Solution");<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = VecSetSizes(mesh->hpx,mesh-><wbr>lxN,mesh->neqns); // size = # of equations; distribution to match mesh<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = VecSetFromOptions(mesh->hpx); <wbr> // allow run time options<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = VecDuplicate(mesh->hpx,&q); <wbr> // create the RHS vector<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> // Create the stiffnexx matrix<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = MatCreate(petscData->mpicomm,&<wbr>K);<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = MatSetSizes(K,mesh->lxN,mesh-><wbr>lxN,mesh->neqns,mesh->neqns);<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = MatSetType(K,MATAIJ); // default sparse type<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> // Do preallocation<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = MatMPIAIJSetPreallocation(K,d_<wbr>nz,NULL,o_nz,NULL);<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = MatSeqAIJSetPreallocation(K,d_<wbr>nz,NULL);<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = MatSetOption(K,MAT_NEW_<wbr>NONZERO_ALLOCATION_ERR,PETSC_<wbr>FALSE);<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = MatSetUp(K);<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> // Create and set up the KSP context as a PreConditioner Only (Direct) Solution<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = KSPCreate(petscData->mpicomm,&<wbr>ksp);<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = KSPSetOperators(ksp,K,K);<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = KSPSetType(ksp,KSPPREONLY);<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> // Set the temperature vector<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = VecSet(mesh->hpx,mesh->Tmin);<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> // Set the default PC method as MUMPS<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = KSPGetPC(ksp,&pc); // extract the preconditioner<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = PCSetType(pc,PCLU); // set pc options<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = PCFactorSetMatSolverPackage(<wbr>pc,MATSOLVERMUMPS);<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = KSPSetFromOptions(ksp);<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> <u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> // Set the values for the K matrix and q vector<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> // which involves a lot of these calls<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = MatSetValues(K,mrows,idxm,<wbr>ncols,idxn,pKe,ADD_VALUES); // 1 call per matrix row (equation)<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = VecSetValues(q,nqe,ixn,pqe,<wbr>ADD_VALUES); // 1 call per element<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = VecAssemblyBegin(q);<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = MatAssemblyBegin(K,MAT_FINAL_<wbr>ASSEMBLY);<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = VecAssemblyEnd(q);<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = MatAssemblyEnd(K,MAT_FINAL_<wbr>ASSEMBLY);<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> // Solve //////////////////////////////<wbr>////////<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ierr = KSPSolve(ksp,q,mesh->hpx);<u></u><u></u></span></p><p class="MsoNormal">...<u></u><u></u></p><p class="MsoNormal">*Note that the code evenly divides the finite elements over the total number of processors, and I am using ghosting of the FE vertices vector to handle the vertices that are needed on more than 1 process.<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Thanks in advance for your help,<u></u><u></u></p><p class="MsoNormal">Matt Overholt<u></u><u></u></p><p class="MsoNormal">CapeSym, Inc.<u></u><u></u></p></div><br>
<table style="border-top:1px solid #d3d4de">
<tbody><tr>
<td style="width:55px;padding-top:13px"><a href="https://www.avast.com/en-us/lp-safe-emailing-2270-b?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=oa-2270-b" target="_blank"><img src="https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-tick-v1.gif" width="46" height="29" style="width:46px;height:29px"></a></td>
<td style="width:470px;padding-top:12px;color:#41424e;font-size:13px;font-family:Arial,Helvetica,sans-serif;line-height:18px">Virus-free. <a href="https://www.avast.com/en-us/lp-safe-emailing-2270-b?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=oa-2270-b" style="color:#4453ea" target="_blank">www.avast.com</a>
</td>
</tr>
</tbody></table></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div>