<div dir="ltr"><div dir="ltr"><div>Hong,</div><div><br></div><div>Thanks for your improvements on PtAP that is critical for MG-type algorithms. </div><br><div class="gmail_quote"><div dir="ltr">On Wed, May 3, 2017 at 10:17 AM Hong <<a href="mailto:hzhang@mcs.anl.gov">hzhang@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Mark,<div>Below is the copy of my email sent to you on Feb 27:</div><div><br></div><div><span style="font-size:12.8px">I implemented scalable MatPtAP and did comparisons of three implementations using </span><span class="gmail-m_3433343712918822110gmail-il" style="font-size:12.8px;background-color:rgb(255,255,255)">ex56</span><span style="font-size:12.8px">.c on alcf </span><span class="gmail-m_3433343712918822110gmail-il" style="font-size:12.8px;background-color:rgb(255,255,255)">cetus</span><span style="font-size:12.8px"> machine (this machine has small memory, 1GB/core):</span><div style="font-size:12.8px">- nonscalable PtAP: use an array of length PN to do dense axpy</div><div style="font-size:12.8px">- scalable PtAP: do sparse axpy without use of PN array</div></div></div></blockquote><div><br></div><div>What PN means here?</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div style="font-size:12.8px">- hypre PtAP.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">The results are attached. Summary:</div><div style="font-size:12.8px">- nonscalable PtAP is 2x faster than scalable, 8x faster than hypre PtAP</div><div style="font-size:12.8px">- scalable PtAP is 4x faster than hypre PtAP</div><div style="font-size:12.8px">- hypre uses less memory (see <a href="http://job.ne399.n63.np1000.sh/" target="_blank">job.ne399.n63.np1000.sh</a>)</div></div></div></blockquote><div><br></div><div>I was wondering how much more memory PETSc PtAP uses than hypre? I am implementing an AMG algorithm based on PETSc right now, and it is working well. But we find some a bottleneck with PtAP. For the same P and A, PETSc PtAP fails to generate a coarse matrix due to out of memory, while hypre still can generates the coarse matrix.</div><div><br></div><div>I do not want to just use the HYPRE one because we had to duplicate matrices if I used HYPRE PtAP.</div><div><br></div><div>It would be nice if you guys already have done some compassions on these implementations for the memory usage.</div><div><br></div><div><br></div><div>Fande,</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Based on above observation, I set the default PtAP algorithm as 'nonscalable'. </div><div style="font-size:12.8px">When PN > local estimated nonzero of C=PtAP, then switch default to 'scalable'.</div><div style="font-size:12.8px">User can overwrite default.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">For the case of np=8000, ne=599 (see <a href="http://job.ne599.n500.np8000.sh/" target="_blank">job.ne599.n500.np8000.sh</a>), I get</div><div style="font-size:12.8px">MatPtAP 3.6224e+01 (nonscalable for small mats, scalable for larger ones)<br></div><div style="font-size:12.8px">scalable MatPtAP 4.6129e+01<br></div><div style="font-size:12.8px">hypre 1.9389e+02 </div></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">This work in on petsc-master. Give it a try. If you encounter any problem, let me know.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Hong</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 3, 2017 at 10:01 AM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">(Hong), what is the current state of optimizing RAP for scaling?<div><br></div><div>Nate, is driving 3D elasticity problems at scaling with GAMG and we are working out performance problems. They are hitting problems at ~1.5B dof problems on a basic Cray (XC30 I think).</div><div><br></div><div>Thanks,</div><div>Mark</div></div>
</blockquote></div><br></div>
</blockquote></div></div></div>