On Fri, Mar 11, 2011 at 9:52 AM, Sebastian Steiger <span dir="ltr"><<a href="mailto:steiger@purdue.edu">steiger@purdue.edu</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hello PETSc developers<br>
<br>
I'm doing some scaling benchmarks and I found that the parallel asm<br>
preconditioner, my favorite preconditioner, has a limit in the number of<br>
cores it can handle.<br>
<br>
I am doing a numerical experiment where I scale up the size of my matrix<br>
by roughly the same factor as the number of CPUs employed. When I look<br>
at which function used how much memory using PETSc's routine<br>
PetscMallocDumpLog, I see the following:<br>
<br>
Function name N=300 N=600 increase<br>
======================================================================<br>
MatGetSubMatrices_MPIAIJ_Local 75'912'016 134'516'928 1.77<br>
MatIncreaseOverlap_MPIAIJ_Once 168'288'288 346'870'832 2.06<br>
MatIncreaseOverlap_MPIAIJ_Receive 2'918'960 5'658'160 1.94<br>
<br>
The matrix sizes are 6'899'904 and 14'224'896, respectively. Above<br>
N~5000 CPUs I am running out of memory.<br></blockquote><div><br></div><div>We have run ASM on 224,000 processors of the XT5 at ORNL, so something else</div><div>is going on here. The best thing to do here is send us -log_summary. For attachments,</div>
<div>we usually recommend <a href="mailto:petsc-maint@mcs.anl.gov">petsc-maint@mcs.anl.gov</a>.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Here's my question now: Is the asm preconditioner limited from the<br>
algorithm point of view, or is it the implementation? I thought that<br>
'only' the local matrices, plus some constant overlap with neighbors,<br>
are solved, so that memory consumption should stay constant when I scale<br>
up with a constant number of rows per process.<br>
<br>
Best<br>
<font color="#888888">Sebastian<br>
<br>
<br>
</font></blockquote></div><br><br clear="all"><br>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<br>