[petsc-users] Scalability of PETSc on vesta.alcf

[Jed Brown]

Please always use "reply-all" so that your messages go to the list.
This is standard mailing list etiquette.  It is important to preserve
threading for people who find this discussion later and so that we do
not waste our time re-answering the same questions that have already
been answered in private side-conversations.  You'll likely get an
answer faster that way too.

Roc Wang <pengxwang at hotmail.com> writes:

> Thanks, Jed
>
>
>> From: jed at jedbrown.org
>> To: pengxwang at hotmail.com; petsc-users at mcs.anl.gov
>> Subject: Re: [petsc-users] Scalability of PETSc on vesta.alcf
>> Date: Mon, 20 Jan 2014 08:33:29 -0700
>> 
>> Roc Wang <pengxwang at hotmail.com> writes:
>> 
>> > Hello, 
>> >
>> >    I am testing a petsc program on vesta.alcf.acl.gov. The scalability
>> >    was fine when the number of ranks is less then 1024. However, when
>> >    the 2048 ranks were used, 
>> 
>> Are you using c64 mode in all cases or did you run smaller fewer
>> processes per node out to 1024?  You can't do fair scaling with
>> different modes because BG/Q has only 16 cores per node. 
> The ranks=2048 was with mode c64 and ranks <1024 were with c1. 

That is completely different.  I recommend running c16 for all sizes;
that should be efficient and reproducible.

>> The four hardware threads per core only cover latency, but do not significantly
>> improve memory bandwidth.  
> So, the bandwidth of c64 mode is kept same as c1, and it makes the computation slow down, right?
>
> I run a case of 1024 cores with c64 mode, the timing is 56.74 s which
> is larger than c1 mode. So, it is still possible to have shorter
> computation time with 4096 ranks in the same mode c64 compared with
> 2048 (c64) and 1024(c64), right?

Yes, that indicates that you're still scaling well.

>>We're seeing most of the time in MatSolve,
>> which does no communication.  (Also MatMult, but your large fill makes
>> the factors much heavier than the matrix itself.)
>
> Which fill did you meant to larger? Is there any solution to make the large fill better?

ILU(3).  Reduce the number of levels to reduce MatSolve time.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20140120/a56a33c1/attachment.pgp>