[petsc-users] Poor weak scaling when solving successive linearsystems

[Junchao Zhang]

I tried periodic boundary conditions but found load-imbalance still
existed. So the boundary may not be a big issue. I am debugging the code to
see why.  Thanks.

--Junchao Zhang

On Fri, Jun 15, 2018 at 2:26 AM, Michael Becker <
michael.becker at physik.uni-giessen.de> wrote:

> Hello,
>
> thanks again for your efforts.
>
> Boundary processors have less nonzeros than interior processors.
>
> Doesn't that mean that boundary processors have less work to do than the
> others? Or does this affect the size of the coarse grids?
>
> I know that defining the outermost nodes as boundary is not the most
> efficient way in this particular case (using Dirichlet boundary conditions
> on a smaller grid would do the same), but I need the solver to be able to
> handle arbitrarily shaped boundaries inside the domain, e.g. to calculate
> the potential inside a spherical capacitor (constant potential on the
> boundaries, charge distribution inside). Is there a better way to do that?
>
> Michael
>
>
>
>
> Am 12.06.2018 um 22:07 schrieb Junchao Zhang:
>
> Hello, Michael,
>   Sorry for the delay. I am actively doing experiments with your example
> code. I tested it on a cluster with 36 cores/node. To distribute MPI ranks
> evenly among nodes, I used 216  and 1728 ranks instead of 125, 1000.  So
> far I have these findings:
>  1) It is not a strict weak scaling test since with 1728 ranks it needs
> more KPS iterations, and more calls to MatSOR etc functions.
>  2) If I use half cores per node but double the nodes (keep MPI ranks the
> same), the performance is 60~70% better. It implies memory bandwidth plays
> an important role in performance.
>  3) I find you define the outermost two layers of nodes of the grid as
> boundary. Boundary processors have less nonzeros than interior processors.
> It is a source of load imbalance. At coarser grids, it gets worse. But I
> need to confirm this caused the poor scaling and big vecscatter delays in
> the experiment.
>
>  Thanks.
>
>
> --Junchao Zhang
>
> On Tue, Jun 12, 2018 at 12:42 AM, Michael Becker <
> michael.becker at physik.uni-giessen.de> wrote:
>
>> Hello,
>>
>> any new insights yet?
>>
>> Michael
>>
>>
>>
>> Am 04.06.2018 um 21:56 schrieb Junchao Zhang:
>>
>> Miachael,  I can compile and run you test.  I am now profiling it. Thanks.
>>
>> --Junchao Zhang
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20180615/032443d5/attachment.html>