[petsc-users] Scaling problem when cores > 600

[Matthew Knepley]

On Wed, Feb 28, 2018 at 9:01 PM, TAY wee-beng <zonexo at gmail.com> wrote:

>
> On 1/3/2018 12:10 AM, Matthew Knepley wrote:
>
> On Wed, Feb 28, 2018 at 10:45 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>
>> Hi,
>>
>> I have a CFD code which uses PETSc and HYPRE. I found that for a certain
>> case with grid size of 192,570,048, I encounter scaling problem when my
>> cores > 600. At 600 cores, the code took 10min for 100 time steps. At 960,
>> 1440 and 2880 cores, it still takes around 10min. At 360 cores, it took
>> 15min.
>>
>> So how can I find the bottleneck? Any recommended steps?
>>
>
> For any performance question, we need to see the output of -log_view for
> all test cases.
>
> Hi,
>
> To be more specific, I use PETSc KSPBCGS and HYPRE geometric multigrid
> (entirely based on HYPRE, no PETSc) for the momentum and Poisson eqns in my
> code.
>
> So can log_view be used in this case to give a meaningful? Since part of
> the code uses HYPRE?
>

Make an event to time the HYPRE solve. It only takes a few lines of code.


> I also program another subroutine in the past which uses PETSc to solve
> the Poisson eqn. It uses either HYPRE's boomeramg, KSPBCGS or KSPGMRES.
>
> If I use boomeramg, can log_view be used in this case?
>

Yes, its automatic.


> Or do I have to use KSPBCGS or KSPGMRES, which is directly from PETSc?
> However, I ran KSPGMRES yesterday with the Poisson eqn and my ans didn't
> converge.
>

Plain GMRES is not good for Poisson. You would be better off with
GMRES/GAMG.

  Thanks,

     Matt


> Thanks.
>
>
>
>> I must also mention that I partition my grid only in the x and y
>> direction. There is no partitioning in the z direction due to limited code
>> development. I wonder if there is a strong effect in this case.
>
>
> Maybe. Usually what happens is you fill up memory with a z-column and
> cannot scale further.
>
>   Thanks,
>
>      Matt
>
>
>>
>> --
>> Thank you very much
>>
>> Yours sincerely,
>>
>> ================================================
>> TAY Wee-Beng 郑伟明 (Zheng Weiming)
>> Personal research webpage: http://tayweebeng.wixsite.com/website
>> Youtube research showcase: https://www.youtube.com/channe
>> l/UC72ZHtvQNMpNs2uRTSToiLA
>> linkedin: www.linkedin.com/in/tay-weebeng
>> ================================================
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/%7Emk51/>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20180228/30c1e238/attachment.html>