[petsc-users] Scaling problem when cores > 600

[TAY wee-beng]

On 1/3/2018 12:10 AM, Matthew Knepley wrote:
> On Wed, Feb 28, 2018 at 10:45 AM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>     Hi,
>
>     I have a CFD code which uses PETSc and HYPRE. I found that for a
>     certain case with grid size of 192,570,048, I encounter scaling
>     problem when my cores > 600. At 600 cores, the code took 10min for
>     100 time steps. At 960, 1440 and 2880 cores, it still takes around
>     10min. At 360 cores, it took 15min.
>
>     So how can I find the bottleneck? Any recommended steps?
>
>
> For any performance question, we need to see the output of -log_view 
> for all test cases.
Hi,

To be more specific, I use PETSc KSPBCGS and HYPRE geometric multigrid 
(entirely based on HYPRE, no PETSc) for the momentum and Poisson eqns in 
my code.

So can log_view be used in this case to give a meaningful? Since part of 
the code uses HYPRE?

I also program another subroutine in the past which uses PETSc to solve 
the Poisson eqn. It uses either HYPRE's boomeramg, KSPBCGS or KSPGMRES.

If I use boomeramg, can log_view be used in this case?

Or do I have to use KSPBCGS or KSPGMRES, which is directly from PETSc? 
However, I ran KSPGMRES yesterday with the Poisson eqn and my ans didn't 
converge.

Thanks.
>
>     I must also mention that I partition my grid only in the x and y
>     direction. There is no partitioning in the z direction due to
>     limited code development. I wonder if there is a strong effect in
>     this case.
>
>
> Maybe. Usually what happens is you fill up memory with a z-column and 
> cannot scale further.
>
>   Thanks,
>
>      Matt
>
>
>     -- 
>     Thank you very much
>
>     Yours sincerely,
>
>     ================================================
>     TAY Wee-Beng 郑伟明 (Zheng Weiming)
>     Personal research webpage: http://tayweebeng.wixsite.com/website
>     <http://tayweebeng.wixsite.com/website>
>     Youtube research showcase:
>     https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA
>     <https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA>
>     linkedin: www.linkedin.com/in/tay-weebeng
>     <http://www.linkedin.com/in/tay-weebeng>
>     ================================================
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/%7Emk51/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20180301/38c7f706/attachment-0001.html>