[petsc-users] Scaling problem when cores > 600

[Smith, Barry F.]

> On Feb 28, 2018, at 8:01 PM, TAY wee-beng <zonexo at gmail.com> wrote:
> 
> 
> On 1/3/2018 12:10 AM, Matthew Knepley wrote:
>> On Wed, Feb 28, 2018 at 10:45 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>> Hi,
>> 
>> I have a CFD code which uses PETSc and HYPRE. I found that for a certain case with grid size of 192,570,048, I encounter scaling problem when my cores > 600. At 600 cores, the code took 10min for 100 time steps. At 960, 1440 and 2880 cores, it still takes around 10min. At 360 cores, it took 15min.
>> 
>> So how can I find the bottleneck? Any recommended steps?
>> 
>> For any performance question, we need to see the output of -log_view for all test cases.
> Hi,
> 
> To be more specific, I use PETSc KSPBCGS and HYPRE geometric multigrid (entirely based on HYPRE, no PETSc) for the momentum and Poisson eqns in my code.
> 
> So can log_view be used in this case to give a meaningful? Since part of the code uses HYPRE?

  Yes, just send the logs.


> 
> I also program another subroutine in the past which uses PETSc to solve the Poisson eqn. It uses either HYPRE's boomeramg, KSPBCGS or KSPGMRES.
> 
> If I use boomeramg, can log_view be used in this case?
> 
> Or do I have to use KSPBCGS or KSPGMRES, which is directly from PETSc? However, I ran KSPGMRES yesterday with the Poisson eqn and my ans didn't converge.
> 
> Thanks.
>>  
>> I must also mention that I partition my grid only in the x and y direction. There is no partitioning in the z direction due to limited code development. I wonder if there is a strong effect in this case.
>> 
>> Maybe. Usually what happens is you fill up memory with a z-column and cannot scale further.
>> 
>>   Thanks,
>> 
>>      Matt
>>  
>> 
>> -- 
>> Thank you very much
>> 
>> Yours sincerely,
>> 
>> ================================================
>> TAY Wee-Beng 郑伟明 (Zheng Weiming)
>> Personal research webpage: http://tayweebeng.wixsite.com/website
>> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA
>> linkedin: www.linkedin.com/in/tay-weebeng
>> ================================================
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
>> 
>> https://www.cse.buffalo.edu/~knepley/
>