<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div dir="ltr">
<div dir="ltr"><br clear="all">
<div>
<div dir="ltr" data-smartmail="gmail_signature">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Jan 27, 2020 at 10:09 AM Felix Huber <<a href="mailto:st107539@stud.uni-stuttgart.de" target="_blank">st107539@stud.uni-stuttgart.de</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Thank you all for you reply!<br>
<br>
> Are you using a KSP/PC configuration which should weak scale?<br>
Yes the system is solved with KSPSolve. There is no preconditioner yet, <br>
but I fixed the number of CG iterations to 3 to ensure an apples to <br>
apples comparison during the scaling measurements.<br>
<br>
>> VecScatter has been greatly refactored (and the default implementation<br>
>> is entirely new) since 3.7.<br>
<br>
I now tried to use PETSc 3.11 and the code runs fine. The communication <br>
seems to show a better weak scaling behavior now.<br>
<br>
I'll see if we can just upgrade to 3.11.<br>
<br>
<br>
<br>
> Anyway, I'm curious about your<br>
> configuration and how you determine that MPI_Alltoallv/MPI_Alltoallw is<br>
> being used.<br>
I used the Extrae profiler which intercepts all MPI calls and logs them <br>
into a file. This showed that Alltoall is being used for the <br>
communication, which I found surprising. With PETSc 3.11 the Alltoall <br>
calls are replaced by MPI_Start(all) and MPI_Wait(all), which sounds <br>
more reasonable to me.<br>
> This has never been a default code path, so I suspect<br>
> something in your environment or code making this happen.<br>
<br>
I attached some log files for some PETSc 3.7 runs on 1,19 and 115 nodes <br>
(24 cores each) which suggest polynomial scaling (vs logarithmic <br>
scaling). Could it be some installation setting of the PETSc version? (I <br>
use a preinstalled PETSc)<br>
</blockquote>
<div>I checked petsc 3.7.6 and did not think the vecscatter type could be set at configure time. Anyway, upgrading petsc is preferred. If that is not possible, we can work together to see what happened.</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> Can you please send representative log files which characterize the <br>
> lack of scaling (include the full log_view)?<br>
<br>
"Stage 1: activation" is the stage of interest, as it wraps the <br>
KSPSolve. The number of unkowns per rank is very small in the <br>
measurement, so most of the time should be communication. However, I <br>
just noticed, that the stage also contains an additional setup step <br>
which might be the reason why the MatMul takes longer than the KSPSolve. <br>
I can repeat the measurements if necessary.<br>
I should add, that I put a MPI_Barrier before the KSPSolve, to avoid any <br>
previous work imbalance to effect the KSPSolve call.<br>
<br>
</blockquote>
You can use -log_sync, which adds an MPI_Barrier at the beginning of each event. Compare log_view files with and without -log_sync. If an event has much higher %T without -log_sync than with -log_sync, it means the code is not balanced. Alternatively, you can
look at the Ratio column in log file without -log_sync.</div>
<div class="gmail_quote"> <br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Best regards,<br>
Felix<br>
<br>
</blockquote>
</div>
</div>
</body>
</html>