<div dir="ltr">Dear Patrick Sanan,<div><br>Thank you very much for your answer, especially for your code.<br>I was able to compile and run your code on 8 nodes with 20 processes per node. Below is the result<br><br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><font color="#0000ff">Testing with 160 MPI ranks<br></font><font color="#0000ff">reducing an array of size 32 (256 bytes)<br></font><font color="#0000ff">Running 5 burnin runs and 100 tests ... Done.<br></font><font color="#0000ff">For 100 runs with 5 burnin runs, on 160 MPI processes, min/max times over all ranks:<br></font><font color="#0000ff">MPI timer resolution: 1.0000e-06 seconds<br></font><font color="#0000ff">MPI timer resolution/#trials: 1.0000e-08 seconds<br></font><font color="#0000ff">B. Red. Only (min/max): 8.850098e-06 / 8.890629e-06 seconds<br></font><font color="#0000ff">N.B. Red. Only (min/max): 1.725912e-05 / 1.733065e-05 seconds<br></font><font color="#0000ff">Loc. Only (min/max): 2.364278e-04 / 2.374697e-04 seconds<br></font><font color="#0000ff">Blocking (min/max): 2.650309e-04 / 2.650595e-04 seconds<br></font><font color="#0000ff">Non-Blocking (min/max): 2.673984e-04 / 2.674508e-04 seconds<br></font><font color="#0000ff">Observe to see if the local time is enough to hide the reduction, and see if the reduction is indeed hidden</font></blockquote><br>It appears that the non-blocking computation with this test is no faster than the blocking computation.<br>I think I am missing some suitable Intel MPI environment settings.<br>I am now thinking about using MPICH, which does not require any environment settings for non-blocking computation.<br>Could you please let me know which MPI (MPICH or OpenMPI) you used in your tests?<br>Thanks again.<br>Viet<br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 25, 2021 at 7:47 PM Patrick Sanan <<a href="mailto:patrick.sanan@gmail.com">patrick.sanan@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;">Sorry about the delay in responding, but I'll add a couple of points here:<div><br></div><div><br></div><div>1) It's important to have some reason to believe that pipelining will actually help your problem. Pipelined Krylov methods work by overlapping reductions with operator and preconditioner applications. So, to see speedup, the time for a reduction needs to be comparable to the time for the operator/preconditioner application. This will only be true in some cases - typical cases are when you have a large number of ranks/nodes, a slow network, or very fast operator/preconditioner applications (assuming that these require the same time on each rank - it's an interesting case when they don't, but unless you say otherwise I'll assume this doesn't apply to your use case). <br><div><div><br></div><div>2) As you're discovering, simply ensuring that asynchronous progress works, at the pure MPI level, isn't as easy as it might be, as it's so dependent on the MPI implementation.</div><div><br></div><div><br></div><div>For both of these reasons, I suggest setting up a test that just directly uses MPI (which you can of course do from a PETSc-style code) and allows you to compare times for blocking and non-blocking reductions, overlapping some (useless) local work. You should make sure to run multiple iterations within the script, and also run the script multiple times on the cluster (bearing in mind that it's possible that the performance will be affected by other users of the system).</div><div><br></div><div>I attach an old script I found that I used to test some of these things, to give a more concrete idea of what I mean. Note that this was used early on in our own exploration of these topics so I'm only offering it to give an idea, not as a meaningful benchmark in its own right.</div><div><br></div><div></div></div></div></div><div style="overflow-wrap: break-word;"><div><div><div></div><div><blockquote type="cite"><div>Am 25.01.2021 um 09:17 schrieb Viet H.Q.H. <<a href="mailto:hqhviet@tohoku.ac.jp" target="_blank">hqhviet@tohoku.ac.jp</a>>:</div><br><div><div dir="ltr"><br><div><font>Dear Barry,</font></div><div><font><br></font></div><div><font>Thank you very much for your information.<br><br>It seems complicated to set environment variables to allow asynchronous progress and pinning threads to cores when using Intel MPI.<br><br></font></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><font color="#0000ff">$ export I_MPI_ASYNC_PROGRESS = 1</font></div><div><font color="#0000ff">$ export I_MPI_ASYNC_PROGRESS_PIN = <CPU list></font></div></blockquote><div><font><br><a href="https://techdecoded.intel.io/resources/hiding-communication-latency-using-mpi-3-non-blocking-collectives/" target="_blank">https://techdecoded.intel.io/resources/hiding-communication-latency-using-mpi-3-non-blocking-collectives/</a><br><br>I'm still not sure how to get an appropriate "CPU list" when running MPI with multiple nodes and multiple processes on one node.<br></font></div><div><span style="background-color:transparent">Best,</span><br></div><div><font>Viet.</font></div><div><font><br></font></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jan 23, 2021 at 3:01 AM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><br></div><a href="https://software.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top/additional-supported-features/asynchronous-progress-control.html" target="_blank">https://software.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top/additional-supported-features/asynchronous-progress-control.html</a><div><br></div><div>It states "<span style="color:rgb(85,85,85);font-family:intel-clear,tahoma,Helvetica,helvetica,Arial,sans-serif;font-size:16px;background-color:rgb(226,231,235)">and a partial support for non-blocking collectives ( </span><span style="box-sizing:border-box;color:rgb(85,85,85);font-family:intel-clear,tahoma,Helvetica,helvetica,Arial,sans-serif;font-size:16px">MPI_Ibcas</span><span style="color:rgb(85,85,85);font-family:intel-clear,tahoma,Helvetica,helvetica,Arial,sans-serif;font-size:16px;background-color:rgb(226,231,235)"> t, </span><span style="box-sizing:border-box;color:rgb(85,85,85);font-family:intel-clear,tahoma,Helvetica,helvetica,Arial,sans-serif;font-size:16px">MPI_Ireduce</span><span style="color:rgb(85,85,85);font-family:intel-clear,tahoma,Helvetica,helvetica,Arial,sans-serif;font-size:16px;background-color:rgb(226,231,235)"> , and </span><span style="box-sizing:border-box;color:rgb(85,85,85);font-family:intel-clear,tahoma,Helvetica,helvetica,Arial,sans-serif;font-size:16px">MPI_Iallreduce</span><span style="color:rgb(85,85,85);font-family:intel-clear,tahoma,Helvetica,helvetica,Arial,sans-serif;font-size:16px;background-color:rgb(226,231,235)"> )." I do not know what partial support means but you can try setting the variables and see if that helps.</span></div><div><font color="#555555" face="intel-clear, tahoma, Helvetica, helvetica, Arial, sans-serif" size="3"><span style="background-color:rgb(226,231,235)"><br></span></font></div><div><font color="#555555" face="intel-clear, tahoma, Helvetica, helvetica, Arial, sans-serif" size="3"><span style="background-color:rgb(226,231,235)"><br></span></font><div><br><blockquote type="cite"><div>On Jan 22, 2021, at 11:20 AM, Viet H.Q.H. <<a href="mailto:hqhviet@tohoku.ac.jp" target="_blank">hqhviet@tohoku.ac.jp</a>> wrote:</div><br><div><div dir="ltr"><div dir="ltr"><br><div>Dear Victor and Berry,<br><br>Thank you so much for your answers.<br><br>I fixed the code with the bug in the PetscCommSplitReductionBegin function as commented by Brave.<br><br> <font color="#0000ff"> ierr = PetscCommSplitReductionBegin (PetscObjectComm ((PetscObject) u));</font><br><br>It was also a mistake to set the vector size too small.<br>I just set a vector size of 100000000 and ran the code on 4 nodes with 2 processors per node. The result is as follows<br><br>The time used for the asynchronous calculation: <font color="#ff0000">0.022043</font><br>+ | u | = 10000.<br>The time used for the synchronous calculation: <font color="#ff0000">0.016188</font><br>+ | b | = 10000.<br></div><div><br></div><div>Asynchronous computation still takes a longer time.</div><div><br></div><div>I also confirmed that PETSC_HAVE_MPI_IALLREDUCE is defined in the file $PETSC_DIR/include/petscconf.h<br><br>I built Petsc by using the following script</div><div><br></div><div><font color="#0000ff">#!/usr/bin/bash<br>set -e<br>DATE="21.01.18"<br>MPIIT_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mpi/intel64"<br>MKL_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mkl"<br>INSTL_DIR="${HOME}/local/petsc-3.14.3"<br>BUILD_DIR="${HOME}/tmp/petsc/build_${DATE}"<br>PETSC_DIR="${HOME}/tmp/petsc"<br><br>cd ${PETSC_DIR}<br>./configure --force --prefix=${INSTL_DIR} --with-mpi-dir=${MPIIT_DIR} --with-fortran-bindings=0 --with-mpiexe=${MPIIT_DIR}/bin/mpiexec --with-valgrind-dir=${HOME}/local/valgrind --with-blaslapack-dir=${MKL_DIR} --download-make --with-debugging=0 COPTFLAGS='-O3 -march=native -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 -march=native -mtune=native'<br><br>make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt all<br>make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt install </font><br></div><div><br><br>Intel 2018 also complies with the MPI-3 standard.<br><br>Are there specific settings for Intel MPI to obtain the performance of the MPI_IALLREDUCE function?<br></div><div><br></div><div>Sincerely,</div><div>Viet.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 22, 2021 at 11:20 AM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><br></div><span style="color:rgb(0,0,255)"> </span><span> ierr = VecNormBegin(u,NORM_2,&norm1);</span><br><span> ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); </span><div><span><br></span></div><div><span>How come you call this on Ax and not on u? For clarity, if nothing else, I think you should call it on u.</span></div><div><span><br></span></div><div><span>comb.c has </span></div><div><span><br></span></div><div><div>/*</div><div> Split phase global vector reductions with support for combining the</div><div> communication portion of several operations. Using MPI-1.1 support only</div><div><br></div><div> The idea for this and much of the initial code is contributed by</div><div> Victor Eijkhout.</div><div><br></div><div> Usage:</div><div> VecDotBegin(Vec,Vec,PetscScalar *);</div><div> VecNormBegin(Vec,NormType,PetscReal *);</div><div> ....</div><div> VecDotEnd(Vec,Vec,PetscScalar *);</div><div> VecNormEnd(Vec,NormType,PetscReal *);</div><div><br></div><div> Limitations:</div><div> - The order of the xxxEnd() functions MUST be in the same order</div><div> as the xxxBegin(). There is extensive error checking to try to</div><div> insure that the user calls the routines in the correct order</div><div>*/</div><div><br></div><div>#include <petsc/private/vecimpl.h> /*I "petscvec.h" I*/</div><div><br></div><div>static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm comm,MPI_Request *request)</div><div>{</div><div> PETSC_UNUSED PetscErrorCode ierr;</div><div><br></div><div> PetscFunctionBegin;</div><div>#if defined(PETSC_HAVE_MPI_IALLREDUCE)</div><div> ierr = MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr);</div><div>#elif defined(PETSC_HAVE_MPIX_IALLREDUCE)</div><div> ierr = MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr);</div><div>#else</div><div> ierr = MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr);</div><div> *request = MPI_REQUEST_NULL;</div><div>#endif</div><div> PetscFunctionReturn(0);</div><div>}</div><div style="color:rgb(0,0,255)"><br></div></div><div><font color="#0000ff"><span><br></span></font><div>So first check if $PETSC_DIR/include/petscconf.h has </div><div><br></div><div><span>PETSC_HAVE_MPI_IALLREDUCE</span></div><div><span><br></span></div><div><span>if it does not then the standard MPI reduce is called. </span></div><div><span><br></span></div><div><span>If this is set then any improvement depends on the implementation of iallreduce inside the MPI you are using. </span></div><div><span><br></span></div><div><span>Barry</span></div><div><span><br></span></div><div><font color="#0000ff"><span><br></span></font><blockquote type="cite"><div>On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. <<a href="mailto:hqhviet@tohoku.ac.jp" target="_blank">hqhviet@tohoku.ac.jp</a>> wrote:</div><br><div><div dir="ltr"><div><br></div><div>Hello Petsc developers and supporters,<br><br>I would like to confirm the performance of asynchronous computations of inner product computation overlapping with matrix-vector multiplication computation by the below code.<br></div><div><br></div><div><br></div><font color="#0000ff"> PetscLogDouble tt1,tt2;<br> KSP ksp;<br> //ierr = VecSet(c,one);<br> ierr = VecSet(c,one);<br> ierr = VecSet(u,one);<br> ierr = VecSet(b,one);<br><br> ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr);<br> ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr);<br><br><br> ierr = PetscTime(&tt1);CHKERRQ(ierr);<br> ierr = VecNormBegin(u,NORM_2,&norm1);<br> ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); <br> ierr = KSP_MatMult(ksp,A,c,Ac); <br> ierr = VecNormEnd(u,NORM_2,&norm1);<br> ierr = PetscTime(&tt2);CHKERRQ(ierr);<br><br> ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr);<br> ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) norm1); CHKERRQ(ierr);<br><br><br> ierr = PetscTime(&tt1);CHKERRQ(ierr);<br> ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr);<br> ierr = KSP_MatMult(ksp,A,c,Ac); <br> ierr = PetscTime(&tt2);CHKERRQ(ierr);<br><br> ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr);<br> ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) norm2); CHKERRQ(ierr);<br></font><div><font color="#0000ff"><br></font></div><div><br></div><div>On a cluster with two or four nodes, the asynchronous computation is always much slower than synchronous computation.<br><br><font color="#ff0000">The time used for the asynchronous calculation: 0.000203<br>+ |u| = 100.<br>The time used for the synchronous calculation: 0.000006<br>+ |b| = 100.</font><br><br>Are there any necessary settings on MPI or Petsc to gain performance of asynchronous computation?<br><br>Thank you very much for anything you can provide.<br>Sincerely,<br>Viet.<br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div>
</div></blockquote></div><br></div></div></blockquote></div></div>
</div></blockquote></div><br></div></div></blockquote></div>
</div></blockquote></div><br></div></div></div></blockquote></div>