<div dir="ltr"><br><div><font color="#000000">Dear Barry,</font></div><div><font color="#000000"><br></font></div><div><font color="#000000">Thank you very much for your information.<br><br>It seems complicated to set environment variables to allow asynchronous progress and pinning threads to cores when using Intel MPI.<br><br></font></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><font color="#0000ff">$ export I_MPI_ASYNC_PROGRESS = 1</font></div><div><font color="#0000ff">$ export I_MPI_ASYNC_PROGRESS_PIN = <CPU list></font></div></blockquote><div><font color="#000000"><br><a href="https://techdecoded.intel.io/resources/hiding-communication-latency-using-mpi-3-non-blocking-collectives/">https://techdecoded.intel.io/resources/hiding-communication-latency-using-mpi-3-non-blocking-collectives/</a><br><br>I'm still not sure how to get an appropriate "CPU list" when running MPI with multiple nodes and multiple processes on one node.<br></font></div><div><span style="color:rgb(0,0,0);background-color:transparent">Best,</span><br></div><div><font color="#000000">Viet.</font></div><div><font color="#000000"><br></font></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jan 23, 2021 at 3:01 AM Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div><br></div><a href="https://software.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top/additional-supported-features/asynchronous-progress-control.html" target="_blank">https://software.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top/additional-supported-features/asynchronous-progress-control.html</a><div><br></div><div>It states "<span style="color:rgb(85,85,85);font-family:intel-clear,tahoma,Helvetica,helvetica,Arial,sans-serif;font-size:16px;background-color:rgb(226,231,235)">and a partial support for non-blocking collectives ( </span><span style="box-sizing:border-box;color:rgb(85,85,85);font-family:intel-clear,tahoma,Helvetica,helvetica,Arial,sans-serif;font-size:16px">MPI_Ibcas</span><span style="color:rgb(85,85,85);font-family:intel-clear,tahoma,Helvetica,helvetica,Arial,sans-serif;font-size:16px;background-color:rgb(226,231,235)"> t, </span><span style="box-sizing:border-box;color:rgb(85,85,85);font-family:intel-clear,tahoma,Helvetica,helvetica,Arial,sans-serif;font-size:16px">MPI_Ireduce</span><span style="color:rgb(85,85,85);font-family:intel-clear,tahoma,Helvetica,helvetica,Arial,sans-serif;font-size:16px;background-color:rgb(226,231,235)"> , and </span><span style="box-sizing:border-box;color:rgb(85,85,85);font-family:intel-clear,tahoma,Helvetica,helvetica,Arial,sans-serif;font-size:16px">MPI_Iallreduce</span><span style="color:rgb(85,85,85);font-family:intel-clear,tahoma,Helvetica,helvetica,Arial,sans-serif;font-size:16px;background-color:rgb(226,231,235)"> )." I do not know what partial support means but you can try setting the variables and see if that helps.</span></div><div><font color="#555555" face="intel-clear, tahoma, Helvetica, helvetica, Arial, sans-serif" size="3"><span style="background-color:rgb(226,231,235)"><br></span></font></div><div><font color="#555555" face="intel-clear, tahoma, Helvetica, helvetica, Arial, sans-serif" size="3"><span style="background-color:rgb(226,231,235)"><br></span></font><div><br><blockquote type="cite"><div>On Jan 22, 2021, at 11:20 AM, Viet H.Q.H. <<a href="mailto:hqhviet@tohoku.ac.jp" target="_blank">hqhviet@tohoku.ac.jp</a>> wrote:</div><br><div><div dir="ltr"><div dir="ltr"><br><div>Dear Victor and Berry,<br><br>Thank you so much for your answers.<br><br>I fixed the code with the bug in the PetscCommSplitReductionBegin function as commented by Brave.<br><br> <font color="#0000ff"> ierr = PetscCommSplitReductionBegin (PetscObjectComm ((PetscObject) u));</font><br><br>It was also a mistake to set the vector size too small.<br>I just set a vector size of 100000000 and ran the code on 4 nodes with 2 processors per node. The result is as follows<br><br>The time used for the asynchronous calculation: <font color="#ff0000">0.022043</font><br>+ | u | = 10000.<br>The time used for the synchronous calculation: <font color="#ff0000">0.016188</font><br>+ | b | = 10000.<br></div><div><br></div><div>Asynchronous computation still takes a longer time.</div><div><br></div><div>I also confirmed that PETSC_HAVE_MPI_IALLREDUCE is defined in the file $PETSC_DIR/include/petscconf.h<br><br>I built Petsc by using the following script</div><div><br></div><div><font color="#0000ff">#!/usr/bin/bash<br>set -e<br>DATE="21.01.18"<br>MPIIT_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mpi/intel64"<br>MKL_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mkl"<br>INSTL_DIR="${HOME}/local/petsc-3.14.3"<br>BUILD_DIR="${HOME}/tmp/petsc/build_${DATE}"<br>PETSC_DIR="${HOME}/tmp/petsc"<br><br>cd ${PETSC_DIR}<br>./configure --force --prefix=${INSTL_DIR} --with-mpi-dir=${MPIIT_DIR} --with-fortran-bindings=0 --with-mpiexe=${MPIIT_DIR}/bin/mpiexec --with-valgrind-dir=${HOME}/local/valgrind --with-blaslapack-dir=${MKL_DIR} --download-make --with-debugging=0 COPTFLAGS='-O3 -march=native -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 -march=native -mtune=native'<br><br>make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt all<br>make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt install </font><br></div><div><br><br>Intel 2018 also complies with the MPI-3 standard.<br><br>Are there specific settings for Intel MPI to obtain the performance of the MPI_IALLREDUCE function?<br></div><div><br></div><div>Sincerely,</div><div>Viet.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 22, 2021 at 11:20 AM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><br></div><span style="color:rgb(0,0,255)"> </span><span> ierr = VecNormBegin(u,NORM_2,&norm1);</span><br><span> ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); </span><div><span><br></span></div><div><span>How come you call this on Ax and not on u? For clarity, if nothing else, I think you should call it on u.</span></div><div><span><br></span></div><div><span>comb.c has </span></div><div><span><br></span></div><div><div>/*</div><div> Split phase global vector reductions with support for combining the</div><div> communication portion of several operations. Using MPI-1.1 support only</div><div><br></div><div> The idea for this and much of the initial code is contributed by</div><div> Victor Eijkhout.</div><div><br></div><div> Usage:</div><div> VecDotBegin(Vec,Vec,PetscScalar *);</div><div> VecNormBegin(Vec,NormType,PetscReal *);</div><div> ....</div><div> VecDotEnd(Vec,Vec,PetscScalar *);</div><div> VecNormEnd(Vec,NormType,PetscReal *);</div><div><br></div><div> Limitations:</div><div> - The order of the xxxEnd() functions MUST be in the same order</div><div> as the xxxBegin(). There is extensive error checking to try to</div><div> insure that the user calls the routines in the correct order</div><div>*/</div><div><br></div><div>#include <petsc/private/vecimpl.h> /*I "petscvec.h" I*/</div><div><br></div><div>static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm comm,MPI_Request *request)</div><div>{</div><div> PETSC_UNUSED PetscErrorCode ierr;</div><div><br></div><div> PetscFunctionBegin;</div><div>#if defined(PETSC_HAVE_MPI_IALLREDUCE)</div><div> ierr = MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr);</div><div>#elif defined(PETSC_HAVE_MPIX_IALLREDUCE)</div><div> ierr = MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr);</div><div>#else</div><div> ierr = MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr);</div><div> *request = MPI_REQUEST_NULL;</div><div>#endif</div><div> PetscFunctionReturn(0);</div><div>}</div><div style="color:rgb(0,0,255)"><br></div></div><div><font color="#0000ff"><span><br></span></font><div>So first check if $PETSC_DIR/include/petscconf.h has </div><div><br></div><div><span>PETSC_HAVE_MPI_IALLREDUCE</span></div><div><span><br></span></div><div><span>if it does not then the standard MPI reduce is called. </span></div><div><span><br></span></div><div><span>If this is set then any improvement depends on the implementation of iallreduce inside the MPI you are using. </span></div><div><span><br></span></div><div><span>Barry</span></div><div><span><br></span></div><div><font color="#0000ff"><span><br></span></font><blockquote type="cite"><div>On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. <<a href="mailto:hqhviet@tohoku.ac.jp" target="_blank">hqhviet@tohoku.ac.jp</a>> wrote:</div><br><div><div dir="ltr"><div><br></div><div>Hello Petsc developers and supporters,<br><br>I would like to confirm the performance of asynchronous computations of inner product computation overlapping with matrix-vector multiplication computation by the below code.<br></div><div><br></div><div><br></div><font color="#0000ff"> PetscLogDouble tt1,tt2;<br> KSP ksp;<br> //ierr = VecSet(c,one);<br> ierr = VecSet(c,one);<br> ierr = VecSet(u,one);<br> ierr = VecSet(b,one);<br><br> ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr);<br> ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr);<br><br><br> ierr = PetscTime(&tt1);CHKERRQ(ierr);<br> ierr = VecNormBegin(u,NORM_2,&norm1);<br> ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); <br> ierr = KSP_MatMult(ksp,A,c,Ac); <br> ierr = VecNormEnd(u,NORM_2,&norm1);<br> ierr = PetscTime(&tt2);CHKERRQ(ierr);<br><br> ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr);<br> ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) norm1); CHKERRQ(ierr);<br><br><br> ierr = PetscTime(&tt1);CHKERRQ(ierr);<br> ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr);<br> ierr = KSP_MatMult(ksp,A,c,Ac); <br> ierr = PetscTime(&tt2);CHKERRQ(ierr);<br><br> ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr);<br> ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) norm2); CHKERRQ(ierr);<br></font><div><font color="#0000ff"><br></font></div><div><br></div><div>On a cluster with two or four nodes, the asynchronous computation is always much slower than synchronous computation.<br><br><font color="#ff0000">The time used for the asynchronous calculation: 0.000203<br>+ |u| = 100.<br>The time used for the synchronous calculation: 0.000006<br>+ |b| = 100.</font><br><br>Are there any necessary settings on MPI or Petsc to gain performance of asynchronous computation?<br><br>Thank you very much for anything you can provide.<br>Sincerely,<br>Viet.<br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div>
</div></blockquote></div><br></div></div></blockquote></div></div>
</div></blockquote></div><br></div></div></blockquote></div>