<div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Wed, Jul 4, 2018 at 4:51 PM Jeff Hammond <<a href="mailto:jeff.science@gmail.com">jeff.science@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>On Wed, Jul 4, 2018 at 6:31 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Tue, Jul 3, 2018 at 10:32 PM Jeff Hammond <<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><br></div><div><br><div class="gmail_quote"><div dir="ltr">On Tue, Jul 3, 2018 at 4:35 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Tue, Jul 3, 2018 at 1:00 PM Richard Tran Mills <<a href="mailto:rtmills@anl.gov" target="_blank">rtmills@anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi Mark,</div><div><br></div><div>I'm glad to see you trying out the AIJMKL stuff. I think you are the first person trying to actually use it, so we are probably going to expose some bugs and also some performance issues. My somewhat limited testing has shown that the MKL sparse routines often perform worse than our own implementations in PETSc. </div></div></blockquote><div><br></div></div></div><div dir="ltr"><div class="gmail_quote"><div>My users just want OpenMP.</div></div></div><div dir="ltr"><div class="gmail_quote"><div> </div></div></div></blockquote><div dir="auto"><br></div><div dir="auto">Why not just add OpenMP to PETSc? I know certain developers hate it, but it is silly to let a principled objection stand in the way of enabling users</div></div></div></blockquote><div> </div><div> "if that would deliver the best performance for NERSC users."</div><div><br></div></div></div><div dir="ltr"><div class="gmail_quote"><div>You have answered your own question.</div><div></div></div></div></blockquote><div dir="auto"><br></div><div dir="auto">Please share the results of your experiments that prove OpenMP does not improve performance for Mark’s users. </div></div></div></blockquote><div><br></div><div>Oh God. I am supremely uninterested in minutely proving yet again that OpenMP is not better than MPI.</div><div>There are already countless experiments. One more will not add anything of merit.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div>Also we are not in the habit of fucking up our codebase in order to follow some fad.</div><div></div></div></div></blockquote><div dir="auto"><br></div><div dir="auto">If you can’t use OpenMP without messing up your code base, you probably don’t know how to design software.</div></div></div></blockquote><div><br></div><div>That is an interesting, if wrong, opinion. It might be your contention that sticking any random paradigm in a library should</div><div>be alright if its "well designed"? I have never encountered such a well-designed library.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="gmail_quote"><div dir="auto"> I guess if you refuse to use _Pragma because C99 is still a fad for you, it is harder, but clearly _Complex is tolerated.</div></div></div></blockquote><div><br></div><div>Yes, littering your code with preprocessor directives improves almost everything. Doing proper resource management</div><div>using Pragmas, in an environment with several layers of libraries, is a dream.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="gmail_quote"><div dir="auto">More seriously, you’ve adopted OpenMP hidden behind MKL </div></div></div></blockquote><div><br></div><div>Nope. We can use MKL with that crap shutoff.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="gmail_quote"><div dir="auto">so I see no reason why you can’t wrap OpenMP implementations of the PETSc sparse kernels in a similar manner. </div></div></div></blockquote><div><br></div><div>We could, its just a colossal waste of time and effort, as well as counterproductive for the codebase :)</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="gmail_quote"><div dir="auto">Jeff</div><div dir="auto"><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><br></div><div> Matt</div></div></div><div dir="ltr"><div class="gmail_quote"><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="gmail_quote"><div dir="auto"><br></div><div dir="auto">Jeff</div><div dir="auto"><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>We need to systematically look at this and let the MKL team know where performance is lagging.</div><div><br></div><div>I'm able to run some GAMG examples with AIJMKL matrices if I use type MPIAIJ for my "parallel" matrices but SEQAIJMKL for the "sequential" matrices that MPIAIJ uses. Does your example work if you provide the option "-mat_seqaij_type seqaijmkl"?</div></div></blockquote><div><br></div></div></div><div dir="ltr"><div class="gmail_quote"><div>Humm, maybe that is a better way to do it. I'll try it. I've been trying to keep the MPI matrix as an MKL matrix but I guess that is not necessary.</div></div></div><div dir="ltr"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>--Richard<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 3, 2018 at 6:28 AM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div dir="ltr">
<div>GAMG drills into AIJ data structures and will need to be fixed up to work with MKL matrices, I guess, but it is failing now from a logic error.</div>
<div><br>
</div>
<div>This example works with one processor but fails with 2 (appended). The code looks like this:</div>
<div><br>
</div>
<div>
<div> ierr = PetscObjectBaseTypeCompare((PetscObject)Amat,MATSEQAIJ,&isseqaij);CHKERRQ(ierr);</div>
<div> ierr = PetscObjectBaseTypeCompare((PetscObject)Amat,MATMPIAIJ,&ismpiaij);CHKERRQ(ierr);</div>
<div> if (isseqaij) {</div>
<div> ....</div>
<div> } else if (ismpiaij) {</div>
<div> .....</div>
<div> ierr = MatMPIAIJGetSeqAIJ(Amat,&Daij,&Oaij,&garray);CHKERRQ(ierr);</div>
</div>
<div><br>
</div>
<div>The failure below is from this call to <span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">MatMPIAIJGetSeqAIJ, on this line:</span></div>
<div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br>
</span></div>
<div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">ierr = PetscObjectTypeCompare((PetscObject)A,MATMPIAIJ,&flg);CHKERRQ(ierr);<br>
</span></div>
<div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br>
</span></div>
<div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">The difference is <span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">PetscObjectTypeCompare
vs <span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">PetscObject<b>Base</b>TypeCompare. GAMG could use <span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">PetscObjectTypeCompare
or <span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">MatMPIAIJGetSeqAIJ
<span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">
could </span>use <span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">PetscObject</span><b style="text-decoration-style:initial;text-decoration-color:initial">Base</b><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">TypeCompare
...</span></span></span></span></span></span></div>
<div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br>
</span></span></span></span></span></span></div>
<div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Mark</span></span></span></span></span></span></div>
<div><br>
</div>
<div>srun -n 2 ./ex19 -pc_type gamg -snes_monitor_short -ksp_monitor_short </div>
<div>lid velocity = 0.0625, prandtl # = 1., grashof # = 1. </div>
<div> 0 SNES Function norm 0.239155 </div>
<div>[0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- </div>
<div>[0]PETSC ERROR: No support for this operation for this object type </div>
<div>[0]PETSC ERROR: This function requires a MATMPIAIJ matrix as input </div>
<div>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">
http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting. </div>
<div>[0]PETSC ERROR: Petsc Development GIT revision: v3.9.2-825-g3a11c7608d GIT Date: 2018-07-01 06:15:09 +0200 </div>
<div>[0]PETSC ERROR: /global/u2/m/madams/petsc_install/petsc/src/snes/examples/tutorials/./ex19 on a named nid02516 by madams Tue Jul 3 04:58:13 2018 </div>
<div>[0]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=64 --known-level1-dcache-assoc=8 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8
--known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-memcmp-ok=1 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-mklspblas-supports-zero-based=0
--known-has-attribute-aligned=1 --with-cc=cc --with-cxx=CC --with-fc=ftn COPTFLAGS=" -g -O1 -mkl -static-intel" CXXOPTFLAGS="-g -O1 -mkl -static-intel" FOPTFLAGS=" -g -O1 -mkl -static-intel" --download-metis=1 --with-hypre-dir=/global/homes/m/madams/tmp/hypre32-2.14.0
--download-parmetis=1 LIBS="-L/opt/intel/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread" --with-blaslapack-include=/opt/intel/compilers_and_libraries_2018.1.163/linux/mkl/include --with-debugging=0
--with-mpiexec=srun --with-batch=1 --known-mpi-shared-libraries=1 --known-64-bit-blas-indices=0 --with-64-bit-indices=0 PETSC_ARCH=arch-cori-knl-opt-intel-omp --with-openmp=1 --download-p4est=0 --with-x=0 --prefix=/global/homes/m/madams/petsc_install/petsc-cori-knl-opt-intel-omp
PETSC_DIR=/global/homes/m/madams/petsc_install/petsc</div>
<div>[0]PETSC ERROR: #1 MatMPIAIJGetSeqAIJ() line 4328 in /global/u2/m/madams/petsc_install/petsc/src/mat/impls/aij/mpi/mpiaij.c</div>
<div>[0]PETSC ERROR: #2 PCGAMGCreateGraph() line 118 in /global/u2/m/madams/petsc_install/petsc/src/ksp/pc/impls/gamg/util.c</div>
<div>[0]PETSC ERROR: #3 PCGAMGGraph_AGG() line 832 in /global/u2/m/madams/petsc_install/petsc/src/ksp/pc/impls/gamg/agg.c</div>
<div>[0]PETSC ERROR: #4 PCSetUp_GAMG() line 517 in /global/u2/m/madams/petsc_install/petsc/src/ksp/pc/impls/gamg/gamg.c</div>
<div>[0]PETSC ERROR: #5 PCSetUp() line 932 in /global/u2/m/madams/petsc_install/petsc/src/ksp/pc/interface/precon.c</div>
<div>[0]PETSC ERROR: #6 KSPSetUp() line 381 in /global/u2/m/madams/petsc_install/petsc/src/ksp/ksp/interface/itfunc.c</div>
<div>[0]PETSC ERROR: #7 KSPSolve() line 612 in /global/u2/m/madams/petsc_install/petsc/src/ksp/ksp/interface/itfunc.c</div>
<div>[0]PETSC ERROR: #8 SNESSolve_NEWTONLS() line 224 in /global/u2/m/madams/petsc_install/petsc/src/snes/impls/ls/ls.c</div>
<div>[0]PETSC ERROR: #9 SNESSolve() line 4350 in /global/u2/m/madams/petsc_install/petsc/src/snes/interface/snes.c</div>
<div>[0]PETSC ERROR: #10 main() line 161 in /global/homes/m/madams/petsc_install/petsc/src/snes/examples/tutorials/ex19.c</div>
<div>[0]PETSC ERROR: PETSc Option Table entries:</div>
<div>[0]PETSC ERROR: -ksp_monitor_short</div>
<div>[0]PETSC ERROR: -mat_type aijmkl</div>
<div>[0]PETSC ERROR: -options_left</div>
<div>[0]PETSC ERROR: -pc_type gamg</div>
<div>[0]PETSC ERROR: -snes_monitor_short</div>
<div><br>
</div>
</div>
</div>
</blockquote></div><br></div>
</blockquote></div></div></blockquote></div></div>-- <br><div dir="ltr" class="m_6911183743079912772m_7487948708552762844m_-2839219068731168839gmail_signature" data-smartmail="gmail_signature">Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>
</blockquote></div></div><div dir="ltr"><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_6911183743079912772m_7487948708552762844gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></blockquote></div></div>-- <br><div dir="ltr" class="m_6911183743079912772gmail_signature" data-smartmail="gmail_signature">Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div>