<div dir="ltr"><div>Hi Mark,</div><div><br></div><div>I'm glad to see you trying out the AIJMKL stuff. I think you are the first person trying to actually use it, so we are probably going to expose some bugs and also some performance issues. My somewhat limited testing has shown that the MKL sparse routines often perform worse than our own implementations in PETSc. We need to systematically look at this and let the MKL team know where performance is lagging.</div><div><br></div><div>I'm able to run some GAMG examples with AIJMKL matrices if I use type MPIAIJ for my "parallel" matrices but SEQAIJMKL for the "sequential" matrices that MPIAIJ uses. Does your example work if you provide the option "-mat_seqaij_type seqaijmkl"?</div><div><br></div><div>--Richard<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 3, 2018 at 6:28 AM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div dir="ltr">
<div>GAMG drills into AIJ data structures and will need to be fixed up to work with MKL matrices, I guess, but it is failing now from a logic error.</div>
<div><br>
</div>
<div>This example works with one processor but fails with 2 (appended). The code looks like this:</div>
<div><br>
</div>
<div>
<div> ierr = PetscObjectBaseTypeCompare((<wbr>PetscObject)Amat,MATSEQAIJ,&<wbr>isseqaij);CHKERRQ(ierr);</div>
<div> ierr = PetscObjectBaseTypeCompare((<wbr>PetscObject)Amat,MATMPIAIJ,&<wbr>ismpiaij);CHKERRQ(ierr);</div>
<div> if (isseqaij) {</div>
<div> ....</div>
<div> } else if (ismpiaij) {</div>
<div> .....</div>
<div> ierr = MatMPIAIJGetSeqAIJ(Amat,&Daij,<wbr>&Oaij,&garray);CHKERRQ(ierr);</div>
</div>
<div><br>
</div>
<div>The failure below is from this call to <span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">MatMPIAIJGetSeqAIJ, on this line:</span></div>
<div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br>
</span></div>
<div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">ierr = PetscObjectTypeCompare((<wbr>PetscObject)A,MATMPIAIJ,&flg);<wbr>CHKERRQ(ierr);<br>
</span></div>
<div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br>
</span></div>
<div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">The difference is <span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">PetscObjectTypeCompare
vs <span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">PetscObject<b>Base</b>TypeCompare. GAMG could use <span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">PetscObjectTypeCompare
or <span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">MatMPIAIJGetSeqAIJ
<span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">
could </span>use <span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">PetscObject</span><b style="text-decoration-style:initial;text-decoration-color:initial">Base</b><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">TypeC<wbr>ompare
...</span></span></span></span></span></span></div>
<div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br>
</span></span></span></span></span></span></div>
<div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Mark</span></span></span></span></span></span></div>
<div><br>
</div>
<div>srun -n 2 ./ex19 -pc_type gamg -snes_monitor_short -ksp_monitor_short </div>
<div>lid velocity = 0.0625, prandtl # = 1., grashof # = 1. </div>
<div> 0 SNES Function norm 0.239155 </div>
<div>[0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>-- </div>
<div>[0]PETSC ERROR: No support for this operation for this object type </div>
<div>[0]PETSC ERROR: This function requires a MATMPIAIJ matrix as input </div>
<div>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">
http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html</a> for trouble shooting. </div>
<div>[0]PETSC ERROR: Petsc Development GIT revision: v3.9.2-825-g3a11c7608d GIT Date: 2018-07-01 06:15:09 +0200 </div>
<div>[0]PETSC ERROR: /global/u2/m/madams/petsc_<wbr>install/petsc/src/snes/<wbr>examples/tutorials/./ex19 on a named nid02516 by madams Tue Jul 3 04:58:13 2018 </div>
<div>[0]PETSC ERROR: Configure options --known-level1-dcache-size=<wbr>32768 --known-level1-dcache-<wbr>linesize=64 --known-level1-dcache-assoc=8 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8
--known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-memcmp-ok=1 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-mklspblas-supports-<wbr>zero-based=0
--known-has-attribute-aligned=<wbr>1 --with-cc=cc --with-cxx=CC --with-fc=ftn COPTFLAGS=" -g -O1 -mkl -static-intel" CXXOPTFLAGS="-g -O1 -mkl -static-intel" FOPTFLAGS=" -g -O1 -mkl -static-intel" --download-metis=1 --with-hypre-dir=/global/<wbr>homes/m/madams/tmp/hypre32-2.<wbr>14.0
--download-parmetis=1 LIBS="-L/opt/intel/compilers_<wbr>and_libraries_2018.1.163/<wbr>linux/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread" --with-blaslapack-include=/<wbr>opt/intel/compilers_and_<wbr>libraries_2018.1.163/linux/<wbr>mkl/include --with-debugging=0
--with-mpiexec=srun --with-batch=1 --known-mpi-shared-libraries=1 --known-64-bit-blas-indices=0 --with-64-bit-indices=0 PETSC_ARCH=arch-cori-knl-opt-<wbr>intel-omp --with-openmp=1 --download-p4est=0 --with-x=0 --prefix=/global/homes/m/<wbr>madams/petsc_install/petsc-<wbr>cori-knl-opt-intel-omp
PETSC_DIR=/global/homes/m/<wbr>madams/petsc_install/petsc</div>
<div>[0]PETSC ERROR: #1 MatMPIAIJGetSeqAIJ() line 4328 in /global/u2/m/madams/petsc_<wbr>install/petsc/src/mat/impls/<wbr>aij/mpi/mpiaij.c</div>
<div>[0]PETSC ERROR: #2 PCGAMGCreateGraph() line 118 in /global/u2/m/madams/petsc_<wbr>install/petsc/src/ksp/pc/<wbr>impls/gamg/util.c</div>
<div>[0]PETSC ERROR: #3 PCGAMGGraph_AGG() line 832 in /global/u2/m/madams/petsc_<wbr>install/petsc/src/ksp/pc/<wbr>impls/gamg/agg.c</div>
<div>[0]PETSC ERROR: #4 PCSetUp_GAMG() line 517 in /global/u2/m/madams/petsc_<wbr>install/petsc/src/ksp/pc/<wbr>impls/gamg/gamg.c</div>
<div>[0]PETSC ERROR: #5 PCSetUp() line 932 in /global/u2/m/madams/petsc_<wbr>install/petsc/src/ksp/pc/<wbr>interface/precon.c</div>
<div>[0]PETSC ERROR: #6 KSPSetUp() line 381 in /global/u2/m/madams/petsc_<wbr>install/petsc/src/ksp/ksp/<wbr>interface/itfunc.c</div>
<div>[0]PETSC ERROR: #7 KSPSolve() line 612 in /global/u2/m/madams/petsc_<wbr>install/petsc/src/ksp/ksp/<wbr>interface/itfunc.c</div>
<div>[0]PETSC ERROR: #8 SNESSolve_NEWTONLS() line 224 in /global/u2/m/madams/petsc_<wbr>install/petsc/src/snes/impls/<wbr>ls/ls.c</div>
<div>[0]PETSC ERROR: #9 SNESSolve() line 4350 in /global/u2/m/madams/petsc_<wbr>install/petsc/src/snes/<wbr>interface/snes.c</div>
<div>[0]PETSC ERROR: #10 main() line 161 in /global/homes/m/madams/petsc_<wbr>install/petsc/src/snes/<wbr>examples/tutorials/ex19.c</div>
<div>[0]PETSC ERROR: PETSc Option Table entries:</div>
<div>[0]PETSC ERROR: -ksp_monitor_short</div>
<div>[0]PETSC ERROR: -mat_type aijmkl</div>
<div>[0]PETSC ERROR: -options_left</div>
<div>[0]PETSC ERROR: -pc_type gamg</div>
<div>[0]PETSC ERROR: -snes_monitor_short</div>
<div><br>
</div>
</div>
</div>
</blockquote></div><br></div>