[petsc-dev] GAMG error with MKL

Matthew Knepley knepley at gmail.com
Wed Jul 4 18:44:36 CDT 2018 

On Wed, Jul 4, 2018 at 4:51 PM Jeff Hammond <jeff.science at gmail.com> wrote:

> On Wed, Jul 4, 2018 at 6:31 AM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Tue, Jul 3, 2018 at 10:32 PM Jeff Hammond <jeff.science at gmail.com>
>> wrote:
>>
>>>
>>>
>>> On Tue, Jul 3, 2018 at 4:35 PM Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>> On Tue, Jul 3, 2018 at 1:00 PM Richard Tran Mills <rtmills at anl.gov>
>>>> wrote:
>>>>
>>>>> Hi Mark,
>>>>>
>>>>> I'm glad to see you trying out the AIJMKL stuff. I think you are the
>>>>> first person trying to actually use it, so we are probably going to expose
>>>>> some bugs and also some performance issues. My somewhat limited testing has
>>>>> shown that the MKL sparse routines often perform worse than our own
>>>>> implementations in PETSc.
>>>>>
>>>>
>>>> My users just want OpenMP.
>>>>
>>>>
>>>
>>> Why not just add OpenMP to PETSc? I know certain developers hate it, but
>>> it is silly to let a principled objection stand in the way of enabling users
>>>
>>
>>  "if that would deliver the best performance for NERSC users."
>>
>> You have answered your own question.
>>
>
> Please share the results of your experiments that prove OpenMP does not
> improve performance for Mark’s users.
>

Oh God. I am supremely uninterested in minutely proving yet again that
OpenMP is not better than MPI.
There are already countless experiments. One more will not add anything of
merit.


> Also we are not in the habit of fucking up our codebase in order to follow
>> some fad.
>>
>
> If you can’t use OpenMP without messing up your code base, you probably
> don’t know how to design software.
>

That is an interesting, if wrong, opinion. It might be your contention that
sticking any random paradigm in a library should
be alright if its "well designed"? I have never encountered such a
well-designed library.


> I guess if you refuse to use _Pragma because C99 is still a fad for you,
> it is harder, but clearly _Complex is tolerated.
>

Yes, littering your code with preprocessor directives improves almost
everything. Doing proper resource management
using Pragmas, in an environment with several layers of libraries, is a
dream.


> More seriously, you’ve adopted OpenMP hidden behind MKL
>

Nope. We can use MKL with that crap shutoff.


> so I see no reason why you can’t wrap OpenMP implementations of the PETSc
> sparse kernels in a similar manner.
>

We could, its just a colossal waste of time and effort, as well as
counterproductive for the codebase :)

  Matt


> Jeff
>
>
>>     Matt
>>
>>
>>> Jeff
>>>
>>> We need to systematically look at this and let the MKL team know where
>>>>> performance is lagging.
>>>>>
>>>>> I'm able to run some GAMG examples with AIJMKL matrices if I use type
>>>>> MPIAIJ for my "parallel" matrices but SEQAIJMKL for the "sequential"
>>>>> matrices that MPIAIJ uses. Does your example work if you provide the option
>>>>> "-mat_seqaij_type seqaijmkl"?
>>>>>
>>>>
>>>> Humm, maybe that is a better way to do it. I'll try it. I've been
>>>> trying to keep the MPI matrix as an MKL matrix but I guess that is not
>>>> necessary.
>>>>
>>>>
>>>>>
>>>>> --Richard
>>>>>
>>>>> On Tue, Jul 3, 2018 at 6:28 AM, Mark Adams <mfadams at lbl.gov> wrote:
>>>>>
>>>>>> GAMG drills into AIJ data structures and will need to be fixed up to
>>>>>> work with MKL matrices, I guess, but it is failing now from a logic error.
>>>>>>
>>>>>> This example works with one processor but fails with 2 (appended).
>>>>>> The code looks like this:
>>>>>>
>>>>>>     ierr =
>>>>>> PetscObjectBaseTypeCompare((PetscObject)Amat,MATSEQAIJ,&isseqaij);CHKERRQ(ierr);
>>>>>>     ierr =
>>>>>> PetscObjectBaseTypeCompare((PetscObject)Amat,MATMPIAIJ,&ismpiaij);CHKERRQ(ierr);
>>>>>>     if (isseqaij) {
>>>>>>        ....
>>>>>>     } else if (ismpiaij) {
>>>>>>       .....
>>>>>>       ierr =
>>>>>> MatMPIAIJGetSeqAIJ(Amat,&Daij,&Oaij,&garray);CHKERRQ(ierr);
>>>>>>
>>>>>> The failure below is from this call to MatMPIAIJGetSeqAIJ, on this
>>>>>> line:
>>>>>>
>>>>>> ierr =
>>>>>> PetscObjectTypeCompare((PetscObject)A,MATMPIAIJ,&flg);CHKERRQ(ierr);
>>>>>>
>>>>>> The difference is PetscObjectTypeCompare vs PetscObject*Base*TypeCompare.
>>>>>> GAMG could use PetscObjectTypeCompare or MatMPIAIJGetSeqAIJ could
>>>>>> use PetscObject*Base*TypeCompare ...
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>> srun -n 2 ./ex19 -pc_type gamg -snes_monitor_short
>>>>>> -ksp_monitor_short
>>>>>>
>>>>>> lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
>>>>>>
>>>>>>
>>>>>>   0 SNES Function norm 0.239155
>>>>>>
>>>>>>
>>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>>> --------------------------------------------------------------
>>>>>>
>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>
>>>>>>
>>>>>> [0]PETSC ERROR: This function requires a MATMPIAIJ matrix as input
>>>>>>
>>>>>>
>>>>>> [0]PETSC ERROR: See
>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>>>>> shooting.
>>>>>>
>>>>>> [0]PETSC ERROR: Petsc Development GIT revision:
>>>>>> v3.9.2-825-g3a11c7608d  GIT Date: 2018-07-01 06:15:09 +0200
>>>>>>
>>>>>> [0]PETSC ERROR:
>>>>>> /global/u2/m/madams/petsc_install/petsc/src/snes/examples/tutorials/./ex19
>>>>>> on a  named nid02516 by madams Tue Jul  3 04:58:13 2018
>>>>>>
>>>>>> [0]PETSC ERROR: Configure options --known-level1-dcache-size=32768
>>>>>> --known-level1-dcache-linesize=64 --known-level1-dcache-assoc=8
>>>>>> --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2
>>>>>> --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8
>>>>>> --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8
>>>>>> --known-bits-per-byte=8 --known-memcmp-ok=1 --known-sizeof-MPI_Comm=4
>>>>>> --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1
>>>>>> --known-mpi-c-double-complex=1 --known-mklspblas-supports-zero-based=0
>>>>>> --known-has-attribute-aligned=1 --with-cc=cc --with-cxx=CC --with-fc=ftn
>>>>>> COPTFLAGS="  -g -O1 -mkl -static-intel" CXXOPTFLAGS="-g -O1 -mkl
>>>>>> -static-intel" FOPTFLAGS="  -g -O1 -mkl -static-intel" --download-metis=1
>>>>>> --with-hypre-dir=/global/homes/m/madams/tmp/hypre32-2.14.0
>>>>>> --download-parmetis=1
>>>>>> LIBS="-L/opt/intel/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64
>>>>>> -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread"
>>>>>> --with-blaslapack-include=/opt/intel/compilers_and_libraries_2018.1.163/linux/mkl/include
>>>>>> --with-debugging=0 --with-mpiexec=srun --with-batch=1
>>>>>> --known-mpi-shared-libraries=1 --known-64-bit-blas-indices=0
>>>>>> --with-64-bit-indices=0 PETSC_ARCH=arch-cori-knl-opt-intel-omp
>>>>>> --with-openmp=1 --download-p4est=0 --with-x=0
>>>>>> --prefix=/global/homes/m/madams/petsc_install/petsc-cori-knl-opt-intel-omp
>>>>>> PETSC_DIR=/global/homes/m/madams/petsc_install/petsc
>>>>>> [0]PETSC ERROR: #1 MatMPIAIJGetSeqAIJ() line 4328 in
>>>>>> /global/u2/m/madams/petsc_install/petsc/src/mat/impls/aij/mpi/mpiaij.c
>>>>>> [0]PETSC ERROR: #2 PCGAMGCreateGraph() line 118 in
>>>>>> /global/u2/m/madams/petsc_install/petsc/src/ksp/pc/impls/gamg/util.c
>>>>>> [0]PETSC ERROR: #3 PCGAMGGraph_AGG() line 832 in
>>>>>> /global/u2/m/madams/petsc_install/petsc/src/ksp/pc/impls/gamg/agg.c
>>>>>> [0]PETSC ERROR: #4 PCSetUp_GAMG() line 517 in
>>>>>> /global/u2/m/madams/petsc_install/petsc/src/ksp/pc/impls/gamg/gamg.c
>>>>>> [0]PETSC ERROR: #5 PCSetUp() line 932 in
>>>>>> /global/u2/m/madams/petsc_install/petsc/src/ksp/pc/interface/precon.c
>>>>>> [0]PETSC ERROR: #6 KSPSetUp() line 381 in
>>>>>> /global/u2/m/madams/petsc_install/petsc/src/ksp/ksp/interface/itfunc.c
>>>>>> [0]PETSC ERROR: #7 KSPSolve() line 612 in
>>>>>> /global/u2/m/madams/petsc_install/petsc/src/ksp/ksp/interface/itfunc.c
>>>>>> [0]PETSC ERROR: #8 SNESSolve_NEWTONLS() line 224 in
>>>>>> /global/u2/m/madams/petsc_install/petsc/src/snes/impls/ls/ls.c
>>>>>> [0]PETSC ERROR: #9 SNESSolve() line 4350 in
>>>>>> /global/u2/m/madams/petsc_install/petsc/src/snes/interface/snes.c
>>>>>> [0]PETSC ERROR: #10 main() line 161 in
>>>>>> /global/homes/m/madams/petsc_install/petsc/src/snes/examples/tutorials/ex19.c
>>>>>> [0]PETSC ERROR: PETSc Option Table entries:
>>>>>> [0]PETSC ERROR: -ksp_monitor_short
>>>>>> [0]PETSC ERROR: -mat_type aijmkl
>>>>>> [0]PETSC ERROR: -options_left
>>>>>> [0]PETSC ERROR: -pc_type gamg
>>>>>> [0]PETSC ERROR: -snes_monitor_short
>>>>>>
>>>>>>
>>>>> --
>>> Jeff Hammond
>>> jeff.science at gmail.com
>>> http://jeffhammond.github.io/
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/>
>>
> --
> Jeff Hammond
> jeff.science at gmail.com
> http://jeffhammond.github.io/
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20180704/a4064ab6/attachment.html>

More information about the petsc-dev mailing list