[petsc-dev] development version for MATAIJMKL type mat

[Mark Adams]

On Tue, Oct 10, 2017 at 2:50 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Oct 10, 2017, at 10:52 AM, Bakytzhan Kallemov <bkallemov at lbl.gov>
> wrote:
> >
> > Hi,
> >
> > My name is Baky Kallemov.
> >
> > Currently, I am working on improving a scalibility of  the Chombo-Petsc
> interface on cori machine at nersc system.
> >
> > I successfully build the libs from master branch with --with-openmp and
> hypre.
> >
> > However, I have not noticed any difference running my test problem on
> single node KNL node using new MATAIJMKL
>
>   hyre uses its own matrix operations so it won't get faster when using
> running PETSc with MATAIJMKL or any other specific matrix type.
>

Yes Baky, you need to use '-pc_type gamg with MKL'. Let's stick with hypre.


> >
> >
> > type for different hybrid mpi+openmp runs  compared to regular released
> version.
>
>    What are you comparing? Are you using say 32 MPI processes and 2
> threads or 16 MPI processes and 4 threads? How are you controlling the
> number of OpenMP threads, OpenMP environmental variable? What parts of the
> time in the code are you comparing? You should just -log_view and compare
> the times for PCApply and PCSetUp() between say 64 MPI process/1 thread and
> 32 MPI processes/2 threads and send us the output for those two cases.
>

These folks don't use many MPI processes. I'm not sure what the optimal
configuration is with Chombo-Crunch when using all of Cori.

Baky: how many MPI processes per socket are you aiming for on Cori-KNL?


>
> >
> > It seems that it made no difference, so perhaps I am doing something
> wrong or my build is not configured right.
> >
> > Do you have any example that makes use of threads when running hybrid
> and show an advantage?
>
>    There is not reason to think that using threads on KNL is faster than
> just using MPI processes. Despite what the NERSc/LBL web pages may say,
> just because a website says something doesn't make it true.
>
>
> >
> > I'd like to test it and make sure that my libs are configured correctly,
> before start to investigate it further.
> >
> >
> > Thanks,
> >
> > Baky
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20171010/22651da7/attachment-0001.html>