[petsc-users] Re-configuring PETSc

Tue May 29 05:40:46 CDT 2018

On Tue, May 29, 2018 at 4:05 AM, Najeeb Ahmad <nahmad16 at ku.edu.tr> wrote:

>
>
> On Mon, May 28, 2018 at 9:32 PM, Smith, Barry F. <bsmith at mcs.anl.gov>
> wrote:
>
>>
>>
>> > On May 28, 2018, at 10:32 AM, Najeeb Ahmad <nahmad16 at ku.edu.tr> wrote:
>> >
>> > Thanks a lot Satish for your prompt reply.
>> >
>> > I just checked that SuperLU_dist package works only for matrices of
>> type aij. and uses lu preconditioner. I am currently working with baij
>> matrix. What is the best preconditioner choice for baij matrices on
>> parallel machines?
>>
>>     The best preconditioner is always problem specific. Where does your
>> problem come from? CFD? Structural mechanics? other apps?
>>
>
>         I am interested in writing solver for reservoir simulation
> employing FVM and unstructured grids. My main objective is to study
> performance of the code with different data structures/data layouts and
> architecture specific optimizations, specifically targeting the multicore
> architectures like KNL for instance. Later the study may be extended to
> include GPUs. The options for switching between AIJ and BAIJ etc. are
> therefore very useful for my study.
>

I would strongly encourage you to create a simple performance model first.
It should demonstrate that the investigation can
produce a positive result. For example, performance for stencil codes like
this is usually limited by memory bandwidth. The
bandwidth advantage would be at most 3x with KNL/GPU vs Skylake, and it
would be nil if the Skylake were to get MCDRAM
as we all expect.

>         The purpose why I wanted to change the preconditioner is that the
> default preconditioner is giving me different iterations count for
> different number of processesors. I would rather like a preconditioner that
> would give me same iteration count for any processor count so that I can
> better compare the performance results.
>
>         Your suggestions in this regard are highly appreciated,
> specifically with reference to the following points:
>
>          - Is it possible to explicitly use high bandwidth memory in PETSc
> for selected object placement (e.g. using memkind library for instance)?
>

I believe Richard has some extensions for this. Richard?

>          - What would it take to take advantage of architecture specific
> compiler flags to achieve good performance on a given platform (e.g.
> -xMIC-AVX512 for AVX512 on KNL, #pragma SIMD etc.).
>

Usually intrinsics. We already have this for MatMult() for some
architectures.

  Thanks,

    Matt

>
>        Sorry for some very basic questions as I am a novice PETSc user.
>
>      Thanks for your time :)
>
>>
>>      Anyways you probably want to make your code be able to switch
>> between AIJ and BAIJ at run time since the different formats support
>> somewhat different solvers. If your code alls MatSetFromOptions then you
>> can switch via the command line option -mat_type aij or baij
>>
>>    Barry
>>
>> >
>> > Thanks
>> >
>> > On Mon, May 28, 2018 at 8:23 PM, Satish Balay <balay at mcs.anl.gov>
>> wrote:
>> > On Mon, 28 May 2018, Najeeb Ahmad wrote:
>> >
>> > > Hi All,
>> > >
>> > > I have Petsc release version 3.9.2 configured with the following
>> options:
>> > >
>> > > Configure options --with-cc=mpiicc --with-cxx=mpiicpc
>> --with-fc=mpiifort
>> > > --download-fblaslapack=1
>> > >
>> > > Now I want to use PCILU in my code and when I set the PC type to
>> PCILU in
>> > > the code, I get the following error:
>> > >
>> > > [0]PETSC ERROR: --------------------- Error Message
>> > > --------------------------------------------------------------
>> > > [0]PETSC ERROR: See
>> > > http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for
>> > > possible LU and Cholesky solvers
>> > > [0]PETSC ERROR: Could not locate a solver package. Perhaps you must
>> > > ./configure with --download-<package>
>> > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d
>> ocumentation/faq.html for
>> > > trouble shooting.
>> > > [0]PETSC ERROR: Petsc Release Version 3.9.2, unknown
>> > > [0]PETSC ERROR: ./main on a arch-linux2-c-debug named Karachi by
>> nahmad Mon
>> > > May 28 17:52:41 2018
>> > > [0]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc
>> > > --with-fc=mpiifort --download-fblaslapack=1
>> > > [0]PETSC ERROR: #1 MatGetFactor() line 4318 in
>> > > /home/nahmad/PETSc/petsc/src/mat/interface/matrix.c
>> > > [0]PETSC ERROR: #2 PCSetUp_ILU() line 142 in
>> > > /home/nahmad/PETSc/petsc/src/ksp/pc/impls/factor/ilu/ilu.c
>> > > [0]PETSC ERROR: #3 PCSetUp() line 923 in
>> > > /home/nahmad/PETSc/petsc/src/ksp/pc/interface/precon.c
>> > > [0]PETSC ERROR: #4 KSPSetUp() line 381 in
>> > > /home/nahmad/PETSc/petsc/src/ksp/ksp/interface/itfunc.c
>> > > [0]PETSC ERROR: #5 KSPSolve() line 612 in
>> > > /home/nahmad/PETSc/petsc/src/ksp/ksp/interface/itfunc.c
>> > > [0]PETSC ERROR: #6 SolveSystem() line 60 in
>> > > /home/nahmad/Aramco/petsc/petsc/BlockSolveTest/src/main.c
>> > >
>> > >
>> > > I assume that I am missing LU package like SuperLU_dist for instance
>> and I
>> > > need to download and configure it with Petsc.
>> >
>> > yes - petsc has sequential LU - but you need superlu_dist/mumps for
>> parallel lu.
>> >
>> > >
>> > > I am wondering what is the best way to reconfigure Petsc to download
>> and
>> > > use the appropriate package to support PCILU?
>> >
>> > Rerun configure with the additional option --download-superlu_dist=1.
>> >
>> > You can do this with current PETSC_ARCH you are using [i.e reinstall
>> > over the current build] - or use a different PETSC_ARCH - so both
>> > builds exist and useable.
>> >
>> > Satish
>> >
>> > >
>> > > You advice is highly appreciated.
>> > >
>> > >
>> >
>> >
>> >
>> >
>> > --
>> > Najeeb Ahmad
>> >
>> > Research and Teaching Assistant
>> > PARallel and MultiCORE Computing Laboratory (ParCoreLab)
>> > Computer Science and Engineering
>> > Koç University, Istanbul, Turkey
>> >
>>
>>
>
>
> --
> *Najeeb Ahmad*
>
>
> *Research and Teaching Assistant*
> *PARallel and MultiCORE Computing Laboratory (ParCoreLab) *
>
> *Computer Science and Engineering*
> *Koç University, Istanbul, Turkey*
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20180529/51cfdd7c/attachment-0001.html>